Brian Warner
08f5bc8e2f
convert UploadResults to a fat init
...
Populate most of UploadResults (except .uri, which is learned later when
using a Helper) in the constructor, instead of allowing creators to
write to attributes later. This will help isolate the fields that we
want to change to use IServers.
2012-05-21 21:14:14 -07:00
Brian Warner
b71234c538
add HelperUploadResults
...
This splits the pb.Copyable on-wire object (HelperUploadResults) out
from the local results object (UploadResults). To maintain compatibility
with older Helpers, we have to leave pb.Copyable classes alone and
unmodified, but we want to change UploadResults to use IServers instead
of serverids. So by using a different class on the wire, and translating
to/from it on either end, we can accomplish both.
2012-05-21 21:14:00 -07:00
Brian Warner
b3af012b13
Uploader cleanup: create results at end, not beginning
...
This will make it easier to populate the UploadResults during __init__,
instead of doing it one-field-at-a-time later.
2012-05-21 21:13:47 -07:00
Brian Warner
0df833eac9
clean up Helper to make later changes easier
...
Fix up control flow inside the Helper, to make it more friendly for
later refactoring.
2012-05-21 21:13:32 -07:00
Brian Warner
e60982c851
helper: remove timings["existence_check"], aka "Already-In-Grid Check"
...
This measured how long the Helper took to do a filecheck before asking
for ciphertext. The "Contacting Helper" report includes both
existence_check and the client-helper RTT.
For non-overlapping uploads, it was being returned correctly. But when
multiple upload requests overlapped, and the file was not already in the
grid, the filecheck would only run once, and its existence_check time
would be reported for all uploaders (even if they didn't have to wait
for that time). Cleaning that up proved too difficult: the only correct
place to report this time is from the initial remote_upload_chk() call,
but the return value of that is too constrained to accomodate it in the
needs-upload case.
So I'm removing it altogether. Eventually I plan to add a proper
events/times field and record more data, including this check, in a form
that can be drawn on a nice zoomable timeline view.
Old clients talking to a new Helper (which doesn't supply the value)
will tolerate the loss (they'll just display an empty field on the web
view).
2012-05-21 21:13:11 -07:00
david-sarah
4ddcde3094
Since we now require Python 2.5, we can use os.SEEK_END.
2012-05-16 21:39:48 +00:00
Brian Warner
9acf5beebd
immutable repairer: populate servers-responding properly
...
If a server did not respond to the pre-repair filecheck, but did respond
to the repair, that server was not correctly added to the
RepairResults.data["servers-responding"] list. (This resulted from a
buggy usage of DictOfSets.union() in filenode.py).
In addition, servers to which filecheck queries were sent, but did not
respond, were incorrectly added to the servers-responding list
anyawys. (This resulted from code in the checker.py not paying attention
to the 'responded' flag).
The first bug was neatly masked by the second: it's pretty rare to have
a server suddenly start responding in the one-second window between a
filecheck and a subsequent repair, and if the server was around for the
filecheck, you'd never notice the problem. I only spotted the smelly
code while I was changing it for IServer cleanup purposes.
I added coverage to test_repairer.py for this. Trying to get that test
to fail before fixing the first bug is what led me to discover the
second bug. I also had to update test_corrupt_file_verno, since it was
incorrectly asserting that 10 servers responded, when in fact one of
them throws an error (but the second bug was causing it to be reported
anyways).
2012-05-16 16:55:09 -07:00
Brian Warner
fcc7e64759
Doc updates and cosmetic fixes for #1115 patch.
...
Removes the caveat from webapi.txt about count-good-share-hosts being wrong.
This series should close #1115 .
2012-05-13 01:15:50 -07:00
Andrew Miller
04eb6086ad
Fixed an error in previous commit where an empty servermap would throw an exception in 'count-good-share-hosts'. Augmented unit test.
...
Signed-off-by: Andrew Miller <amiller@dappervision.com>
2012-05-13 00:59:30 -07:00
Andrew Miller
4b7f34d179
Added tests for count-good-share-hosts under check and repair conditions. Patched the incorrect computation in immutable/filenode.py
...
Signed-off-by: Andrew Miller <amiller@dappervision.com>
Fixed missing import statements
Signed-off-by: Andrew Miller <amiller@dappervision.com>
2012-05-13 00:59:30 -07:00
Brian Warner
4b8876c5da
checker.py: minor simplifications
2012-04-04 12:05:31 -07:00
Brian Warner
7da112ccef
Rewrite download-status-timeline visualizer ('viz') with d3.js
...
* use d3.js v2.4.6
* add a "toggle misc events" button, to get hash/bitmap-checking details
* only draw data that's on screen, for speed
* add fragment-arg to fetch timeline data.json from somewhere else
2011-10-31 23:18:21 -07:00
Brian Warner
5bf1ffbc87
IServer refactoring: pass IServer instances around, instead of peerids
...
refs #1363
This collapses 88 small incremental changes (each of which passes all tests)
into one big patch. The development process for the long path started with
adding some temporary scaffolding, changing one method at a time, then
removing the scaffolding. The individual pieces are as follows, in reverse
chronological order (the first patch is at the end of this comment):
commit 9bbe4174fd0d98a6cf47a8ef96e85d9ef34b2f9a
Author: Brian Warner <warner@lothar.com>
Date: Tue Oct 4 16:05:00 2011 -0400
immutable/downloader/status.py: correct comment
src/allmydata/immutable/downloader/status.py | 2 +-
1 files changed, 1 insertions(+), 1 deletions(-)
commit 72146a7c7c91eac2f7c3ceb801eb7a1721376889
Author: Brian Warner <warner@lothar.com>
Date: Tue Oct 4 15:46:20 2011 -0400
remove temporary ServerMap._storage_broker
src/allmydata/mutable/checker.py | 2 +-
src/allmydata/mutable/filenode.py | 2 +-
src/allmydata/mutable/publish.py | 2 +-
src/allmydata/mutable/servermap.py | 5 ++---
src/allmydata/test/test_mutable.py | 8 ++++----
5 files changed, 9 insertions(+), 10 deletions(-)
commit d703096b41632c47d76414b12672e076a422ff5c
Author: Brian Warner <warner@lothar.com>
Date: Tue Oct 4 15:37:05 2011 -0400
remove temporary storage_broker.get_server_for_id()
src/allmydata/storage_client.py | 3 ---
src/allmydata/test/no_network.py | 13 -------------
2 files changed, 0 insertions(+), 16 deletions(-)
commit 620cc5d80882ef6f7decfd26af8a6c7c1ddf80d1
Author: Brian Warner <warner@lothar.com>
Date: Tue Oct 4 12:50:06 2011 -0400
API of Retrieve._try_to_validate_privkey(), trying to remove reader.server
src/allmydata/mutable/retrieve.py | 10 +++++-----
1 files changed, 5 insertions(+), 5 deletions(-)
commit 92f43f856f4a8b36c207d1b190ed8699b5a4ecb4
Author: Brian Warner <warner@lothar.com>
Date: Tue Oct 4 12:48:08 2011 -0400
API of Retrieve._validate_block(), trying to remove reader.server
src/allmydata/mutable/retrieve.py | 14 +++++++-------
1 files changed, 7 insertions(+), 7 deletions(-)
commit 572d5070761861a2190349d1ed8d85dbc25698a5
Author: Brian Warner <warner@lothar.com>
Date: Tue Oct 4 12:36:58 2011 -0400
API of Retrieve._mark_bad_share(), trying to remove reader.server
src/allmydata/mutable/retrieve.py | 21 +++++++++------------
1 files changed, 9 insertions(+), 12 deletions(-)
commit a793ff00c0de1e2eec7b46288fdf388c7a2bec89
Author: Brian Warner <warner@lothar.com>
Date: Tue Oct 4 12:06:13 2011 -0400
remove now-unused get_rref_for_serverid()
src/allmydata/mutable/servermap.py | 3 ---
1 files changed, 0 insertions(+), 3 deletions(-)
commit 1b9827cc9366bf90b93297fdd6832f2ad0480ce7
Author: Brian Warner <warner@lothar.com>
Date: Tue Oct 4 12:03:09 2011 -0400
Retrieve: stop adding .serverid attributes to readers
src/allmydata/mutable/retrieve.py | 1 -
1 files changed, 0 insertions(+), 1 deletions(-)
commit 5d4e9d491b19e49d2e443a1dfff2c672842c36ef
Author: Brian Warner <warner@lothar.com>
Date: Tue Oct 4 12:03:34 2011 -0400
return value of Retrieve(verify=True)
src/allmydata/mutable/checker.py | 11 ++++++-----
src/allmydata/mutable/retrieve.py | 3 +--
2 files changed, 7 insertions(+), 7 deletions(-)
commit e9ab7978c384e1f677cb7779dc449b1044face82
Author: Brian Warner <warner@lothar.com>
Date: Tue Oct 4 11:54:23 2011 -0400
Retrieve._bad_shares (but not return value, used by Verifier)
src/allmydata/mutable/retrieve.py | 7 ++++---
1 files changed, 4 insertions(+), 3 deletions(-)
commit 2d91926de233ec5c881f30e36b4a30ad92ab42a9
Author: Brian Warner <warner@lothar.com>
Date: Tue Oct 4 11:51:23 2011 -0400
Publish: stop adding .serverid attributes to writers
src/allmydata/mutable/publish.py | 9 ++-------
1 files changed, 2 insertions(+), 7 deletions(-)
commit 47c7a0105dec7cbf4f7e0a3ce800bbb85b15df4a
Author: Brian Warner <warner@lothar.com>
Date: Tue Oct 4 11:56:33 2011 -0400
API of get_write_enabler()
src/allmydata/mutable/filenode.py | 7 ++++---
src/allmydata/mutable/publish.py | 4 ++--
src/allmydata/test/no_network.py | 3 +++
3 files changed, 9 insertions(+), 5 deletions(-)
commit 9196a5c6590fdbfd660325ea8358b345887d3db0
Author: Brian Warner <warner@lothar.com>
Date: Tue Oct 4 11:46:24 2011 -0400
API of get_(renewal|cancel)_secret()
src/allmydata/mutable/filenode.py | 14 ++++++++------
src/allmydata/mutable/publish.py | 8 ++++----
src/allmydata/mutable/servermap.py | 5 ++---
3 files changed, 14 insertions(+), 13 deletions(-)
commit de7c1552f8c163eff5b6d820b5fb3b21c1b47cb5
Author: Brian Warner <warner@lothar.com>
Date: Tue Oct 4 11:41:52 2011 -0400
API of CorruptShareError. Also comment out some related+unused test_web.py code
src/allmydata/mutable/common.py | 13 +++++--------
src/allmydata/mutable/retrieve.py | 10 +++++-----
src/allmydata/mutable/servermap.py | 8 +++-----
src/allmydata/test/common.py | 13 ++++++++-----
4 files changed, 21 insertions(+), 23 deletions(-)
commit 2c1c314046b620c16f1e66d030c150d768b7d01e
Author: Brian Warner <warner@lothar.com>
Date: Tue Oct 4 12:01:46 2011 -0400
API of ServerMap.mark_bad_share()
src/allmydata/mutable/publish.py | 2 +-
src/allmydata/mutable/retrieve.py | 6 +++---
src/allmydata/mutable/servermap.py | 6 ++----
src/allmydata/test/test_mutable.py | 3 +--
4 files changed, 7 insertions(+), 10 deletions(-)
commit 1bed349030779fd0c378ae4e821384f953c6f6ff
Author: Brian Warner <warner@lothar.com>
Date: Tue Oct 4 11:11:17 2011 -0400
API+name of ServerMap.shares_on_server() : only for tests, so debug_ prefix
src/allmydata/mutable/servermap.py | 7 ++-----
src/allmydata/test/test_mutable.py | 6 +++---
2 files changed, 5 insertions(+), 8 deletions(-)
commit 2d32e448677d6b818692e801045d4115b29abf21
Author: Brian Warner <warner@lothar.com>
Date: Tue Oct 4 11:07:10 2011 -0400
API of ServerMap.all_servers_for_version()
src/allmydata/mutable/servermap.py | 4 ++--
1 files changed, 2 insertions(+), 2 deletions(-)
commit 48f3204d1889c3e7179578125c4bdef515af3d6a
Author: Brian Warner <warner@lothar.com>
Date: Tue Oct 4 11:04:50 2011 -0400
internals of ServerMap methods that use make_versionmap(), remove temp copy
src/allmydata/mutable/servermap.py | 28 +++++++++----------------
1 files changed, 10 insertions(+), 18 deletions(-)
commit 5c3da77b6c777a145bd5ddfaa4db849dc9495548
Author: Brian Warner <warner@lothar.com>
Date: Tue Oct 4 11:01:28 2011 -0400
API of ServerMap.make_versionmap()
src/allmydata/mutable/checker.py | 4 ++--
src/allmydata/mutable/retrieve.py | 5 ++---
src/allmydata/mutable/servermap.py | 4 ++--
src/allmydata/test/test_mutable.py | 7 ++++---
4 files changed, 10 insertions(+), 10 deletions(-)
commit b6882ece49afb4c507d118af2db346fa329209dc
Author: Brian Warner <warner@lothar.com>
Date: Tue Oct 4 10:53:38 2011 -0400
make a copy of ServerMap.make_versionmap() (_make_versionmap2) for internal use
src/allmydata/mutable/servermap.py | 18 +++++++++++++-----
1 files changed, 13 insertions(+), 5 deletions(-)
commit 963f8e63faf32b950eb1b8103cd2ff16fe8f0151
Author: Brian Warner <warner@lothar.com>
Date: Tue Oct 4 00:45:58 2011 -0400
API of RetrieveStatus.add_problem()
src/allmydata/mutable/retrieve.py | 5 +++--
1 files changed, 3 insertions(+), 2 deletions(-)
commit 4976d29ffae565a048851601c29013bbae2976d8
Author: Brian Warner <warner@lothar.com>
Date: Tue Oct 4 00:45:05 2011 -0400
API of RetrieveStatus.add_fetch_timing()
src/allmydata/mutable/retrieve.py | 5 +++--
1 files changed, 3 insertions(+), 2 deletions(-)
commit d057d3bbba72663ee148a8b916bc2d52be2e3982
Author: Brian Warner <warner@lothar.com>
Date: Tue Oct 4 00:44:04 2011 -0400
API of Retrieve.notify_server_corruption()
src/allmydata/mutable/retrieve.py | 6 +++---
1 files changed, 3 insertions(+), 3 deletions(-)
commit 8a2a81e46671c860610e0e96d6add1a57551f22d
Author: Brian Warner <warner@lothar.com>
Date: Tue Oct 4 00:42:32 2011 -0400
remove unused _outstanding_queries
src/allmydata/mutable/retrieve.py | 1 -
1 files changed, 0 insertions(+), 1 deletions(-)
commit 56d12cc9968d03ccd53764455c671122c4f391d1
Author: Brian Warner <warner@lothar.com>
Date: Tue Oct 4 00:40:57 2011 -0400
change Retrieve.remaining_sharemap
src/allmydata/mutable/retrieve.py | 4 ++--
1 files changed, 2 insertions(+), 2 deletions(-)
commit 4f0b7af4821f43290bfc70f2b1fc30149ad81281
Author: Brian Warner <warner@lothar.com>
Date: Tue Oct 4 10:40:18 2011 -0400
accessor for PublishStatus._problems
src/allmydata/mutable/publish.py | 4 +++-
src/allmydata/web/status.py | 2 +-
2 files changed, 4 insertions(+), 2 deletions(-)
commit 627087cf66d0b8cc519f4d551a967a7bd9b6a741
Author: Brian Warner <warner@lothar.com>
Date: Tue Oct 4 10:36:39 2011 -0400
accessor for RetrieveStatus._problems
src/allmydata/mutable/retrieve.py | 8 ++++++--
src/allmydata/web/status.py | 2 +-
2 files changed, 7 insertions(+), 3 deletions(-)
commit ca7dea81f03801b1c7353fc00ecba689268109cf
Author: Brian Warner <warner@lothar.com>
Date: Tue Oct 4 00:35:32 2011 -0400
add .server to "reader", so we can get at it later
src/allmydata/mutable/retrieve.py | 5 +++--
1 files changed, 3 insertions(+), 2 deletions(-)
commit 6ef516e24908ec195af084a7550d1921a5e983b0
Author: Brian Warner <warner@lothar.com>
Date: Tue Oct 4 00:32:32 2011 -0400
temporarily give Retrieve a _storage_broker, so it can map serverids to servers
src/allmydata/mutable/checker.py | 3 ++-
src/allmydata/mutable/filenode.py | 6 ++++--
src/allmydata/mutable/retrieve.py | 5 +++--
src/allmydata/test/test_mutable.py | 4 ++--
4 files changed, 11 insertions(+), 7 deletions(-)
commit afe08e4dd3f4ff9ff7e8a2a8d28b181e3625bcc9
Author: Brian Warner <warner@lothar.com>
Date: Tue Oct 4 00:21:51 2011 -0400
mutable/retrieve.py: s/peer/server/
src/allmydata/mutable/retrieve.py | 82 +++++++++++++-------------
src/allmydata/test/test_mutable.py | 6 +-
2 files changed, 44 insertions(+), 44 deletions(-)
commit 910afcb5d7f274880f68dd6cdb5b05f2bbc29adc
Author: Brian Warner <warner@lothar.com>
Date: Tue Oct 4 00:16:01 2011 -0400
web.status.PublishStatusPage: add comment, I think .problems isn't exercised
src/allmydata/web/status.py | 2 ++
1 files changed, 2 insertions(+), 0 deletions(-)
commit 311466dd8c931bbba40d590ade867704282e7f1a
Author: Brian Warner <warner@lothar.com>
Date: Mon Oct 3 23:48:16 2011 -0400
API of PublishStatus.add_per_server_time()
src/allmydata/mutable/publish.py | 5 +++--
1 files changed, 3 insertions(+), 2 deletions(-)
commit 2df5faa1b6cbfbaded520d2320305a62fe961118
Author: Brian Warner <warner@lothar.com>
Date: Mon Oct 3 23:46:37 2011 -0400
more simplifications
src/allmydata/mutable/publish.py | 4 +---
1 files changed, 1 insertions(+), 3 deletions(-)
commit 6ac4544a3da385f2aad9392f906b90192f4f919a
Author: Brian Warner <warner@lothar.com>
Date: Mon Oct 3 23:44:08 2011 -0400
API of ServerMap.version_on_server()
src/allmydata/mutable/publish.py | 2 +-
src/allmydata/mutable/servermap.py | 4 ++--
src/allmydata/test/test_mutable.py | 5 ++---
3 files changed, 5 insertions(+), 6 deletions(-)
commit 3e187e322511072e4683329df6b2c6c733a66dba
Author: Brian Warner <warner@lothar.com>
Date: Tue Oct 4 00:16:32 2011 -0400
API of ServerMap.make_sharemap()
src/allmydata/mutable/servermap.py | 4 ++--
src/allmydata/test/test_mutable.py | 7 ++++---
src/allmydata/web/status.py | 4 ++--
3 files changed, 8 insertions(+), 7 deletions(-)
commit 318feed8437bdd8d4943c6569d38f7b54b6313cc
Author: Brian Warner <warner@lothar.com>
Date: Mon Oct 3 23:36:19 2011 -0400
small cleanups
src/allmydata/mutable/publish.py | 4 ++--
1 files changed, 2 insertions(+), 2 deletions(-)
commit bd459ed5714e1db5a7163935c54b7b0b56db8349
Author: Brian Warner <warner@lothar.com>
Date: Mon Oct 3 23:33:39 2011 -0400
API of ServerMap.add_new_share()
src/allmydata/mutable/publish.py | 4 ++--
src/allmydata/mutable/servermap.py | 6 ++----
2 files changed, 4 insertions(+), 6 deletions(-)
commit f2804fb6ed11d80088e0da8ed48e6c2922f2ffef
Author: Brian Warner <warner@lothar.com>
Date: Mon Oct 3 23:30:26 2011 -0400
API of ServerMap.get_bad_shares()
src/allmydata/mutable/publish.py | 3 +--
src/allmydata/mutable/servermap.py | 9 ++++-----
2 files changed, 5 insertions(+), 7 deletions(-)
commit 965074a47b3ce1431cb46d9a233840afcf9105f5
Author: Brian Warner <warner@lothar.com>
Date: Mon Oct 3 23:26:58 2011 -0400
more small cleanups
src/allmydata/mutable/publish.py | 6 +++---
1 files changed, 3 insertions(+), 3 deletions(-)
commit 38020da34f034f8889947dd3dc05e087ffff7106
Author: Brian Warner <warner@lothar.com>
Date: Mon Oct 3 23:18:47 2011 -0400
change Publish.bad_share_checkstrings
src/allmydata/mutable/publish.py | 6 +++---
1 files changed, 3 insertions(+), 3 deletions(-)
commit 5efebcbd2ee0c2f299ea86f7591d856c0f265304
Author: Brian Warner <warner@lothar.com>
Date: Mon Oct 3 23:16:31 2011 -0400
change internals of Publish.update_goal()
src/allmydata/mutable/publish.py | 8 +++-----
1 files changed, 3 insertions(+), 5 deletions(-)
commit e91b55ff4c2a69165b71f2c7b217ac319ff4c527
Author: Brian Warner <warner@lothar.com>
Date: Mon Oct 3 23:11:42 2011 -0400
get rid of Publish.connections
src/allmydata/mutable/publish.py | 27 +++++----------------------
1 files changed, 5 insertions(+), 22 deletions(-)
commit 64e9a53b3229ebe2f9ebf7ed502d539311d0e037
Author: Brian Warner <warner@lothar.com>
Date: Mon Oct 3 23:05:32 2011 -0400
change Publish.bad_servers
src/allmydata/mutable/publish.py | 10 +++++-----
1 files changed, 5 insertions(+), 5 deletions(-)
commit b85a934bef315a06bcfe00c9c12a3627fed2b918
Author: Brian Warner <warner@lothar.com>
Date: Mon Oct 3 23:03:07 2011 -0400
Publish.bad_servers: fix bug, this should be a set of serverids, not writers
src/allmydata/mutable/publish.py | 2 +-
1 files changed, 1 insertions(+), 1 deletions(-)
commit 605ea15ec15ed671513819003ccd211cdb9761e0
Author: Brian Warner <warner@lothar.com>
Date: Mon Oct 3 23:00:21 2011 -0400
change .placed
src/allmydata/mutable/publish.py | 6 +++---
1 files changed, 3 insertions(+), 3 deletions(-)
commit f7aba37b1b345d5b6d5cb16e3b3f6f3c1afb658e
Author: Brian Warner <warner@lothar.com>
Date: Mon Oct 3 22:59:22 2011 -0400
temporarily stash IServer as .server on the "writer" object
src/allmydata/mutable/publish.py | 2 ++
1 files changed, 2 insertions(+), 0 deletions(-)
commit f9b551d788e7db1f187fce5ab98ab5d5fe4e1c36
Author: Brian Warner <warner@lothar.com>
Date: Mon Oct 3 22:48:18 2011 -0400
change Publish.goal and API of log_goal() to use IServer, not serverid
src/allmydata/mutable/publish.py | 48 ++++++++++++++--------------
1 files changed, 24 insertions(+), 24 deletions(-)
commit 75f20616558e4900b8b1f685dd99aa838de6d452
Author: Brian Warner <warner@lothar.com>
Date: Mon Oct 3 15:27:02 2011 -0400
API of ServerMap.get_known_shares()
src/allmydata/mutable/publish.py | 16 ++++++++++------
src/allmydata/mutable/servermap.py | 7 ++-----
2 files changed, 12 insertions(+), 11 deletions(-)
commit 1c38c9d37bb08221b4418762234b1a62397b3b4b
Author: Brian Warner <warner@lothar.com>
Date: Mon Oct 3 15:20:29 2011 -0400
Publish.full_serverlist
src/allmydata/mutable/publish.py | 10 +++++-----
1 files changed, 5 insertions(+), 5 deletions(-)
commit b6cbd215a04b9cde31a7d92a97a7f048622b16f1
Author: Brian Warner <warner@lothar.com>
Date: Mon Oct 3 15:12:31 2011 -0400
API of ServerMap.all_servers()
src/allmydata/mutable/servermap.py | 19 ++++++-------------
1 files changed, 6 insertions(+), 13 deletions(-)
commit e63cd0315fae65357b1727ec6d5ff3c6e0d27c98
Author: Brian Warner <warner@lothar.com>
Date: Mon Oct 3 15:10:18 2011 -0400
remove ServerMap.connections, set_rref_for_serverid()
src/allmydata/mutable/servermap.py | 11 +----------
1 files changed, 1 insertions(+), 10 deletions(-)
commit 4df52db2f80eb12eefa5d57103c24893cde89553
Author: Brian Warner <warner@lothar.com>
Date: Mon Oct 3 15:04:06 2011 -0400
API of ServerMap.mark_server_reachable()
src/allmydata/mutable/servermap.py | 7 ++-----
1 files changed, 2 insertions(+), 5 deletions(-)
commit 69c715bde77944dc25181b3dbbeb042c816f9a1b
Author: Brian Warner <warner@lothar.com>
Date: Mon Oct 3 15:03:21 2011 -0400
API of ServerMap.mark_server_unreachable()
src/allmydata/mutable/servermap.py | 9 +++------
1 files changed, 3 insertions(+), 6 deletions(-)
commit 3d784d60eec1c508858e3a617e4411ffbcc3c1fa
Author: Brian Warner <warner@lothar.com>
Date: Mon Oct 3 15:02:03 2011 -0400
API of status.set_privkey_from()
src/allmydata/mutable/servermap.py | 7 +++----
1 files changed, 3 insertions(+), 4 deletions(-)
commit 544ed3ea29bed7e66da7fd29ca3f6f076f27a9e6
Author: Brian Warner <warner@lothar.com>
Date: Mon Oct 3 15:01:15 2011 -0400
API of status.add_per_server_time()
src/allmydata/mutable/servermap.py | 7 ++++---
1 files changed, 4 insertions(+), 3 deletions(-)
commit fffe5008b6320bd1e04c3c68389a2bf2ee383fa8
Author: Brian Warner <warner@lothar.com>
Date: Mon Oct 3 14:59:02 2011 -0400
remove unused .versionmap
src/allmydata/mutable/servermap.py | 7 -------
1 files changed, 0 insertions(+), 7 deletions(-)
commit 2816562e090d2294179db3588dafcca18de1bc2b
Author: Brian Warner <warner@lothar.com>
Date: Mon Oct 3 14:57:51 2011 -0400
remove serverid from all log messages. Also one unused lambda.
src/allmydata/mutable/servermap.py | 30 +++++++++++++-------------
1 files changed, 15 insertions(+), 15 deletions(-)
commit 28fa6b1a2738fa98c1f1dbd3d0e01ae98912d11f
Author: Brian Warner <warner@lothar.com>
Date: Mon Oct 3 14:54:30 2011 -0400
removed unused _readers
src/allmydata/mutable/servermap.py | 3 ---
1 files changed, 0 insertions(+), 3 deletions(-)
commit a8e4ed3d645ab592d1add6a1e69b6d1ebfb77817
Author: Brian Warner <warner@lothar.com>
Date: Mon Oct 3 14:54:16 2011 -0400
remove unused _sharemap
src/allmydata/mutable/servermap.py | 1 -
1 files changed, 0 insertions(+), 1 deletions(-)
commit 3f072e55cf1d0700f9fffe23f8f3a475725df588
Author: Brian Warner <warner@lothar.com>
Date: Mon Oct 3 14:49:03 2011 -0400
_must_query
src/allmydata/mutable/servermap.py | 8 ++++----
1 files changed, 4 insertions(+), 4 deletions(-)
commit c599a059b8df3f5785e4bf89fb6ecc6d8dcd708b
Author: Brian Warner <warner@lothar.com>
Date: Mon Oct 3 14:48:05 2011 -0400
_queries_outstanding
src/allmydata/mutable/servermap.py | 16 +++++++---------
1 files changed, 7 insertions(+), 9 deletions(-)
commit 7743759f98ac2c07926b2fdbd80bf52dfab33085
Author: Brian Warner <warner@lothar.com>
Date: Mon Oct 3 14:46:17 2011 -0400
_empty_servers
src/allmydata/mutable/servermap.py | 5 ++---
1 files changed, 2 insertions(+), 3 deletions(-)
commit 6bb1825916828a713a32cdf7f7411fa3ea2e1e5d
Author: Brian Warner <warner@lothar.com>
Date: Mon Oct 3 14:45:39 2011 -0400
_good_servers
src/allmydata/mutable/servermap.py | 4 ++--
1 files changed, 2 insertions(+), 2 deletions(-)
commit 1768fab1b51d8dd93ecabbaaabfadfa20cf6c3d4
Author: Brian Warner <warner@lothar.com>
Date: Mon Oct 3 14:44:59 2011 -0400
_bad_servers
src/allmydata/mutable/servermap.py | 14 +++++++-------
1 files changed, 7 insertions(+), 7 deletions(-)
commit dccbaef30f0ba714c746bf6d4a1a803c36e17b65
Author: Brian Warner <warner@lothar.com>
Date: Mon Oct 3 14:41:54 2011 -0400
API of _try_to_set_pubkey()
src/allmydata/mutable/servermap.py | 7 ++++---
1 files changed, 4 insertions(+), 3 deletions(-)
commit 0481ea70042ba3575f15eac7fd0780f8ece580cc
Author: Brian Warner <warner@lothar.com>
Date: Mon Oct 3 14:35:02 2011 -0400
API of notify_server_corruption()
src/allmydata/mutable/servermap.py | 6 +++---
1 files changed, 3 insertions(+), 3 deletions(-)
commit bea9cba18fb3b9c11bb22f18356a263ecec7351e
Author: Brian Warner <warner@lothar.com>
Date: Mon Oct 3 14:34:09 2011 -0400
API of _got_signature_one_share()
src/allmydata/mutable/servermap.py | 9 +++++----
1 files changed, 5 insertions(+), 4 deletions(-)
commit 1520123583cf78650706e114b15bb5b0ac1f4a14
Author: Brian Warner <warner@lothar.com>
Date: Mon Oct 3 14:32:33 2011 -0400
API of _try_to_validate_privkey()
src/allmydata/mutable/servermap.py | 9 +++++----
1 files changed, 5 insertions(+), 4 deletions(-)
commit 938852c9c8519c7a078f58a9b1f4dd8ec8b6715e
Author: Brian Warner <warner@lothar.com>
Date: Mon Oct 3 14:31:48 2011 -0400
API and internals of _add_lease_failed()
src/allmydata/mutable/servermap.py | 8 ++++----
1 files changed, 4 insertions(+), 4 deletions(-)
commit 3843dba367e3c19e176a622ab853cb51d2472ddf
Author: Brian Warner <warner@lothar.com>
Date: Mon Oct 3 14:30:37 2011 -0400
API of _privkey_query_failed()
src/allmydata/mutable/servermap.py | 5 +++--
1 files changed, 3 insertions(+), 2 deletions(-)
commit 2219a710e1633cd57d0ca0786490de87b3e19ba7
Author: Brian Warner <warner@lothar.com>
Date: Mon Oct 3 14:29:43 2011 -0400
fix bug in call to _privkey_query_failed, unrelated to refactoring
src/allmydata/mutable/servermap.py | 2 +-
1 files changed, 1 insertions(+), 1 deletions(-)
commit ae615bec7d0d1b269710b6902797b12f9592ad62
Author: Brian Warner <warner@lothar.com>
Date: Mon Oct 3 14:27:17 2011 -0400
API of _got_corrupt_share()
src/allmydata/mutable/servermap.py | 17 +++++++++--------
1 files changed, 9 insertions(+), 8 deletions(-)
commit cb51c95a6f4e077278157a77dab060c8c1ad7a81
Author: Brian Warner <warner@lothar.com>
Date: Mon Oct 3 14:23:16 2011 -0400
API of _got_results()
src/allmydata/mutable/servermap.py | 9 +++++----
1 files changed, 5 insertions(+), 4 deletions(-)
commit bac9154fe0af18f226999a58ffc2362d8cf4b802
Author: Brian Warner <warner@lothar.com>
Date: Mon Oct 3 14:19:19 2011 -0400
API of _query_failed()
src/allmydata/mutable/servermap.py | 5 +++--
1 files changed, 3 insertions(+), 2 deletions(-)
commit fdc29a8ca95d4b5c503e5382b9e5d4d02141ba12
Author: Brian Warner <warner@lothar.com>
Date: Mon Oct 3 14:17:20 2011 -0400
API of _do_read()
src/allmydata/mutable/servermap.py | 6 ++++--
1 files changed, 4 insertions(+), 2 deletions(-)
commit e7e9e338f28d004aa4d423d11c65f1e271ac7322
Author: Brian Warner <warner@lothar.com>
Date: Mon Oct 3 14:20:21 2011 -0400
API of _do_query()
src/allmydata/mutable/servermap.py | 15 +++++++--------
1 files changed, 7 insertions(+), 8 deletions(-)
commit 330625b9dac4cdbe72a11464a893065b9aeed453
Author: Brian Warner <warner@lothar.com>
Date: Mon Oct 3 14:43:05 2011 -0400
next step: first batch of updates to ServermapUpdater
updates:
most method-local variables in update()
API of _build_initial_querylist()
API of _send_initial_requests()
.full_serverlist
.extra_servers
src/allmydata/mutable/servermap.py | 39 ++++++++++++++------------
1 files changed, 21 insertions(+), 18 deletions(-)
commit 4aadc584fa7dcb2daa86b048c81dee0049ba26d9
Author: Brian Warner <warner@lothar.com>
Date: Mon Oct 3 15:07:00 2011 -0400
internal change: index _bad_shares with IServer
src/allmydata/mutable/servermap.py | 20 ++++++++++----------
1 files changed, 10 insertions(+), 10 deletions(-)
commit 16d4e6fa82a9907dbdc92094213387c6a4164e41
Author: Brian Warner <warner@lothar.com>
Date: Mon Oct 3 18:20:47 2011 +0100
internal change: index _known_shares with IServer instead of serverid
callers are unchanged
src/allmydata/mutable/servermap.py | 42 +++++++++++++++----------
1 files changed, 25 insertions(+), 17 deletions(-)
commit ceeb5f4938cc814a0c75d1b8f4018aed965c2176
Author: Brian Warner <warner@lothar.com>
Date: Mon Oct 3 18:11:43 2011 +0100
accessors and name cleanup for servermap.Servermap.last_update_mode/time
src/allmydata/mutable/filenode.py | 6 +++---
src/allmydata/mutable/publish.py | 4 ++--
src/allmydata/mutable/servermap.py | 17 +++++++++++------
3 files changed, 16 insertions(+), 11 deletions(-)
commit 8d3cbda82661c0a7e5c3d3b65cf7a5d5ab7e32c0
Author: Brian Warner <warner@lothar.com>
Date: Mon Oct 3 18:11:14 2011 +0100
accessors and name cleanup for servermap.Servermap.problems
src/allmydata/mutable/servermap.py | 21 +++++++++++++--------
src/allmydata/test/test_mutable.py | 6 +++---
2 files changed, 16 insertions(+), 11 deletions(-)
commit 348f57988f79389db0aab7672e6eaa9a6d8e3219
Author: Brian Warner <warner@lothar.com>
Date: Mon Oct 3 18:10:41 2011 +0100
accessors and name cleanup for servermap.Servermap.bad_shares
src/allmydata/mutable/publish.py | 2 +-
src/allmydata/mutable/servermap.py | 30 ++++++++++++++-----------
2 files changed, 18 insertions(+), 14 deletions(-)
commit 520c9368134673cdf76c653c5e1bb91c2ab5d51e
Author: Brian Warner <warner@lothar.com>
Date: Mon Oct 3 18:10:05 2011 +0100
accessors and name cleanup for servermap.Servermap.servermap .
src/allmydata/mutable/publish.py | 14 +++++----
src/allmydata/mutable/servermap.py | 38 ++++++++++++++-----------
2 files changed, 29 insertions(+), 23 deletions(-)
commit b8b8dc38287a91dbdf494426ac801d9381ce5841
Author: Brian Warner <warner@lothar.com>
Date: Mon Oct 3 18:08:02 2011 +0100
fix reachable_servers
src/allmydata/mutable/checker.py | 3 ++-
src/allmydata/mutable/publish.py | 4 +++-
src/allmydata/mutable/servermap.py | 12 ++++++++++--
3 files changed, 15 insertions(+), 4 deletions(-)
commit cb0cfd1adfefad357c187aaaf690c3df68b622bc
Author: Brian Warner <warner@lothar.com>
Date: Mon Oct 3 18:06:03 2011 +0100
fix Servermap.unreachable_servers
src/allmydata/mutable/servermap.py | 11 ++++++++---
1 files changed, 8 insertions(+), 3 deletions(-)
commit 2d9ea79b94bd4db674d40386fda90825785ac495
Author: Brian Warner <warner@lothar.com>
Date: Mon Oct 3 18:03:48 2011 +0100
give ServerMap a StorageFarmBroker, temporary
this makes it possible for the ServerMap to accept bare serverids and still
build data structures with IServers
src/allmydata/mutable/checker.py | 2 +-
src/allmydata/mutable/filenode.py | 2 +-
src/allmydata/mutable/publish.py | 2 +-
src/allmydata/mutable/servermap.py | 5 +++--
src/allmydata/test/test_mutable.py | 8 ++++----
5 files changed, 10 insertions(+), 9 deletions(-)
commit 718d1aeff6fded893f65397806d22ece928b0dd4
Author: Brian Warner <warner@lothar.com>
Date: Mon Oct 3 13:43:30 2011 -0400
add StorageFarmBroker.get_server_for_id(), temporary helper
This will go away once we're passing IServers everywhere.
src/allmydata/storage_client.py | 2 ++
src/allmydata/test/no_network.py | 13 +++++++++++++
2 files changed, 15 insertions(+), 0 deletions(-)
commit ece20231d7fda0d503704842a4aa068dfbc2e54e
Author: Brian Warner <warner@lothar.com>
Date: Sun Oct 2 01:11:50 2011 +0100
add proper accessors for Servermap.connections, to make refactoring easier
src/allmydata/mutable/publish.py | 6 +++---
src/allmydata/mutable/retrieve.py | 10 +++++-----
src/allmydata/mutable/servermap.py | 17 +++++++++++------
3 files changed, 19 insertions(+), 14 deletions(-)
commit 3b943d6bf302ff702668081a612fc4fe2604cf9c
Author: Brian Warner <warner@lothar.com>
Date: Fri Sep 23 10:34:30 2011 -0700
mutable/servermap.py and neighbors: s/peer/server/
src/allmydata/mutable/checker.py | 22 +-
src/allmydata/mutable/publish.py | 204 +++++++-------
src/allmydata/mutable/servermap.py | 402 +++++++++++++-------------
src/allmydata/test/test_mutable.py | 18 +-
4 files changed, 323 insertions(+), 323 deletions(-)
IServer refactoring: pass IServer instances around, instead of peerids
refs #1363
This collapses 88 small incremental changes (each of which passes all tests)
into one big patch. The development process for the long path started with
adding some temporary scaffolding, changing one method at a time, then
removing the scaffolding. The individual pieces are as follows, in reverse
chronological order (the first patch is at the end of this comment):
commit 9bbe4174fd0d98a6cf47a8ef96e85d9ef34b2f9a
Author: Brian Warner <warner@lothar.com>
Date: Tue Oct 4 16:05:00 2011 -0400
immutable/downloader/status.py: correct comment
src/allmydata/immutable/downloader/status.py | 2 +-
1 files changed, 1 insertions(+), 1 deletions(-)
commit 72146a7c7c91eac2f7c3ceb801eb7a1721376889
Author: Brian Warner <warner@lothar.com>
Date: Tue Oct 4 15:46:20 2011 -0400
remove temporary ServerMap._storage_broker
src/allmydata/mutable/checker.py | 2 +-
src/allmydata/mutable/filenode.py | 2 +-
src/allmydata/mutable/publish.py | 2 +-
src/allmydata/mutable/servermap.py | 5 ++---
src/allmydata/test/test_mutable.py | 8 ++++----
5 files changed, 9 insertions(+), 10 deletions(-)
commit d703096b41632c47d76414b12672e076a422ff5c
Author: Brian Warner <warner@lothar.com>
Date: Tue Oct 4 15:37:05 2011 -0400
remove temporary storage_broker.get_server_for_id()
src/allmydata/storage_client.py | 3 ---
src/allmydata/test/no_network.py | 13 -------------
2 files changed, 0 insertions(+), 16 deletions(-)
commit 620cc5d80882ef6f7decfd26af8a6c7c1ddf80d1
Author: Brian Warner <warner@lothar.com>
Date: Tue Oct 4 12:50:06 2011 -0400
API of Retrieve._try_to_validate_privkey(), trying to remove reader.server
src/allmydata/mutable/retrieve.py | 10 +++++-----
1 files changed, 5 insertions(+), 5 deletions(-)
commit 92f43f856f4a8b36c207d1b190ed8699b5a4ecb4
Author: Brian Warner <warner@lothar.com>
Date: Tue Oct 4 12:48:08 2011 -0400
API of Retrieve._validate_block(), trying to remove reader.server
src/allmydata/mutable/retrieve.py | 14 +++++++-------
1 files changed, 7 insertions(+), 7 deletions(-)
commit 572d5070761861a2190349d1ed8d85dbc25698a5
Author: Brian Warner <warner@lothar.com>
Date: Tue Oct 4 12:36:58 2011 -0400
API of Retrieve._mark_bad_share(), trying to remove reader.server
src/allmydata/mutable/retrieve.py | 21 +++++++++------------
1 files changed, 9 insertions(+), 12 deletions(-)
commit a793ff00c0de1e2eec7b46288fdf388c7a2bec89
Author: Brian Warner <warner@lothar.com>
Date: Tue Oct 4 12:06:13 2011 -0400
remove now-unused get_rref_for_serverid()
src/allmydata/mutable/servermap.py | 3 ---
1 files changed, 0 insertions(+), 3 deletions(-)
commit 1b9827cc9366bf90b93297fdd6832f2ad0480ce7
Author: Brian Warner <warner@lothar.com>
Date: Tue Oct 4 12:03:09 2011 -0400
Retrieve: stop adding .serverid attributes to readers
src/allmydata/mutable/retrieve.py | 1 -
1 files changed, 0 insertions(+), 1 deletions(-)
commit 5d4e9d491b19e49d2e443a1dfff2c672842c36ef
Author: Brian Warner <warner@lothar.com>
Date: Tue Oct 4 12:03:34 2011 -0400
return value of Retrieve(verify=True)
src/allmydata/mutable/checker.py | 11 ++++++-----
src/allmydata/mutable/retrieve.py | 3 +--
2 files changed, 7 insertions(+), 7 deletions(-)
commit e9ab7978c384e1f677cb7779dc449b1044face82
Author: Brian Warner <warner@lothar.com>
Date: Tue Oct 4 11:54:23 2011 -0400
Retrieve._bad_shares (but not return value, used by Verifier)
src/allmydata/mutable/retrieve.py | 7 ++++---
1 files changed, 4 insertions(+), 3 deletions(-)
commit 2d91926de233ec5c881f30e36b4a30ad92ab42a9
Author: Brian Warner <warner@lothar.com>
Date: Tue Oct 4 11:51:23 2011 -0400
Publish: stop adding .serverid attributes to writers
src/allmydata/mutable/publish.py | 9 ++-------
1 files changed, 2 insertions(+), 7 deletions(-)
commit 47c7a0105dec7cbf4f7e0a3ce800bbb85b15df4a
Author: Brian Warner <warner@lothar.com>
Date: Tue Oct 4 11:56:33 2011 -0400
API of get_write_enabler()
src/allmydata/mutable/filenode.py | 7 ++++---
src/allmydata/mutable/publish.py | 4 ++--
src/allmydata/test/no_network.py | 3 +++
3 files changed, 9 insertions(+), 5 deletions(-)
commit 9196a5c6590fdbfd660325ea8358b345887d3db0
Author: Brian Warner <warner@lothar.com>
Date: Tue Oct 4 11:46:24 2011 -0400
API of get_(renewal|cancel)_secret()
src/allmydata/mutable/filenode.py | 14 ++++++++------
src/allmydata/mutable/publish.py | 8 ++++----
src/allmydata/mutable/servermap.py | 5 ++---
3 files changed, 14 insertions(+), 13 deletions(-)
commit de7c1552f8c163eff5b6d820b5fb3b21c1b47cb5
Author: Brian Warner <warner@lothar.com>
Date: Tue Oct 4 11:41:52 2011 -0400
API of CorruptShareError. Also comment out some related+unused test_web.py code
src/allmydata/mutable/common.py | 13 +++++--------
src/allmydata/mutable/retrieve.py | 10 +++++-----
src/allmydata/mutable/servermap.py | 8 +++-----
src/allmydata/test/common.py | 13 ++++++++-----
4 files changed, 21 insertions(+), 23 deletions(-)
commit 2c1c314046b620c16f1e66d030c150d768b7d01e
Author: Brian Warner <warner@lothar.com>
Date: Tue Oct 4 12:01:46 2011 -0400
API of ServerMap.mark_bad_share()
src/allmydata/mutable/publish.py | 2 +-
src/allmydata/mutable/retrieve.py | 6 +++---
src/allmydata/mutable/servermap.py | 6 ++----
src/allmydata/test/test_mutable.py | 3 +--
4 files changed, 7 insertions(+), 10 deletions(-)
commit 1bed349030779fd0c378ae4e821384f953c6f6ff
Author: Brian Warner <warner@lothar.com>
Date: Tue Oct 4 11:11:17 2011 -0400
API+name of ServerMap.shares_on_server() : only for tests, so debug_ prefix
src/allmydata/mutable/servermap.py | 7 ++-----
src/allmydata/test/test_mutable.py | 6 +++---
2 files changed, 5 insertions(+), 8 deletions(-)
commit 2d32e448677d6b818692e801045d4115b29abf21
Author: Brian Warner <warner@lothar.com>
Date: Tue Oct 4 11:07:10 2011 -0400
API of ServerMap.all_servers_for_version()
src/allmydata/mutable/servermap.py | 4 ++--
1 files changed, 2 insertions(+), 2 deletions(-)
commit 48f3204d1889c3e7179578125c4bdef515af3d6a
Author: Brian Warner <warner@lothar.com>
Date: Tue Oct 4 11:04:50 2011 -0400
internals of ServerMap methods that use make_versionmap(), remove temp copy
src/allmydata/mutable/servermap.py | 28 +++++++++----------------
1 files changed, 10 insertions(+), 18 deletions(-)
commit 5c3da77b6c777a145bd5ddfaa4db849dc9495548
Author: Brian Warner <warner@lothar.com>
Date: Tue Oct 4 11:01:28 2011 -0400
API of ServerMap.make_versionmap()
src/allmydata/mutable/checker.py | 4 ++--
src/allmydata/mutable/retrieve.py | 5 ++---
src/allmydata/mutable/servermap.py | 4 ++--
src/allmydata/test/test_mutable.py | 7 ++++---
4 files changed, 10 insertions(+), 10 deletions(-)
commit b6882ece49afb4c507d118af2db346fa329209dc
Author: Brian Warner <warner@lothar.com>
Date: Tue Oct 4 10:53:38 2011 -0400
make a copy of ServerMap.make_versionmap() (_make_versionmap2) for internal use
src/allmydata/mutable/servermap.py | 18 +++++++++++++-----
1 files changed, 13 insertions(+), 5 deletions(-)
commit 963f8e63faf32b950eb1b8103cd2ff16fe8f0151
Author: Brian Warner <warner@lothar.com>
Date: Tue Oct 4 00:45:58 2011 -0400
API of RetrieveStatus.add_problem()
src/allmydata/mutable/retrieve.py | 5 +++--
1 files changed, 3 insertions(+), 2 deletions(-)
commit 4976d29ffae565a048851601c29013bbae2976d8
Author: Brian Warner <warner@lothar.com>
Date: Tue Oct 4 00:45:05 2011 -0400
API of RetrieveStatus.add_fetch_timing()
src/allmydata/mutable/retrieve.py | 5 +++--
1 files changed, 3 insertions(+), 2 deletions(-)
commit d057d3bbba72663ee148a8b916bc2d52be2e3982
Author: Brian Warner <warner@lothar.com>
Date: Tue Oct 4 00:44:04 2011 -0400
API of Retrieve.notify_server_corruption()
src/allmydata/mutable/retrieve.py | 6 +++---
1 files changed, 3 insertions(+), 3 deletions(-)
commit 8a2a81e46671c860610e0e96d6add1a57551f22d
Author: Brian Warner <warner@lothar.com>
Date: Tue Oct 4 00:42:32 2011 -0400
remove unused _outstanding_queries
src/allmydata/mutable/retrieve.py | 1 -
1 files changed, 0 insertions(+), 1 deletions(-)
commit 56d12cc9968d03ccd53764455c671122c4f391d1
Author: Brian Warner <warner@lothar.com>
Date: Tue Oct 4 00:40:57 2011 -0400
change Retrieve.remaining_sharemap
src/allmydata/mutable/retrieve.py | 4 ++--
1 files changed, 2 insertions(+), 2 deletions(-)
commit 4f0b7af4821f43290bfc70f2b1fc30149ad81281
Author: Brian Warner <warner@lothar.com>
Date: Tue Oct 4 10:40:18 2011 -0400
accessor for PublishStatus._problems
src/allmydata/mutable/publish.py | 4 +++-
src/allmydata/web/status.py | 2 +-
2 files changed, 4 insertions(+), 2 deletions(-)
commit 627087cf66d0b8cc519f4d551a967a7bd9b6a741
Author: Brian Warner <warner@lothar.com>
Date: Tue Oct 4 10:36:39 2011 -0400
accessor for RetrieveStatus._problems
src/allmydata/mutable/retrieve.py | 8 ++++++--
src/allmydata/web/status.py | 2 +-
2 files changed, 7 insertions(+), 3 deletions(-)
commit ca7dea81f03801b1c7353fc00ecba689268109cf
Author: Brian Warner <warner@lothar.com>
Date: Tue Oct 4 00:35:32 2011 -0400
add .server to "reader", so we can get at it later
src/allmydata/mutable/retrieve.py | 5 +++--
1 files changed, 3 insertions(+), 2 deletions(-)
commit 6ef516e24908ec195af084a7550d1921a5e983b0
Author: Brian Warner <warner@lothar.com>
Date: Tue Oct 4 00:32:32 2011 -0400
temporarily give Retrieve a _storage_broker, so it can map serverids to servers
src/allmydata/mutable/checker.py | 3 ++-
src/allmydata/mutable/filenode.py | 6 ++++--
src/allmydata/mutable/retrieve.py | 5 +++--
src/allmydata/test/test_mutable.py | 4 ++--
4 files changed, 11 insertions(+), 7 deletions(-)
commit afe08e4dd3f4ff9ff7e8a2a8d28b181e3625bcc9
Author: Brian Warner <warner@lothar.com>
Date: Tue Oct 4 00:21:51 2011 -0400
mutable/retrieve.py: s/peer/server/
src/allmydata/mutable/retrieve.py | 82 +++++++++++++-------------
src/allmydata/test/test_mutable.py | 6 +-
2 files changed, 44 insertions(+), 44 deletions(-)
commit 910afcb5d7f274880f68dd6cdb5b05f2bbc29adc
Author: Brian Warner <warner@lothar.com>
Date: Tue Oct 4 00:16:01 2011 -0400
web.status.PublishStatusPage: add comment, I think .problems isn't exercised
src/allmydata/web/status.py | 2 ++
1 files changed, 2 insertions(+), 0 deletions(-)
commit 311466dd8c931bbba40d590ade867704282e7f1a
Author: Brian Warner <warner@lothar.com>
Date: Mon Oct 3 23:48:16 2011 -0400
API of PublishStatus.add_per_server_time()
src/allmydata/mutable/publish.py | 5 +++--
1 files changed, 3 insertions(+), 2 deletions(-)
commit 2df5faa1b6cbfbaded520d2320305a62fe961118
Author: Brian Warner <warner@lothar.com>
Date: Mon Oct 3 23:46:37 2011 -0400
more simplifications
src/allmydata/mutable/publish.py | 4 +---
1 files changed, 1 insertions(+), 3 deletions(-)
commit 6ac4544a3da385f2aad9392f906b90192f4f919a
Author: Brian Warner <warner@lothar.com>
Date: Mon Oct 3 23:44:08 2011 -0400
API of ServerMap.version_on_server()
src/allmydata/mutable/publish.py | 2 +-
src/allmydata/mutable/servermap.py | 4 ++--
src/allmydata/test/test_mutable.py | 5 ++---
3 files changed, 5 insertions(+), 6 deletions(-)
commit 3e187e322511072e4683329df6b2c6c733a66dba
Author: Brian Warner <warner@lothar.com>
Date: Tue Oct 4 00:16:32 2011 -0400
API of ServerMap.make_sharemap()
src/allmydata/mutable/servermap.py | 4 ++--
src/allmydata/test/test_mutable.py | 7 ++++---
src/allmydata/web/status.py | 4 ++--
3 files changed, 8 insertions(+), 7 deletions(-)
commit 318feed8437bdd8d4943c6569d38f7b54b6313cc
Author: Brian Warner <warner@lothar.com>
Date: Mon Oct 3 23:36:19 2011 -0400
small cleanups
src/allmydata/mutable/publish.py | 4 ++--
1 files changed, 2 insertions(+), 2 deletions(-)
commit bd459ed5714e1db5a7163935c54b7b0b56db8349
Author: Brian Warner <warner@lothar.com>
Date: Mon Oct 3 23:33:39 2011 -0400
API of ServerMap.add_new_share()
src/allmydata/mutable/publish.py | 4 ++--
src/allmydata/mutable/servermap.py | 6 ++----
2 files changed, 4 insertions(+), 6 deletions(-)
commit f2804fb6ed11d80088e0da8ed48e6c2922f2ffef
Author: Brian Warner <warner@lothar.com>
Date: Mon Oct 3 23:30:26 2011 -0400
API of ServerMap.get_bad_shares()
src/allmydata/mutable/publish.py | 3 +--
src/allmydata/mutable/servermap.py | 9 ++++-----
2 files changed, 5 insertions(+), 7 deletions(-)
commit 965074a47b3ce1431cb46d9a233840afcf9105f5
Author: Brian Warner <warner@lothar.com>
Date: Mon Oct 3 23:26:58 2011 -0400
more small cleanups
src/allmydata/mutable/publish.py | 6 +++---
1 files changed, 3 insertions(+), 3 deletions(-)
commit 38020da34f034f8889947dd3dc05e087ffff7106
Author: Brian Warner <warner@lothar.com>
Date: Mon Oct 3 23:18:47 2011 -0400
change Publish.bad_share_checkstrings
src/allmydata/mutable/publish.py | 6 +++---
1 files changed, 3 insertions(+), 3 deletions(-)
commit 5efebcbd2ee0c2f299ea86f7591d856c0f265304
Author: Brian Warner <warner@lothar.com>
Date: Mon Oct 3 23:16:31 2011 -0400
change internals of Publish.update_goal()
src/allmydata/mutable/publish.py | 8 +++-----
1 files changed, 3 insertions(+), 5 deletions(-)
commit e91b55ff4c2a69165b71f2c7b217ac319ff4c527
Author: Brian Warner <warner@lothar.com>
Date: Mon Oct 3 23:11:42 2011 -0400
get rid of Publish.connections
src/allmydata/mutable/publish.py | 27 +++++----------------------
1 files changed, 5 insertions(+), 22 deletions(-)
commit 64e9a53b3229ebe2f9ebf7ed502d539311d0e037
Author: Brian Warner <warner@lothar.com>
Date: Mon Oct 3 23:05:32 2011 -0400
change Publish.bad_servers
src/allmydata/mutable/publish.py | 10 +++++-----
1 files changed, 5 insertions(+), 5 deletions(-)
commit b85a934bef315a06bcfe00c9c12a3627fed2b918
Author: Brian Warner <warner@lothar.com>
Date: Mon Oct 3 23:03:07 2011 -0400
Publish.bad_servers: fix bug, this should be a set of serverids, not writers
src/allmydata/mutable/publish.py | 2 +-
1 files changed, 1 insertions(+), 1 deletions(-)
commit 605ea15ec15ed671513819003ccd211cdb9761e0
Author: Brian Warner <warner@lothar.com>
Date: Mon Oct 3 23:00:21 2011 -0400
change .placed
src/allmydata/mutable/publish.py | 6 +++---
1 files changed, 3 insertions(+), 3 deletions(-)
commit f7aba37b1b345d5b6d5cb16e3b3f6f3c1afb658e
Author: Brian Warner <warner@lothar.com>
Date: Mon Oct 3 22:59:22 2011 -0400
temporarily stash IServer as .server on the "writer" object
src/allmydata/mutable/publish.py | 2 ++
1 files changed, 2 insertions(+), 0 deletions(-)
commit f9b551d788e7db1f187fce5ab98ab5d5fe4e1c36
Author: Brian Warner <warner@lothar.com>
Date: Mon Oct 3 22:48:18 2011 -0400
change Publish.goal and API of log_goal() to use IServer, not serverid
src/allmydata/mutable/publish.py | 48 ++++++++++++++--------------
1 files changed, 24 insertions(+), 24 deletions(-)
commit 75f20616558e4900b8b1f685dd99aa838de6d452
Author: Brian Warner <warner@lothar.com>
Date: Mon Oct 3 15:27:02 2011 -0400
API of ServerMap.get_known_shares()
src/allmydata/mutable/publish.py | 16 ++++++++++------
src/allmydata/mutable/servermap.py | 7 ++-----
2 files changed, 12 insertions(+), 11 deletions(-)
commit 1c38c9d37bb08221b4418762234b1a62397b3b4b
Author: Brian Warner <warner@lothar.com>
Date: Mon Oct 3 15:20:29 2011 -0400
Publish.full_serverlist
src/allmydata/mutable/publish.py | 10 +++++-----
1 files changed, 5 insertions(+), 5 deletions(-)
commit b6cbd215a04b9cde31a7d92a97a7f048622b16f1
Author: Brian Warner <warner@lothar.com>
Date: Mon Oct 3 15:12:31 2011 -0400
API of ServerMap.all_servers()
src/allmydata/mutable/servermap.py | 19 ++++++-------------
1 files changed, 6 insertions(+), 13 deletions(-)
commit e63cd0315fae65357b1727ec6d5ff3c6e0d27c98
Author: Brian Warner <warner@lothar.com>
Date: Mon Oct 3 15:10:18 2011 -0400
remove ServerMap.connections, set_rref_for_serverid()
src/allmydata/mutable/servermap.py | 11 +----------
1 files changed, 1 insertions(+), 10 deletions(-)
commit 4df52db2f80eb12eefa5d57103c24893cde89553
Author: Brian Warner <warner@lothar.com>
Date: Mon Oct 3 15:04:06 2011 -0400
API of ServerMap.mark_server_reachable()
src/allmydata/mutable/servermap.py | 7 ++-----
1 files changed, 2 insertions(+), 5 deletions(-)
commit 69c715bde77944dc25181b3dbbeb042c816f9a1b
Author: Brian Warner <warner@lothar.com>
Date: Mon Oct 3 15:03:21 2011 -0400
API of ServerMap.mark_server_unreachable()
src/allmydata/mutable/servermap.py | 9 +++------
1 files changed, 3 insertions(+), 6 deletions(-)
commit 3d784d60eec1c508858e3a617e4411ffbcc3c1fa
Author: Brian Warner <warner@lothar.com>
Date: Mon Oct 3 15:02:03 2011 -0400
API of status.set_privkey_from()
src/allmydata/mutable/servermap.py | 7 +++----
1 files changed, 3 insertions(+), 4 deletions(-)
commit 544ed3ea29bed7e66da7fd29ca3f6f076f27a9e6
Author: Brian Warner <warner@lothar.com>
Date: Mon Oct 3 15:01:15 2011 -0400
API of status.add_per_server_time()
src/allmydata/mutable/servermap.py | 7 ++++---
1 files changed, 4 insertions(+), 3 deletions(-)
commit fffe5008b6320bd1e04c3c68389a2bf2ee383fa8
Author: Brian Warner <warner@lothar.com>
Date: Mon Oct 3 14:59:02 2011 -0400
remove unused .versionmap
src/allmydata/mutable/servermap.py | 7 -------
1 files changed, 0 insertions(+), 7 deletions(-)
commit 2816562e090d2294179db3588dafcca18de1bc2b
Author: Brian Warner <warner@lothar.com>
Date: Mon Oct 3 14:57:51 2011 -0400
remove serverid from all log messages. Also one unused lambda.
src/allmydata/mutable/servermap.py | 30 +++++++++++++-------------
1 files changed, 15 insertions(+), 15 deletions(-)
commit 28fa6b1a2738fa98c1f1dbd3d0e01ae98912d11f
Author: Brian Warner <warner@lothar.com>
Date: Mon Oct 3 14:54:30 2011 -0400
removed unused _readers
src/allmydata/mutable/servermap.py | 3 ---
1 files changed, 0 insertions(+), 3 deletions(-)
commit a8e4ed3d645ab592d1add6a1e69b6d1ebfb77817
Author: Brian Warner <warner@lothar.com>
Date: Mon Oct 3 14:54:16 2011 -0400
remove unused _sharemap
src/allmydata/mutable/servermap.py | 1 -
1 files changed, 0 insertions(+), 1 deletions(-)
commit 3f072e55cf1d0700f9fffe23f8f3a475725df588
Author: Brian Warner <warner@lothar.com>
Date: Mon Oct 3 14:49:03 2011 -0400
_must_query
src/allmydata/mutable/servermap.py | 8 ++++----
1 files changed, 4 insertions(+), 4 deletions(-)
commit c599a059b8df3f5785e4bf89fb6ecc6d8dcd708b
Author: Brian Warner <warner@lothar.com>
Date: Mon Oct 3 14:48:05 2011 -0400
_queries_outstanding
src/allmydata/mutable/servermap.py | 16 +++++++---------
1 files changed, 7 insertions(+), 9 deletions(-)
commit 7743759f98ac2c07926b2fdbd80bf52dfab33085
Author: Brian Warner <warner@lothar.com>
Date: Mon Oct 3 14:46:17 2011 -0400
_empty_servers
src/allmydata/mutable/servermap.py | 5 ++---
1 files changed, 2 insertions(+), 3 deletions(-)
commit 6bb1825916828a713a32cdf7f7411fa3ea2e1e5d
Author: Brian Warner <warner@lothar.com>
Date: Mon Oct 3 14:45:39 2011 -0400
_good_servers
src/allmydata/mutable/servermap.py | 4 ++--
1 files changed, 2 insertions(+), 2 deletions(-)
commit 1768fab1b51d8dd93ecabbaaabfadfa20cf6c3d4
Author: Brian Warner <warner@lothar.com>
Date: Mon Oct 3 14:44:59 2011 -0400
_bad_servers
src/allmydata/mutable/servermap.py | 14 +++++++-------
1 files changed, 7 insertions(+), 7 deletions(-)
commit dccbaef30f0ba714c746bf6d4a1a803c36e17b65
Author: Brian Warner <warner@lothar.com>
Date: Mon Oct 3 14:41:54 2011 -0400
API of _try_to_set_pubkey()
src/allmydata/mutable/servermap.py | 7 ++++---
1 files changed, 4 insertions(+), 3 deletions(-)
commit 0481ea70042ba3575f15eac7fd0780f8ece580cc
Author: Brian Warner <warner@lothar.com>
Date: Mon Oct 3 14:35:02 2011 -0400
API of notify_server_corruption()
src/allmydata/mutable/servermap.py | 6 +++---
1 files changed, 3 insertions(+), 3 deletions(-)
commit bea9cba18fb3b9c11bb22f18356a263ecec7351e
Author: Brian Warner <warner@lothar.com>
Date: Mon Oct 3 14:34:09 2011 -0400
API of _got_signature_one_share()
src/allmydata/mutable/servermap.py | 9 +++++----
1 files changed, 5 insertions(+), 4 deletions(-)
commit 1520123583cf78650706e114b15bb5b0ac1f4a14
Author: Brian Warner <warner@lothar.com>
Date: Mon Oct 3 14:32:33 2011 -0400
API of _try_to_validate_privkey()
src/allmydata/mutable/servermap.py | 9 +++++----
1 files changed, 5 insertions(+), 4 deletions(-)
commit 938852c9c8519c7a078f58a9b1f4dd8ec8b6715e
Author: Brian Warner <warner@lothar.com>
Date: Mon Oct 3 14:31:48 2011 -0400
API and internals of _add_lease_failed()
src/allmydata/mutable/servermap.py | 8 ++++----
1 files changed, 4 insertions(+), 4 deletions(-)
commit 3843dba367e3c19e176a622ab853cb51d2472ddf
Author: Brian Warner <warner@lothar.com>
Date: Mon Oct 3 14:30:37 2011 -0400
API of _privkey_query_failed()
src/allmydata/mutable/servermap.py | 5 +++--
1 files changed, 3 insertions(+), 2 deletions(-)
commit 2219a710e1633cd57d0ca0786490de87b3e19ba7
Author: Brian Warner <warner@lothar.com>
Date: Mon Oct 3 14:29:43 2011 -0400
fix bug in call to _privkey_query_failed, unrelated to refactoring
src/allmydata/mutable/servermap.py | 2 +-
1 files changed, 1 insertions(+), 1 deletions(-)
commit ae615bec7d0d1b269710b6902797b12f9592ad62
Author: Brian Warner <warner@lothar.com>
Date: Mon Oct 3 14:27:17 2011 -0400
API of _got_corrupt_share()
src/allmydata/mutable/servermap.py | 17 +++++++++--------
1 files changed, 9 insertions(+), 8 deletions(-)
commit cb51c95a6f4e077278157a77dab060c8c1ad7a81
Author: Brian Warner <warner@lothar.com>
Date: Mon Oct 3 14:23:16 2011 -0400
API of _got_results()
src/allmydata/mutable/servermap.py | 9 +++++----
1 files changed, 5 insertions(+), 4 deletions(-)
commit bac9154fe0af18f226999a58ffc2362d8cf4b802
Author: Brian Warner <warner@lothar.com>
Date: Mon Oct 3 14:19:19 2011 -0400
API of _query_failed()
src/allmydata/mutable/servermap.py | 5 +++--
1 files changed, 3 insertions(+), 2 deletions(-)
commit fdc29a8ca95d4b5c503e5382b9e5d4d02141ba12
Author: Brian Warner <warner@lothar.com>
Date: Mon Oct 3 14:17:20 2011 -0400
API of _do_read()
src/allmydata/mutable/servermap.py | 6 ++++--
1 files changed, 4 insertions(+), 2 deletions(-)
commit e7e9e338f28d004aa4d423d11c65f1e271ac7322
Author: Brian Warner <warner@lothar.com>
Date: Mon Oct 3 14:20:21 2011 -0400
API of _do_query()
src/allmydata/mutable/servermap.py | 15 +++++++--------
1 files changed, 7 insertions(+), 8 deletions(-)
commit 330625b9dac4cdbe72a11464a893065b9aeed453
Author: Brian Warner <warner@lothar.com>
Date: Mon Oct 3 14:43:05 2011 -0400
next step: first batch of updates to ServermapUpdater
updates:
most method-local variables in update()
API of _build_initial_querylist()
API of _send_initial_requests()
.full_serverlist
.extra_servers
src/allmydata/mutable/servermap.py | 39 ++++++++++++++------------
1 files changed, 21 insertions(+), 18 deletions(-)
commit 4aadc584fa7dcb2daa86b048c81dee0049ba26d9
Author: Brian Warner <warner@lothar.com>
Date: Mon Oct 3 15:07:00 2011 -0400
internal change: index _bad_shares with IServer
src/allmydata/mutable/servermap.py | 20 ++++++++++----------
1 files changed, 10 insertions(+), 10 deletions(-)
commit 16d4e6fa82a9907dbdc92094213387c6a4164e41
Author: Brian Warner <warner@lothar.com>
Date: Mon Oct 3 18:20:47 2011 +0100
internal change: index _known_shares with IServer instead of serverid
callers are unchanged
src/allmydata/mutable/servermap.py | 42 +++++++++++++++----------
1 files changed, 25 insertions(+), 17 deletions(-)
commit ceeb5f4938cc814a0c75d1b8f4018aed965c2176
Author: Brian Warner <warner@lothar.com>
Date: Mon Oct 3 18:11:43 2011 +0100
accessors and name cleanup for servermap.Servermap.last_update_mode/time
src/allmydata/mutable/filenode.py | 6 +++---
src/allmydata/mutable/publish.py | 4 ++--
src/allmydata/mutable/servermap.py | 17 +++++++++++------
3 files changed, 16 insertions(+), 11 deletions(-)
commit 8d3cbda82661c0a7e5c3d3b65cf7a5d5ab7e32c0
Author: Brian Warner <warner@lothar.com>
Date: Mon Oct 3 18:11:14 2011 +0100
accessors and name cleanup for servermap.Servermap.problems
src/allmydata/mutable/servermap.py | 21 +++++++++++++--------
src/allmydata/test/test_mutable.py | 6 +++---
2 files changed, 16 insertions(+), 11 deletions(-)
commit 348f57988f79389db0aab7672e6eaa9a6d8e3219
Author: Brian Warner <warner@lothar.com>
Date: Mon Oct 3 18:10:41 2011 +0100
accessors and name cleanup for servermap.Servermap.bad_shares
src/allmydata/mutable/publish.py | 2 +-
src/allmydata/mutable/servermap.py | 30 ++++++++++++++-----------
2 files changed, 18 insertions(+), 14 deletions(-)
commit 520c9368134673cdf76c653c5e1bb91c2ab5d51e
Author: Brian Warner <warner@lothar.com>
Date: Mon Oct 3 18:10:05 2011 +0100
accessors and name cleanup for servermap.Servermap.servermap .
src/allmydata/mutable/publish.py | 14 +++++----
src/allmydata/mutable/servermap.py | 38 ++++++++++++++-----------
2 files changed, 29 insertions(+), 23 deletions(-)
commit b8b8dc38287a91dbdf494426ac801d9381ce5841
Author: Brian Warner <warner@lothar.com>
Date: Mon Oct 3 18:08:02 2011 +0100
fix reachable_servers
src/allmydata/mutable/checker.py | 3 ++-
src/allmydata/mutable/publish.py | 4 +++-
src/allmydata/mutable/servermap.py | 12 ++++++++++--
3 files changed, 15 insertions(+), 4 deletions(-)
commit cb0cfd1adfefad357c187aaaf690c3df68b622bc
Author: Brian Warner <warner@lothar.com>
Date: Mon Oct 3 18:06:03 2011 +0100
fix Servermap.unreachable_servers
src/allmydata/mutable/servermap.py | 11 ++++++++---
1 files changed, 8 insertions(+), 3 deletions(-)
commit 2d9ea79b94bd4db674d40386fda90825785ac495
Author: Brian Warner <warner@lothar.com>
Date: Mon Oct 3 18:03:48 2011 +0100
give ServerMap a StorageFarmBroker, temporary
this makes it possible for the ServerMap to accept bare serverids and still
build data structures with IServers
src/allmydata/mutable/checker.py | 2 +-
src/allmydata/mutable/filenode.py | 2 +-
src/allmydata/mutable/publish.py | 2 +-
src/allmydata/mutable/servermap.py | 5 +++--
src/allmydata/test/test_mutable.py | 8 ++++----
5 files changed, 10 insertions(+), 9 deletions(-)
commit 718d1aeff6fded893f65397806d22ece928b0dd4
Author: Brian Warner <warner@lothar.com>
Date: Mon Oct 3 13:43:30 2011 -0400
add StorageFarmBroker.get_server_for_id(), temporary helper
This will go away once we're passing IServers everywhere.
src/allmydata/storage_client.py | 2 ++
src/allmydata/test/no_network.py | 13 +++++++++++++
2 files changed, 15 insertions(+), 0 deletions(-)
commit ece20231d7fda0d503704842a4aa068dfbc2e54e
Author: Brian Warner <warner@lothar.com>
Date: Sun Oct 2 01:11:50 2011 +0100
add proper accessors for Servermap.connections, to make refactoring easier
src/allmydata/mutable/publish.py | 6 +++---
src/allmydata/mutable/retrieve.py | 10 +++++-----
src/allmydata/mutable/servermap.py | 17 +++++++++++------
3 files changed, 19 insertions(+), 14 deletions(-)
commit 3b943d6bf302ff702668081a612fc4fe2604cf9c
Author: Brian Warner <warner@lothar.com>
Date: Fri Sep 23 10:34:30 2011 -0700
mutable/servermap.py and neighbors: s/peer/server/
src/allmydata/mutable/checker.py | 22 +-
src/allmydata/mutable/publish.py | 204 +++++++-------
src/allmydata/mutable/servermap.py | 402 +++++++++++++-------------
src/allmydata/test/test_mutable.py | 18 +-
4 files changed, 323 insertions(+), 323 deletions(-)
2011-10-31 21:03:19 -07:00
david-sarah
bc0d9b682e
immutable/literal.py: add pauseProducing method to LiteralProducer. refs #1537
2011-10-03 12:52:39 -07:00
Brian Warner
748e419a9b
move DownloadStopped from download.common to interfaces
2011-09-09 11:11:50 -07:00
Brian Warner
fd676a5846
Let Uploader retain History instead of passing it into upload(). Fixes #1079 .
...
This consistently records all immutable uploads in the Recent Uploads And
Downloads page, regardless of code path. Previously, certain webapi upload
operations (like PUT /uri/$DIRCAP/newchildname) failed to pass the History
object and were left out.
2011-08-28 23:32:46 -07:00
Brian Warner
522868d5e9
MDMF: more writable/writeable consistentifications
2011-08-27 12:06:02 -07:00
Kevan Carstensen
c413a8fae1
immutable/filenode: fix pyflakes warnings
2011-08-06 17:45:14 -07:00
Kevan Carstensen
edf9858fb8
immutable/filenode: implement unified filenode interface
2011-08-01 19:09:05 -07:00
Kevan Carstensen
3e996e0c94
immutable/literal.py: Implement interface changes in literal nodes.
2011-08-01 19:08:14 -07:00
david-sarah
c7f65ee8ad
verifier: correct a bug introduced in changeset [5106] that caused us to only verify the first block of a file. refs #1395
2011-08-02 10:24:37 -07:00
Zooko O'Whielacronx
f426e82287
verifier: serialize the fetching of blocks within a share so that we don't use too much RAM
...
Shares are still verified in parallel, but within a share, don't request a
block until the previous block has been verified and the memory we used to hold
it has been freed up.
Patch originally due to Brian. This version has a mockery-patchery-style test
which is "low tech" (it implements the patching inline in the test code instead
of using an extension of the mock.patch() function from the mock library) and
which unpatches in case of exception.
fixes #1395
2011-08-01 23:37:03 -07:00
Brian Warner
3668cb3d06
remove nodeid from WriteBucketProxy classes and customers
...
refs #1363
2011-08-01 15:43:17 -07:00
Brian Warner
550d67f51f
remove get_serverid() from ReadBucketProxy and customers, including Checker
...
and debug.py dump-share commands
refs #1363
2011-08-01 15:43:07 -07:00
Zooko O'Whielacronx
6b2e798595
remove get_serverid from DownloadStatus.add_dyhb_request and customers
...
This patch is a rebase of a patch originally written by Brian. I didn't change any of the intent of Brian's patch, just ported it to current trunk.
refs #1363
2011-08-01 11:54:01 -07:00
Zooko O'Whielacronx
dc66875479
remove get_serverid from DownloadStatus.add_block_request and customers
...
This is a rebase of a patch Brian originally wrote. I haven't changed the intent of that patch, just ported it to trunk.
refs #1363
2011-08-01 11:53:44 -07:00
Brian Warner
b07af5e1a2
DownloadStatus.add_known_share wants to be used by Finder, web.status
...
refs #1363
2011-08-01 10:44:36 -07:00
Brian Warner
0f11d35f85
replace IServer.name() with get_name(), and get_longname()
...
This patch was originally written by Brian, but was re-recorded by Zooko to use
darcs replace instead of hunks for any file in which it would result in fewer
total hunks.
refs #1363
2011-08-01 10:44:28 -07:00
Zooko O'Whielacronx
880758340f
upload.py: apply David-Sarah's advice rename (un)contacted(2) trackers to first_pass/second_pass/next_pass
...
This patch was written by Brian but was re-recorded by Zooko (with David-Sarah looking on) to use darcs replace instead of editing to rename the three variables to their new names.
refs #1363
2011-08-01 10:41:43 -07:00
Brian Warner
c9def76977
Coalesce multiple Share.loop() calls, make downloads faster. Closes #1268 .
2011-08-01 08:18:34 -07:00
Brian Warner
d8358f2863
add more download-status data, fix tests
2011-06-29 15:25:55 -07:00
Brian Warner
fc5c2208fb
prepare for viz: improve DownloadStatus events
...
consolidate IDownloadStatusHandlingConsumer stuff into DownloadNode
2011-06-29 15:25:42 -07:00
Brian Warner
3834192455
immutable/downloader/fetcher.py: remove all get_serverid() calls
2011-02-26 19:11:56 -07:00
Brian Warner
a4068dd1e0
immutable/downloader/fetcher.py: fix diversity bug in server-response handling
...
When blocks terminate (either COMPLETE or CORRUPT/DEAD/BADSEGNUM), the
_shares_from_server dict was being popped incorrectly (using shnum as the
index instead of serverid). I'm still thinking through the consequences of
this bug. It was probably benign and really hard to detect. I think it would
cause us to incorrectly believe that we're pulling too many shares from a
server, and thus prefer a different server rather than asking for a second
share from the first server. The diversity code is intended to spread out the
number of shares simultaneously being requested from each server, but with
this bug, it might be spreading out the total number of shares requested at
all, not just simultaneously. (note that SegmentFetcher is scoped to a single
segment, so the effect doesn't last very long).
2011-02-26 19:11:53 -07:00
Brian Warner
9ae026d9f4
immutable/downloader/share.py: reduce get_serverid(), one left, update ext deps
...
test_download.py: create+check MyShare instances better, make sure they share
Server objects, now that finder.py cares
2011-02-26 19:11:50 -07:00
Brian Warner
baf1506b9b
immutable/downloader/finder.py: reduce use of get_serverid(), one left
2011-02-26 19:11:46 -07:00
Brian Warner
d9d55ad006
immutable/offloaded.py: reduce use of get_serverid() a bit more
2011-02-26 19:11:42 -07:00
Brian Warner
2f63d9b522
immutable/upload.py: reduce use of get_serverid()
2011-02-26 19:11:38 -07:00
Brian Warner
bdc5cfbdad
immutable/checker.py: remove some uses of s.get_serverid(), not all
2011-02-26 19:11:34 -07:00
Brian Warner
36fabb4f4a
upload.py: rearrange _make_trackers a bit, no behavior changes
2011-02-26 19:11:28 -07:00
Brian Warner
ee14f2ad67
happinessutil.py: finally rename merge_peers to merge_servers
2011-02-26 19:11:24 -07:00
Brian Warner
43488b025c
upload.py: more tracker-vs-server cleanup
2011-02-26 19:11:07 -07:00
Brian Warner
0cf9e3b150
upload.py: fix var names to avoid confusion between 'trackers' and 'servers'
2011-02-26 19:11:03 -07:00
Brian Warner
ebfcb649f9
refactor: s/peer/server/ in immutable/upload, happinessutil.py, test_upload
...
No behavioral changes, just updating variable/method names and log messages.
The effects outside these three files should be minimal: some exception
messages changed (to say "server" instead of "peer"), and some internal class
names were changed. A few things still use "peer" to minimize external
changes, like UploadResults.timings["peer_selection"] and
happinessutil.merge_peers, which can be changed later.
2011-02-26 19:11:00 -07:00
Brian Warner
cba3de04a4
test_client.py, upload.py:: remove KiB/MiB/etc constants, and other dead code
2011-02-26 19:10:51 -07:00
Brian Warner
ffd296fc5a
Refactor StorageFarmBroker handling of servers
...
Pass around IServer instance instead of (peerid, rref) tuple. Replace
"descriptor" with "server". Other replacements:
get_all_servers -> get_connected_servers/get_known_servers
get_servers_for_index -> get_servers_for_psi (now returns IServers)
This change still needs to be pushed further down: lots of code is now
getting the IServer and then distributing (peerid, rref) internally.
Instead, it ought to distribute the IServer internally and delay
extracting a serverid or rref until the last moment.
no_network.py was updated to retain parallelism.
2011-02-20 17:58:04 -08:00
Brian Warner
c18953c169
fix #1223 , crash+inefficiency during repair due to read overrun
...
* repairer (really the uploader) reads beyond end of input file (Uploadable)
* new-downloader does not tolerate overreads
* uploader does lots of tiny reads (inefficient)
This fixes the last two. The uploader still does a single overread at the end
of the input file, but now that's ok so we can leave it in place. The
uploader now expects the Uploadable to behave like a normal disk
file (reading beyond EOF will return less data than was asked for), and now
the new-downloadable behaves that way.
2010-10-29 01:20:36 -07:00
Kevan Carstensen
ec4f87a98c
immutable/repairer.py: don't use the default happiness setting when repairing
2010-09-27 13:01:02 -07:00
Brian Warner
0475bd8e27
immutable download: have the finder inform its share consumer "no more shares" in a subsequent tick, thus avoiding accidentally telling it "no more shares" now and then telling it "here's another share" in a subsequent tick
...
fixes #1191
Patch by Brian. This patch description was actually written by Zooko, but I forged Brian's name on the "author" field so that he would get credit for this patch in revision control history.
2010-09-09 21:30:38 -07:00
Zooko O'Whielacronx
63fb687a44
download status: fix bug from me committing the wrong one of François's #1172 patches, fixes #1172
2010-09-02 09:15:41 -07:00
francois
485bfc0fd6
DownloadStatus: show active immutable downloads in Active Operations, Fix #1172
2010-09-02 03:17:28 -07:00
Brian Warner
00e9e4e676
SegmentFetcher: use new diversity-seeking share-selection algorithm, and
...
deliver all shares at once instead of feeding them out one-at-a-time.
Also fix distribution of real-number-of-segments information: now all
CommonShares (not just the ones used for the first segment) get a
correctly-sized hashtree. Previously, the late ones might not, which would
make them crash and get dropped (causing the download to fail if the initial
set were insufficient, perhaps because one of their servers went away).
Update tests, add some TODO notes, improve variable names and comments.
Improve logging: add logparents, set more appropriate levels.
2010-08-31 18:37:02 -07:00
Brian Warner
c89a464510
Share: drop received data after each block finishes. Quick fix for the #1170 spans.py complexity bug.
2010-08-31 18:35:58 -07:00
Brian Warner
a3cf3331d2
DownloadStatus: put real numbers in progress/status rows, not placeholders.
...
Improve tests.
2010-08-09 23:06:03 -07:00
Brian Warner
5267fc17fe
immutable/filenode.py: put off DownloadStatus creation until first read() call
...
This avoids spamming the "recent uploads and downloads" /status page from
FileNode instances that were created for a directory read but which nobody is
ever going to read from. I also cleaned up the way DownloadStatus instances
are made to only ever do it in the CiphertextFileNode, not in the
higher-level plaintext FileNode. Also fixed DownloadStatus handling of read
size, thanks to David-Sarah for the catch.
2010-08-09 15:50:55 -07:00
Brian Warner
ed821d1504
Share: hush log entries in the main loop() after the fetch has been completed.
2010-08-09 13:43:59 -07:00
Brian Warner
4b7c94ece0
downloader.Segmentation: unregisterProducer when asked to stopProducing, this
...
seems to avoid the #1155 log message which reveals the URI (and filecap).
Also add an [ERROR] marker to the flog entry, since unregisterProducer also
makes interrupted downloads appear "200 OK"; this makes it more obvious that
the download did not complete.
2010-08-06 00:07:05 -07:00
Brian Warner
f6f9a97627
DownloadNode: fix lost-progress in fetch_failed, tolerate cancel when no segment-fetch is active. Fixes #1154 .
...
The lost-progress bug occurred when two simultanous read() calls fetched
different segments, and the first one failed (due to corruption, or the other
bugs in #1154 ): the second read() would never complete. While in this state,
cancelling the second read by having its consumer call stopProducing) would
trigger the cancel-intolerance bug. Finally, in downloader.node.Cancel,
prevent late cancels by adding an 'active' flag
2010-08-05 11:55:07 -07:00
Brian Warner
8844655705
One fix for bug #1154 : webapi GETs with a 'Range' header broke new-downloader.
...
The Range header causes n.read() to be called with an offset= of type 'long',
which eventually got used in a Spans/DataSpans object's __len__ method.
Apparently python doesn't permit __len__() to return longs, only ints.
Rewrote Spans/DataSpans to use s.len() instead of len(s) aka s.__len__() .
Added a test in test_download. Note that test_web didn't catch this because
it uses mock FileNodes for speed: it's probably time to rewrite that.
There is still an unresolved error-recovery problem in #1154 , so I'm not
closing the ticket quite yet.
2010-08-04 11:45:49 -07:00
Brian Warner
fa34e4dd16
fetcher.py: improve comments
2010-08-04 00:28:14 -07:00
Brian Warner
2a05aa2d91
lazily create DownloadNode upon first read()/get_segment()
2010-08-04 00:28:08 -07:00
Brian Warner
20847dd876
ShareFinder: add 10s OVERDUE timer, send new requests to replace overdue ones
...
The fixed 10-second timer will eventually be replaced with a per-server
value, calculated based on observed response times.
test_hung_server.py: enhance to exercise DYHB=OVERDUE state. Split existing
mutable+immutable tests into two pieces for clarity. Reenabled several tests.
Deleted the now-obsolete "test_failover_during_stage_4".
2010-08-04 00:27:41 -07:00
Brian Warner
797828f47f
Rewrite immutable downloader ( #798 ). This patch rearranges the rest of src/allmydata/immutable/ .
2010-08-04 00:26:39 -07:00
Brian Warner
22a07e9bbe
Rewrite immutable downloader ( #798 ). This patch adds the new downloader itself.
2010-08-04 00:26:29 -07:00
Zooko O'Whielacronx
24c72f2942
upload: tidy up logging messages
...
reformat code to be less than 100 chars wide, refactor formatting of logging messages, add log levels to some logging messages, M-x whitespace-cleanup
2010-08-02 00:02:12 -07:00
Zooko O'Whielacronx
69c48ebde6
immutable: tidy-up some code by using a set instead of list to hold homeless_shares
2010-08-01 23:20:04 -07:00
Zooko O'Whielacronx
0f94923f22
immutable: use PrefixingLogMixin to organize logging in Tahoe2PeerSelector and add more detailed messages about peer
2010-07-19 01:20:00 -07:00
Zooko O'Whielacronx
9b4f412e49
trivial: rename and add in-line doc to clarify "used_peers" => "upload_servers"
2010-07-18 21:47:44 -07:00
david-sarah
461f946457
upload.py: fix #1118 by aborting newly-homeless buckets when reassignment runs. This makes a previously failing assert correct. This version refactors 'abort' into two methods, rather than using a default argument.
2010-07-18 21:46:55 -07:00
Zooko O'Whielacronx
2e83f243c2
immutable: test for #1118
2010-07-18 15:15:37 -07:00
Kevan Carstensen
6d2c216b8f
immutable/upload.py: abort buckets if peer selection fails
2010-07-15 16:17:14 -07:00
Kevan Carstensen
d0706d27ea
Hush pyflakes warnings
2010-05-15 11:43:44 -07:00
Kevan Carstensen
55a2849f38
Remove a comment that no longer makes sense.
2010-05-14 13:35:16 -07:00
Kevan Carstensen
e225f573b9
Fix up the behavior of #778 , per reviewers' comments
...
- Make some important utility functions clearer and more thoroughly
documented.
- Assert in upload.servers_of_happiness that the buckets attributes
of PeerTrackers passed to it are mutually disjoint.
- Get rid of some silly non-Pythonisms that I didn't see when I first
wrote these patches.
- Make sure that should_add_server returns true when queried about a
shnum that it doesn't know about yet.
- Change Tahoe2PeerSelector.preexisting_shares to map a shareid to a set
of peerids, alter dependencies to deal with that.
- Remove upload.should_add_servers, because it is no longer necessary
- Move upload.shares_of_happiness and upload.shares_by_server to a utility
file.
- Change some points in Tahoe2PeerSelector.
- Compute servers_of_happiness using a bipartite matching algorithm that
we know is optimal instead of an ad-hoc greedy algorithm that isn't.
- Change servers_of_happiness to just take a sharemap as an argument,
change its callers to merge existing_shares and used_peers before
calling it.
- Change an error message in the encoder to be more appropriate for
servers of happiness.
- Clarify the wording of an error message in immutable/upload.py
- Refactor a happiness failure message to happinessutil.py, and make
immutable/upload.py and immutable/encode.py use it.
- Move the word "only" as far to the right as possible in failure
messages.
- Use a better definition of progress during peer selection.
- Do read-only peer share detection queries in parallel, not sequentially.
- Clean up logging semantics; print the query statistics whenever an
upload is unsuccessful, not just in one case.
2010-05-13 17:49:17 -07:00
Kevan Carstensen
9bc71d3da0
Alter the error message when an upload fails, per some comments in #778 .
...
When I first implemented #778 , I just altered the error messages to refer to
servers where they referred to shares. The resulting error messages weren't
very good. These are a bit better.
2009-12-30 15:03:44 -07:00
Kevan Carstensen
8bcc771e26
Change "UploadHappinessError" to "UploadUnhappinessError"
2009-12-04 22:30:37 -07:00
Kevan Carstensen
68fb556e93
Alter the error message returned when peer selection fails
...
The Tahoe2PeerSelector returned either NoSharesError or NotEnoughSharesError
for a variety of error conditions that weren't informatively described by them.
This patch creates a new error, UploadHappinessError, replaces uses of
NoSharesError and NotEnoughSharesError with it, and alters the error message
raised with the errors to be more in line with the new servers_of_happiness
behavior. See ticket #834 for more information.
2009-11-22 18:24:05 -07:00
Kevan Carstensen
320582be5a
Eliminate overcounting iof servers_of_happiness in Tahoe2PeerSelector; also reorganize some things.
2009-11-17 19:45:42 -07:00
Kevan Carstensen
4e29060847
Change stray "shares_of_happiness" to "servers_of_happiness"
2009-11-16 15:24:59 -07:00
Kevan Carstensen
a816de3f23
Alter Tahoe2PeerSelector to make sure that it recognizes existing shares on readonly servers, fixing an issue in #778
2009-11-16 13:28:05 -07:00
Kevan Carstensen
8c71df53f9
Alter 'immutable/encode.py' and 'immutable/upload.py' to use servers_of_happiness instead of shares_of_happiness.
2009-11-04 05:12:22 -07:00
Kevan Carstensen
89a8da9dd2
Alter CiphertextDownloader to work with servers_of_happiness
2009-09-23 21:19:32 -07:00
david-sarah
973f0afdd3
Change direct accesses to an_uri.storage_index to calls to .get_storage_index() ( fixes #948 )
2010-02-21 18:45:04 -08:00
Zooko O'Whielacronx
3e4342ecb3
immutable: downloader accepts notifications of buckets even if those notifications arrive after he has begun downloading shares.
...
This can be useful if one of the ones that he has already begun downloading fails. See #287 for discussion. This fixes part of #287 which part was a regression caused by #928 , namely this fixes fail-over in case a share is corrupted (or the server returns an error or disconnects). This does not fix the related issue mentioned in #287 if a server hangs and doesn't reply to requests for blocks.
2010-01-31 22:16:10 -08:00
david-sarah
37a242e01a
Improvements to test_hung_server, and fix for status updates in download.py
2010-01-29 22:43:03 -08:00
Zooko O'Whielacronx
d62428c1e6
immutable: fix bug in tests, change line-endings to unix style, add comment
2010-01-29 10:42:37 -08:00
david-sarah
baa11a0ad4
New tests for #928
2010-01-29 04:38:45 -08:00
Zooko O'Whielacronx
2bd9dfa5bd
immutable: download from the first servers which provide at least K buckets instead of waiting for all servers to reply
...
This should put an end to the phenomenon I've been seeing that a single hung server can cause all downloads on a grid to hang. Also it should speed up all downloads by (a) not-waiting for responses to queries that it doesn't need, and (b) downloading shares from the servers which answered the initial query the fastest.
Also, do not count how many buckets you've gotten when deciding whether the download has enough shares or not -- instead count how many buckets to *unique* shares that you've gotten. This appears to improve a slightly weird behavior in the current download code in which receiving >= K different buckets all to the same sharenumber would make it think it had enough to download the file when in fact it hadn't.
This patch needs tests before it is actually ready for trunk.
2010-01-27 15:34:17 -08:00
david-sarah
6057bc02cc
Prevent mutable objects from being retrieved from an immutable directory, and associated forward-compatibility improvements.
2010-01-26 22:44:30 -08:00
Brian Warner
731d15e56f
hush pyflakes-0.4.0 warnings: remove trivial unused variables. For #900 .
2010-01-14 14:15:29 -08:00
Brian Warner
d888bf3377
Clean up log.err calls, for one of the issues in #889 .
...
allmydata.util.log.err() either takes a Failure as the first positional
argument, or takes no positional arguments and must be invoked in an
exception handler. Fixed its signature to match both foolscap.logging.log.err
and twisted.python.log.err . Included a brief unit test.
2010-01-11 17:33:43 -08:00
Brian Warner
bacb6fe5aa
tidy up DeadReferenceError handling, ignore them in add_lease calls
...
Stop checking separately for ConnectionDone/ConnectionLost, since those have
been folded into DeadReferenceError since foolscap-0.3.1 . Write
rrefutil.trap_deadref() in terms of rrefutil.trap_and_discard() to improve
code coverage.
2010-01-11 16:07:23 -08:00
Brian Warner
db19b62702
immutable/checker.py: oops, forgot some imports. Also hush pyflakes.
2009-12-29 15:39:09 -08:00
Brian Warner
794e32738f
checker: don't let failures in add-lease affect checker results. Closes #875 .
...
Mutable servermap updates and the immutable checker, when run with
add_lease=True, send both the do-you-have-block and add-lease commands in
parallel, to avoid an extra round trip time. Many older servers have problems
with add-lease and raise various exceptions, which don't generally matter.
The client-side code was catching+ignoring some of them, but unrecognized
exceptions were passed through to the DYHB code, concealing the DYHB results
from the checker, making it think the server had no shares.
The fix is to separate the code paths. Both commands are sent at the same
time, but the errback path from add-lease is handled separately. Known
exceptions are ignored, the others (both unknown-remote and all-local) are
logged (log.WEIRD, which will trigger an Incident), but neither will affect
the DYHB results.
The add-lease message is sent first, and we know that the server handles them
synchronously. So when the checker is done, we can be sure that all the
add-lease messages have been retired. This makes life easier for unit tests.
2009-12-29 15:01:08 -08:00
Brian Warner
96834da0a2
Simplify immutable download API: use just filenode.read(consumer, offset, size)
...
* remove Downloader.download_to_data/download_to_filename/download_to_filehandle
* remove download.Data/FileName/FileHandle targets
* remove filenode.download/download_to_data/download_to_filename methods
* leave Downloader.download (the whole Downloader will go away eventually)
* add util.consumer.MemoryConsumer/download_to_data, for convenience
(this is mostly used by unit tests, but it gets used by enough non-test
code to warrant putting it in allmydata.util)
* update tests
* removes about 180 lines of code. Yay negative code days!
Overall plan is to rewrite immutable/download.py and leave filenode.read() as
the sole read-side API.
2009-12-01 17:53:30 -05:00
david-sarah
c4d38ad4c5
make status of finished operations consistently "Finished"
2009-11-20 22:15:43 -08:00
Brian Warner
0cf320c2ab
interface name cleanups: IFileNode, IImmutableFileNode, IMutableFileNode
...
The proper hierarchy is:
IFilesystemNode
+IFileNode
++IMutableFileNode
++IImmutableFileNode
+IDirectoryNode
Also expand test_client.py (NodeMaker) to hit all IFilesystemNode types.
2009-11-19 23:52:55 -08:00
Brian Warner
d2badbea78
class name cleanups: s/FileNode/ImmutableFileNode/
...
also fix test/bench_dirnode.py for recent dirnode changes
2009-11-19 23:22:39 -08:00
Brian Warner
e046744f40
make get_size/get_current_size consistent for all IFilesystemNode classes
...
* stop caching most_recent_size in dirnode, rely upon backing filenode for it
* start caching most_recent_size in MutableFileNode
* return None when you don't know, not "?"
* only render None as "?" in the web "more info" page
* add get_size/get_current_size to UnknownNode
2009-11-18 11:16:24 -08:00
Brian Warner
131e05b155
clean up uri-vs-cap terminology, emphasize cap instances instead of URI strings
...
* "cap" means a python instance which encapsulates a filecap/dircap (uri.py)
* "uri" means a string with a "URI:" prefix
* FileNode instances are created with (and retain) a cap instance, and
generate uri strings on demand
* .get_cap/get_readcap/get_verifycap/get_repaircap return cap instances
* .get_uri/get_readonly_uri return uri strings
* add filenode.download_to_filename() for control.py, should find a better way
* use MutableFileNode.init_from_cap, not .init_from_uri
* directory URI instances: use get_filenode_cap, not get_filenode_uri
* update/cleanup bench_dirnode.py to match, add Makefile target to run it
2009-11-11 14:26:19 -08:00
Brian Warner
f4aa418086
Verifier: check the full cryptext-hash tree on each share. Removed .todos
...
from the last few test_repairer tests that were waiting on this.
2009-10-05 15:18:49 -07:00
Brian Warner
504c767d03
Verifier: check the full block-hash-tree on each share
...
Removed the .todo from two test_repairer tests that check this. The only
remaining .todos are on the three crypttext-hash-tree tests.
2009-10-05 14:48:44 -07:00
Brian Warner
e8f56af5a7
Verifier: check the full share-hash chain on each share
...
Removed the .todo from two test_repairer tests that check this.
2009-10-05 14:34:43 -07:00
Brian Warner
be95129833
immutable/checker.py: rearrange code a little bit, make it easier to follow
2009-10-05 13:02:52 -07:00
Brian Warner
19d336513c
immutable/download.py: wrap to 80cols, no functional changes
2009-10-05 12:25:42 -07:00
Brian Warner
5283d4c19e
de-Service-ify Helper, pass in storage_broker and secret_holder directly.
...
This makes it more obvious that the Helper currently generates leases with
the Helper's own secrets, rather than getting values from the client, which
is arguably a bug that will likely be resolved with the Accounting project.
2009-08-15 13:17:37 -07:00
Brian Warner
4a4a4f9520
immutable.Downloader: pass StorageBroker to constructor, stop being a Service
...
child of the client, access with client.downloader instead of
client.getServiceNamed("downloader"). The single "Downloader" instance is
scheduled for demolition anyways, to be replaced by individual
filenode.download calls.
2009-08-15 12:25:43 -07:00
Brian Warner
0d5dc51617
Overhaul IFilesystemNode handling, to simplify tests and use POLA internally.
...
* stop using IURI as an adapter
* pass cap strings around instead of URI instances
* move filenode/dirnode creation duties from Client to new NodeMaker class
* move other Client duties to KeyGenerator, SecretHolder, History classes
* stop passing Client reference to dirnode/filenode constructors
- pass less-powerful references instead, like StorageBroker or Uploader
* always create DirectoryNodes by wrapping a filenode (mutable for now)
* remove some specialized mock classes from unit tests
Detailed list of changes (done one at a time, then merged together)
always pass a string to create_node_from_uri(), not an IURI instance
always pass a string to IFilesystemNode constructors, not an IURI instance
stop using IURI() as an adapter, switch on cap prefix in create_node_from_uri()
client.py: move SecretHolder code out to a separate class
test_web.py: hush pyflakes
client.py: move NodeMaker functionality out into a separate object
LiteralFileNode: stop storing a Client reference
immutable Checker: remove Client reference, it only needs a SecretHolder
immutable Upload: remove Client reference, leave SecretHolder and StorageBroker
immutable Repairer: replace Client reference with StorageBroker and SecretHolder
immutable FileNode: remove Client reference
mutable.Publish: stop passing Client
mutable.ServermapUpdater: get StorageBroker in constructor, not by peeking into Client reference
MutableChecker: reference StorageBroker and History directly, not through Client
mutable.FileNode: removed unused indirection to checker classes
mutable.FileNode: remove Client reference
client.py: move RSA key generation into a separate class, so it can be passed to the nodemaker
move create_mutable_file() into NodeMaker
test_dirnode.py: stop using FakeClient mockups, use NoNetworkGrid instead. This simplifies the code, but takes longer to run (17s instead of 6s). This should come down later when other cleanups make it possible to use simpler (non-RSA) fake mutable files for dirnode tests.
test_mutable.py: clean up basedir names
client.py: move create_empty_dirnode() into NodeMaker
dirnode.py: get rid of DirectoryNode.create
remove DirectoryNode.init_from_uri, refactor NodeMaker for customization, simplify test_web's mock Client to match
stop passing Client to DirectoryNode, make DirectoryNode.create_with_mutablefile the normal DirectoryNode constructor, start removing client from NodeMaker
remove Client from NodeMaker
move helper status into History, pass History to web.Status instead of Client
test_mutable.py: fix minor typo
2009-08-15 04:28:46 -07:00
Brian Warner
1192b61dfe
upload: fix #758 recursion-loop in peer-selection when servers report errors.
...
The bug was in the code that handles a third-or-later pass, and was
previously untested.
2009-07-17 00:07:09 -05:00
Zooko O'Whielacronx
22d390acbb
immutable: base32-encode the keys to generate cache filenames that will work on all platforms
2009-07-08 08:26:33 -07:00
Zooko O'Whielacronx
c0d1e7deae
directories: make initialization of the download cache lazy
...
If you open up a directory containing thousands of files, it currently computes the cache filename and checks for the cache file on disk immediately for each immutble file in that directory. With this patch, it delays those steps until you try to do something with an immutable file that could use the cache.
2009-07-07 17:40:40 -07:00
Brian Warner
8fca155a66
repairer.py: wrap to 80cols. No code changes.
2009-06-30 17:00:47 -07:00
Brian Warner
bd6ecc9f44
Split out NoSharesError, stop adding attributes to NotEnoughSharesError, change humanize_failure to include the original exception string, update tests, behave better if humanize_failure fails.
2009-06-24 19:17:07 -07:00
Brian Warner
8df15e9f30
big rework of introducer client: change local API, split division of responsibilites better, remove old-code testing, improve error logging
2009-06-22 19:10:47 -07:00
Brian Warner
711c09bc5d
clean up storage_broker interface: should fix #732
2009-06-21 16:51:19 -07:00
Brian Warner
a6caae9b5d
immutable/download: instrument do-you-have-block responses to investigate #732
2009-06-20 21:12:09 -07:00
Brian Warner
b1290633b8
more storage_broker refactoring: downloader gets a broker instead of a client,
...
use Client.get_storage_broker() accessor instead of direct attribute access.
2009-06-01 19:25:11 -07:00
Brian Warner
4177a3616b
remove plaintext-hashing code from the helper interface, to close #722
...
and deny the Helper the ability to mount a partial-information-guessing
attack. This will probably break compatibility between new clients and very
old (pre-1.0) helpers.
2009-06-01 15:49:16 -07:00
Brian Warner
c516361fd2
start to factor server-connection-management into a distinct 'StorageServerFarmBroker' object, separate from the client and the introducer. This is the starting point for #467 : static server selection
2009-06-01 14:06:04 -07:00
Brian Warner
de83526acd
immutable/encode.py: tolerate immediate _remove_shareholder by copying the
...
landlord list before iterating over it. This can probably only happen in unit
tests, but cleaning it up makes certain test failures easier to analyze.
2009-05-22 11:44:24 -07:00
Brian Warner
1863aee0aa
switch to using RemoteException instead of 'wrapped' RemoteReferences. Should fix #653 , the rref-EQ problem
2009-05-21 17:46:32 -07:00
Brian Warner
c9803d5217
switch all foolscap imports to use foolscap.api or foolscap.logging
2009-05-21 17:38:23 -07:00
Brian Warner
79437baade
immutable WriteBucketProxy: use pipeline to speed up uploads by overlapping roundtrips, for #392
2009-05-18 16:44:22 -07:00
Brian Warner
67571eb033
add more information to NotEnoughSharesError, split out new exceptions for no-servers and no-source-of-ueb-hash
2009-03-03 19:37:15 -07:00
Brian Warner
400c04c19a
immutable checker add-lease: catch remote IndexError here too
2009-02-27 01:17:24 -07:00
Brian Warner
f95e9b5964
immutable/checker.py: trap ShareVersionIncompatible too. Also, use f.check
...
instead of examining the value returned by f.trap, because the latter appears
to squash exception types down into their base classes (i.e. since
ShareVersionIncompatible is a subclass of LayoutInvalid,
f.trap(Failure(ShareVersionIncompatible)) == LayoutInvalid).
All this resulted in 'incompatible' shares being misclassified as 'corrupt'.
2009-02-23 22:14:05 -07:00
Brian Warner
9af9d8ae35
immutable/layout.py: wrap to 80 cols, no functional changes
2009-02-23 18:58:37 -07:00
Brian Warner
ef53da2b12
break storage.py into smaller pieces in storage/*.py . No behavioral changes.
2009-02-18 14:46:55 -07:00
Brian Warner
a0c5f92cbd
immutable/layout: minor change to repr name
2009-02-18 14:46:48 -07:00
Brian Warner
bce4a5385b
add --add-lease to 'tahoe check', 'tahoe deep-check', and webapi.
2009-02-17 19:32:43 -07:00
Brian Warner
fde2289e7b
CLI #590 : convert 'tahoe deep-check' to streaming form, improve display, add tests
2009-02-17 17:15:11 -07:00
Zooko O'Whielacronx
d7dbd6675e
immutable repairer: fix DownUpConnector so that it satisfies short reads the were requested after the last write and before the close
...
This is probably the cause of the very rare "loss of progress" bug. This is tested by unit tests. A recent patch changed this to errback instead of losing progress, and now this patch is changing it again to return a short read instead of errbacking. Returning a short read is what the uploader (in encode.py) is expecting, when it is reading the last block of the ciphertext, which might be shorter than the other blocks.
2009-02-12 17:04:47 -07:00
Zooko O'Whielacronx
bdb992467c
immutable repairer: add an assertion that a certain value in this tricky function is always what I think it is
2009-02-12 16:31:32 -07:00
Zooko O'Whielacronx
76d7cc4404
immutable repairer: errback any pending readers of DownUpConnectorwhen it runs out of bytes, and test that fact
2009-02-11 20:11:29 -07:00
Zooko O'Whielacronx
7eb260a9cf
versioning: include an "appname" in the application version string in the versioning protocol, and make that appname be controlled by setup.py
...
It is currently hardcoded in setup.py to be 'allmydata-tahoe'. Ticket #556 is to make it configurable by a runtime command-line argument to setup.py: "--appname=foo", but I suddenly wondered if we really wanted that and at the same time realized that we don't need that for tahoe-1.3.0 release, so this patch just hardcodes it in setup.py.
setup.py inspects a file named 'src/allmydata/_appname.py' and assert that it contains the string "__appname__ = 'allmydata-tahoe'", and creates it if it isn't already present. src/allmydata/__init__.py import _appname and reads __appname__ from it. The rest of the Python code imports allmydata and inspects "allmydata.__appname__", although actually every use it uses "allmydata.__full_version__" instead, where "allmydata.__full_version__" is created in src/allmydata/__init__.py to be:
__full_version__ = __appname + '-' + str(__version__).
All the code that emits an "application version string" when describing what version of a protocol it supports (introducer server, storage server, upload helper), or when describing itself in general (introducer client), usese allmydata.__full_version__.
This fixes ticket #556 at least well enough for tahoe-1.3.0 release.
2009-02-11 17:18:16 -07:00
Zooko O'Whielacronx
ef1bfdd2bf
immutable: repairer: add a simple test to exercise the "leftover" code path, fix the bug (and rename the variable "leftover" to "extra")
2009-02-10 12:12:45 -07:00
Zooko O'Whielacronx
75e4e67ed7
immutable: tighten preconditions -- you can write empty strings or read zero bytes, and add the first simple unit test of DownUpConnector
2009-02-10 00:56:47 -07:00
Zooko O'Whielacronx
c59940852b
immutable: defensive programming: assert that the encrypted readable gave you no more than the number of bytes you asked for
...
(There is a bug in the current DownUpConnector which can cause it to give more bytes than you asked for on one request, and then less on the next, effectively shifting some of the bytes to an earlier request, but I think this bug never gets triggered in practice.)
2009-02-09 23:46:05 -07:00
Brian Warner
a9a3b509df
upload: add a think-of-the-compatibility note to UploadResults
2009-02-09 14:50:04 -07:00
Brian Warner
a5ab6c060d
helper #609 : uploading client should ignore old helper's UploadResults, which were in a different format
2009-02-09 14:45:43 -07:00
Brian Warner
38ee95fec4
immutable/checker: wrap comments to 80cols, my laptop does not have a wide screen. No functional changes.
2009-02-07 14:04:39 -07:00
Brian Warner
d8b3505cf5
filenode: add get_repair_cap(), which uses the read-write filecap for immutable files, and the verifycap for immutable files
2009-01-22 21:38:36 -07:00
Brian Warner
aa50c30aa2
download: tiny cleanup of history code
2009-01-14 16:41:51 -07:00
Brian Warner
10268a4f7f
upload: move upload history into History object
2009-01-14 16:41:06 -07:00
Brian Warner
3920e6d1e7
immutable/download.py move recent-downloads history out of Downloader and into a separate class. upload/etc will follow soon.
2009-01-14 16:14:24 -07:00
Brian Warner
cc50e2f4aa
upload: use WriteBucketProxy_v2 when uploading a large file (with shares larger than 4GiB). This finally closes #346 . I think we can now handle immutable files up to 48EiB.
2009-01-12 20:14:42 -07:00
Brian Warner
bf56e2bb51
deep-check-and-repair: improve results and their HTML representation
2009-01-12 18:56:19 -07:00
Brian Warner
fe362c0021
hush pyflakes by removing unused imports
2009-01-12 15:41:20 -07:00
Zooko O'Whielacronx
25063688b4
immutable repairer
...
This implements an immutable repairer by marrying a CiphertextDownloader to a CHKUploader. It extends the IDownloadTarget interface so that the downloader can provide some metadata that the uploader requires.
The processing is incremental -- it uploads the first segments before it finishes downloading the whole file. This is necessary so that you can repair large files without running out of RAM or using a temporary file on the repairer.
It requires only a verifycap, not a readcap. That is: it doesn't need or use the decryption key, only the integrity check codes.
There are several tests marked TODO and several instances of XXX in the source code. I intend to open tickets to document further improvements to functionality and testing, but the current version is probably good enough for Tahoe-1.3.0.
2009-01-12 11:00:22 -07:00
Zooko O'Whielacronx
b496eba072
trivial: minor changes to in-line comments -- mark plaintext-hash-tree as obsolete
2009-01-10 14:56:01 -07:00
