Commit Graph

1761 Commits

Author SHA1 Message Date
Brian Warner
b2dcbbb62d test/common.py: fix race condition waiting for the helper connection
The wait_for_connections() method, which is used at the start of
test_system to make sure that all the clients are connected to all the
servers, did not also wait for clients to be connected to their Helpers.
Every once in a while, the helper connection would take a bit longer,
and then
test_system.SystemTest.test_filesystem._test_web._got_welcome_helper
would fail, because we'd check for a helper connection before it was
ready.

The fix is to modify wait_for_connections's polling predicate to look
for helper connections (if configured) as well as the regular
introducer- and server- connections.

Tested by temporarily adding a large (30s) delay to the connectTo() call
in Uploader.startService, simulating a long helper
connection-establishment delay. This makes the test fail consistently.
Then I fixed wait_for_connections(), and the test passed (slowly). Then
I removed the delay.

Closes #1467
2012-06-14 12:18:35 -07:00
Brian Warner
c073565ecc Display serverids consistently as 8-char pubkey, or 6-char tubid.
This makes it easy to distinguish between old V1-Introducer
nodes (identified by their Foolscap TubID) and new V2 nodes (identified
by their ed25519 pubkey).

This fixes a few places where we used to display a tubid even if we had
a pubkey, making it hard to visually correlate servers in two different
displays. It also cleans up the way we pass serverids to the JS-based
download timeline.

The "introweb" subscribed-clients list still shows tubids.
2012-06-12 14:30:34 -07:00
Brian Warner
51a5aaa639 test_system.py: wait for the Helper connection properly before uploading 2012-06-11 23:19:30 -07:00
Brian Warner
5e432b12a1 test_system.py: clean up control flow, reduce use of stall()
The _upload_resumable() test interrupts a Helper upload partway
through (by shutting down the Helper), then restarts the Helper and
resumes the upload. The control flow is kind of tricky: to do anything
"partway through" requires adding a hook to the Uploadable. The previous
flow depended upon a (fragile) call to self.stall(), which waits a fixed
number of seconds.

This removes one of those stall() calls (the remainder is in
test/common.py and I'll try removing it in a subsequent revision). It
also removes some now-redundant wait_for_connections() calls, since
bounce_client() doesn't fire its Deferred until the client has finished
coming back up (and uses wait_for_connections() internally to do so).
2012-06-11 18:22:35 -07:00
Brian Warner
82519ef93f test_system.py: fix minor typo 2012-06-11 18:16:36 -07:00
Brian Warner
e1093cbb33 introducer: add sequence-numbers to announcements, ignore replays
This will support revocation of Accounting recommendation records,
assuming the gossip-based broadcast channel isn't easily jammed.
2012-06-10 19:10:22 -07:00
Brian Warner
26d3869076 node.py: add get_private_config()
Also add tests for this and the pre-existing private-config methods.
2012-06-10 17:46:38 -07:00
Brian Warner
518e4cec98 Fix text in Publish Status results. Closes #1762. 2012-06-08 15:21:46 -07:00
Brian Warner
188c7fecf5 CheckResults corrupt/incompatible shares now return IServers
DeepResultsBase also has a get_corrupt_shares(), and it is populated
from CheckResults.get_corrupt_shares(). It has been updated too, along
with get_remaining_corrupt_shares().

Remove temporary get_new_corrupt_shares() and
get_new_incompatible_shares().
2012-06-02 11:39:12 -07:00
Brian Warner
da9ac55294 CheckResults.get_servers_responding() now returns IServers
Remove temporary get_new_servers_responding().
2012-06-02 11:39:12 -07:00
Brian Warner
957a5315aa CheckResults.get_sharemap() now returns IServers
Remove temporary get_new_sharemap().
2012-06-02 11:39:11 -07:00
Brian Warner
76fca000df CheckResults: pass IServer to corrupt/incompatible share locators
Getters still return serverid. Adds temporary get_new_corrupt_shares()
and get_new_incompatible_shares().
2012-06-02 11:39:11 -07:00
Brian Warner
dd8178ee6d CheckResults: pass IServer to servers_responding=, getter returns serverid
Add temporary get_new_servers_responding().
2012-06-02 11:39:11 -07:00
Brian Warner
a4c95609c7 CheckResults: pass IServer to sharemap=, but get_sharemap() returns serverids
This changes all code which feeds CheckResults(sharemap=) to provide
IServer instances, but CheckResults converts these to old-style
serverids during output, so downstream code doesn't have to change yet.

It adds a temporary get_new_sharemap(), which *does* return IServer
instances, so the immutable repairer can build new CheckResults from an
old one. This will go away when get_sharemap() is updated to return
IServer (and downstream code is updated too).
2012-06-02 11:39:11 -07:00
Brian Warner
0fcc054a61 CheckResults: use fat init, add type-checking assertions
Added assertions for sharemap, servermap, servers_responding,
list_corrupt_shares, and list_incompatible_shares.
2012-06-02 11:39:10 -07:00
Brian Warner
8daacbcf69 CheckResults: replace get_data() with as_dict(), use getters in web status 2012-06-02 11:39:10 -07:00
Brian Warner
4867dca3f0 use the new CheckResult getters almost everywhere
The remaining get_data() calls are either in
web.check_results.json_check_results(), or functioning as repr()s in
various unit test failure cases.
2012-06-02 11:39:10 -07:00
Brian Warner
ccfcd4de37 change CheckResults to use a fat set_data()
i.e. change set_data() to accept lots of parameters, instead of taking
a single dictionary with lots of keys. Also Convert all CheckResults
creators to use it.
2012-06-02 11:39:10 -07:00
Brian Warner
d446897282 CheckResults: simplify self._data 2012-06-02 11:39:09 -07:00
Brian Warner
e313cf6406 CheckResults: start hiding .data, first step to clean it up
The goal is to make CheckResults more strongly typed, and remove the
ambiguous ".data" field in favor of a bunch of specific counters and
sharelists, so I can changes .sharemap and .servermap to use IServer
instances instead of string serverids. By cleaning this up first, I hope
to get that task done with less debugging.
2012-06-02 11:39:09 -07:00
Brian Warner
bfee999e20 test_web.py: fix memory leak when run with --until-failure
The Fake*Node classes in test/common.py were accumulating share data in
a class-level dictionary, which persisted from one test run to the next.
As a result, running test_web.py over and over (with trial's
--until-failure feature) made this dictionary grow without bound,
eventually running out of memory.

This fix moves that dictionary into the FakeClient built fresh for each
test, so it doesn't build up. It does the same thing for "file_types",
which was much smaller but still lived at the class level.

Closes #1729
2012-05-22 15:39:49 -07:00
Brian Warner
bcdfb5802e test/check_memory.py: oops, fix one last ur.uri -> ur.get_uri() 2012-05-22 08:50:36 -07:00
Brian Warner
843739486a UploadResults: store IServers internally, but still return serverids
This stores IDisplayableServer-providing instances (StubServers or
NativeStorageServers) in the .servermap and .sharemap dictionaries. But
get_servermap()/get_sharemap() still return data structures with
serverids, not IServers, by translating their data on the way out. This
lets us put off changing the callers for a little bit longer.
2012-05-21 21:18:25 -07:00
Brian Warner
3d771132a8 switch UploadResults to use get_uri(), hide internal ._uri
Complete the getter-based transformation, by hiding ".uri" and updating
callers to use get_uri(). Also don't set a dummy self._uri, leave it
undefined until someone calls set_uri().
2012-05-21 21:14:44 -07:00
Brian Warner
29b11531b5 switch UploadResults to use getters, hide internal data, for all but .uri
This hides attributes with e.g. _sharemap, and creates getters like
get_sharemap() to access them, for every field except .uri . This will
make it easier to modify the internal representation of .sharemap
without requiring callers to adjust quite yet.

".uri" has so many users that it seemed better to update it in a
subsequent patch.
2012-05-21 21:14:28 -07:00
Brian Warner
08f5bc8e2f convert UploadResults to a fat init
Populate most of UploadResults (except .uri, which is learned later when
using a Helper) in the constructor, instead of allowing creators to
write to attributes later. This will help isolate the fields that we
want to change to use IServers.
2012-05-21 21:14:14 -07:00
Brian Warner
b71234c538 add HelperUploadResults
This splits the pb.Copyable on-wire object (HelperUploadResults) out
from the local results object (UploadResults). To maintain compatibility
with older Helpers, we have to leave pb.Copyable classes alone and
unmodified, but we want to change UploadResults to use IServers instead
of serverids. So by using a different class on the wire, and translating
to/from it on either end, we can accomplish both.
2012-05-21 21:14:00 -07:00
Brian Warner
0df833eac9 clean up Helper to make later changes easier
Fix up control flow inside the Helper, to make it more friendly for
later refactoring.
2012-05-21 21:13:32 -07:00
Brian Warner
393c0729de test_checker: minor improvement in fake-server setup
This prepares for testing the differences between tubid and pubkey-based
name/longname.
2012-05-21 19:49:36 -07:00
david-sarah
4ddcde3094 Since we now require Python 2.5, we can use os.SEEK_END. 2012-05-16 21:39:48 +00:00
david-sarah
a1a1b5bf8a Simplifications resulting from requiring Python 2.5 and therefore being able to use sqlite3 from the standard library. This also drops sqlite3 from the set of versions and paths we report. 2012-05-16 02:47:25 +00:00
Brian Warner
cc366903ce dictutil.DictOfSets: remove .union() method, it was misleading
Unlike set.union(), which returns a new set, DictOfSets.union() modified
the DictOfSets in-place. The name collision bit me when I changed some
code from using DictOfSets to a normal set, and expected that
set.union() would modify the set in-place. Since there was only one user
of DictOfSets.union, I figured it was safer to just get rid of it.
2012-05-16 16:55:09 -07:00
Brian Warner
9acf5beebd immutable repairer: populate servers-responding properly
If a server did not respond to the pre-repair filecheck, but did respond
to the repair, that server was not correctly added to the
RepairResults.data["servers-responding"] list. (This resulted from a
buggy usage of DictOfSets.union() in filenode.py).

In addition, servers to which filecheck queries were sent, but did not
respond, were incorrectly added to the servers-responding list
anyawys. (This resulted from code in the checker.py not paying attention
to the 'responded' flag).

The first bug was neatly masked by the second: it's pretty rare to have
a server suddenly start responding in the one-second window between a
filecheck and a subsequent repair, and if the server was around for the
filecheck, you'd never notice the problem. I only spotted the smelly
code while I was changing it for IServer cleanup purposes.

I added coverage to test_repairer.py for this. Trying to get that test
to fail before fixing the first bug is what led me to discover the
second bug. I also had to update test_corrupt_file_verno, since it was
incorrectly asserting that 10 servers responded, when in fact one of
them throws an error (but the second bug was causing it to be reported
anyways).
2012-05-16 16:55:09 -07:00
Brian Warner
c4d7b7b109 write node.url and portnum files atomically, to fix race in test_runner
Previously, test_runner sometimes fails because the _node_has_started()
poller fires after the portnum file has been opened, but before it has
actually been filled, allowing the test process to observe an empty file,
which flunks the test.

This adds a new fileutil.write_atomically() function (using the usual
write-to-.tmp-then-rename approach), and uses it for both node.url and
client.port . These files are written a bit before the node is really up and
running, but they're late enough for test_runner's purposes, which is to know
when it's safe to read client.port and use 'tahoe restart' (and therefore
SIGINT) to restart the node.

The current node/client code doesn't offer any better "are you really done
with startup" indicator.. the ideal approach would be to either watch the
logfile, or connect to its flogport, but both are a hassle. Changing the node
to write out a new "all done" file would be intrusive for regular
operations.
2012-05-14 15:03:14 -07:00
Brian Warner
fcc7e64759 Doc updates and cosmetic fixes for #1115 patch.
Removes the caveat from webapi.txt about count-good-share-hosts being wrong.

This series should close #1115.
2012-05-13 01:15:50 -07:00
Andrew Miller
04eb6086ad Fixed an error in previous commit where an empty servermap would throw an exception in 'count-good-share-hosts'. Augmented unit test.
Signed-off-by: Andrew Miller <amiller@dappervision.com>
2012-05-13 00:59:30 -07:00
Andrew Miller
4b7f34d179 Added tests for count-good-share-hosts under check and repair conditions. Patched the incorrect computation in immutable/filenode.py
Signed-off-by: Andrew Miller <amiller@dappervision.com>

Fixed missing import statements

Signed-off-by: Andrew Miller <amiller@dappervision.com>
2012-05-13 00:59:30 -07:00
Brian Warner
8f630a76fe test_web: fix use of headers= that's been wrong for a while 2012-05-13 00:45:12 -07:00
Brian Warner
5d404db898 webapi: don't allow ETags in t=info or t=rename-form, both are variable
t=info contains randomly-generated ophandles, and t=rename-form contains the
name of the child being renamed, so neither is eligible for a
short-circuiting ETag. Enhanced test_web to exercise this. Had to improve
FakeCHKFileNode slightly to let it participate. Refs #443.
2012-05-13 00:45:11 -07:00
Brian Warner
d437a9b33e test_web: improve ETag tests, add If-None-Match test 2012-05-13 00:45:11 -07:00
Andrew Miller
16eddc35e8 Added unit tests covering #466:comment-15. Refactored the 'etag' behavior for immutable files to respond to all GET '?t=' flags, not just t=None
Signed-off-by: Andrew Miller <amiller@dappervision.com>
2012-05-13 00:45:11 -07:00
david-sarah
379901bf8f Add 'tahoe debug flogtool' command, test for --help, and docs. This version gets the help synopses more correct, and changes the doc to say that this command is added in 1.10.0 rather than 1.9.2. fixes #1693 2012-03-31 22:41:22 +00:00
Brian Warner
eb2a4ef246 webapi: remove undocumented t=mkdir-p operation
Closes #380
2012-05-12 19:19:43 -07:00
Brian Warner
8417579147 webui: merge 'move' form with 'rename' form 2012-05-09 14:21:37 -07:00
Brian Warner
81c30b1cd1 test_web: improve shouldFail2() error reporting 2012-05-09 14:18:37 -07:00
Brian Warner
a92385b54b webapi 'move'-button cleanups
test_web.py: use shouldFail2(), safer than old shouldFail()
directory.py: forbid slashes in from_name=, return BAD_REQUEST instead of
              GONE when trying to move into a non-directory
2012-05-09 14:18:27 -07:00
Marcus Wanner
e89bce21a5 Add unit test for moving a directory
My gut tells me this case sould be tested. The rename suite tests it, so
move's will too.
2012-05-09 13:07:14 -07:00
Marcus Wanner
e58a01270b Change the arbitrary URI support from implied to explicit
The move webapi function now takes a target_type argument which lets it
know whether the target is a subdirectory name or URI. This is an
improvement over the old system in which the move handler tried to guess
whether the target was a name or a URI. Also fixed a little docs
copypaste problem and tweaked some line wrapping.
2012-05-09 13:07:14 -07:00
Marcus Wanner
b29d0920d3 Adding 'move' button to web UI, closes #1579
This adds "move file" capability to the web UI's directory display. The
support and test framework is heavily based on the similar "rename file"
feature. Unit tests and documentation are included. Multiple in-progress
versions of this patch may be found in ticket 1579. This version
includes arbitrary URI target support and is compatible with the change
from tahoe_css to tahoe.css.
2012-05-09 13:07:13 -07:00
david-sarah
a555336910 Make sure that foolscap.logging.log.setLogDir is called with a str (not unicode) path, v2. Includes test. fixes #1725 2012-04-29 02:28:44 +00:00