Brian Warner
ff0b9e2549
test_mutable.py: test replacing a file that has one new outlier share present: closes #272
2008-05-14 13:10:41 -07:00
Brian Warner
26187bfc81
use a weakref cache in the client to manage singleton filenodes/dirnodes, to avoid autocollision. Should close #391 .
2008-05-08 18:02:55 -07:00
Brian Warner
8f4e409677
check_memory: oops, fix HTTP-based modes: trailing slash in node.url vs no slash
2008-05-08 18:01:31 -07:00
Brian Warner
7323ced8e4
test_dirnode: improve coverage of not-mutable-error a bit
2008-05-08 16:53:35 -07:00
Brian Warner
411f578c19
check_memory: don't use fixed webports, they cause occasional test failures
2008-05-08 16:49:27 -07:00
Brian Warner
fabdc28c06
deep-stats: add file-size histogram
2008-05-08 16:19:42 -07:00
Brian Warner
a1ea3d9b37
test_dirnode: the size of the directories varies, tolerate it as low as 500 bytes
2008-05-08 13:42:42 -07:00
Brian Warner
e6ae7a2c60
dirnode: refactor deep-stats a bit
2008-05-08 13:33:07 -07:00
Brian Warner
6c00a70dbc
dirnode: add a deep_stats(), like deep-size but with more information. webish adds t=deeps-size too.
2008-05-08 13:21:14 -07:00
Brian Warner
462ef2a0ac
run a stats provider even if there's no gatherer, since the HTTP /statistics page is then useful. Only run the once-per-second load-monitor if there is a gatherer configured
2008-05-08 11:37:30 -07:00
Brian Warner
3cb361e233
dirnode: use the concurrency limiter in t=manifest and t=deep-size, allow 10 retrievals in parallel
2008-05-07 18:36:37 -07:00
Brian Warner
d76cf6f748
limiter: fix bug that ignores the limit= argument
2008-05-07 18:35:53 -07:00
Brian Warner
15a9f3d01b
test_util: improve ConcurrencyLimiter tests
2008-05-07 17:00:47 -07:00
Brian Warner
c7e441309d
add a basic concurrency limiter utility
2008-05-07 16:53:30 -07:00
Brian Warner
1e4b904b1a
web status: abbreviate file size in the upload+download summary page
2008-05-07 13:53:58 -07:00
Zooko O'Whielacronx
c1184f51e4
docs: fix a few stale comments in code
2008-05-07 08:39:03 -07:00
Brian Warner
fc774803ce
test_stats.py: improve test coverage
2008-04-30 11:52:31 -07:00
Brian Warner
a5a7ba24ef
stats: add tests for CPUUsageMonitor, modify it a bit to facilitate testing
2008-04-30 11:39:13 -07:00
Zooko O'Whielacronx
fc271a0ee9
setup: don't catch ImportError when importing _auto_deps in allmydata/__init__.py
...
Nowadays pkg_resources is a runtime requirement, and if there is something screwed up in the installation, we want an explicit ImportError exception as early as possible.
2008-04-30 13:22:04 -07:00
Brian Warner
88951548ee
test_mutable: update notify_publish() to match new signature
2008-04-29 18:24:57 -07:00
Brian Warner
4b3adb3fcf
mutable stats: track mutable bytes published too
2008-04-29 18:20:05 -07:00
Brian Warner
3be921174b
stats: add CPU-percentage monitor, with 1min/5min/15min moving-window averages, using time.clock()
2008-04-29 18:12:53 -07:00
Zooko O'Whielacronx
f842656ec2
wui: reorganize the welcome.xhtml page
...
Jake Edge tried Tahoe out and didn't notice the /status page. Hopefully with this new organization people like he will see that link more easily. This also addresses drewp's suggestion that the controls appear above the list of servers instead of below. (I think that was his suggestion.) I also reordered the controls.
2008-04-29 15:10:14 -07:00
Brian Warner
351a7f4632
offloaded: add 'resumes' counter to stats: how many upload requests are resuming an interrupted upload?
2008-04-28 13:27:55 -07:00
ben
c85869fc08
support freebsd 6
2008-04-28 00:41:40 -07:00
Brian Warner
186492e620
introducer: only record one announcement per (tubid,service) tuple. Fixes #343 .
2008-04-23 15:05:39 -07:00
Brian Warner
55dfb697a4
client: don't start the IntroducerClient until the Tub is ready, otherwise we will sometimes connect to the introducer (or other clients) before we've done Tub.setLocation, which loses some information on the introducer status page
2008-04-23 14:52:34 -07:00
Brian Warner
a557d44877
test_mutable: test that all servers refusing our share means a publish fails
2008-04-22 18:53:20 -07:00
Brian Warner
13eb00145e
mutable: improve test coverage in Retrieve, when shares change after mapupdate
2008-04-22 17:25:14 -07:00
Brian Warner
acdd98432d
hush pyflakes warning about code that got moved in the recent StallMixin refactoring
2008-04-22 17:14:26 -07:00
Brian Warner
a7632a345e
mutable/servermap: improve test coverage
2008-04-22 16:47:52 -07:00
Brian Warner
81ab1ec785
testutil: factor stall() out into a common location
2008-04-22 16:47:15 -07:00
Brian Warner
57d5341715
test_introducer.py: don't log nodeids as binary goop
2008-04-22 12:54:16 -07:00
Brian Warner
9ad6ed3777
web/status: remove trailing whitespace
2008-04-22 11:51:14 -07:00
Brian Warner
e6074f5dfc
mutable: test write failures, uncoordinated write detection
2008-04-22 11:49:53 -07:00
Brian Warner
7e1f9761e8
mutable: remove some dead code, rearrange use of populate_pubkey
2008-04-21 17:51:22 -07:00
Brian Warner
db4ddfc1bd
mutable: improve testing to exercise the non-cached case
2008-04-21 17:50:50 -07:00
Brian Warner
f39f8ae032
check_speed: rearrange a bit to make it easy to disable certain portions
2008-04-21 17:29:59 -07:00
Brian Warner
8377244c46
mutable status: make the 'total' timing value optional: unit tests fail otherwise
2008-04-21 17:29:12 -07:00
Brian Warner
6af124dc3e
mutable read: enable the cache (written during mapupdate, read during retrieve). This speeds up small-file reads by about 30% over a link with an average 25ms RTT
2008-04-21 17:27:50 -07:00
Brian Warner
e8b0f3d3cb
mutable status: use google-chart-API to draw server response times for servermap update
2008-04-21 16:16:55 -07:00
Brian Warner
a04c2866b1
mutable: improve test coverage slightly
2008-04-21 15:10:50 -07:00
Brian Warner
f16f48a909
download status: add time spent paused by the client (when we're serving over a slow HTTP link)
2008-04-21 12:19:17 -07:00
Zooko O'Whielacronx
78829dcf08
key_generator: make default key size be a constructor argument instead of a class variable, pass default key size of 522 (the smallest that we can do) in unit tests to make them faster
2008-04-22 12:28:18 -07:00
Brian Warner
c375b1503a
mutable/servermap: improve test coverage
2008-04-18 20:13:28 -07:00
Brian Warner
09dcfeae22
mutable: improve test coverage, fix bug in privkey fetching, add .finished to stats, remove dead code
2008-04-18 19:55:12 -07:00
Brian Warner
fc7b75e55f
mutable.node: avoid reentrancy problems in Deferred code on twisted2.5, by adding an eventual-send call
2008-04-18 00:43:29 -07:00
Zooko O'Whielacronx
a504f3395f
let MutableFileNode.__repr__() return something useful even if it isn't not initialize yet
...
This is because I sometimes log or examine a repr of an object from a function which is being called from that object's own __init__()...
(I'm committing this patch in order to test our patch management infrastructure.)
2008-04-17 21:32:38 -07:00
Brian Warner
d7b82f73c5
dirnode: return to 'delete fails if the child wasn't actually there' semantics, to make tests pass. There's a switch to enable/disable this
2008-04-17 20:06:06 -07:00
Brian Warner
d5b182c183
test_mutable: hush pyflakes
2008-04-17 19:57:55 -07:00
Brian Warner
1bff08a7df
dirnode: update to use MutableFileNode.modify
2008-04-17 19:57:04 -07:00
Brian Warner
b6b3b6b1bd
testutil: minor reformatting
2008-04-17 19:56:45 -07:00
Brian Warner
945003ea0f
test_mutable: factor out ShouldFailMixin
2008-04-17 19:55:51 -07:00
Brian Warner
004d3c14bf
mutable: implement MutableFileNode.modify, plus tests
2008-04-17 19:12:42 -07:00
Brian Warner
a379690b04
mutable: replace MutableFileNode API, update tests. Changed all callers to use overwrite(), but that will change soon
2008-04-17 17:51:38 -07:00
Brian Warner
157073d8d8
mutable WIP: merge in patches from current trunk
2008-04-17 13:09:22 -07:00
Brian Warner
ee218c262d
mutable WIP: use fireOnOneErrback when using a DeferredList
2008-04-17 13:11:48 -07:00
Brian Warner
e1838ba217
mutable WIP: clean up status handling, shrink the code a lot, improve test coverage
2008-04-17 13:02:22 -07:00
Brian Warner
a1670497a8
mutable WIP: add servermap update status pages
2008-04-16 19:05:41 -07:00
Brian Warner
842b1f1a70
mutable WIP: oops, fix test_mutable
2008-04-16 18:06:54 -07:00
Brian Warner
f35f20e10b
mutable WIP: publish status doesn't know its size early enough to update the stats_provider
2008-04-16 17:55:17 -07:00
Brian Warner
749c42fa2c
mutable WIP: re-enable publish/retrieve status
2008-04-16 17:49:06 -07:00
Brian Warner
4f07d96667
mutable WIP: improve logging a bit
2008-04-16 15:22:30 -07:00
Brian Warner
c0c8d72b44
mutable WIP: rewrite ServerMap data structure, add tests
2008-04-16 14:49:47 -07:00
Brian Warner
1b4b4cbd4a
mutable WIP: rename NotEnoughPeersError to NotEnoughSharesError
2008-04-15 16:08:32 -07:00
Brian Warner
ca14b99397
mutable WIP: if corrupt shares cause a retrieve to fail, restart it once, ignoring those shares and using different ones
2008-04-15 15:58:02 -07:00
Brian Warner
cf84bae850
mutable WIP: merge conflicts in test_system.py
2008-04-15 15:57:28 -07:00
Brian Warner
d4230d1781
mutable WIP: split mutable.py into separate files. All tests pass.
2008-04-11 14:31:16 -07:00
Brian Warner
0515b1099c
mutable WIP: all tests pass, but publish/retrieve status is still stubbed out
2008-04-10 19:26:35 -07:00
Brian Warner
418407ee5c
mutable: WIP. make Publish work, remove some test scaffolding. test_system still fails.
2008-04-10 18:44:06 -07:00
Brian Warner
b53bbaa43c
mutable.py: checkpointing #303 work part 2, Publish is sketched out
2008-04-07 19:01:28 -07:00
Brian Warner
2c939bfdd3
mutable.py: checkpointing #303 work: retrieve does what I want, now starting in on publish
2008-04-04 17:09:26 -07:00
Brian Warner
68527b25fc
client: add 'node.uptime' to the stats we collect
2008-04-17 11:13:39 -07:00
Brian Warner
7385ddaf30
download: DownloadStopped isn't SCARY, lower the log severity
2008-04-15 16:06:09 -07:00
Brian Warner
944f8babec
test_mutable.py: remove spurious Retrieve during a publish test
2008-04-15 11:20:38 -07:00
Brian Warner
b5d19456f4
web/statistics: fix typo that make immutable-download stats missing
2008-04-15 11:20:04 -07:00
Brian Warner
aebd80feec
upload.py: add some notes: extensions are more like 419 than 1000 bytes
2008-04-15 11:14:58 -07:00
Brian Warner
f153dafaa0
web: return a proper error upon POST with a bad t= value
2008-04-15 11:11:29 -07:00
Brian Warner
6de2a0794f
web: don't break status page when there is no helper running
2008-04-14 23:42:20 -07:00
Brian Warner
a0fa21e442
helper: log SI in ciphertext-read-failed message
2008-04-14 19:41:23 -07:00
Brian Warner
a719b5060f
helper: add SI to logged progress messages
2008-04-14 19:26:53 -07:00
Brian Warner
a2dd4d14a6
helper status: include percentage fetched+pushed, add helper-uploads to the upload/download list
2008-04-14 18:36:27 -07:00
Brian Warner
86578bc3cd
webish: fix 'not running helper' status indicator on welcome page, add tests
2008-04-14 16:28:11 -07:00
Brian Warner
5b8320442a
stats: add /statistics web page to show them, add tests
2008-04-14 14:17:08 -07:00
Brian Warner
8bea663fb2
offloaded: add fetched-percentage to the log message
2008-04-14 14:16:38 -07:00
Brian Warner
500934b72f
helper stats: fix the /helper_status page, the recent conflict merging missed some uses. Added tests, updated the munin plugins to match
2008-04-14 13:18:53 -07:00
Brian Warner
1334a251ca
remove size constraint on ShareData: large directories caused errors which triggered massive memory usage. See #379 for details
2008-04-11 22:51:54 -07:00
Brian Warner
d82f90faaa
mutable.py: catch errors during publish.. previously they were ignored. oops.
2008-04-11 22:51:02 -07:00
Zooko O'Whielacronx
bdda08df8c
copy pyutil.time_format into src/allmydata/util
...
time_format() provides a good format for logging timestamps
2007-08-14 09:33:49 -07:00
Zooko O'Whielacronx
0a6b2b5ff2
doc: slightly clarify an error message
2008-04-14 11:43:05 -07:00
robk-tahoe
a32275771a
stats_gatherer: fix typo in helper stats gathering.
2008-04-10 17:41:42 -07:00
robk-tahoe
766deaa9b6
stats_gatherer: reconcile helper stats gathering
...
I'd implemented stats gathering hooks in the helper a while back.
Brian did the same without reference to my changes. This reconciles
those two changes, encompassing all the stats in both changes,
implemented through the stats_provider interface.
this also provide templates for all 10 helper graphs in the
tahoe-stats munin plugin.
2008-04-10 17:25:44 -07:00
robk-tahoe
f47ebd0814
conflict reconciliation (part 1, stats gathering in helper)
2008-03-27 17:25:16 -07:00
robk-tahoe
5446ea5d67
stats: added stats reporting to the upload helper
...
adds a stats_producer for the upload helper, which provides a series of counters
to the stats gatherer, under the name 'chk_upload_helper'.
it examines both the 'incoming' directory, and the 'encoding' dir, providing
inc_count inc_size inc_size_old enc_count enc_size enc_size_old, respectively
the number of files in each dir, the total size thereof, and the aggregate
size of all files older than 48hrs
2008-03-25 18:19:08 -07:00
robk-tahoe
8620c7271d
key_generator: fix edge case found on windows
...
the windows (cygwin) buildslave has been failing the key generator test
it turns out that the time check on whether to refill the pool, and the
reactor, are interacting such that when the maybe_refill_pool call posted
on the reactor fires, the test on whether to fill the pool fails.
this adds a loop in the failure case to retry each 1s until it is time
to refill the pool, thus mitigating this timing accuracy problem on
windows.
2008-04-10 14:27:35 -07:00
robk-tahoe
35319c3380
stats gathering: fix storage server stats if not tracking consumed
...
the RIStatsProvider interface requires that counter and stat values be
ChoiceOf(float, int, long) the recent changes to storage server to not
track 'consumed' led to returning None as the value of a counter.
this causes violations to be experienced by nodes whose stats are being
gathered.
this patch simply omits that stat if 'consumed' is not being tracked.
2008-04-09 18:23:06 -07:00
robk-tahoe
ef0f523fe0
stats gathering: added counters to upload,download,mutablewatcher
...
counting number of operations, and for immutable files, bytes transferred
2008-04-09 18:08:59 -07:00
robk-tahoe
0d2eb1edf6
stats_gatherer: verbose debug logging
...
one of the storage servers is throwing foolscap violations about the
return value of get_stats(). this adds a log of the data returned
to the foolscap log event stream at the debug level '12' (between
NOISY(10) and OPERATIONAL(20)) hopefully this will facilitate
finding the cause of this problem.
2008-04-09 16:10:53 -07:00
robk-tahoe
27ac71458f
key_generator: remove poll timeouts from test
...
the timeouts on uses of 'poll' were there purely to make sure a test doesn't
poll indefinitely. however having such timeouts makes tests susceptible
to premature timeouts under high load, or on slow machines. (e.g. cygwin
slaves running in virtual machines on loaded hosts)
purportedly trial by default applies a timeout to tests to prevent them
hanging out indefinitely, so these poll timeouts are redundant and cause
intermittent failures on slow hosts. hence they're more bother than they're
worth, and should be culled.
2008-04-08 11:06:58 -07:00
robk-tahoe
8e9e56fdd0
key_generator: fix a typo in the .tac generating create-key-generator
...
verbose is an object attribute, no longer settable via init args
2008-04-08 11:06:06 -07:00
