Brian Warner
411f578c19
check_memory: don't use fixed webports, they cause occasional test failures
2008-05-08 16:49:27 -07:00
Brian Warner
fabdc28c06
deep-stats: add file-size histogram
2008-05-08 16:19:42 -07:00
Brian Warner
a1ea3d9b37
test_dirnode: the size of the directories varies, tolerate it as low as 500 bytes
2008-05-08 13:42:42 -07:00
Brian Warner
e6ae7a2c60
dirnode: refactor deep-stats a bit
2008-05-08 13:33:07 -07:00
Brian Warner
6c00a70dbc
dirnode: add a deep_stats(), like deep-size but with more information. webish adds t=deeps-size too.
2008-05-08 13:21:14 -07:00
Brian Warner
15a9f3d01b
test_util: improve ConcurrencyLimiter tests
2008-05-07 17:00:47 -07:00
Brian Warner
c7e441309d
add a basic concurrency limiter utility
2008-05-07 16:53:30 -07:00
Brian Warner
fc774803ce
test_stats.py: improve test coverage
2008-04-30 11:52:31 -07:00
Brian Warner
a5a7ba24ef
stats: add tests for CPUUsageMonitor, modify it a bit to facilitate testing
2008-04-30 11:39:13 -07:00
Brian Warner
88951548ee
test_mutable: update notify_publish() to match new signature
2008-04-29 18:24:57 -07:00
Brian Warner
186492e620
introducer: only record one announcement per (tubid,service) tuple. Fixes #343 .
2008-04-23 15:05:39 -07:00
Brian Warner
a557d44877
test_mutable: test that all servers refusing our share means a publish fails
2008-04-22 18:53:20 -07:00
Brian Warner
13eb00145e
mutable: improve test coverage in Retrieve, when shares change after mapupdate
2008-04-22 17:25:14 -07:00
Brian Warner
acdd98432d
hush pyflakes warning about code that got moved in the recent StallMixin refactoring
2008-04-22 17:14:26 -07:00
Brian Warner
a7632a345e
mutable/servermap: improve test coverage
2008-04-22 16:47:52 -07:00
Brian Warner
81ab1ec785
testutil: factor stall() out into a common location
2008-04-22 16:47:15 -07:00
Brian Warner
57d5341715
test_introducer.py: don't log nodeids as binary goop
2008-04-22 12:54:16 -07:00
Brian Warner
e6074f5dfc
mutable: test write failures, uncoordinated write detection
2008-04-22 11:49:53 -07:00
Brian Warner
db4ddfc1bd
mutable: improve testing to exercise the non-cached case
2008-04-21 17:50:50 -07:00
Brian Warner
f39f8ae032
check_speed: rearrange a bit to make it easy to disable certain portions
2008-04-21 17:29:59 -07:00
Brian Warner
a04c2866b1
mutable: improve test coverage slightly
2008-04-21 15:10:50 -07:00
Zooko O'Whielacronx
78829dcf08
key_generator: make default key size be a constructor argument instead of a class variable, pass default key size of 522 (the smallest that we can do) in unit tests to make them faster
2008-04-22 12:28:18 -07:00
Brian Warner
c375b1503a
mutable/servermap: improve test coverage
2008-04-18 20:13:28 -07:00
Brian Warner
09dcfeae22
mutable: improve test coverage, fix bug in privkey fetching, add .finished to stats, remove dead code
2008-04-18 19:55:12 -07:00
Brian Warner
d7b82f73c5
dirnode: return to 'delete fails if the child wasn't actually there' semantics, to make tests pass. There's a switch to enable/disable this
2008-04-17 20:06:06 -07:00
Brian Warner
d5b182c183
test_mutable: hush pyflakes
2008-04-17 19:57:55 -07:00
Brian Warner
1bff08a7df
dirnode: update to use MutableFileNode.modify
2008-04-17 19:57:04 -07:00
Brian Warner
945003ea0f
test_mutable: factor out ShouldFailMixin
2008-04-17 19:55:51 -07:00
Brian Warner
004d3c14bf
mutable: implement MutableFileNode.modify, plus tests
2008-04-17 19:12:42 -07:00
Brian Warner
a379690b04
mutable: replace MutableFileNode API, update tests. Changed all callers to use overwrite(), but that will change soon
2008-04-17 17:51:38 -07:00
Brian Warner
157073d8d8
mutable WIP: merge in patches from current trunk
2008-04-17 13:09:22 -07:00
Brian Warner
e1838ba217
mutable WIP: clean up status handling, shrink the code a lot, improve test coverage
2008-04-17 13:02:22 -07:00
Brian Warner
a1670497a8
mutable WIP: add servermap update status pages
2008-04-16 19:05:41 -07:00
Brian Warner
842b1f1a70
mutable WIP: oops, fix test_mutable
2008-04-16 18:06:54 -07:00
Brian Warner
749c42fa2c
mutable WIP: re-enable publish/retrieve status
2008-04-16 17:49:06 -07:00
Brian Warner
4f07d96667
mutable WIP: improve logging a bit
2008-04-16 15:22:30 -07:00
Brian Warner
c0c8d72b44
mutable WIP: rewrite ServerMap data structure, add tests
2008-04-16 14:49:47 -07:00
Brian Warner
1b4b4cbd4a
mutable WIP: rename NotEnoughPeersError to NotEnoughSharesError
2008-04-15 16:08:32 -07:00
Brian Warner
cf84bae850
mutable WIP: merge conflicts in test_system.py
2008-04-15 15:57:28 -07:00
Brian Warner
d4230d1781
mutable WIP: split mutable.py into separate files. All tests pass.
2008-04-11 14:31:16 -07:00
Brian Warner
0515b1099c
mutable WIP: all tests pass, but publish/retrieve status is still stubbed out
2008-04-10 19:26:35 -07:00
Brian Warner
418407ee5c
mutable: WIP. make Publish work, remove some test scaffolding. test_system still fails.
2008-04-10 18:44:06 -07:00
Brian Warner
2c939bfdd3
mutable.py: checkpointing #303 work: retrieve does what I want, now starting in on publish
2008-04-04 17:09:26 -07:00
Brian Warner
68527b25fc
client: add 'node.uptime' to the stats we collect
2008-04-17 11:13:39 -07:00
Brian Warner
944f8babec
test_mutable.py: remove spurious Retrieve during a publish test
2008-04-15 11:20:38 -07:00
Brian Warner
f153dafaa0
web: return a proper error upon POST with a bad t= value
2008-04-15 11:11:29 -07:00
Brian Warner
6de2a0794f
web: don't break status page when there is no helper running
2008-04-14 23:42:20 -07:00
Brian Warner
a2dd4d14a6
helper status: include percentage fetched+pushed, add helper-uploads to the upload/download list
2008-04-14 18:36:27 -07:00
Brian Warner
86578bc3cd
webish: fix 'not running helper' status indicator on welcome page, add tests
2008-04-14 16:28:11 -07:00
Brian Warner
5b8320442a
stats: add /statistics web page to show them, add tests
2008-04-14 14:17:08 -07:00
Brian Warner
500934b72f
helper stats: fix the /helper_status page, the recent conflict merging missed some uses. Added tests, updated the munin plugins to match
2008-04-14 13:18:53 -07:00
robk-tahoe
27ac71458f
key_generator: remove poll timeouts from test
...
the timeouts on uses of 'poll' were there purely to make sure a test doesn't
poll indefinitely. however having such timeouts makes tests susceptible
to premature timeouts under high load, or on slow machines. (e.g. cygwin
slaves running in virtual machines on loaded hosts)
purportedly trial by default applies a timeout to tests to prevent them
hanging out indefinitely, so these poll timeouts are redundant and cause
intermittent failures on slow hosts. hence they're more bother than they're
worth, and should be culled.
2008-04-08 11:06:58 -07:00
robk-tahoe
c838cfef44
key_generator: up timeouts on keygen test
...
in both these cases, the timeout only serves to abort a stuck test, and
the key_generator should respond more quickly, but seeing test failures
in buildbot on some platforms suggests that the test is too susceptible
to timing issues on loaded buildslaves.
2008-04-03 16:26:24 -07:00
robk-tahoe
1ae2c39862
key_generator: service related cleanups, incorporation into system test
...
this cleans up KeyGenerator to be a service (a subservice of the
KeyGeneratorService as instantiated by the key-generator.tac app)
this means that the timer which replenishes the keypool will be
shutdown cleanly when the service is stopped.
adds checks on the key_generator service and client into the system
test 'test_mutable' such that one of the nodes (clients[3]) uses
the key_generator service, and checks that mutable file creation
in that node, via a variety of means, are all consuming keys from
the key_generator.
2008-04-03 15:57:07 -07:00
robk-tahoe
ccda06b061
key_generator: added a unit test
...
implemented a unit test of basic KeyGenService functionality,
fixed a bug in the timing of pool refreshes
2008-04-03 13:01:43 -07:00
Brian Warner
06dae8fd67
trial_figleaf.py: make our figleaf code compatible with both Twisted-8.x and Twisted-2.5.x
2008-04-02 17:48:55 -07:00
robk-tahoe
5578559b85
added offloaded key generation
...
this adds a new service to pre-generate RSA key pairs. This allows
the expensive (i.e. slow) key generation to be placed into a process
outside the node, so that the node's reactor will not block when it
needs a key pair, but instead can retrieve them from a pool of already
generated key pairs in the key-generator service.
it adds a tahoe create-key-generator command which initialises an
empty dir with a tahoe-key-generator.tac file which can then be run
via twistd. it stashes its .pem and portnum for furl stability and
writes the furl of the key gen service to key_generator.furl, also
printing it to stdout.
by placing a key_generator.furl file into the nodes config directory
(e.g. ~/.tahoe) a node will attempt to connect to such a service, and
will use that when creating mutable files (i.e. directories) whenever
possible. if the keygen service is unavailable, it will perform the
key generation locally instead, as before.
2008-04-01 18:45:13 -07:00
Brian Warner
36f5c025a6
helper: add stats for the gatherer, show some on the webish welcome page
2008-03-27 15:55:32 -07:00
Brian Warner
9b3a32d0b3
add GET /uri/URI/?t=deep-size, to compute the total size of immutable files reachable from a given directory
2008-03-27 11:33:42 -07:00
Brian Warner
4531d1e953
introweb.py: add ?t=json, to provide machine-readable subscriber counts
2008-03-25 12:56:12 -07:00
Brian Warner
1e097766c9
disable plaintext hashes in shares, but leave a switch to turn it back on
2008-03-24 13:39:51 -07:00
Brian Warner
7b21054c33
UNDO: upload: stop putting plaintext and ciphertext hashes in shares.
...
This removes the guess-partial-information attack vector, and reduces
the amount of overhead that we consume with each file. It also introduces
a forwards-compability break: older versions of the code (before the
previous download-time "make hashes optional" patch) will be unable
to read files uploaded by this version, as they will complain about the
missing hashes. This patch is experimental, and is being pushed into
trunk to obtain test coverage. We may undo it before releasing 1.0.
2008-03-23 15:35:54 -07:00
Zooko O'Whielacronx
dfdbb95648
fix check-memory to use new upload API (which requires a "convergence" argument), and change it to measure convergence instead of random-key, since convergence is the use case we care about more
2008-03-24 15:28:04 -07:00
Zooko O'Whielacronx
fc3bd0c987
use added secret to protect convergent encryption
...
Now upload or encode methods take a required argument named "convergence" which can be either None, indicating no convergent encryption at all, or a string, which is the "added secret" to be mixed in to the content hash key. If you want traditional convergent encryption behavior, set the added secret to be the empty string.
This patch also renames "content hash key" to "convergent encryption" in a argument names and variable names. (A different and larger renaming is needed in order to clarify that Tahoe supports immutable files which are not encrypted content-hash-key a.k.a. convergent encryption.)
This patch also changes a few unit tests to use non-convergent encryption, because it doesn't matter for what they are testing and non-convergent encryption is slightly faster.
2008-03-24 09:46:06 -07:00
Brian Warner
7996131a0a
upload: stop putting plaintext and ciphertext hashes in shares.
...
This removes the guess-partial-information attack vector, and reduces
the amount of overhead that we consume with each file. It also introduces
a forwards-compability break: older versions of the code (before the
previous download-time "make hashes optional" patch) will be unable
to read files uploaded by this version, as they will complain about the
missing hashes. This patch is experimental, and is being pushed into
trunk to obtain test coverage. We may undo it before releasing 1.0.
2008-03-23 15:35:54 -07:00
Brian Warner
553367d567
download: make plaintext and ciphertext hashes in the UEB optional.
...
Removing the plaintext hashes can help with the guess-partial-information
attack. This does not affect compatibility, but if and when we actually
remove any hashes from the share, that will introduce a
forwards-compatibility break: tahoe-0.9 will not be able to read such files.
2008-03-23 14:46:49 -07:00
robk-tahoe
e75c417bb5
webish mkdir-p: added unit test
...
added a test for the simple mkdir-p hack I added yesterday
checks that mkdir-p can create a directory hierarchy, and that resubmitting
a request for the same path yields the existing dir's uri
2008-03-18 13:11:08 -07:00
Brian Warner
2ef70ab814
mutable.py: split replace() into update() and overwrite(). Addresses #328 .
2008-03-12 18:00:43 -07:00
Brian Warner
89be2e1bea
introweb: combine announcement and subscriber information to show version+nickname for each client
2008-03-11 19:21:29 -07:00
Brian Warner
810ba68343
add a webserver for the Introducer, showing service announcements and subscriber lists
2008-03-11 17:36:25 -07:00
Brian Warner
c727348d85
mutable: tolerate multiple encodings, using whichever version is recoverable first. Closes #312
2008-03-11 00:26:00 -07:00
Brian Warner
791482cf8d
test_mutable: improve multiple-encodings test coverage
2008-03-10 23:47:35 -07:00
Brian Warner
9ca55b8b79
test_mutable: make test-multiple-encodings work
2008-03-10 23:16:28 -07:00
Brian Warner
57bd23f35f
test_mutable: more test coverage, building up a framework to cause reads to occur in a specific order
2008-03-10 22:15:43 -07:00
Brian Warner
11445b5109
test_mutable: exercise short reads too
2008-03-10 18:08:23 -07:00
Brian Warner
be5a6147b4
test_mutable: test all hash-failure cases except a corrupted encrypted private key
2008-03-10 17:46:52 -07:00
Brian Warner
647734cd3b
test_mutable: add Roundtrip test, suitable for new share-mangling tests
2008-03-10 16:14:08 -07:00
Brian Warner
9e460cd22e
mutable: cosmetic changes
2008-03-10 15:44:05 -07:00
Brian Warner
d6cde55c63
mutable: minor refactoring of _do_read, to make other tests easier
2008-03-10 15:42:56 -07:00
Brian Warner
4fa622f9b1
webish: this file is too big, start breaking it into pieces, beginning with status
2008-03-05 14:59:56 -07:00
Brian Warner
7d9bfbcabe
mutable: oops, .download *is* in use, by the speedtest. Restore it and add a test.
2008-03-04 15:11:40 -07:00
Brian Warner
4daa817502
test_util.py: get 100% test coverage for hashutil.py
2008-03-04 14:42:25 -07:00
Brian Warner
95a085763f
uri.py: get 100% test coverage, fix a few bugs in the process
2008-03-04 14:27:45 -07:00
Brian Warner
e6489b1ef2
test_filenode.py : improve coverage of mutable filenode, fix a bug in __hash__
2008-03-04 14:01:28 -07:00
Brian Warner
f6ca62df31
test_system: improve test coverage of publish/retrieve status
2008-03-04 01:24:35 -07:00
Brian Warner
68fbd89e66
webish: add primitive publish/retrieve status pages
2008-03-04 01:07:44 -07:00
Brian Warner
7e159feb27
stats: make StatsGatherer happy about sharing a process with other services, add one during system test to get some test coverage
2008-03-03 23:55:58 -07:00
Brian Warner
abb51e70f3
test_web: improve upload/download status coverage
2008-03-03 21:56:23 -07:00
Brian Warner
3ecb483e35
test_system: add test coverage for download-status and upload-status
2008-03-03 21:37:17 -07:00
Brian Warner
886ef22335
webish: download-results: add server_problems
2008-03-03 20:30:35 -07:00
Brian Warner
c8e24f0904
webish: make upload timings visible on the recent uploads/downloads status page
2008-03-03 14:48:52 -07:00
Brian Warner
436baa1b19
webish: add per-file upload/download status pages
2008-02-29 23:03:00 -07:00
Brian Warner
1a7651ce82
retain 10 most recent upload/download status objects, show them in /status . Prep for showing individual status objects
2008-02-29 22:19:03 -07:00
Zooko O'Whielacronx
99f006c584
wapi: add POST /uri/$DIRECTORY?t=set_children
...
Unfinished bits: doc in webapi.txt, test handling of badly formed JSON, return reasonable HTTP response, examination of the effect of this patch on code coverage -- but I'm committing it anyway because MikeB can use it and I'm being called to dinner...
2008-02-29 18:40:27 -07:00
Zooko O'Whielacronx
4287079455
test: update todo string in test_nevow
2008-02-29 13:10:08 -07:00
Brian Warner
301dd3d489
webish status: distinguish active uploads/downloads from recent ones
2008-02-26 15:35:28 -07:00
Brian Warner
1af3bfb1d4
test_runner.py: test launching an introducer too
2008-02-18 00:28:56 -07:00
Brian Warner
6ff98a3c0b
'tahoe start': allow this to start arbitrary .tac file -bearing directories (like the log gatherer)
2008-02-18 00:28:17 -07:00
Brian Warner
c3a1491cf4
test_system.py: improve coverage of webish.py
2008-02-15 04:02:50 -07:00
Brian Warner
72874390fe
BIG COMPATIBILITY BREAK: update hash tags, switch to SHA-256d everywhere
2008-02-14 19:58:01 -07:00
Zooko O'Whielacronx
fc0d637523
docs: update install and usage docs, improve cli "usage" output, make new example directories, add unit test that fails code which prints out sentences that don't end with punctuation marks
2008-02-15 13:11:02 -07:00
Zooko O'Whielacronx
5942fca1e1
add unit test asserting that allmydata.util.base32 matches rfc 3548
2008-02-14 20:20:19 -07:00
Zooko O'Whielacronx
3eb95c1fad
merge patch which adds base32 to test_util with patch which removes bencode from test_util
2008-02-14 19:35:32 -07:00
Brian Warner
09bfa3bea2
remove bencode, yay
2008-02-14 18:06:41 -07:00
Brian Warner
7927495cbe
unicode handling: declare dirnodes to contain unicode child names, update webish to match
2008-02-14 15:45:56 -07:00
Zooko O'Whielacronx
7c6de95bc6
switch from base62 to base32 for storage indices, switch from z-base-32 to rfc 3548 base-32 for everything, separate out base32 encoding from idlib
2008-02-14 19:27:47 -07:00
Brian Warner
a050204833
add test coverage for the /stats web page
2008-02-13 13:57:39 -07:00
Brian Warner
871e7756f7
test_web: remove test_GET_DIRURL_large, it failed to exercise the problem in #237 and consumes a disproportionate amount of time
2008-02-13 13:08:26 -07:00
Zooko O'Whielacronx
e89edf8803
fix a few unused imports and suchlike, discovered by pyflakes
2008-02-13 07:38:08 -07:00
Brian Warner
a7867aeea2
introducer: remove encoding-parameter config, for now
2008-02-12 18:59:54 -07:00
Zooko O'Whielacronx
3f8df27063
use base62 encoding for storage indexes, on disk and in verifier caps, and in logging and diagnostic tools
...
base62 encoding fits more information into alphanumeric chars while avoiding the troublesome non-alphanumeric chars of base64 encoding. In particular, this allows us to work around the ext3 "32,000 entries in a directory" limit while retaining the convenient property that the intermediate directory names are leading prefixes of the storage index file names.
2008-02-12 20:48:37 -07:00
Brian Warner
a70fba4ef1
webish: add edge metadata to t=json output, including timestamps
2008-02-11 19:14:10 -07:00
Brian Warner
ca27b8e5ad
add 'tahoe catalog-shares' tool, to make a one-line summary of each share file. This can help do cross-server correlation of sharefiles, looking for anomalies
2008-02-11 18:17:01 -07:00
Brian Warner
5f50543c3b
test_dirnode.py: improve error messages in case timestamp tests fail
2008-02-11 17:37:51 -07:00
Brian Warner
09cf1e2a8a
test_dirnode.py: simplejson-1.7.1 incorrectly rounds floats to two decimal places. Don't let this bug flunk the timestamp test.
2008-02-11 17:35:17 -07:00
Brian Warner
7123c9d875
test_system.py: refactor bounce_client, probably make it stop failing on cygwin
2008-02-11 15:26:58 -07:00
Brian Warner
3a1db45579
test_dirnode.py: assert that we update mtime and preserve ctime
2008-02-11 15:12:55 -07:00
Brian Warner
e6ddd9f3da
dirnode.py: add metadata= to add_file(), add tests
2008-02-11 14:53:28 -07:00
Brian Warner
b90eaf75da
test_dirnode.py: add diag output to test-ctime/mtime tests
2008-02-11 14:13:07 -07:00
Brian Warner
622c477e31
dirnode: add ctime/mtime to metadata, update metadata-modifying APIs. Needs more testing and sanity checking.
2008-02-08 18:43:47 -07:00
Brian Warner
873dee53e9
test_system: remove the hackish debug_interrupt= attribute magic used to exercise interrupted-upload resumption, instead just make the Uploadable bounce the helper halfway through the upload
2008-02-07 20:15:37 -07:00
Brian Warner
3bd79917b2
test_system.py: remove that ugly debug_stash_RemoteencryptedUploadable hack, now that UploadResults give us a better approach
2008-02-07 17:27:30 -07:00
Brian Warner
96ca6c2033
test_upload.py: hush pyflakes
2008-02-06 20:04:31 -07:00
Brian Warner
f78c80d7aa
test_upload.py: add test to exercise CHK hashing variations
2008-02-06 20:03:35 -07:00
Brian Warner
02e7c56563
test_cli: oops, need to update this when the CHK hash changes
2008-02-06 19:58:53 -07:00
Brian Warner
da9610e90a
change encryption-key hash to include encoding parameters. This is a minor compatibility break: CHK files encoded (with convergence) before and after this will have different keys and ciphertexts. Also switched to SHA-256d for both the data-to-key hash and the key-to-storageindex hash
2008-02-06 19:50:47 -07:00
Brian Warner
9f9a458249
hashutil: add tagged_hash_256d and tagged_hasher_256d
2008-02-06 19:36:43 -07:00
Brian Warner
81c5ceae16
upload: rework passing of default encoding parameters: move more responsibility into BaseUploadable
2008-02-06 18:39:03 -07:00
Brian Warner
75e4332462
offloaded: oops, need more tricks to make the unit tests pass
2008-02-06 17:51:11 -07:00
Brian Warner
d87bfa9eb2
test_util: add full coverage for allmydata.util.deferredutil
2008-02-06 16:41:04 -07:00
Brian Warner
4d4073fb16
test_system: deferred-handling logic in test_upload_and_download was broken
2008-02-06 14:05:11 -07:00
Brian Warner
2dcac796e9
add 'tahoe find-shares' command, to locate share files on a local node's disk
2008-02-06 13:19:51 -07:00
Brian Warner
e92a2b5ab2
dump-cap: include UEB_hash in output
2008-02-06 12:48:19 -07:00
Brian Warner
6e0d3059a3
webish: add when_done= to POST /uri?t=upload . I did not add a 'recent uploads' section to the welcome page, but I think the new upload-results page provides the desired data
2008-02-06 02:38:16 -07:00
Brian Warner
124fb5ecdf
add upload-results timing info for helper uploads. This changes the Helper protocol, and introduces a compatibility break
2008-02-06 01:52:25 -07:00
Brian Warner
1e4504463c
webish: make POST /uri?t=upload deposit you on an 'Upload Results' page
2008-02-05 23:01:37 -07:00
Brian Warner
27f46f2aaf
test_web.py: hush pyflakes
2008-02-05 22:37:51 -07:00
Brian Warner
23c2c8d660
webish: add PUT /uri?mutable=true
2008-02-05 22:18:02 -07:00
Brian Warner
e5dc9a8486
webish: add POST /uri?t=upload&mutable=true
2008-02-05 22:10:22 -07:00
Brian Warner
6afe50e2aa
webish: add upload/view-uri forms (not associated with any particular directory) to the welcome page. Document POST /uri?t=upload .
2008-02-05 21:44:40 -07:00
Brian Warner
66f33ee504
upload: return an UploadResults instance (with .uri) instead of just a URI
2008-02-05 21:01:38 -07:00
Brian Warner
5103bf8148
storage: change service name from 'storageserver' to 'storage'
2008-02-05 20:28:59 -07:00
Brian Warner
d146ef7e09
webish: add extra introducer data (version, timestamps) to Welcome page
2008-02-05 17:32:27 -07:00
Brian Warner
a7ea39e626
test_introducer.py: increase timeouts on poll() calls
2008-02-05 16:37:58 -07:00
Brian Warner
d98fde952c
introducer: remove remaining bits of 'push-to-myself' flags. The uploading/downloading node is no longer special.
2008-02-05 14:16:01 -07:00
Brian Warner
daecca6589
big introducer refactoring: separate publish+subscribe. Addresses #271 .
2008-02-05 13:05:13 -07:00
Brian Warner
3a5ba35215
PollMixin: add timeout= argument, rewrite to avoid tail-recursion problems
2008-02-04 20:35:07 -07:00
Brian Warner
a01f9ce9cc
introducer: allow nodes to refrain from publishing themselves, by passing furl=None. This would be useful for clients who do not run storage servers.
2008-02-01 19:48:38 -07:00
Zooko O'Whielacronx
1d1628e525
rename storage_index_chk_hash() to storage_index_hash() and add TODO about how our use of it now includes keys that are not CHKs
2008-02-01 12:27:37 -07:00
Brian Warner
670933ecee
storage: clean up use of si_s vs si_dir, add test for BadWriterEnabler message, add some logging
2008-01-31 17:48:48 -07:00
Zooko O'Whielacronx
7bf21082f7
remove unused import (thanks, pyflakes)
2008-01-31 17:00:59 -07:00
Zooko O'Whielacronx
79c439d026
storage: make two levels of share directories so as not to exceed certain filesystems's limitations on directory size
...
The filesystem which gets my vote for most undeservedly popular is ext3, and it has a hard limit of 32,000 entries in a directory. Many other filesystems (even ones that I like more than I like ext3) have either hard limits or bad performance consequences or weird edge cases when you get too many entries in a single directory.
This patch makes it so that there is a layer of intermediate directories between the "shares" directory and the actual storage-index directory (the one whose name contains the entire storage index (z-base-32 encoded) and which contains one or more share files named by their share number).
The intermediate directories are named by the first 14 bits of the storage index, which means there are at most 16384 of them. (This also means that the intermediate directory names are not a leading prefix of the storage-index directory names -- to do that would have required us to have intermediate directories limited to either 1024 (2-char), which is too few, or 32768 (3-chars of a full 5 bits each), which would overrun ext3's funny hard limit of 32,000.))
This closes #150 , and please see the "convertshares.py" script attached to #150 to convert your old tahoe-0.7.0 storage/shares directory into a new tahoe-0.8.0 storage/shares directory.
2008-01-31 16:26:28 -07:00
Brian Warner
22071c00e0
upload: oops, fix breakage after removing upload_file/upload_data/etc
2008-01-30 19:41:43 -07:00
Brian Warner
909d848071
remove upload.upload_(data,filename,filehandle) convenience functions
2008-01-30 19:03:19 -07:00
Brian Warner
81eeafc574
upload-helper: avoid duplicate uploads: check the grid to see if the file already exists
2008-01-30 18:49:02 -07:00
Zooko O'Whielacronx
ca971559e6
make content-hash-key encryption a parameter of uploading
...
fixes #293
2008-01-30 12:24:50 -07:00
Zooko O'Whielacronx
424d338c82
tests: add a test that nevow can find its default css file
2008-01-30 11:41:58 -07:00
Brian Warner
492cb92dc8
speedcheck: track SSK creation time separately
2008-01-29 20:44:32 -07:00
Brian Warner
d9d0920177
webish: link to directory URIs rather than a child path. Addresses #103 .
2008-01-29 18:04:32 -07:00
Brian Warner
8063aa8b86
WriteBucketProxy: improve __repr__
2008-01-28 18:53:51 -07:00
Brian Warner
4c5518faef
webish: upload+localdir=missing should give an error
2008-01-28 14:48:06 -07:00
Brian Warner
69a0b5cc00
CHK upload helper: don't let one failed upload prevent us from trying again
2008-01-28 12:58:13 -07:00
Brian Warner
8f1212edac
encode.py: don't allow a shareholder which dies in start() to kill the whole upload
2008-01-28 12:14:48 -07:00
Brian Warner
1ff21d1d64
test_upload.py: implement remote_abort on our fake BucketWriter
2008-01-23 18:07:34 -07:00
robk-tahoe
7c34658438
offloaded: fix failure in unit test on windows
...
in trying to test my fix for the failure of the offloaded unit test on windows
(by closing the reader before unlinking the encoding file - which, perhaps
disturbingly doesn't actually make a difference in my windows environment)
I was unable too because the unit test failed every time with a connection lost
error.
after much more time than I'd like to admit it took, I eventually managed to
track that down to a part of the unit test which is supposed to be be dropping
a connection. it looks like the exceptions that get thrown on unix, or at
least all the specific environments brian tested in, for that dropped
connection are different from what is thrown on my box (which is running py2.4
and twisted 2.4.0, for reference) adding ConnectionLost to the list of
expected exceptions makes the test pass.
though curiously still my test logs a NotEnoughWritersError error, and I'm not
currently able to fathom why that exception isn't leading to any overall
failure of the unit test itself.
for general interest, a large part of the time spent trying to track this down
was lost to the state of logging. I added a whole bunch of logging to try
and track down where the tests were failing, but then spent a bunch of time
searching in vain for that log output. as far as I can tell at this point
the unit tests are themselves logging to foolscap's log module, but that isn't
being directed anywhere, so all the test's logging is being black holed.
2008-01-17 20:57:29 -07:00
Brian Warner
fd0dc3013c
offloaded: update unit tests: assert that interrupt/resume works, and that the helper deletes tempfiles
2008-01-17 01:18:10 -07:00
Brian Warner
51321944f0
megapatch: overhaul encoding_parameters handling: now it comes from the Uploadable, or the Client. Removed options= too. Also move helper towards resumability.
2008-01-16 03:03:35 -07:00
Brian Warner
821521cc3e
test_storage: fix pyflakes warnings
2008-01-14 21:26:48 -07:00
Brian Warner
9f0ee4b758
test_system: fix pyflakes warnings
2008-01-14 21:26:28 -07:00
Brian Warner
168a8c3b73
offloaded: improve logging, pass through options, get ready for testing interrupted uploads. test_system: add (disabled) interrupted-upload test
2008-01-14 21:24:26 -07:00
Brian Warner
a6ca98ac53
upload: add Encoder.abort(), to abandon the upload in progress. Add some debug hooks to enable unit tests.
2008-01-14 21:22:55 -07:00
Brian Warner
cb76ed36f9
logging: enable flogging in more places, replace Node.log with flogging
2008-01-14 21:16:58 -07:00
Brian Warner
7ac2b94aba
remove wait_for_numpeers and the when_enough_peers call in mutable.Publish
2008-01-14 14:55:59 -07:00
Brian Warner
222718f13c
test_GET_DIRURL_large: reduce from 400 to 200 children: the test fails to
...
fail anyways, and 200 ought to be enough to trigger the problem, so 400 is
overkill, and just wastes CPU.
2008-01-14 22:31:41 -07:00
Brian Warner
37b7f136ce
'tahoe dump-cap': accept http:// -prefixed URLs too
2008-01-14 14:12:27 -07:00
Brian Warner
6ca0efeef6
add 'tahoe dump-cap' command, to show storage index, lease secrets, etc
2008-01-14 13:43:25 -07:00
Brian Warner
76ee9cccfe
storage: improve logging a bit
2008-01-14 11:58:58 -07:00
Zooko O'Whielacronx
676ded6a7e
tests: put back skipped and todo tests
...
closes #258 -- "put back skipped and todo tests"
2008-01-14 21:02:41 -07:00
Brian Warner
964edadf44
offloaded: add a system test, make it pass. files are now being uploaded through the helper.
2008-01-11 05:42:55 -07:00
Brian Warner
6ac01fde4c
offloaded: more test coverage on client side, change interfaces a bit
2008-01-11 04:53:37 -07:00
Brian Warner
11179bb7d7
test_mutable.py: accomodate changes to mutable.py logging
2008-01-10 22:18:34 -07:00
Zooko O'Whielacronx
69dba9a704
tests: increase the timeout on a test which failed on the overloaded virtual buildslaves
2008-01-11 14:27:54 -07:00
Brian Warner
e825406fc2
offloaded: move interfaces to interfaces.py, start implementing backend
2008-01-09 21:25:47 -07:00
Brian Warner
da7778774b
offloaded: basic test for client-side of AssistedUploader
2008-01-09 20:25:50 -07:00
Brian Warner
0e2ddb00be
test_system: slight refactoring to eventually make it easier to configure some nodes with the output of others
2008-01-09 20:23:54 -07:00
Brian Warner
d4e9e3b9c4
check_speed.py: re-enable 100x200B and 1x100MB CHK tests
2008-01-08 20:59:48 -07:00
Brian Warner
464637605e
test_system: assert that BASEDIR/node.url is created properly
2008-01-07 17:46:22 -07:00
Zooko O'Whielacronx
4223f7216d
rename "dir-uri" to "dir-cap"
2008-01-08 10:41:27 -07:00
Zooko O'Whielacronx
bf14613b6a
suppress the skipped and the todo unit tests in order to make unit test results prettier (the install.html instructs users to run the unit tests)
2008-01-04 00:36:18 -07:00
Zooko O'Whielacronx
32d2cc8aba
fix test_cli to put the root_dir.cap in the private subdir
2008-01-03 17:48:53 -07:00
Zooko O'Whielacronx
23961448da
a few formatting tidy-ups
2008-01-03 17:14:19 -07:00
Brian Warner
acfb11d26f
test_client.py: validate more versioning code
2008-01-03 14:38:24 -07:00
Brian Warner
6a2e5d4aea
test_client.py: assert allmydata.__version__ is not unknown
2008-01-03 14:34:59 -07:00
Zooko O'Whielacronx
5995d11a3c
remove automatic private dir
...
* rename my_private_dir.cap to root_dir.cap
* move it into the private subdir
* change the cmdline argument "--root-uri=[private]" to "--dir-uri=[root]"
2008-01-03 17:02:05 -07:00
Zooko O'Whielacronx
eb373c0001
add human-encodings of caps, refactor encodings of caps, tighten checks, fix unit tests to use values of the right size
2008-01-03 16:55:43 -07:00
Brian Warner
6b81ebfd8e
refactor webish.py slightly, improve the test coverage a bit. One test is marked SKIP pending improvements in webish.py
2007-12-25 03:48:57 -07:00
Brian Warner
fae5de8d0d
test_web.py: fix pyflakes error
2007-12-25 01:56:04 -07:00
Brian Warner
4ba1af37a6
test_web: more coverage of URIPOSTHandler error cases
2007-12-24 22:49:35 -07:00
Brian Warner
7d311b8dd7
test_web: test POST_upload_no_link against /uri, not /uri/
2007-12-24 17:55:38 -07:00
Brian Warner
ae760b3ee4
test_web.py: use /uri?t=mkdir instead of /uri/?t=mkdir, and confirm that the redirection target is correct
2007-12-24 17:46:52 -07:00
