Brian Warner
4fa622f9b1
webish: this file is too big, start breaking it into pieces, beginning with status
2008-03-05 14:59:56 -07:00
Brian Warner
7d9bfbcabe
mutable: oops, .download *is* in use, by the speedtest. Restore it and add a test.
2008-03-04 15:11:40 -07:00
Brian Warner
4daa817502
test_util.py: get 100% test coverage for hashutil.py
2008-03-04 14:42:25 -07:00
Brian Warner
95a085763f
uri.py: get 100% test coverage, fix a few bugs in the process
2008-03-04 14:27:45 -07:00
Brian Warner
e6489b1ef2
test_filenode.py : improve coverage of mutable filenode, fix a bug in __hash__
2008-03-04 14:01:28 -07:00
Brian Warner
f6ca62df31
test_system: improve test coverage of publish/retrieve status
2008-03-04 01:24:35 -07:00
Brian Warner
68fbd89e66
webish: add primitive publish/retrieve status pages
2008-03-04 01:07:44 -07:00
Brian Warner
7e159feb27
stats: make StatsGatherer happy about sharing a process with other services, add one during system test to get some test coverage
2008-03-03 23:55:58 -07:00
Brian Warner
abb51e70f3
test_web: improve upload/download status coverage
2008-03-03 21:56:23 -07:00
Brian Warner
3ecb483e35
test_system: add test coverage for download-status and upload-status
2008-03-03 21:37:17 -07:00
Brian Warner
886ef22335
webish: download-results: add server_problems
2008-03-03 20:30:35 -07:00
Brian Warner
c8e24f0904
webish: make upload timings visible on the recent uploads/downloads status page
2008-03-03 14:48:52 -07:00
Brian Warner
436baa1b19
webish: add per-file upload/download status pages
2008-02-29 23:03:00 -07:00
Brian Warner
1a7651ce82
retain 10 most recent upload/download status objects, show them in /status . Prep for showing individual status objects
2008-02-29 22:19:03 -07:00
Zooko O'Whielacronx
99f006c584
wapi: add POST /uri/$DIRECTORY?t=set_children
...
Unfinished bits: doc in webapi.txt, test handling of badly formed JSON, return reasonable HTTP response, examination of the effect of this patch on code coverage -- but I'm committing it anyway because MikeB can use it and I'm being called to dinner...
2008-02-29 18:40:27 -07:00
Zooko O'Whielacronx
4287079455
test: update todo string in test_nevow
2008-02-29 13:10:08 -07:00
Brian Warner
301dd3d489
webish status: distinguish active uploads/downloads from recent ones
2008-02-26 15:35:28 -07:00
Brian Warner
1af3bfb1d4
test_runner.py: test launching an introducer too
2008-02-18 00:28:56 -07:00
Brian Warner
6ff98a3c0b
'tahoe start': allow this to start arbitrary .tac file -bearing directories (like the log gatherer)
2008-02-18 00:28:17 -07:00
Brian Warner
c3a1491cf4
test_system.py: improve coverage of webish.py
2008-02-15 04:02:50 -07:00
Brian Warner
72874390fe
BIG COMPATIBILITY BREAK: update hash tags, switch to SHA-256d everywhere
2008-02-14 19:58:01 -07:00
Zooko O'Whielacronx
fc0d637523
docs: update install and usage docs, improve cli "usage" output, make new example directories, add unit test that fails code which prints out sentences that don't end with punctuation marks
2008-02-15 13:11:02 -07:00
Zooko O'Whielacronx
5942fca1e1
add unit test asserting that allmydata.util.base32 matches rfc 3548
2008-02-14 20:20:19 -07:00
Zooko O'Whielacronx
3eb95c1fad
merge patch which adds base32 to test_util with patch which removes bencode from test_util
2008-02-14 19:35:32 -07:00
Brian Warner
09bfa3bea2
remove bencode, yay
2008-02-14 18:06:41 -07:00
Brian Warner
7927495cbe
unicode handling: declare dirnodes to contain unicode child names, update webish to match
2008-02-14 15:45:56 -07:00
Zooko O'Whielacronx
7c6de95bc6
switch from base62 to base32 for storage indices, switch from z-base-32 to rfc 3548 base-32 for everything, separate out base32 encoding from idlib
2008-02-14 19:27:47 -07:00
Brian Warner
a050204833
add test coverage for the /stats web page
2008-02-13 13:57:39 -07:00
Brian Warner
871e7756f7
test_web: remove test_GET_DIRURL_large, it failed to exercise the problem in #237 and consumes a disproportionate amount of time
2008-02-13 13:08:26 -07:00
Zooko O'Whielacronx
e89edf8803
fix a few unused imports and suchlike, discovered by pyflakes
2008-02-13 07:38:08 -07:00
Brian Warner
a7867aeea2
introducer: remove encoding-parameter config, for now
2008-02-12 18:59:54 -07:00
Zooko O'Whielacronx
3f8df27063
use base62 encoding for storage indexes, on disk and in verifier caps, and in logging and diagnostic tools
...
base62 encoding fits more information into alphanumeric chars while avoiding the troublesome non-alphanumeric chars of base64 encoding. In particular, this allows us to work around the ext3 "32,000 entries in a directory" limit while retaining the convenient property that the intermediate directory names are leading prefixes of the storage index file names.
2008-02-12 20:48:37 -07:00
Brian Warner
a70fba4ef1
webish: add edge metadata to t=json output, including timestamps
2008-02-11 19:14:10 -07:00
Brian Warner
ca27b8e5ad
add 'tahoe catalog-shares' tool, to make a one-line summary of each share file. This can help do cross-server correlation of sharefiles, looking for anomalies
2008-02-11 18:17:01 -07:00
Brian Warner
5f50543c3b
test_dirnode.py: improve error messages in case timestamp tests fail
2008-02-11 17:37:51 -07:00
Brian Warner
09cf1e2a8a
test_dirnode.py: simplejson-1.7.1 incorrectly rounds floats to two decimal places. Don't let this bug flunk the timestamp test.
2008-02-11 17:35:17 -07:00
Brian Warner
7123c9d875
test_system.py: refactor bounce_client, probably make it stop failing on cygwin
2008-02-11 15:26:58 -07:00
Brian Warner
3a1db45579
test_dirnode.py: assert that we update mtime and preserve ctime
2008-02-11 15:12:55 -07:00
Brian Warner
e6ddd9f3da
dirnode.py: add metadata= to add_file(), add tests
2008-02-11 14:53:28 -07:00
Brian Warner
b90eaf75da
test_dirnode.py: add diag output to test-ctime/mtime tests
2008-02-11 14:13:07 -07:00
Brian Warner
622c477e31
dirnode: add ctime/mtime to metadata, update metadata-modifying APIs. Needs more testing and sanity checking.
2008-02-08 18:43:47 -07:00
Brian Warner
873dee53e9
test_system: remove the hackish debug_interrupt= attribute magic used to exercise interrupted-upload resumption, instead just make the Uploadable bounce the helper halfway through the upload
2008-02-07 20:15:37 -07:00
Brian Warner
3bd79917b2
test_system.py: remove that ugly debug_stash_RemoteencryptedUploadable hack, now that UploadResults give us a better approach
2008-02-07 17:27:30 -07:00
Brian Warner
96ca6c2033
test_upload.py: hush pyflakes
2008-02-06 20:04:31 -07:00
Brian Warner
f78c80d7aa
test_upload.py: add test to exercise CHK hashing variations
2008-02-06 20:03:35 -07:00
Brian Warner
02e7c56563
test_cli: oops, need to update this when the CHK hash changes
2008-02-06 19:58:53 -07:00
Brian Warner
da9610e90a
change encryption-key hash to include encoding parameters. This is a minor compatibility break: CHK files encoded (with convergence) before and after this will have different keys and ciphertexts. Also switched to SHA-256d for both the data-to-key hash and the key-to-storageindex hash
2008-02-06 19:50:47 -07:00
Brian Warner
9f9a458249
hashutil: add tagged_hash_256d and tagged_hasher_256d
2008-02-06 19:36:43 -07:00
Brian Warner
81c5ceae16
upload: rework passing of default encoding parameters: move more responsibility into BaseUploadable
2008-02-06 18:39:03 -07:00
Brian Warner
75e4332462
offloaded: oops, need more tricks to make the unit tests pass
2008-02-06 17:51:11 -07:00
Brian Warner
d87bfa9eb2
test_util: add full coverage for allmydata.util.deferredutil
2008-02-06 16:41:04 -07:00
Brian Warner
4d4073fb16
test_system: deferred-handling logic in test_upload_and_download was broken
2008-02-06 14:05:11 -07:00
Brian Warner
2dcac796e9
add 'tahoe find-shares' command, to locate share files on a local node's disk
2008-02-06 13:19:51 -07:00
Brian Warner
e92a2b5ab2
dump-cap: include UEB_hash in output
2008-02-06 12:48:19 -07:00
Brian Warner
6e0d3059a3
webish: add when_done= to POST /uri?t=upload . I did not add a 'recent uploads' section to the welcome page, but I think the new upload-results page provides the desired data
2008-02-06 02:38:16 -07:00
Brian Warner
124fb5ecdf
add upload-results timing info for helper uploads. This changes the Helper protocol, and introduces a compatibility break
2008-02-06 01:52:25 -07:00
Brian Warner
1e4504463c
webish: make POST /uri?t=upload deposit you on an 'Upload Results' page
2008-02-05 23:01:37 -07:00
Brian Warner
27f46f2aaf
test_web.py: hush pyflakes
2008-02-05 22:37:51 -07:00
Brian Warner
23c2c8d660
webish: add PUT /uri?mutable=true
2008-02-05 22:18:02 -07:00
Brian Warner
e5dc9a8486
webish: add POST /uri?t=upload&mutable=true
2008-02-05 22:10:22 -07:00
Brian Warner
6afe50e2aa
webish: add upload/view-uri forms (not associated with any particular directory) to the welcome page. Document POST /uri?t=upload .
2008-02-05 21:44:40 -07:00
Brian Warner
66f33ee504
upload: return an UploadResults instance (with .uri) instead of just a URI
2008-02-05 21:01:38 -07:00
Brian Warner
5103bf8148
storage: change service name from 'storageserver' to 'storage'
2008-02-05 20:28:59 -07:00
Brian Warner
d146ef7e09
webish: add extra introducer data (version, timestamps) to Welcome page
2008-02-05 17:32:27 -07:00
Brian Warner
a7ea39e626
test_introducer.py: increase timeouts on poll() calls
2008-02-05 16:37:58 -07:00
Brian Warner
d98fde952c
introducer: remove remaining bits of 'push-to-myself' flags. The uploading/downloading node is no longer special.
2008-02-05 14:16:01 -07:00
Brian Warner
daecca6589
big introducer refactoring: separate publish+subscribe. Addresses #271 .
2008-02-05 13:05:13 -07:00
Brian Warner
3a5ba35215
PollMixin: add timeout= argument, rewrite to avoid tail-recursion problems
2008-02-04 20:35:07 -07:00
Brian Warner
a01f9ce9cc
introducer: allow nodes to refrain from publishing themselves, by passing furl=None. This would be useful for clients who do not run storage servers.
2008-02-01 19:48:38 -07:00
Zooko O'Whielacronx
1d1628e525
rename storage_index_chk_hash() to storage_index_hash() and add TODO about how our use of it now includes keys that are not CHKs
2008-02-01 12:27:37 -07:00
Brian Warner
670933ecee
storage: clean up use of si_s vs si_dir, add test for BadWriterEnabler message, add some logging
2008-01-31 17:48:48 -07:00
Zooko O'Whielacronx
7bf21082f7
remove unused import (thanks, pyflakes)
2008-01-31 17:00:59 -07:00
Zooko O'Whielacronx
79c439d026
storage: make two levels of share directories so as not to exceed certain filesystems's limitations on directory size
...
The filesystem which gets my vote for most undeservedly popular is ext3, and it has a hard limit of 32,000 entries in a directory. Many other filesystems (even ones that I like more than I like ext3) have either hard limits or bad performance consequences or weird edge cases when you get too many entries in a single directory.
This patch makes it so that there is a layer of intermediate directories between the "shares" directory and the actual storage-index directory (the one whose name contains the entire storage index (z-base-32 encoded) and which contains one or more share files named by their share number).
The intermediate directories are named by the first 14 bits of the storage index, which means there are at most 16384 of them. (This also means that the intermediate directory names are not a leading prefix of the storage-index directory names -- to do that would have required us to have intermediate directories limited to either 1024 (2-char), which is too few, or 32768 (3-chars of a full 5 bits each), which would overrun ext3's funny hard limit of 32,000.))
This closes #150 , and please see the "convertshares.py" script attached to #150 to convert your old tahoe-0.7.0 storage/shares directory into a new tahoe-0.8.0 storage/shares directory.
2008-01-31 16:26:28 -07:00
Brian Warner
22071c00e0
upload: oops, fix breakage after removing upload_file/upload_data/etc
2008-01-30 19:41:43 -07:00
Brian Warner
909d848071
remove upload.upload_(data,filename,filehandle) convenience functions
2008-01-30 19:03:19 -07:00
Brian Warner
81eeafc574
upload-helper: avoid duplicate uploads: check the grid to see if the file already exists
2008-01-30 18:49:02 -07:00
Zooko O'Whielacronx
ca971559e6
make content-hash-key encryption a parameter of uploading
...
fixes #293
2008-01-30 12:24:50 -07:00
Zooko O'Whielacronx
424d338c82
tests: add a test that nevow can find its default css file
2008-01-30 11:41:58 -07:00
Brian Warner
492cb92dc8
speedcheck: track SSK creation time separately
2008-01-29 20:44:32 -07:00
Brian Warner
d9d0920177
webish: link to directory URIs rather than a child path. Addresses #103 .
2008-01-29 18:04:32 -07:00
Brian Warner
8063aa8b86
WriteBucketProxy: improve __repr__
2008-01-28 18:53:51 -07:00
Brian Warner
4c5518faef
webish: upload+localdir=missing should give an error
2008-01-28 14:48:06 -07:00
Brian Warner
69a0b5cc00
CHK upload helper: don't let one failed upload prevent us from trying again
2008-01-28 12:58:13 -07:00
Brian Warner
8f1212edac
encode.py: don't allow a shareholder which dies in start() to kill the whole upload
2008-01-28 12:14:48 -07:00
Brian Warner
1ff21d1d64
test_upload.py: implement remote_abort on our fake BucketWriter
2008-01-23 18:07:34 -07:00
robk-tahoe
7c34658438
offloaded: fix failure in unit test on windows
...
in trying to test my fix for the failure of the offloaded unit test on windows
(by closing the reader before unlinking the encoding file - which, perhaps
disturbingly doesn't actually make a difference in my windows environment)
I was unable too because the unit test failed every time with a connection lost
error.
after much more time than I'd like to admit it took, I eventually managed to
track that down to a part of the unit test which is supposed to be be dropping
a connection. it looks like the exceptions that get thrown on unix, or at
least all the specific environments brian tested in, for that dropped
connection are different from what is thrown on my box (which is running py2.4
and twisted 2.4.0, for reference) adding ConnectionLost to the list of
expected exceptions makes the test pass.
though curiously still my test logs a NotEnoughWritersError error, and I'm not
currently able to fathom why that exception isn't leading to any overall
failure of the unit test itself.
for general interest, a large part of the time spent trying to track this down
was lost to the state of logging. I added a whole bunch of logging to try
and track down where the tests were failing, but then spent a bunch of time
searching in vain for that log output. as far as I can tell at this point
the unit tests are themselves logging to foolscap's log module, but that isn't
being directed anywhere, so all the test's logging is being black holed.
2008-01-17 20:57:29 -07:00
Brian Warner
fd0dc3013c
offloaded: update unit tests: assert that interrupt/resume works, and that the helper deletes tempfiles
2008-01-17 01:18:10 -07:00
Brian Warner
51321944f0
megapatch: overhaul encoding_parameters handling: now it comes from the Uploadable, or the Client. Removed options= too. Also move helper towards resumability.
2008-01-16 03:03:35 -07:00
Brian Warner
821521cc3e
test_storage: fix pyflakes warnings
2008-01-14 21:26:48 -07:00
Brian Warner
9f0ee4b758
test_system: fix pyflakes warnings
2008-01-14 21:26:28 -07:00
Brian Warner
168a8c3b73
offloaded: improve logging, pass through options, get ready for testing interrupted uploads. test_system: add (disabled) interrupted-upload test
2008-01-14 21:24:26 -07:00
Brian Warner
a6ca98ac53
upload: add Encoder.abort(), to abandon the upload in progress. Add some debug hooks to enable unit tests.
2008-01-14 21:22:55 -07:00
Brian Warner
cb76ed36f9
logging: enable flogging in more places, replace Node.log with flogging
2008-01-14 21:16:58 -07:00
Brian Warner
7ac2b94aba
remove wait_for_numpeers and the when_enough_peers call in mutable.Publish
2008-01-14 14:55:59 -07:00
Brian Warner
222718f13c
test_GET_DIRURL_large: reduce from 400 to 200 children: the test fails to
...
fail anyways, and 200 ought to be enough to trigger the problem, so 400 is
overkill, and just wastes CPU.
2008-01-14 22:31:41 -07:00
Brian Warner
37b7f136ce
'tahoe dump-cap': accept http:// -prefixed URLs too
2008-01-14 14:12:27 -07:00
Brian Warner
6ca0efeef6
add 'tahoe dump-cap' command, to show storage index, lease secrets, etc
2008-01-14 13:43:25 -07:00
Brian Warner
76ee9cccfe
storage: improve logging a bit
2008-01-14 11:58:58 -07:00
Zooko O'Whielacronx
676ded6a7e
tests: put back skipped and todo tests
...
closes #258 -- "put back skipped and todo tests"
2008-01-14 21:02:41 -07:00
Brian Warner
964edadf44
offloaded: add a system test, make it pass. files are now being uploaded through the helper.
2008-01-11 05:42:55 -07:00
