Brian Warner
7b21054c33
UNDO: upload: stop putting plaintext and ciphertext hashes in shares.
...
This removes the guess-partial-information attack vector, and reduces
the amount of overhead that we consume with each file. It also introduces
a forwards-compability break: older versions of the code (before the
previous download-time "make hashes optional" patch) will be unable
to read files uploaded by this version, as they will complain about the
missing hashes. This patch is experimental, and is being pushed into
trunk to obtain test coverage. We may undo it before releasing 1.0.
2008-03-23 15:35:54 -07:00
Zooko O'Whielacronx
dfdbb95648
fix check-memory to use new upload API (which requires a "convergence" argument), and change it to measure convergence instead of random-key, since convergence is the use case we care about more
2008-03-24 15:28:04 -07:00
Zooko O'Whielacronx
fc3bd0c987
use added secret to protect convergent encryption
...
Now upload or encode methods take a required argument named "convergence" which can be either None, indicating no convergent encryption at all, or a string, which is the "added secret" to be mixed in to the content hash key. If you want traditional convergent encryption behavior, set the added secret to be the empty string.
This patch also renames "content hash key" to "convergent encryption" in a argument names and variable names. (A different and larger renaming is needed in order to clarify that Tahoe supports immutable files which are not encrypted content-hash-key a.k.a. convergent encryption.)
This patch also changes a few unit tests to use non-convergent encryption, because it doesn't matter for what they are testing and non-convergent encryption is slightly faster.
2008-03-24 09:46:06 -07:00
Brian Warner
7996131a0a
upload: stop putting plaintext and ciphertext hashes in shares.
...
This removes the guess-partial-information attack vector, and reduces
the amount of overhead that we consume with each file. It also introduces
a forwards-compability break: older versions of the code (before the
previous download-time "make hashes optional" patch) will be unable
to read files uploaded by this version, as they will complain about the
missing hashes. This patch is experimental, and is being pushed into
trunk to obtain test coverage. We may undo it before releasing 1.0.
2008-03-23 15:35:54 -07:00
Brian Warner
553367d567
download: make plaintext and ciphertext hashes in the UEB optional.
...
Removing the plaintext hashes can help with the guess-partial-information
attack. This does not affect compatibility, but if and when we actually
remove any hashes from the share, that will introduce a
forwards-compatibility break: tahoe-0.9 will not be able to read such files.
2008-03-23 14:46:49 -07:00
robk-tahoe
e75c417bb5
webish mkdir-p: added unit test
...
added a test for the simple mkdir-p hack I added yesterday
checks that mkdir-p can create a directory hierarchy, and that resubmitting
a request for the same path yields the existing dir's uri
2008-03-18 13:11:08 -07:00
Brian Warner
2ef70ab814
mutable.py: split replace() into update() and overwrite(). Addresses #328 .
2008-03-12 18:00:43 -07:00
Brian Warner
89be2e1bea
introweb: combine announcement and subscriber information to show version+nickname for each client
2008-03-11 19:21:29 -07:00
Brian Warner
810ba68343
add a webserver for the Introducer, showing service announcements and subscriber lists
2008-03-11 17:36:25 -07:00
Brian Warner
c727348d85
mutable: tolerate multiple encodings, using whichever version is recoverable first. Closes #312
2008-03-11 00:26:00 -07:00
Brian Warner
791482cf8d
test_mutable: improve multiple-encodings test coverage
2008-03-10 23:47:35 -07:00
Brian Warner
9ca55b8b79
test_mutable: make test-multiple-encodings work
2008-03-10 23:16:28 -07:00
Brian Warner
57bd23f35f
test_mutable: more test coverage, building up a framework to cause reads to occur in a specific order
2008-03-10 22:15:43 -07:00
Brian Warner
11445b5109
test_mutable: exercise short reads too
2008-03-10 18:08:23 -07:00
Brian Warner
be5a6147b4
test_mutable: test all hash-failure cases except a corrupted encrypted private key
2008-03-10 17:46:52 -07:00
Brian Warner
647734cd3b
test_mutable: add Roundtrip test, suitable for new share-mangling tests
2008-03-10 16:14:08 -07:00
Brian Warner
9e460cd22e
mutable: cosmetic changes
2008-03-10 15:44:05 -07:00
Brian Warner
d6cde55c63
mutable: minor refactoring of _do_read, to make other tests easier
2008-03-10 15:42:56 -07:00
Brian Warner
4fa622f9b1
webish: this file is too big, start breaking it into pieces, beginning with status
2008-03-05 14:59:56 -07:00
Brian Warner
7d9bfbcabe
mutable: oops, .download *is* in use, by the speedtest. Restore it and add a test.
2008-03-04 15:11:40 -07:00
Brian Warner
4daa817502
test_util.py: get 100% test coverage for hashutil.py
2008-03-04 14:42:25 -07:00
Brian Warner
95a085763f
uri.py: get 100% test coverage, fix a few bugs in the process
2008-03-04 14:27:45 -07:00
Brian Warner
e6489b1ef2
test_filenode.py : improve coverage of mutable filenode, fix a bug in __hash__
2008-03-04 14:01:28 -07:00
Brian Warner
f6ca62df31
test_system: improve test coverage of publish/retrieve status
2008-03-04 01:24:35 -07:00
Brian Warner
68fbd89e66
webish: add primitive publish/retrieve status pages
2008-03-04 01:07:44 -07:00
Brian Warner
7e159feb27
stats: make StatsGatherer happy about sharing a process with other services, add one during system test to get some test coverage
2008-03-03 23:55:58 -07:00
Brian Warner
abb51e70f3
test_web: improve upload/download status coverage
2008-03-03 21:56:23 -07:00
Brian Warner
3ecb483e35
test_system: add test coverage for download-status and upload-status
2008-03-03 21:37:17 -07:00
Brian Warner
886ef22335
webish: download-results: add server_problems
2008-03-03 20:30:35 -07:00
Brian Warner
c8e24f0904
webish: make upload timings visible on the recent uploads/downloads status page
2008-03-03 14:48:52 -07:00
Brian Warner
436baa1b19
webish: add per-file upload/download status pages
2008-02-29 23:03:00 -07:00
Brian Warner
1a7651ce82
retain 10 most recent upload/download status objects, show them in /status . Prep for showing individual status objects
2008-02-29 22:19:03 -07:00
Zooko O'Whielacronx
99f006c584
wapi: add POST /uri/$DIRECTORY?t=set_children
...
Unfinished bits: doc in webapi.txt, test handling of badly formed JSON, return reasonable HTTP response, examination of the effect of this patch on code coverage -- but I'm committing it anyway because MikeB can use it and I'm being called to dinner...
2008-02-29 18:40:27 -07:00
Zooko O'Whielacronx
4287079455
test: update todo string in test_nevow
2008-02-29 13:10:08 -07:00
Brian Warner
301dd3d489
webish status: distinguish active uploads/downloads from recent ones
2008-02-26 15:35:28 -07:00
Brian Warner
1af3bfb1d4
test_runner.py: test launching an introducer too
2008-02-18 00:28:56 -07:00
Brian Warner
6ff98a3c0b
'tahoe start': allow this to start arbitrary .tac file -bearing directories (like the log gatherer)
2008-02-18 00:28:17 -07:00
Brian Warner
c3a1491cf4
test_system.py: improve coverage of webish.py
2008-02-15 04:02:50 -07:00
Brian Warner
72874390fe
BIG COMPATIBILITY BREAK: update hash tags, switch to SHA-256d everywhere
2008-02-14 19:58:01 -07:00
Zooko O'Whielacronx
fc0d637523
docs: update install and usage docs, improve cli "usage" output, make new example directories, add unit test that fails code which prints out sentences that don't end with punctuation marks
2008-02-15 13:11:02 -07:00
Zooko O'Whielacronx
5942fca1e1
add unit test asserting that allmydata.util.base32 matches rfc 3548
2008-02-14 20:20:19 -07:00
Zooko O'Whielacronx
3eb95c1fad
merge patch which adds base32 to test_util with patch which removes bencode from test_util
2008-02-14 19:35:32 -07:00
Brian Warner
09bfa3bea2
remove bencode, yay
2008-02-14 18:06:41 -07:00
Brian Warner
7927495cbe
unicode handling: declare dirnodes to contain unicode child names, update webish to match
2008-02-14 15:45:56 -07:00
Zooko O'Whielacronx
7c6de95bc6
switch from base62 to base32 for storage indices, switch from z-base-32 to rfc 3548 base-32 for everything, separate out base32 encoding from idlib
2008-02-14 19:27:47 -07:00
Brian Warner
a050204833
add test coverage for the /stats web page
2008-02-13 13:57:39 -07:00
Brian Warner
871e7756f7
test_web: remove test_GET_DIRURL_large, it failed to exercise the problem in #237 and consumes a disproportionate amount of time
2008-02-13 13:08:26 -07:00
Zooko O'Whielacronx
e89edf8803
fix a few unused imports and suchlike, discovered by pyflakes
2008-02-13 07:38:08 -07:00
Brian Warner
a7867aeea2
introducer: remove encoding-parameter config, for now
2008-02-12 18:59:54 -07:00
Zooko O'Whielacronx
3f8df27063
use base62 encoding for storage indexes, on disk and in verifier caps, and in logging and diagnostic tools
...
base62 encoding fits more information into alphanumeric chars while avoiding the troublesome non-alphanumeric chars of base64 encoding. In particular, this allows us to work around the ext3 "32,000 entries in a directory" limit while retaining the convenient property that the intermediate directory names are leading prefixes of the storage index file names.
2008-02-12 20:48:37 -07:00
Brian Warner
a70fba4ef1
webish: add edge metadata to t=json output, including timestamps
2008-02-11 19:14:10 -07:00
Brian Warner
ca27b8e5ad
add 'tahoe catalog-shares' tool, to make a one-line summary of each share file. This can help do cross-server correlation of sharefiles, looking for anomalies
2008-02-11 18:17:01 -07:00
Brian Warner
5f50543c3b
test_dirnode.py: improve error messages in case timestamp tests fail
2008-02-11 17:37:51 -07:00
Brian Warner
09cf1e2a8a
test_dirnode.py: simplejson-1.7.1 incorrectly rounds floats to two decimal places. Don't let this bug flunk the timestamp test.
2008-02-11 17:35:17 -07:00
Brian Warner
7123c9d875
test_system.py: refactor bounce_client, probably make it stop failing on cygwin
2008-02-11 15:26:58 -07:00
Brian Warner
3a1db45579
test_dirnode.py: assert that we update mtime and preserve ctime
2008-02-11 15:12:55 -07:00
Brian Warner
e6ddd9f3da
dirnode.py: add metadata= to add_file(), add tests
2008-02-11 14:53:28 -07:00
Brian Warner
b90eaf75da
test_dirnode.py: add diag output to test-ctime/mtime tests
2008-02-11 14:13:07 -07:00
Brian Warner
622c477e31
dirnode: add ctime/mtime to metadata, update metadata-modifying APIs. Needs more testing and sanity checking.
2008-02-08 18:43:47 -07:00
Brian Warner
873dee53e9
test_system: remove the hackish debug_interrupt= attribute magic used to exercise interrupted-upload resumption, instead just make the Uploadable bounce the helper halfway through the upload
2008-02-07 20:15:37 -07:00
Brian Warner
3bd79917b2
test_system.py: remove that ugly debug_stash_RemoteencryptedUploadable hack, now that UploadResults give us a better approach
2008-02-07 17:27:30 -07:00
Brian Warner
96ca6c2033
test_upload.py: hush pyflakes
2008-02-06 20:04:31 -07:00
Brian Warner
f78c80d7aa
test_upload.py: add test to exercise CHK hashing variations
2008-02-06 20:03:35 -07:00
Brian Warner
02e7c56563
test_cli: oops, need to update this when the CHK hash changes
2008-02-06 19:58:53 -07:00
Brian Warner
da9610e90a
change encryption-key hash to include encoding parameters. This is a minor compatibility break: CHK files encoded (with convergence) before and after this will have different keys and ciphertexts. Also switched to SHA-256d for both the data-to-key hash and the key-to-storageindex hash
2008-02-06 19:50:47 -07:00
Brian Warner
9f9a458249
hashutil: add tagged_hash_256d and tagged_hasher_256d
2008-02-06 19:36:43 -07:00
Brian Warner
81c5ceae16
upload: rework passing of default encoding parameters: move more responsibility into BaseUploadable
2008-02-06 18:39:03 -07:00
Brian Warner
75e4332462
offloaded: oops, need more tricks to make the unit tests pass
2008-02-06 17:51:11 -07:00
Brian Warner
d87bfa9eb2
test_util: add full coverage for allmydata.util.deferredutil
2008-02-06 16:41:04 -07:00
Brian Warner
4d4073fb16
test_system: deferred-handling logic in test_upload_and_download was broken
2008-02-06 14:05:11 -07:00
Brian Warner
2dcac796e9
add 'tahoe find-shares' command, to locate share files on a local node's disk
2008-02-06 13:19:51 -07:00
Brian Warner
e92a2b5ab2
dump-cap: include UEB_hash in output
2008-02-06 12:48:19 -07:00
Brian Warner
6e0d3059a3
webish: add when_done= to POST /uri?t=upload . I did not add a 'recent uploads' section to the welcome page, but I think the new upload-results page provides the desired data
2008-02-06 02:38:16 -07:00
Brian Warner
124fb5ecdf
add upload-results timing info for helper uploads. This changes the Helper protocol, and introduces a compatibility break
2008-02-06 01:52:25 -07:00
Brian Warner
1e4504463c
webish: make POST /uri?t=upload deposit you on an 'Upload Results' page
2008-02-05 23:01:37 -07:00
Brian Warner
27f46f2aaf
test_web.py: hush pyflakes
2008-02-05 22:37:51 -07:00
Brian Warner
23c2c8d660
webish: add PUT /uri?mutable=true
2008-02-05 22:18:02 -07:00
Brian Warner
e5dc9a8486
webish: add POST /uri?t=upload&mutable=true
2008-02-05 22:10:22 -07:00
Brian Warner
6afe50e2aa
webish: add upload/view-uri forms (not associated with any particular directory) to the welcome page. Document POST /uri?t=upload .
2008-02-05 21:44:40 -07:00
Brian Warner
66f33ee504
upload: return an UploadResults instance (with .uri) instead of just a URI
2008-02-05 21:01:38 -07:00
Brian Warner
5103bf8148
storage: change service name from 'storageserver' to 'storage'
2008-02-05 20:28:59 -07:00
Brian Warner
d146ef7e09
webish: add extra introducer data (version, timestamps) to Welcome page
2008-02-05 17:32:27 -07:00
Brian Warner
a7ea39e626
test_introducer.py: increase timeouts on poll() calls
2008-02-05 16:37:58 -07:00
Brian Warner
d98fde952c
introducer: remove remaining bits of 'push-to-myself' flags. The uploading/downloading node is no longer special.
2008-02-05 14:16:01 -07:00
Brian Warner
daecca6589
big introducer refactoring: separate publish+subscribe. Addresses #271 .
2008-02-05 13:05:13 -07:00
Brian Warner
3a5ba35215
PollMixin: add timeout= argument, rewrite to avoid tail-recursion problems
2008-02-04 20:35:07 -07:00
Brian Warner
a01f9ce9cc
introducer: allow nodes to refrain from publishing themselves, by passing furl=None. This would be useful for clients who do not run storage servers.
2008-02-01 19:48:38 -07:00
Zooko O'Whielacronx
1d1628e525
rename storage_index_chk_hash() to storage_index_hash() and add TODO about how our use of it now includes keys that are not CHKs
2008-02-01 12:27:37 -07:00
Brian Warner
670933ecee
storage: clean up use of si_s vs si_dir, add test for BadWriterEnabler message, add some logging
2008-01-31 17:48:48 -07:00
Zooko O'Whielacronx
7bf21082f7
remove unused import (thanks, pyflakes)
2008-01-31 17:00:59 -07:00
Zooko O'Whielacronx
79c439d026
storage: make two levels of share directories so as not to exceed certain filesystems's limitations on directory size
...
The filesystem which gets my vote for most undeservedly popular is ext3, and it has a hard limit of 32,000 entries in a directory. Many other filesystems (even ones that I like more than I like ext3) have either hard limits or bad performance consequences or weird edge cases when you get too many entries in a single directory.
This patch makes it so that there is a layer of intermediate directories between the "shares" directory and the actual storage-index directory (the one whose name contains the entire storage index (z-base-32 encoded) and which contains one or more share files named by their share number).
The intermediate directories are named by the first 14 bits of the storage index, which means there are at most 16384 of them. (This also means that the intermediate directory names are not a leading prefix of the storage-index directory names -- to do that would have required us to have intermediate directories limited to either 1024 (2-char), which is too few, or 32768 (3-chars of a full 5 bits each), which would overrun ext3's funny hard limit of 32,000.))
This closes #150 , and please see the "convertshares.py" script attached to #150 to convert your old tahoe-0.7.0 storage/shares directory into a new tahoe-0.8.0 storage/shares directory.
2008-01-31 16:26:28 -07:00
Brian Warner
22071c00e0
upload: oops, fix breakage after removing upload_file/upload_data/etc
2008-01-30 19:41:43 -07:00
Brian Warner
909d848071
remove upload.upload_(data,filename,filehandle) convenience functions
2008-01-30 19:03:19 -07:00
Brian Warner
81eeafc574
upload-helper: avoid duplicate uploads: check the grid to see if the file already exists
2008-01-30 18:49:02 -07:00
Zooko O'Whielacronx
ca971559e6
make content-hash-key encryption a parameter of uploading
...
fixes #293
2008-01-30 12:24:50 -07:00
Zooko O'Whielacronx
424d338c82
tests: add a test that nevow can find its default css file
2008-01-30 11:41:58 -07:00
Brian Warner
492cb92dc8
speedcheck: track SSK creation time separately
2008-01-29 20:44:32 -07:00
Brian Warner
d9d0920177
webish: link to directory URIs rather than a child path. Addresses #103 .
2008-01-29 18:04:32 -07:00
Brian Warner
8063aa8b86
WriteBucketProxy: improve __repr__
2008-01-28 18:53:51 -07:00
Brian Warner
4c5518faef
webish: upload+localdir=missing should give an error
2008-01-28 14:48:06 -07:00
