Zooko O'Whielacronx
b58875fe43
mutable: rename mutable/node.py to mutable/filenode.py and mutable/repair.py to mutable/repairer.py
...
To be more consistent with the immutable layout that I am working on.
2008-12-07 08:20:08 -07:00
Brian Warner
735b3493f1
web/directory.py: really really fix #553 . Unfortunately it's tricky to simulate the behavior of a brower's relative-url handling in a unit test.
2008-12-05 23:14:12 -07:00
francois
db7ad6da12
filenode.py: Fix partial HTTP Range header handling according to RFC2616
...
Tahoe webapi was failing on HTTP request containing a partial Range header.
This change allows movies players like mplayer to seek in movie files stored in
tahoe.
Associated tests for GET and HEAD methods are also included
2008-11-18 07:41:35 -07:00
Brian Warner
ffb5985146
mutable.modify(): after UCWE, publish even if the second invocation of the modifier didn't modify anything. For #551 .
2008-12-05 22:49:23 -07:00
Brian Warner
7a0afb59a4
dirnode.py: dirnode.delete which hits UCWE should not fail with NoSuchChildError. Fixes #550 .
2008-12-05 22:08:37 -07:00
Brian Warner
fb9af2c7a0
MutableFileNode.modify: pass first_time= and servermap= to the modifier callback
2008-12-05 22:07:10 -07:00
Zooko O'Whielacronx
21d88a0cba
web: fix more info links again
...
Really, *really* closes #553 .
2008-12-05 15:39:39 -07:00
Zooko O'Whielacronx
dd834cf7e9
web: fix moreinfo link
2008-12-05 15:29:39 -07:00
Zooko O'Whielacronx
93cd0f58a9
web: "More Info" link describes the same file that the "file" link points to, rather than to the file under the same name in this directory
...
It's a subtle but real difference.
Fixes #553 -- "More Info" link should point to a file/dir, not a dir+childname .
2008-12-05 15:05:02 -07:00
Zooko O'Whielacronx
60bbc46a53
minor: fix unused imports -- thanks, pyflakes
2008-12-05 13:07:23 -07:00
Zooko O'Whielacronx
b315619d6b
download: refactor handling of URI Extension Block and crypttext hash tree, simplify things
...
Refactor into a class the logic of asking each server in turn until one of them gives an answer
that validates. It is called ValidatedThingObtainer.
Refactor the downloading and verification of the URI Extension Block into a class named
ValidatedExtendedURIProxy.
The new logic of validating UEBs is minimalist: it doesn't require the UEB to contain any
unncessary information, but of course it still accepts such information for backwards
compatibility (so that this new download code is able to download files uploaded with old, and
for that matter with current, upload code).
The new logic of validating UEBs follows the practice of doing all validation up front. This
practice advises one to isolate the validation of incoming data into one place, so that all of
the rest of the code can assume only valid data.
If any redundant information is present in the UEB+URI, the new code cross-checks and asserts
that it is all fully consistent. This closes some issues where the uploader could have
uploaded inconsistent redundant data, which would probably have caused the old downloader to
simply reject that download after getting a Python exception, but perhaps could have caused
greater harm to the old downloader.
I removed the notion of selecting an erasure codec from codec.py based on the string that was
passed in the UEB. Currently "crs" is the only such string that works, so
"_assert(codec_name == 'crs')" is simpler and more explicit. This is also in keeping with the
"validate up front" strategy -- now if someone sets a different string than "crs" in their UEB,
the downloader will reject the download in the "validate this UEB" function instead of in a
separate "select the codec instance" function.
I removed the code to check plaintext hashes and plaintext Merkle Trees. Uploaders do not
produce this information any more (since it potentially exposes confidential information about
the file), and the unit tests for it were disabled. The downloader before this patch would
check that plaintext hash or plaintext merkle tree if they were present, but not complain if
they were absent. The new downloader in this patch complains if they are present and doesn't
check them. (We might in the future re-introduce such hashes over the plaintext, but encrypt
the hashes which are stored in the UEB to preserve confidentiality. This would be a double-
check on the correctness of our own source code -- the current Merkle Tree over the ciphertext
is already sufficient to guarantee the integrity of the download unless there is a bug in our
Merkle Tree or AES implementation.)
This patch increases the lines-of-code count by 8 (from 17,770 to 17,778), and reduces the
uncovered-by-tests lines-of-code count by 24 (from 1408 to 1384). Those numbers would be more
meaningful if we omitted src/allmydata/util/ from the test-coverage statistics.
2008-12-05 08:17:54 -07:00
Brian Warner
a0b5b5ab2b
test_web: add get_permuted_peers, to unbreak recent checker_results change
2008-12-05 02:12:10 -07:00
Brian Warner
42f8aece05
web checker_results: include a table of servers in permuted order, so you can see the places where new servers have been inserted
2008-12-05 02:03:09 -07:00
Brian Warner
e942ab141b
test_system.py: assert less about the stats we get, since shares (and thus allocate() calls) are distributed randomly
2008-12-04 17:27:04 -07:00
Brian Warner
7c4edac582
stats: don't return booleans: it violates the schema. Add a test.
2008-12-04 15:01:24 -07:00
Brian Warner
7cfc74bcc9
test_system.py: don't ask the stats-gatherer to poll: it tolerates failures, so it isn't really giving us enough test coverage. Removing the call will make it more clear that we need to improve the tests later
2008-12-04 15:00:53 -07:00
Peter Secor
a8e515a8bb
confwiz.py - removing hardcoded version number
2008-12-02 20:38:31 -07:00
Brian Warner
a2ce315074
CLI: check for pre-existing aliases in 'tahoe create-alias' and 'tahoe add-alias'
2008-12-02 20:20:22 -07:00
Brian Warner
cdbb72d5ff
test_cli: pass rc out of do_cli() too
2008-12-02 20:08:28 -07:00
Brian Warner
6958b7fa90
test_storage.py: more windows-vs-readonly-storage fixes
2008-12-02 19:41:02 -07:00
Brian Warner
fe9c3e9529
storage.py: oops, fix windows again, readonly_storage wasn't getting picked up properly
2008-12-02 19:03:17 -07:00
Brian Warner
d3783475b4
test_download.py: remove extra base32 import
2008-12-02 18:31:26 -07:00
Brian Warner
b997e6ca7c
test_download: test both mutable and immutable pre-generated shares
2008-12-02 18:30:07 -07:00
Brian Warner
2d70f7fc0b
test_download.py: added 'known-answer-tests', to make sure current code can download a file that was created by earlier code
2008-12-02 18:22:08 -07:00
Brian Warner
d295a7afb9
storage.py: unbreak readonly_storage=True on windows
2008-12-01 19:49:46 -07:00
Brian Warner
71c62f8fcb
#542 'tahoe create-key-generator': fix the .tac file this creates to be compatible with modern code, add a test
2008-12-01 17:47:21 -07:00
Brian Warner
c5222558f4
storage.py: fix minor typo in comment
2008-12-01 17:25:40 -07:00
Brian Warner
cfba882b30
storage: replace sizelimit with reserved_space, make the stats 'disk_avail' number incorporate this reservation
2008-12-01 17:24:21 -07:00
Brian Warner
39a41f1d85
util/abbreviate: add abbreviated-size parser
2008-12-01 17:24:12 -07:00
Zooko O'Whielacronx
fe6abac87b
wui/wapi: change the default port number from 8123 to 3456 to avoid conflict with TorButton
...
See ticket #536 for details.
2008-11-25 17:57:37 -07:00
Brian Warner
7ea0a1316a
mutable publish: reinstate the foolscap-reference-token-bug workaround, both for the original reasons and because of an apparent new foolscap bug that's triggered by reference tokens. See #541 for details.
2008-11-25 14:27:35 -07:00
Zooko O'Whielacronx
51c7580fc8
setup: fix missing import -- thanks, pyflakes
2008-11-25 09:55:28 -07:00
Zooko O'Whielacronx
de9edde8f5
setup: correctly detect Arch Linux in platform description
2008-11-25 09:51:18 -07:00
Brian Warner
bc53c24003
dirnode manifest: add verifycaps, both to internal API and to webapi. This will give the manual-GC tools more to work with, so they can estimate how much space will be freed.
2008-11-24 14:40:46 -07:00
Brian Warner
2308440fb6
control.py: use get_buckets() instead of get_version() to measure ping time, because the latter changed recently
2008-11-22 23:13:23 -07:00
Brian Warner
3e25efc010
upload: when using a Helper, insist that it provide protocols/helper/v1 . Related to #538 .
2008-11-21 20:29:32 -07:00
Brian Warner
0fab511be5
upload: don't use servers which can't support the share size we need. This ought to avoid #439 problems. Some day we'll have a storage server which advertises support for a larger share size. No tests yet.
2008-11-21 20:28:12 -07:00
Brian Warner
bf06492a90
#538 : fetch version and attach to the rref. Make IntroducerClient demand v1 support.
2008-11-21 20:07:27 -07:00
Brian Warner
0eb6b324a4
#538 : add remote_get_version() to four main Referenceable objects: Introducer Service, Storage Server, Helper, CHK Upload Helper. Remove unused storage-server get_versions().
2008-11-21 17:43:52 -07:00
Brian Warner
4ada26101b
test_dirnode: add an explainError call
2008-11-19 16:02:12 -07:00
Brian Warner
b84c2c6541
manifest: add storage-index strings to the json results
2008-11-19 16:00:27 -07:00
Brian Warner
815e0673e6
manifest: include stats in results. webapi is unchanged.
2008-11-19 15:03:47 -07:00
Brian Warner
6f6556da22
mutable: respect the new tahoe.cfg 'shares.needed' and 'shares.total' settings
2008-11-19 14:05:01 -07:00
Brian Warner
82fe7ba360
oops, update tests to match 'tahoe stats' change
2008-11-18 20:32:59 -07:00
Brian Warner
677928a1da
cli: tahoe stats: abbreviate total sizes too
2008-11-18 20:28:16 -07:00
Brian Warner
9871f3f1b7
cli: 'tahoe stats': add abbreviated size to the histogram. Not sure this actually improves things.
2008-11-18 20:17:36 -07:00
Brian Warner
7391f63fc1
util/abbreviate: little utility to abbreviate seconds and bytes
2008-11-18 20:11:42 -07:00
Brian Warner
0cdf526b50
cli: add 'tahoe check' and 'tahoe deep-check' commands, with primitive reporting code
2008-11-18 19:12:10 -07:00
Brian Warner
9c866ada71
cli: factor out slow-http-operation to a separate module
2008-11-18 19:11:13 -07:00
Brian Warner
5c3e153d0e
cli: tahoe stats/manifest: change --verbose to --raw, since I want -v for --verify for check/deep-check/repair
2008-11-18 18:36:08 -07:00
Brian Warner
d657d22c0c
test_system: make 'where' strings more helpful, to track down test failures better
2008-11-18 18:29:50 -07:00
Brian Warner
7932fadb5e
webapi: add 'summary' string to checker results JSON
2008-11-18 18:28:26 -07:00
Brian Warner
994d97c644
webapi: introducer stats: add 'announcement_distinct_hosts' to the t=json form, to show how many distinct hosts are providing e.g. storage services
2008-11-18 15:30:15 -07:00
Brian Warner
918970b63a
'tahoe create-key-generator': fix help text
2008-11-18 01:47:58 -07:00
Brian Warner
8473a96ada
#330 : convert stats-gatherer into a .tac file service, add 'tahoe create-stats-gatherer'
2008-11-18 01:46:20 -07:00
Brian Warner
cae54ab118
tahoe.cfg: add controls for k and N (and shares-of-happiness)
2008-11-18 00:29:44 -07:00
Brian Warner
ead0e4d6ca
cli: add tests for 'tahoe stats --verbose'
2008-11-17 22:11:14 -07:00
Brian Warner
94de73ddc0
cli: add --verbose to 'tahoe manifest', to show the raw JSON data
2008-11-17 22:02:19 -07:00
Brian Warner
d6a67cd566
dirnode manifest/stats: process more than one LIT file per tree; we were accidentally ignoring all but the first
2008-11-14 22:50:49 -07:00
Brian Warner
fc7cd23bd1
limiter.py: fix stack blowout by inserting an eventual-send between _done and maybe_start_task. This was causing failures during a 'tahoe manifest' of a large set of directories
2008-11-14 21:11:44 -07:00
francois
1a3a464534
test_cli.py: Ensure that we can read our uploaded files back
2008-11-14 07:44:58 -07:00
francois
32b5bc6330
test_cli.py: use str objects instead of unicode ones
...
This will hopefully fix failing tests with LC_ALL=C
2008-11-14 07:41:37 -07:00
Brian Warner
5a60086dbc
CLI: add 'tahoe stats', to run start-deep-stats and print the results
2008-11-13 19:43:50 -07:00
Brian Warner
9c9994300e
test_system.py: fix new 'tahoe manifest' tests to not break on windows, by providing --node-directory instead of --node-url
2008-11-13 15:27:48 -07:00
francois
5c0c5bfc81
test for bug #534 , unicode filenames
...
This test assure that uploading a file whose name contains unicode character
doesn't prevent further uploads in the same directory.
2008-11-13 05:19:51 -07:00
francois
c1f639d230
Fix an filename encoding issue with "tahoe cp"
2008-11-11 14:08:03 -07:00
Brian Warner
0d5cc38a35
web/info.py: use 128-bit ophandles instead of 64-bit
2008-11-12 20:18:42 -07:00
Brian Warner
0d93d6244e
CLI: add 'tahoe manifest', which takes a directory and returns a list of things you can reach from it
2008-11-12 20:17:25 -07:00
Brian Warner
c9175de0f2
create_node.py: also remove now-unused import of pkg_resources
2008-11-12 18:47:16 -07:00
Brian Warner
9976bd439a
tahoe.cfg: add tub.location, to override the location hints we include in our FURL. This replaces advertised_ip_addresses, which doesn't remain useful enough to retain it. Helps with #517 (Tor).
2008-11-12 18:44:58 -07:00
Zooko O'Whielacronx
74ce1cdc00
setup: remove pkg_resources.require() from create_node.py and add it to runner.py
...
Brian correctly points out that the latter is an entry point.
2008-11-12 15:25:03 -07:00
Zooko O'Whielacronx
14444dbcc0
tests: fix comment
1970-01-05 04:10:55 -07:00
Zooko O'Whielacronx
3a0dc8e4ec
tests: add tahoe_cp to the list of scripts that we don't actually have tests for yet
1970-01-05 04:00:58 -07:00
Zooko O'Whielacronx
1aed9fcfa1
setup: the .tac files created by create_node.py call pkg_resources.require() so that they can load tahoe and twisted packages which were installed with setuptools multi-version mode
...
Also the create_node.py script itself uses pkg_resources.require() for the same reason.
1970-01-01 17:50:05 -07:00
Brian Warner
68f366b584
web/info: don't let an unrecoverable file break the page (show ? instead of a size)
2008-11-06 22:51:17 -07:00
Brian Warner
dfa2408157
checker: add is_recoverable() to checker results, make our stub immutable-verifier not throw an exception on unrecoverable files, add tests
2008-11-06 22:35:47 -07:00
Brian Warner
b7f7b2f43e
monitor: update interface definition: get_status() can return a Failure
2008-11-06 21:54:52 -07:00
Brian Warner
d7f1f9fd63
web/operations.py: if the operation failed, render the Failure
2008-11-06 21:53:09 -07:00
Peter Secor
ccd029bf15
undoing test change for native_client.php
2008-11-06 16:03:10 -07:00
Brian Warner
926f3958a2
client.py: oops, update FTP/SFTP config names to match current docs
2008-11-05 19:34:42 -07:00
Brian Warner
7c4856c222
SFTP/FTP: merge user/account code, merge docs
2008-11-05 19:25:58 -07:00
Brian Warner
c489a05235
ftp/sftp: move to a new frontends/ directory in preparation for factoring out password-auth component
2008-11-05 14:07:33 -07:00
Brian Warner
4e68faebb4
sftpd: minor debug-logging tweak
2008-11-05 13:45:11 -07:00
Peter Secor
510cb6d8e6
confwiz.py - trying out a new configuration site
2008-11-04 19:18:30 -07:00
Brian Warner
3fc69308eb
ftpd: include an (unused) avatar logout callback
2008-11-04 18:01:04 -07:00
Brian Warner
9f908de9e2
#531 : implement an SFTP frontend. Mostly works, still lots of debug messages. Still needs tests and auth-by-pubkey in accounts.file
2008-11-04 18:00:22 -07:00
Zooko O'Whielacronx
eae5403e5a
shebang: replace "/usr/bin/python" with "/usr/bin/env python"
2008-11-04 18:03:06 -07:00
Zooko O'Whielacronx
9c35ca76a4
util: copy in pyutil.fileutil.ReopenableNamedTemporaryFile
2008-11-04 17:47:15 -07:00
Brian Warner
6fa41e738b
immutable: tolerate filenode.read() with a size= that's too big, rather than hanging
2008-11-04 15:29:19 -07:00
Zooko O'Whielacronx
cef24792e7
util: copy in nummedobj from pyutil
2008-11-04 13:55:50 -07:00
Zooko O'Whielacronx
fb85a6280f
util: copy in dictutil from pyutil
2008-11-04 13:53:27 -07:00
Brian Warner
a2e195df3e
debug catalog-shares: tolerate even more errors on bad files/directories
2008-10-30 14:54:47 -07:00
Brian Warner
ba019bfd3a
#527 : expire the cached files that are used to support Range: headers, every hour, when the file is unused and older than an hour
2008-10-30 13:39:09 -07:00
Brian Warner
c205a54965
util/cachedir.py: add a cache-directory manager class, which expires+deletes unused files after a while
2008-10-30 13:01:20 -07:00
Brian Warner
1419db6ac7
test_cli: try to fix windows again
2008-10-30 12:32:04 -07:00
Brian Warner
d97c34d622
debug/test_cli: fix error handling for catalog-shares, to make the test stop failing on windows
2008-10-30 12:06:51 -07:00
Brian Warner
b1db6d9ff2
web: add 'Repair' button to checker results when they indicate unhealthyness. Also add the object's uri to the CheckerResults instance.
2008-10-29 18:09:17 -07:00
Brian Warner
6a7d5b893b
create_node.py: add 'web.static = public_html' to the initial tahoe.cfg
2008-10-29 17:13:36 -07:00
Brian Warner
d3f2df00be
webapi: serve the /static URL tree from /public_html (configurable)
2008-10-29 15:34:31 -07:00
Brian Warner
a9101112a4
catalog-shares command: tolerate errors, log them to stderr, handle v2-immutable shares
2008-10-29 15:10:10 -07:00
