Commit Graph

3735 Commits

Author SHA1 Message Date
cgalvan
58743b2b76 prevent --site-dirs from being passed to the 'install' command 2009-01-16 13:57:32 -07:00
cgalvan
7bfaa74ef1 add option to show version and path to the tahoe executable 2009-01-16 12:47:51 -07:00
Zooko O'Whielacronx
7fc784fc55 setup: put back configuration of the PYTHONPATH which is necessary to build the Windows packages 2009-01-14 20:37:51 -07:00
Brian Warner
26ca53fa3d node.py: use NODEDIR/tmp for the 'tempfile' module's temporary directory, so webapi upload tempfiles are put there instead of /tmp . You can set it to something else by setting [node]tempdir in tahoe.cfg 2009-01-14 20:00:15 -07:00
Brian Warner
e9d4334159 web/operations: undo the disable-ophandle-expiration change that inadvertently got included in zooko's recent 'rename wapi.txt to webapi.txt' patch, causing test failures 2009-01-14 19:14:59 -07:00
Zooko O'Whielacronx
f9ee60a88e cli: tests: skip symlink test if there is no os.symlink 2009-01-14 18:10:10 -07:00
Brian Warner
4334ee056b webapi.txt: explain POST /uri/?t=upload, thanks to Toby Murray for the catch 2009-01-14 18:08:03 -07:00
Brian Warner
2fe099a0b3 mutable: move recent operation history management code (MutableWatcher) into history.py, have History provide stats 2009-01-14 17:36:20 -07:00
Brian Warner
aa50c30aa2 download: tiny cleanup of history code 2009-01-14 16:41:51 -07:00
Brian Warner
10268a4f7f upload: move upload history into History object 2009-01-14 16:41:06 -07:00
Brian Warner
3920e6d1e7 immutable/download.py move recent-downloads history out of Downloader and into a separate class. upload/etc will follow soon. 2009-01-14 16:14:24 -07:00
Zooko O'Whielacronx
0a2895e17a docs: rename wapi.txt to webapi.txt
Because Brian argues that the file contains a description of the wui as well as of the wapi, and because the name "webapi.txt" might be more obvious to the untrained eye.
2009-01-14 13:53:48 -07:00
Zooko O'Whielacronx
e38dc8f310 setup: configure setup.cfg to point setuptools at the index.html page instead of straight at the tahoe directory listing
This is necessary, because if setuptools looks at the directory listing, then it follows the link named "More Info" in attempt to download the file, and this fails.
2009-01-14 12:50:23 -07:00
Zooko O'Whielacronx
08aada50ee setup: fix previous patch to set reactor to poll reactor on linux or cygwin 2009-01-14 10:40:22 -07:00
Zooko O'Whielacronx
6db8f38c3b setup: use poll reactor for trial if on linux2 or cygwin 2009-01-14 09:15:46 -07:00
Zooko O'Whielacronx
c360bf604a docs: rename frontends/webapi.txt to frontends/wapi.txt
rename CLI.txt to frontends/CLI.txt
change a few mentions of "webapi" to "wapi"
fixes #582
2009-01-13 20:51:43 -07:00
Brian Warner
cc50e2f4aa upload: use WriteBucketProxy_v2 when uploading a large file (with shares larger than 4GiB). This finally closes #346. I think we can now handle immutable files up to 48EiB. 2009-01-12 20:14:42 -07:00
Brian Warner
bf56e2bb51 deep-check-and-repair: improve results and their HTML representation 2009-01-12 18:56:19 -07:00
Brian Warner
8f2ec16b2c test_repairer.py: hush pyflakes: remove duplicate/shadowed function name, by using the earlier definition (which is identical) 2009-01-12 15:45:09 -07:00
Brian Warner
fe362c0021 hush pyflakes by removing unused imports 2009-01-12 15:41:20 -07:00
Zooko O'Whielacronx
25063688b4 immutable repairer
This implements an immutable repairer by marrying a CiphertextDownloader to a CHKUploader.  It extends the IDownloadTarget interface so that the downloader can provide some metadata that the uploader requires.
The processing is incremental -- it uploads the first segments before it finishes downloading the whole file.  This is necessary so that you can repair large files without running out of RAM or using a temporary file on the repairer.
It requires only a verifycap, not a readcap.  That is: it doesn't need or use the decryption key, only the integrity check codes.
There are several tests marked TODO and several instances of XXX in the source code.  I intend to open tickets to document further improvements to functionality and testing, but the current version is probably good enough for Tahoe-1.3.0.
2009-01-12 11:00:22 -07:00
Zooko O'Whielacronx
e449052a17 util: dictutil: add DictOfSets.union(key, values) and DictOfSets.update(otherdictofsets) 2009-01-12 10:55:39 -07:00
Zooko O'Whielacronx
c1e3c66db2 setup: update doc in setup.cfg 2009-01-11 09:13:19 -07:00
Zooko O'Whielacronx
ba0c6d1fd1 setup: Point setuptools at a directory on the allmydata.org test grid to find dependencies.
Don't include an unrouteable IP address in find_links (fixes #574).
2009-01-11 09:11:26 -07:00
Zooko O'Whielacronx
8dd3b3185f immutable: separate tests of immutable upload/download from tests of immutable checking/repair 2009-01-10 15:07:39 -07:00
Zooko O'Whielacronx
b496eba072 trivial: minor changes to in-line comments -- mark plaintext-hash-tree as obsolete 2009-01-10 14:56:01 -07:00
Zooko O'Whielacronx
38bb3e464b immutable: make the web display of upload results more human-friendly, like they were before my recent change to the meaning of the "sharemap" 2009-01-10 14:02:09 -07:00
Zooko O'Whielacronx
0f9c11cfde immutable: fix edit-o in interfaces.py documentation introduced in recent patch 2009-01-10 12:54:08 -07:00
Zooko O'Whielacronx
6e3396fb88 immutable: redefine the "sharemap" member of the upload results to be a map from shnum to set of serverids
It used to be a map from shnum to a string saying "placed this share on XYZ server".  The new definition is more in keeping with the "sharemap" object that results from immutable file checking and repair, and it is more useful to the repairer, which is a consumer of immutable upload results.
2009-01-10 11:46:23 -07:00
Zooko O'Whielacronx
ef60e85ec6 naming: finish renaming "CheckerResults" to "CheckResults" 2009-01-09 18:00:52 -07:00
Brian Warner
167742c2b3 storage.py : replace 4294967295 with 2**32-1: python does constant folding, I measured this statement as taking 50ns, versus the 400ns for the call to min(), or the 9us required for the 'assert not os.path.exists' syscall 2009-01-09 19:52:22 -07:00
Brian Warner
c7cd3f38e7 storage.py: announce a maximum-immutable-share-size based upon a 'df' of the disk. Fixes #569, and this should be the last requirement for #346 (remove 12GiB filesize limit) 2009-01-09 19:37:36 -07:00
cgalvan
cc9a6e14fc set bin/tahoe executable permissions and leave build_tahoe in sys.argv 2009-01-09 15:06:40 -07:00
Zooko O'Whielacronx
ea77d27cc5 setup: merge relaxation of the version of setuptools that we require at runtime with an indentation change 2009-01-09 13:09:49 -07:00
Zooko O'Whielacronx
a56df2587a setup: remove custom Trial class inside our setup.py and use the setuptools_trial plugin 2008-12-05 17:22:07 -07:00
Zooko O'Whielacronx
2d90087420 setup: we require pywin32 if building on Windows (plus some formatting and comment fixes) 2008-12-05 17:19:11 -07:00
cgalvan
bff038ab2f fix bin/tahoe executable for Windows 2009-01-09 12:42:22 -07:00
cgalvan
db8194c37c use subprocess.call instead of os.execve in bin/tahoe 2009-01-09 12:03:00 -07:00
Zooko O'Whielacronx
b35d70b1f2 setup: attempt to remove the custom setuptools-ish logic in setup.py -- the result works on my Windows box but doesn't yield a working ./bin/tahoe on Windows, and hasn't been tested yet on other platforms 2008-12-05 17:30:54 -07:00
Zooko O'Whielacronx
fcd4d4d2e9 setup: integrate the bundled setuptools_trial plugin with Chris Galvan's patch to use that plugin 2008-12-01 11:48:04 -07:00
cgalvan
5cd6ed17fb use_setuptools_trial.patch 2008-11-21 14:57:59 -07:00
Zooko O'Whielacronx
4cd02cc12f setup: bundle setuptools_trial in misc/dependencies/ 2008-12-01 11:44:38 -07:00
Brian Warner
45fc3819a2 test_helper: hush pyflakes by avoiding use of 'uri' as a variable, since it shadows an import of the same name 2009-01-08 20:59:41 -07:00
Brian Warner
f8de336039 immutable/checker: include a summary (with 'Healthy' or 'Not Healthy' and a count of shares) in the checker results 2009-01-08 20:01:45 -07:00
Brian Warner
7ee336b274 webapi/deep-manifest t=JSON: don't return the (large) manifest/SI/verifycap lists unless the operation has completed, to avoid the considerable CPU+memory cost of creating the JSON (for 330k dirnodes, it could take two minutes to generate 275MB of JSON). They must be paid eventually, but not on every poll 2009-01-08 19:59:32 -07:00
Brian Warner
39a089dc7e dirnode deep-traversal: remove use of Limiter, stick with strict depth-first-traversal, to reduce memory usage during very large (300k+ dirnode) traversals 2009-01-08 19:41:16 -07:00
Zooko O'Whielacronx
ade6a4fa74 immutable: add a monitor API to CiphertextDownloader with which to tell it to stop its work 2009-01-08 14:42:15 -07:00
Zooko O'Whielacronx
157e365d2b naming: Rename a few things which I touched or changed in the recent patch to download-without-decrypting.
Rename "downloadable" to "target".
Rename "u" to "v" in FileDownloader.__init__().
Rename "_uri" to "_verifycap" in FileDownloader.
Rename "_downloadable" to "_target" in FileDownloader.
Rename "FileDownloader" to "CiphertextDownloader".
2009-01-08 12:13:07 -07:00
Zooko O'Whielacronx
600196f571 immutable: refactor download to do only download-and-decode, not decryption
FileDownloader takes a verify cap and produces ciphertext, instead of taking a read cap and producing plaintext.
FileDownloader does all integrity checking including the mandatory ciphertext hash tree and the optional ciphertext flat hash, rather than expecting its target to do some of that checking.
Rename immutable.download.Output to immutable.download.DecryptingOutput. An instance of DecryptingOutput can be passed to FileDownloader to use as the latter's target.  Text pushed to the DecryptingOutput is decrypted and then pushed to *its* target.
DecryptingOutput satisfies the IConsumer interface, and if its target also satisfies IConsumer, then it forwards and pause/unpause signals to its producer (which is the FileDownloader).
This patch also changes some logging code to use the new logging mixin class.
Check integrity of a segment and decrypt the segment one block-sized buffer at a time instead of copying the buffers together into one segment-sized buffer (reduces peak memory usage, I think, and is probably a tad faster/less CPU, depending on your encoding parameters).
Refactor FileDownloader so that processing of segments and of tail-segment share as much code is possible.
FileDownloader and FileNode take caps as instances of URI (Python objects), not as strings.
2009-01-08 11:53:49 -07:00
Zooko O'Whielacronx
9bba578776 trivial: tiny changes to test code 2009-01-08 11:20:48 -07:00