Commit Graph

444 Commits

Author SHA1 Message Date
4646451de6 change StorageServer to take nodeid in the constructor, instead of assigning it later, since it's cleaner and because the original problem (Tubs not being ready until later) went away 2009-02-18 16:23:01 -07:00
ef53da2b12 break storage.py into smaller pieces in storage/*.py . No behavioral changes. 2009-02-18 14:46:55 -07:00
0409bc5628 client.py: improve docstring 2009-02-16 17:15:32 -07:00
ebe0f2ac08 test/no_network: new test harness, like system-test but doesn't use the network so it's faster 2009-02-16 14:58:44 -07:00
8303f90da0 nodeadmin: node stops itself if a hotline file hasn't been touched in 60 seconds now, instead of in 40 seconds
A test failed on draco (MacPPC) because it took 49 seconds to get around to running the test, and the node had already stopped itself when the hotline file was 40 seconds old.
2009-02-12 15:21:31 -07:00
7eb260a9cf versioning: include an "appname" in the application version string in the versioning protocol, and make that appname be controlled by setup.py
It is currently hardcoded in setup.py to be 'allmydata-tahoe'.  Ticket #556 is to make it configurable by a runtime command-line argument to setup.py: "--appname=foo", but I suddenly wondered if we really wanted that and at the same time realized that we don't need that for tahoe-1.3.0 release, so this patch just hardcodes it in setup.py.
setup.py inspects a file named 'src/allmydata/_appname.py' and assert that it contains the string "__appname__ = 'allmydata-tahoe'", and creates it if it isn't already present.  src/allmydata/__init__.py import _appname and reads __appname__ from it.  The rest of the Python code imports allmydata and inspects "allmydata.__appname__", although actually every use it uses "allmydata.__full_version__" instead, where "allmydata.__full_version__" is created in src/allmydata/__init__.py to be:

__full_version__ = __appname + '-' + str(__version__).

All the code that emits an "application version string" when describing what version of a protocol it supports (introducer server, storage server, upload helper), or when describing itself in general (introducer client), usese allmydata.__full_version__.

This fixes ticket #556 at least well enough for tahoe-1.3.0 release.
2009-02-11 17:18:16 -07:00
7d91dc5efe rollback the feature of making "ambient upload authority" configurable
This reverses some, but not all, of the changes that were committed in the following set of patches.

rolling back:

Sun Jan 18 09:54:30 MST 2009  toby.murray
  * add 'web.ambient_upload_authority' as a paramater to tahoe.cfg

    M ./src/allmydata/client.py -1 +3
    M ./src/allmydata/test/common.py -7 +9
    A ./src/allmydata/test/test_ambient_upload_authority.py
    M ./src/allmydata/web/root.py +12
    M ./src/allmydata/webish.py -1 +4
Sun Jan 18 09:56:08 MST 2009  zooko@zooko.com
  * trivial: whitespace
  I ran emacs's "M-x whitespace-cleanup" on the files that Toby's recent patch had touched that had trailing whitespace on some lines.

    M ./src/allmydata/test/test_ambient_upload_authority.py -9 +8
    M ./src/allmydata/web/root.py -2 +1
    M ./src/allmydata/webish.py -2 +1
Mon Jan 19 14:16:19 MST 2009  zooko@zooko.com
  * trivial: remove unused import noticed by pyflakes

    M ./src/allmydata/test/test_ambient_upload_authority.py -1
Mon Jan 19 21:38:35 MST 2009  toby.murray
  * doc: describe web.ambient_upload_authority

    M ./docs/configuration.txt +14
    M ./docs/frontends/webapi.txt +11
Mon Jan 19 21:38:57 MST 2009  zooko@zooko.com
  * doc: add Toby Murray to the CREDITS

    M ./CREDITS +4
2009-01-20 20:47:35 -07:00
4771e1c9d5 trivial: whitespace
Ran "M-x whitespace-cleanup" on files that Toby's recent patch touched, even though they didn't have trailing whitespace.
2009-01-18 10:58:15 -07:00
66f83c7356 add 'web.ambient_upload_authority' as a paramater to tahoe.cfg 2009-01-18 10:54:30 -07:00
2fe099a0b3 mutable: move recent operation history management code (MutableWatcher) into history.py, have History provide stats 2009-01-14 17:36:20 -07:00
10268a4f7f upload: move upload history into History object 2009-01-14 16:41:06 -07:00
3920e6d1e7 immutable/download.py move recent-downloads history out of Downloader and into a separate class. upload/etc will follow soon. 2009-01-14 16:14:24 -07:00
600196f571 immutable: refactor download to do only download-and-decode, not decryption
FileDownloader takes a verify cap and produces ciphertext, instead of taking a read cap and producing plaintext.
FileDownloader does all integrity checking including the mandatory ciphertext hash tree and the optional ciphertext flat hash, rather than expecting its target to do some of that checking.
Rename immutable.download.Output to immutable.download.DecryptingOutput. An instance of DecryptingOutput can be passed to FileDownloader to use as the latter's target.  Text pushed to the DecryptingOutput is decrypted and then pushed to *its* target.
DecryptingOutput satisfies the IConsumer interface, and if its target also satisfies IConsumer, then it forwards and pause/unpause signals to its producer (which is the FileDownloader).
This patch also changes some logging code to use the new logging mixin class.
Check integrity of a segment and decrypt the segment one block-sized buffer at a time instead of copying the buffers together into one segment-sized buffer (reduces peak memory usage, I think, and is probably a tad faster/less CPU, depending on your encoding parameters).
Refactor FileDownloader so that processing of segments and of tail-segment share as much code is possible.
FileDownloader and FileNode take caps as instances of URI (Python objects), not as strings.
2009-01-08 11:53:49 -07:00
ecabcc674c immutable: Make more parts of download use logging mixins and know what their "parent msg id" is. 2009-01-08 11:25:30 -07:00
c85f75bb08 immutable: refactor uploader to do just encoding-and-uploading, not encryption
This makes Uploader take an EncryptedUploadable object instead of an Uploadable object.  I also changed it to return a verify cap instead of a tuple of the bits of data that one finds in a verify cap.
This will facilitate hooking together an Uploader and a Downloader to make a Repairer.
Also move offloaded.py into src/allmydata/immutable/.
2009-01-06 21:48:22 -07:00
25d962c67d test: extend timeout on the hotline file that prevents the client from stopping itself
The 20-second timeout was apparently tripped on my Powerbook G4 "draco".
2008-12-21 21:06:29 -07:00
1d84606d52 client: add get_servers() 2008-12-08 17:04:00 -07:00
b58875fe43 mutable: rename mutable/node.py to mutable/filenode.py and mutable/repair.py to mutable/repairer.py
To be more consistent with the immutable layout that I am working on.
2008-12-07 08:20:08 -07:00
cfba882b30 storage: replace sizelimit with reserved_space, make the stats 'disk_avail' number incorporate this reservation 2008-12-01 17:24:21 -07:00
cae54ab118 tahoe.cfg: add controls for k and N (and shares-of-happiness) 2008-11-18 00:29:44 -07:00
926f3958a2 client.py: oops, update FTP/SFTP config names to match current docs 2008-11-05 19:34:42 -07:00
c489a05235 ftp/sftp: move to a new frontends/ directory in preparation for factoring out password-auth component 2008-11-05 14:07:33 -07:00
9f908de9e2 #531: implement an SFTP frontend. Mostly works, still lots of debug messages. Still needs tests and auth-by-pubkey in accounts.file 2008-11-04 18:00:22 -07:00
ba019bfd3a #527: expire the cached files that are used to support Range: headers, every hour, when the file is unused and older than an hour 2008-10-30 13:39:09 -07:00
d3f2df00be webapi: serve the /static URL tree from /public_html (configurable) 2008-10-29 15:34:31 -07:00
294e3fb682 util: move PollMixin to a separate file (pollmixin.py), so testutil can be moved into test/ 2008-10-28 21:15:48 -07:00
37e3d8e47c #527: support HTTP 'Range:' requests, using a cachefile. Adds filenode.read(consumer, offset, size) method. Still needs: cache expiration, reduced alacrity. 2008-10-28 13:41:04 -07:00
05a8360177 ftp: change the twisted hack necessary for async-write-close, to one more agreeable to the twisted-dev folks, add a copy of the necessary patch to docs/ftp.txt 2008-10-06 18:06:05 -07:00
bc237b3956 ftp server: initial implementation. Still needs unit tests, custom Twisted patches. For #512 2008-10-06 12:52:36 -07:00
cd26f58305 #518: replace various BASEDIR/* config files with a single BASEDIR/tahoe.cfg, with backwards-compatibility of course 2008-09-30 16:21:49 -07:00
063f85d157 BASEDIR/nickname is now UTF-8 encoded 2008-09-20 11:37:13 -07:00
e2c484a761 introducer: add get_nickname_for_peerid 2008-09-05 22:07:00 -07:00
a94af879ff logging: add 'unique-message-ids' (or 'umids') to each WEIRD-or-higher log.msg call, to make it easier to correlate log message with source code 2008-08-25 18:57:59 -07:00
29255568df storage: make storage servers declare oldest supported version == 1.0, and storage clients declare oldest supported version == 1.0
See comments in patch for intended semantics.
2008-07-30 15:51:07 -07:00
7394607141 move encode/upload/download/checker.py into a new immutable/ directory. No behavior changes expected. 2008-07-16 13:14:39 -07:00
94e619c1f6 overhaul checker invocation
Removed the Checker service, removed checker results storage (both in-memory
and the tiny stub of sqlite-based storage). Added ICheckable, all
check/verify is now done by calling the check() method on filenodes and
dirnodes (immutable files, literal files, mutable files, and directory
instances).

Checker results are returned in a Results instance, with an html() method for
display. Checker results have been temporarily removed from the wui directory
listing until we make some other fixes.

Also fixed client.create_node_from_uri() to create LiteralFileNodes properly,
since they have different checking behavior. Previously we were creating full
FileNodes with LIT uris inside, which were downloadable but not checkable.
2008-07-15 17:23:25 -07:00
28f4652b96 break introducer up into separate modules in the new allmydata.introducer package 2008-06-18 12:24:16 -07:00
26187bfc81 use a weakref cache in the client to manage singleton filenodes/dirnodes, to avoid autocollision. Should close #391. 2008-05-08 18:02:55 -07:00
462ef2a0ac run a stats provider even if there's no gatherer, since the HTTP /statistics page is then useful. Only run the once-per-second load-monitor if there is a gatherer configured 2008-05-08 11:37:30 -07:00
4b3adb3fcf mutable stats: track mutable bytes published too 2008-04-29 18:20:05 -07:00
55dfb697a4 client: don't start the IntroducerClient until the Tub is ready, otherwise we will sometimes connect to the introducer (or other clients) before we've done Tub.setLocation, which loses some information on the introducer status page 2008-04-23 14:52:34 -07:00
e1838ba217 mutable WIP: clean up status handling, shrink the code a lot, improve test coverage 2008-04-17 13:02:22 -07:00
a1670497a8 mutable WIP: add servermap update status pages 2008-04-16 19:05:41 -07:00
749c42fa2c mutable WIP: re-enable publish/retrieve status 2008-04-16 17:49:06 -07:00
d4230d1781 mutable WIP: split mutable.py into separate files. All tests pass. 2008-04-11 14:31:16 -07:00
68527b25fc client: add 'node.uptime' to the stats we collect 2008-04-17 11:13:39 -07:00
6de2a0794f web: don't break status page when there is no helper running 2008-04-14 23:42:20 -07:00
a2dd4d14a6 helper status: include percentage fetched+pushed, add helper-uploads to the upload/download list 2008-04-14 18:36:27 -07:00
5446ea5d67 stats: added stats reporting to the upload helper
adds a stats_producer for the upload helper, which provides a series of counters
to the stats gatherer, under the name 'chk_upload_helper'.

it examines both the 'incoming' directory, and the 'encoding' dir, providing
inc_count inc_size inc_size_old enc_count enc_size enc_size_old, respectively
the number of files in each dir, the total size thereof, and the aggregate
size of all files older than 48hrs
2008-03-25 18:19:08 -07:00
ef0f523fe0 stats gathering: added counters to upload,download,mutablewatcher
counting number of operations, and for immutable files, bytes transferred
2008-04-09 18:08:59 -07:00