Commit Graph

183 Commits

Author SHA1 Message Date
david-sarah
3d7a32647c Implementation, tests and docs for blacklists. This version allows listing directories containing a blacklisted child. Inclusion of blacklist.py fixed. fixes #1425 2011-08-24 08:59:28 -07:00
david-sarah
fbe0e42bbf Replace the hard-coded 522-bit RSA key size used for tests with a TEST_RSA_KEY_SIZE constant defined in test/common.py (part 1). refs #393 2011-08-10 13:22:43 -07:00
Kevan Carstensen
d3f98498c0 client: teach client how to create and work with MDMF files 2011-08-01 18:48:11 -07:00
david-sarah
10ee22f50e drop-upload: rename 'start' method to 'startService', which is what you're supposed to use to start a Service. refs #1429 2011-08-09 20:03:45 -07:00
Zooko O'Whielacronx
b7683d9b83 drop-upload: rename the 'upload.uri' parameter to 'upload.dircap', and a couple of cleanups to error messages. refs #1429
I rerecorded this patch, originally by David-Sarah, to use "darcs replace" instead of editing to do the renames. This uncovered one missed rename in Client.init_drop_uploader. (Which also means that code isn't exercised by the current unit tests.)
refs #1429
2011-08-09 15:05:08 -07:00
david-sarah
32a7717205 Drop-upload frontend, rerecorded for 1.9 beta (and correcting a minor mistake). Includes some fixes for Windows but not the Windows inotify implementation. fixes #1429 2011-08-08 16:40:49 -07:00
david-sarah
2d16a16ee3 Fix the bug that prevents an introducer from starting when introducer.furl already exists. Also remove some dead code that used to read old config files, and rename 'warn_about_old_config_files' to reflect that it's not a warning. refs #1385 2011-08-02 18:32:12 -07:00
Brian Warner
ffd296fc5a Refactor StorageFarmBroker handling of servers
Pass around IServer instance instead of (peerid, rref) tuple. Replace
"descriptor" with "server". Other replacements:

 get_all_servers -> get_connected_servers/get_known_servers
 get_servers_for_index -> get_servers_for_psi (now returns IServers)

This change still needs to be pushed further down: lots of code is now
getting the IServer and then distributing (peerid, rref) internally.
Instead, it ought to distribute the IServer internally and delay
extracting a serverid or rref until the last moment.

no_network.py was updated to retain parallelism.
2011-02-20 17:58:04 -08:00
Brian Warner
7b7b0c9709 Rewrite immutable downloader (#798). This patch includes higher-level
integration into the NodeMaker, and updates the web-status display to handle
the new download events.
2010-08-04 00:27:02 -07:00
david-sarah
f036dfaa4b Fix test failures due to Unicode basedir patches. 2010-07-24 18:03:18 -07:00
david-sarah
6057bc02cc Prevent mutable objects from being retrieved from an immutable directory, and associated forward-compatibility improvements. 2010-01-26 22:44:30 -08:00
Brian Warner
fb879ddea4 client.py: fix/update comments on KeyGenerator 2010-01-12 16:42:26 -08:00
Brian Warner
f85690697a Add t=mkdir-immutable to the webapi. Closes #607.
* change t=mkdir-with-children to not use multipart/form encoding. Instead,
  the request body is all JSON. t=mkdir-immutable uses this format too.
* make nodemaker.create_immutable_dirnode() get convergence from SecretHolder,
  but let callers override it
* raise NotDeepImmutableError instead of using assert()
* add mutable= argument to DirectoryNode.create_subdirectory(), default True
2009-11-17 23:09:00 -08:00
Brian Warner
d2426ea3bd move convergence secret into SecretHolder, next to lease secret 2009-11-17 17:54:44 -08:00
Brian Warner
5fe713fc52 nodemaker: implement immutable directories (internal interface), for #607
* nodemaker.create_from_cap() now handles DIR2-CHK and DIR2-LIT
* client.create_immutable_dirnode() is used to create them
* no webapi yet
2009-11-11 16:22:33 -08:00
Brian Warner
b4ec86c95a update many dirnode interfaces to accept dict-of-nodes instead of dict-of-caps
interfaces.py: define INodeMaker, document argument values, change
               create_new_mutable_directory() to take dict-of-nodes. Change
               dirnode.set_nodes() and dirnode.create_subdirectory() too.
nodemaker.py: use INodeMaker, update create_new_mutable_directory()
client.py: have create_dirnode() delegate initial_children= to nodemaker
dirnode.py (Adder): take dict-of-nodes instead of list-of-nodes, which
                    updates set_nodes() and create_subdirectory()
web/common.py (convert_initial_children_json): create dict-of-nodes
web/directory.py: same
web/unlinked.py: same
test_dirnode.py: update tests to match
2009-10-17 12:28:29 -07:00
Brian Warner
c2520e4ec7 client.create_mutable_file(contents=) now accepts a callable, which is
invoked with the new MutableFileNode and is supposed to return the initial
contents. This can be used by e.g. a new dirnode which needs the filenode's
writekey to encrypt its initial children.

create_mutable_file() still accepts a bytestring too, or None for an empty
file.
2009-10-12 20:12:32 -07:00
Brian Warner
b30041c5ec webapi: t=mkdir now accepts initial children, using the same JSON that t=json
emits.

client.create_dirnode(initial_children=) now works.
2009-10-12 19:34:44 -07:00
Brian Warner
3ee740628a replace Client.create_empty_dirnode() with create_dirnode(), in anticipation
of adding initial_children= argument.

Includes stubbed-out initial_children= support.
2009-10-12 15:45:06 -07:00
Brian Warner
5283d4c19e de-Service-ify Helper, pass in storage_broker and secret_holder directly.
This makes it more obvious that the Helper currently generates leases with
the Helper's own secrets, rather than getting values from the client, which
is arguably a bug that will likely be resolved with the Accounting project.
2009-08-15 13:17:37 -07:00
Brian Warner
4a4a4f9520 immutable.Downloader: pass StorageBroker to constructor, stop being a Service
child of the client, access with client.downloader instead of
client.getServiceNamed("downloader"). The single "Downloader" instance is
scheduled for demolition anyways, to be replaced by individual
filenode.download calls.
2009-08-15 12:25:43 -07:00
Brian Warner
d214f7eebe stop making History be a Service, it wasn't necessary 2009-08-15 04:44:15 -07:00
Brian Warner
0d5dc51617 Overhaul IFilesystemNode handling, to simplify tests and use POLA internally.
* stop using IURI as an adapter
* pass cap strings around instead of URI instances
* move filenode/dirnode creation duties from Client to new NodeMaker class
* move other Client duties to KeyGenerator, SecretHolder, History classes
* stop passing Client reference to dirnode/filenode constructors
  - pass less-powerful references instead, like StorageBroker or Uploader
* always create DirectoryNodes by wrapping a filenode (mutable for now)
* remove some specialized mock classes from unit tests

Detailed list of changes (done one at a time, then merged together)

always pass a string to create_node_from_uri(), not an IURI instance
always pass a string to IFilesystemNode constructors, not an IURI instance
stop using IURI() as an adapter, switch on cap prefix in create_node_from_uri()
client.py: move SecretHolder code out to a separate class
test_web.py: hush pyflakes
client.py: move NodeMaker functionality out into a separate object
LiteralFileNode: stop storing a Client reference
immutable Checker: remove Client reference, it only needs a SecretHolder
immutable Upload: remove Client reference, leave SecretHolder and StorageBroker
immutable Repairer: replace Client reference with StorageBroker and SecretHolder
immutable FileNode: remove Client reference
mutable.Publish: stop passing Client
mutable.ServermapUpdater: get StorageBroker in constructor, not by peeking into Client reference
MutableChecker: reference StorageBroker and History directly, not through Client
mutable.FileNode: removed unused indirection to checker classes
mutable.FileNode: remove Client reference
client.py: move RSA key generation into a separate class, so it can be passed to the nodemaker
move create_mutable_file() into NodeMaker
test_dirnode.py: stop using FakeClient mockups, use NoNetworkGrid instead. This simplifies the code, but takes longer to run (17s instead of 6s). This should come down later when other cleanups make it possible to use simpler (non-RSA) fake mutable files for dirnode tests.
test_mutable.py: clean up basedir names
client.py: move create_empty_dirnode() into NodeMaker
dirnode.py: get rid of DirectoryNode.create
remove DirectoryNode.init_from_uri, refactor NodeMaker for customization, simplify test_web's mock Client to match
stop passing Client to DirectoryNode, make DirectoryNode.create_with_mutablefile the normal DirectoryNode constructor, start removing client from NodeMaker
remove Client from NodeMaker
move helper status into History, pass History to web.Status instead of Client
test_mutable.py: fix minor typo
2009-08-15 04:28:46 -07:00
Brian Warner
531cc7899f rename NewDirectoryNode to DirectoryNode, NewDirectoryURI to DirectoryURI 2009-07-17 17:15:49 -05:00
Brian Warner
d8ba8c2eb5 Allow tests to pass with -OO by turning some AssertionErrors (the ones that
we actually exercise during tests) into more specific exceptions, so they
don't get optimized away. The best rule to follow is probably this: if an
exception is worth testing, then it's part of the API, and AssertionError
should never be part of the API. Closes #749.
2009-07-14 23:45:10 -07:00
Zooko O'Whielacronx
c0d1e7deae directories: make initialization of the download cache lazy
If you open up a directory containing thousands of files, it currently computes the cache filename and checks for the cache file on disk immediately for each immutble file in that directory.  With this patch, it delays those steps until you try to do something with an immutable file that could use the cache.
2009-07-07 17:40:40 -07:00
Zooko O'Whielacronx
c678e8c77a directories: rename internal data member download_cache to download_cache_dirman (benchmarks set this internal member and use the new name, so changing this makes the bench_dirnode.py work again) 2009-07-07 07:34:04 -07:00
Brian Warner
ef1b6ae8e3 Tolerate unknown URI types in directory structures. Part of #683.
The idea is that future versions of Tahoe will add new URI types that this
version won't recognize, but might store them in directories that we *can*
read. We should handle these "objects from the future" as best we can.
Previous releases of Tahoe would just explode. With this change, we'll
continue to be able to work with everything else in the directory.

The code change is to wrap anything we don't recognize as an UnknownNode
instance (as opposed to a FileNode or DirectoryNode). Then webapi knows how
to render these (mostly by leaving fields blank), deep-check knows to skip
over them, deep-stats counts them in "count-unknown". You can rename and
delete these things, but you can't add new ones (because we wouldn't know how
to generate a readcap to put into the dirnode's rocap slot, and because this
lets us catch typos better).
2009-07-02 18:07:49 -07:00
Brian Warner
6237aeabd7 create_node_from_uri: take both writecap+readcap, move logic out of dirnode.py 2009-07-02 15:25:37 -07:00
Brian Warner
52fa421430 use 522-bit RSA keys in all unit tests (except one)
This reduces the total test time on my laptop from 400s to 283s.
* src/allmydata/test/test_system.py (SystemTest.test_mutable._test_debug):
  Remove assertion about container_size/data_size, this changes with keysize
  and was too variable anyways.
* src/allmydata/mutable/filenode.py (MutableFileNode.create): add keysize=
* src/allmydata/dirnode.py (NewDirectoryNode.create): same
* src/allmydata/client.py (Client.DEFAULT_MUTABLE_KEYSIZE): add default,
  this overrides the one in MutableFileNode
2009-06-29 15:31:24 -07:00
Brian Warner
8df15e9f30 big rework of introducer client: change local API, split division of responsibilites better, remove old-code testing, improve error logging 2009-06-22 19:10:47 -07:00
Brian Warner
711c09bc5d clean up storage_broker interface: should fix #732 2009-06-21 16:51:19 -07:00
Brian Warner
35b3f7f426 more refactoring: move get_all_serverids() and get_nickname_for_serverid() from Client to storage_broker 2009-06-01 20:07:50 -07:00
Brian Warner
b1290633b8 more storage_broker refactoring: downloader gets a broker instead of a client,
use Client.get_storage_broker() accessor instead of direct attribute access.
2009-06-01 19:25:11 -07:00
Brian Warner
c516361fd2 start to factor server-connection-management into a distinct 'StorageServerFarmBroker' object, separate from the client and the introducer. This is the starting point for #467: static server selection 2009-06-01 14:06:04 -07:00
Brian Warner
c9803d5217 switch all foolscap imports to use foolscap.api or foolscap.logging 2009-05-21 17:38:23 -07:00
Brian Warner
f0071c2571 expirer: clean up constructor args, add tahoe.cfg controls, use cutoff_date instead of date_cutoff 2009-03-18 18:00:09 -07:00
Zooko O'Whielacronx
7e7167f2e5 nodeadmin: node stops itself if a hotline file hasn't been touched in 120 seconds now, instead of in 60 seconds
A test failed on draco (MacPPC) because it took 67.1 seconds to get around to running the test, and the node had already stopped itself when the hotline file was 60 seconds old.
2009-03-07 20:50:39 -07:00
Brian Warner
c6c30ac5d4 webapi: pass client through constructor arguments, remove IClient, should make it easier to test web renderers in isolation 2009-02-20 12:15:54 -07:00
Brian Warner
4646451de6 change StorageServer to take nodeid in the constructor, instead of assigning it later, since it's cleaner and because the original problem (Tubs not being ready until later) went away 2009-02-18 16:23:01 -07:00
Brian Warner
ef53da2b12 break storage.py into smaller pieces in storage/*.py . No behavioral changes. 2009-02-18 14:46:55 -07:00
Brian Warner
0409bc5628 client.py: improve docstring 2009-02-16 17:15:32 -07:00
Brian Warner
ebe0f2ac08 test/no_network: new test harness, like system-test but doesn't use the network so it's faster 2009-02-16 14:58:44 -07:00
Zooko O'Whielacronx
8303f90da0 nodeadmin: node stops itself if a hotline file hasn't been touched in 60 seconds now, instead of in 40 seconds
A test failed on draco (MacPPC) because it took 49 seconds to get around to running the test, and the node had already stopped itself when the hotline file was 40 seconds old.
2009-02-12 15:21:31 -07:00
Zooko O'Whielacronx
7eb260a9cf versioning: include an "appname" in the application version string in the versioning protocol, and make that appname be controlled by setup.py
It is currently hardcoded in setup.py to be 'allmydata-tahoe'.  Ticket #556 is to make it configurable by a runtime command-line argument to setup.py: "--appname=foo", but I suddenly wondered if we really wanted that and at the same time realized that we don't need that for tahoe-1.3.0 release, so this patch just hardcodes it in setup.py.
setup.py inspects a file named 'src/allmydata/_appname.py' and assert that it contains the string "__appname__ = 'allmydata-tahoe'", and creates it if it isn't already present.  src/allmydata/__init__.py import _appname and reads __appname__ from it.  The rest of the Python code imports allmydata and inspects "allmydata.__appname__", although actually every use it uses "allmydata.__full_version__" instead, where "allmydata.__full_version__" is created in src/allmydata/__init__.py to be:

__full_version__ = __appname + '-' + str(__version__).

All the code that emits an "application version string" when describing what version of a protocol it supports (introducer server, storage server, upload helper), or when describing itself in general (introducer client), usese allmydata.__full_version__.

This fixes ticket #556 at least well enough for tahoe-1.3.0 release.
2009-02-11 17:18:16 -07:00
Zooko O'Whielacronx
7d91dc5efe rollback the feature of making "ambient upload authority" configurable
This reverses some, but not all, of the changes that were committed in the following set of patches.

rolling back:

Sun Jan 18 09:54:30 MST 2009  toby.murray
  * add 'web.ambient_upload_authority' as a paramater to tahoe.cfg

    M ./src/allmydata/client.py -1 +3
    M ./src/allmydata/test/common.py -7 +9
    A ./src/allmydata/test/test_ambient_upload_authority.py
    M ./src/allmydata/web/root.py +12
    M ./src/allmydata/webish.py -1 +4
Sun Jan 18 09:56:08 MST 2009  zooko@zooko.com
  * trivial: whitespace
  I ran emacs's "M-x whitespace-cleanup" on the files that Toby's recent patch had touched that had trailing whitespace on some lines.

    M ./src/allmydata/test/test_ambient_upload_authority.py -9 +8
    M ./src/allmydata/web/root.py -2 +1
    M ./src/allmydata/webish.py -2 +1
Mon Jan 19 14:16:19 MST 2009  zooko@zooko.com
  * trivial: remove unused import noticed by pyflakes

    M ./src/allmydata/test/test_ambient_upload_authority.py -1
Mon Jan 19 21:38:35 MST 2009  toby.murray
  * doc: describe web.ambient_upload_authority

    M ./docs/configuration.txt +14
    M ./docs/frontends/webapi.txt +11
Mon Jan 19 21:38:57 MST 2009  zooko@zooko.com
  * doc: add Toby Murray to the CREDITS

    M ./CREDITS +4
2009-01-20 20:47:35 -07:00
Zooko O'Whielacronx
4771e1c9d5 trivial: whitespace
Ran "M-x whitespace-cleanup" on files that Toby's recent patch touched, even though they didn't have trailing whitespace.
2009-01-18 10:58:15 -07:00
toby.murray
66f83c7356 add 'web.ambient_upload_authority' as a paramater to tahoe.cfg 2009-01-18 10:54:30 -07:00
Brian Warner
2fe099a0b3 mutable: move recent operation history management code (MutableWatcher) into history.py, have History provide stats 2009-01-14 17:36:20 -07:00
Brian Warner
10268a4f7f upload: move upload history into History object 2009-01-14 16:41:06 -07:00