Commit Graph

3167 Commits

Author SHA1 Message Date
Brian Warner
7338440d86 scripts/create_node.py: remove empty-string defaults for --introducer= and --nickname= 2008-10-23 16:02:35 -07:00
Brian Warner
c455d52453 deep-check: add webapi links to detailed per-file/dir results 2008-10-23 16:00:31 -07:00
Brian Warner
819d6b3d03 interface.py: fix typo 2008-10-23 15:59:36 -07:00
Brian Warner
76b7b9d835 webapi: make the /operations/ 't=status' qualifier optional, remove it from examples 2008-10-23 15:56:58 -07:00
Zooko O'Whielacronx
03fb3a0160 setup: require the latest version of the setuptools bootstrap egg 2008-10-25 08:28:58 -07:00
Zooko O'Whielacronx
9f299ad55b setup: include _pkgutil.py in setuptools bootstrap egg so that it will work on Python 2.4 2008-10-25 08:28:39 -07:00
Zooko O'Whielacronx
31a364e8df setup: pretend the tahoe requires twisted to set up, so that twisted will be there for nevow 2008-10-25 06:50:42 -07:00
Zooko O'Whielacronx
426b7e9869 setup: require the SVN snapshot of setuptools to build 2008-10-25 06:49:59 -07:00
Zooko O'Whielacronx
8f260826b4 setup: remove old bundled setuptools-0.6c9 2008-10-25 06:49:47 -07:00
Zooko O'Whielacronx
3c974763ab setup: bundle an SVN snapshot of setuptools instead of the most recent stable release of setuptools
This SVN snapshot fixes a problem that prevents the setting up of nevow:
http://bugs.python.org/setuptools/issue20
2008-10-25 06:48:37 -07:00
Zooko O'Whielacronx
7912f9ff27 setup: reorder dependencies to be sort of increasing order of how much they depend on other stuff
Not that the order makes any different to how it gets installed, as far as I can tell.
2008-10-25 06:47:39 -07:00
Zooko O'Whielacronx
b26ad316bb docs: add a note that when you make a new tahoe release, you should send the announcement to fuse-devel@lists.sourceforge.net 2008-10-23 14:36:58 -07:00
Brian Warner
4025f29b8a web/info.py: fix 'Check This Object' link, for files it was checking the parent directory by mistake 2008-10-22 10:10:56 -07:00
Brian Warner
0129e9931f #514: add meta-refresh=60 tag to t=status page for incomplete operations 2008-10-22 09:48:42 -07:00
Brian Warner
2ee6356c89 test_dirnode.py: oops, missed a Monitor(), unbreak tests 2008-10-22 01:50:54 -07:00
Brian Warner
1566a9474c immutable/filenode.py: add TODO note about the #514 monitor to check(), rather than going through the checker/verifier code and adding it, since Zooko is currently working on that code 2008-10-22 01:42:37 -07:00
Brian Warner
977c6ac510 more #514: pass a Monitor to all checker operations, make mutable-checker honor the cancel flag 2008-10-22 01:38:18 -07:00
Brian Warner
8178b10ef1 dirnode.py: check for cancel during deep-traverse operations, and don't initiate any new ones if we've been cancelled. Gets us closer to #514. 2008-10-22 00:55:52 -07:00
Brian Warner
34ab4e3de3 more #514 log-webop status/cancel: add handle-expiration, test coverage 2008-10-21 22:13:54 -07:00
Brian Warner
1d89c846b9 webapi.txt: improve t=deep-size output docs 2008-10-21 17:53:31 -07:00
Brian Warner
0f7c1fd6fd #514: improve test coverage 2008-10-21 17:52:56 -07:00
Brian Warner
ad3d9207a9 Change deep-size/stats/check/manifest to a start+poll model instead of a single long-running synchronous operation. No cancel or handle-expiration yet. #514. 2008-10-21 17:03:07 -07:00
Zooko O'Whielacronx
ce42244382 setup: change ez_setup.py to install setuptools-0.6c9 2008-09-30 13:05:02 -07:00
Zooko O'Whielacronx
cd015228d5 setup: bundle setuptools-0.6c9 2008-09-30 13:04:48 -07:00
Zooko O'Whielacronx
8cad310dc8 setup: remove bundled setuptools-0.6c8 2008-09-30 13:03:36 -07:00
Zooko O'Whielacronx
26dfc33263 setup: remove the developer note about doing without GNU make (the GNU make requirement is about to hurt Peter if he tries to follow this doc, by the way)
add classifiers showing with which versions of Python it is known to work.
2008-10-21 09:32:00 -07:00
robk-tahoe
06b05a868d * fuse/runtests: added --catch-up-pause option
On linux, write tests are failing because data written to fuse isn't showing
up in tahoe by the time it's checked.  it's not clear where this is originating,
since the fuse implementation [should be] waiting for completion of tahoe 
operations before returning from its calls.  This adds an option to control the
duration of a pause between the fuse write and the check of tahoe, which is by
default set to 2s on linux, which - somewhat inexplicably - seems to 'fix' the 
problem, in as far as it allows tests to complete.
2008-10-20 17:29:02 -07:00
robk-tahoe
fafb584ef6 fuse/runtests: include length in drepr() output 2008-10-20 17:01:59 -07:00
robk-tahoe
3fbfd9fce3 fuse/runtests: make exceptions in 'read_in_random_order' into TestFailures 2008-10-20 16:52:35 -07:00
robk-tahoe
f08d181764 fuse/blackmatch: added asynchronous (background) file download
previously, upon opening a file for reading, the open() call would block
while the entire file was retrieved from tahoe into the cache directory.
This change adds a DownloaderWithReadQueue class, and associated plumbing,
such that an open() will return promptly with the download initiated 'in
the background'.  Subsequent read() operations will block until enough
data has been downloaded to satisfy that request.  This provides a behaviour
similar to streaming, i.e. the client application will be able to read
data from the fuse interface while the remainder of the file is still being
downloaded.
2008-10-20 16:33:33 -07:00
robk-tahoe
1cdfecb446 fuse/runtests: added 'read_in_random_order' test
this test uploads a test file to tahoe, and then reads the file from fuse,
but reads the blocks of the file in a random order; this is designed to
exercise the asynchronous download feature of blackmatch - where the file
is downloaded from tahoe asynchronously, and rather than blocking open()
for the entirety of the download, instead individual read() calls are
blocked until enough of the file has been downloaded to satisfy them
2008-10-20 16:24:27 -07:00
robk-tahoe
c6319dde05 fuse/runtests: added a --no-cleanup option
the code had a 'fullcleanup' flag internally which controlled whether
working directories were cleaned up.  this promotes that to a command
line option (negated) '--no-cleanup' defaulting to False, i.e. do cleanup
2008-10-20 08:51:20 -07:00
robk-tahoe
23cce79d9f fuse/runtests: truncate expected file contents in reported error message
this avoids dumping the repr of 1Mb of random data to stdout in the event
of a test failure, but rather just dumps the start/end of the errant strings
if the amount of data is > 200 chars repr'd
2008-10-20 07:45:23 -07:00
robk-tahoe
e0fb7735bc fuse/blackmatch: fix platform specific problems in repr_flags
the repr_flags debug/logging function had a list of fields from the os 
module that might be passed into an open() call, but it included at 
least one which was available on the mac but not on linux. symmetrically
linux has numerous flags which are not present on the mac. the repr_flags
function is now tolerant of flags not being present, and has an expanded
list of flags
2008-10-20 07:30:52 -07:00
robk-tahoe
e1f7008e47 makefile: added 'fuse-test' target to makefile, to run 'runtests' 2008-10-19 06:25:18 -07:00
robk-tahoe
8566c9751d fuse/runtests: added a 'todo' flag, surpressing failure for implementations not expected to pass
since the current tests assume that the implementation responds to changes made
to tahoe after mount, and impl_b prefetches and cached directory data, impl_b
fails the current 'read' test suite.

rather than reflect that problem in the overall failure of the runtests exit
code, this adds a 'todo' flag to the implementations table, and sets the todo
flag for impl_b.  Thus errors will therein be reported in output, but not cause
a failing exit code.
2008-10-19 06:16:00 -07:00
robk-tahoe
70222ecf1e fuse/runtests: made runtests exit code depend on success
return an exit code of 0 only if no tests failed, and 1 in the case of
linkage error, test setup failure, or individual test case failure
2008-10-17 11:00:58 -07:00
Brian Warner
0832334c0d storage.py: assert that immutable share size will fit in the 4-byte v1 container (see #346). The struct module in py2.4 raises an error on overflow, but py2.5 merely emits a warning 2008-10-20 10:22:08 -07:00
Brian Warner
c08b0eb1ea NEWS: update to summarize all changes since the last update 2008-10-20 09:40:47 -07:00
robk-tahoe
734e4c2343 fuse/runtest: make removal of webport file soft
previously the runtests suite removed the webport file created by
tahoe create-client in all but the first node.  now that the node config
is in tahoe.cfg by default this file might not exist.
2008-10-16 20:01:54 -07:00
robk-tahoe
e679d48660 fuse/blackmatch: update json handling to support simplejson v2
simplejson v2 returns strings as either unicode or str, depending upon its
mood.  thus the interpretation of the node's json repr of a directory, and
the serialisation of strings in the json based rpc both exploded when built
against simplejson v2.  this makes both of these places liberal in their
acceptance of either str or unicode.
2008-10-16 19:59:31 -07:00
robk-tahoe
7ee6882884 fuse/blackmatch: log exception in server startup
humphf.  my build runs the fuse stuff fine, but the build from the buildslave
doesn't seem to start up properly.  hopefully this will elicit some useful info
2008-10-16 18:46:50 -07:00
robk-tahoe
032cac36ed fuse/blackmatch: add readability to some logging, fix a permissions problem
adds a couple of functions to unpack 'mode' and 'flags' for open() calls, to
facilitate debugging.

adds a fix to ensure that all tmp files created for writing are opened with
permissions 0600 - one problem I had with testing with the Finder was that
files were being opened write only (0200) and were then failing to upload
to tahoe due to internal permission denied errors.

there remain a variety of problems with finder access which I'm unable to
comprehend at this time.  sometimes copies to tahoe will work fine, sometimes
they yield "the finder cannot complete the operation because some data ...
could not be read or written. (Error code -36)" sometimes "You may need to
enter the name and password for an administrator on this computer to change
the item" sometimes "The operation cannot be completed because an item with
the name ... already exists." and sometimes "The operation cannot be completed
because the item ... is locked."  What seems to be absent is rhyme or reason.

unix operations (cp, mv) work fine, rsync works fine.
2008-10-16 17:44:21 -07:00
robk-tahoe
e3bc3148c9 fuse/blackmatch: fix linkage problems with daemonize
the daemonize() function imported from twisted was causing problems when 
run from a frozen (py2app) build.  I simply copied the daemonize function
into this file, and that fixes the problem.

also removed a couple of lines of debugging spam that slipped through.
2008-10-16 09:36:37 -07:00
robk-tahoe
4785299980 gui/macapp: minor bugfixes
though it seemed to work before the 'fstype' passed to fuse of 'allmydata' was
today throwing errors that len(fstype) must be at most 7.

fixed a typo in changes to 'mount_filesystem()' args

bumped the delay between mounting a filesystem and 'open'ing it in Finder to
4s, as it seems to take a little longer to mount now the client and server
fuse processes need to coordinate.
2008-10-16 09:30:52 -07:00
robk-tahoe
4bc57f19c9 fuse/blackmatch: split into client/server (twisted server)
This implements a client/server split for blackmatch, where the client 
implements the fuse_main bindings and a simple blocking rpc client mechanism.
The server implements the other half of that rpc mechanism, and contains all
the actual logic for interpreting fuse requests in the context of the on disk
cache and requests to the tahoe node.  The server is based on a twisted reactor.

The rpc mechanism implements a simple method dispatch including marshalling,
using json, of basic inert data types, in a flat namespace (no objects).
The client side is written in a blocking idiom, to interface with the threading
model used by the fuse_main bindings, whereas the server side is written for a
twisted reactor-based environment, intended to facilitate implementing more 
sophisticated logic in that paradigm.  The two communicate over a unix domain
socket, allocated within the nodedir.

Command line usage is unchanged; the server is launched automatically by the
client. The server daemonizes itself, to avoid preventing the original parent
process (e.g. 'runtests') from waiting upon the server exiting.

The client keeps open a 'keepalive' connection to the server; upon loss thereof
the server will exit. This addresses the fact that the python-fuse bindings 
provide no notification of exit of the client process upon unmount.

The client thus provides a relatively thin 'shim' proxying requests from the
fuse_main bindings across the rpc to the server process, which handles the 
logic behind each request.  

For the time being, a '--no-split' option is provided to surpress the splitting
into client/server, yielding the prior behaviour.  Once the server logic gets
more complex and more entrenched in a twisted idiom, this might be removed.
The 'runtests' test harness currently tests both modes, as 'impl_c' and 
'impl_c_no_split'
2008-10-16 08:08:46 -07:00
robk-tahoe
c0b2aae0d4 fuse/blackmatch: 'flatten' the fuse api implementation
the previous revision of blackmatch used a file_class to delegate all fuse
api operations on files to a specific per-file class, which is an option
given by the python-fuse bindings.

this is a pre-cursor to the 'split' client/server version, which uses a
simple, moreover flat, rpc mechanism to broker access to methods.
2008-10-16 07:35:47 -07:00
robk-tahoe
6a78000b55 fuse/runtests: disable impl_a/impl_b on mac, as they don't actually work. 2008-10-16 07:32:32 -07:00
robk-tahoe
85d1ec1c44 fuse/runtests: added write_partial_overwrite test
this tests opening a file for update, overwriting a small part of it, and
ensuring that the end result constitutes an overwrite of the original file.
This tests, e.g. the implementation doesn' open a 'fresh' file but does in
fact initialise the file to be uploaded with the contents of any extant
file before applying updates
2008-10-16 07:29:26 -07:00
robk-tahoe
5af8eeeb3d fuse/runtests: added --tests, renamed --suites
changed the --tests option to be --suites, as it takes a prefix, e.g. 'read'
'write' (or 'all', the default) and runs those suites which are applicable to
each implementation being tested.

added a --tests option, which takes a list of tests, e.g. 'read_file_contents'
'write_overlapping_large_writes' and runs all tests specified without regard
to whether the implementation(s) under test are declared to support them.

this is basically to allow a specific test or two to be run, saving time 
during development and debugging by not running the entire suite
2008-10-16 07:28:36 -07:00