Commit Graph

70 Commits

Author SHA1 Message Date
robk-tahoe
8566c9751d fuse/runtests: added a 'todo' flag, surpressing failure for implementations not expected to pass
since the current tests assume that the implementation responds to changes made
to tahoe after mount, and impl_b prefetches and cached directory data, impl_b
fails the current 'read' test suite.

rather than reflect that problem in the overall failure of the runtests exit
code, this adds a 'todo' flag to the implementations table, and sets the todo
flag for impl_b.  Thus errors will therein be reported in output, but not cause
a failing exit code.
2008-10-19 06:16:00 -07:00
robk-tahoe
70222ecf1e fuse/runtests: made runtests exit code depend on success
return an exit code of 0 only if no tests failed, and 1 in the case of
linkage error, test setup failure, or individual test case failure
2008-10-17 11:00:58 -07:00
robk-tahoe
734e4c2343 fuse/runtest: make removal of webport file soft
previously the runtests suite removed the webport file created by
tahoe create-client in all but the first node.  now that the node config
is in tahoe.cfg by default this file might not exist.
2008-10-16 20:01:54 -07:00
robk-tahoe
e679d48660 fuse/blackmatch: update json handling to support simplejson v2
simplejson v2 returns strings as either unicode or str, depending upon its
mood.  thus the interpretation of the node's json repr of a directory, and
the serialisation of strings in the json based rpc both exploded when built
against simplejson v2.  this makes both of these places liberal in their
acceptance of either str or unicode.
2008-10-16 19:59:31 -07:00
robk-tahoe
7ee6882884 fuse/blackmatch: log exception in server startup
humphf.  my build runs the fuse stuff fine, but the build from the buildslave
doesn't seem to start up properly.  hopefully this will elicit some useful info
2008-10-16 18:46:50 -07:00
robk-tahoe
032cac36ed fuse/blackmatch: add readability to some logging, fix a permissions problem
adds a couple of functions to unpack 'mode' and 'flags' for open() calls, to
facilitate debugging.

adds a fix to ensure that all tmp files created for writing are opened with
permissions 0600 - one problem I had with testing with the Finder was that
files were being opened write only (0200) and were then failing to upload
to tahoe due to internal permission denied errors.

there remain a variety of problems with finder access which I'm unable to
comprehend at this time.  sometimes copies to tahoe will work fine, sometimes
they yield "the finder cannot complete the operation because some data ...
could not be read or written. (Error code -36)" sometimes "You may need to
enter the name and password for an administrator on this computer to change
the item" sometimes "The operation cannot be completed because an item with
the name ... already exists." and sometimes "The operation cannot be completed
because the item ... is locked."  What seems to be absent is rhyme or reason.

unix operations (cp, mv) work fine, rsync works fine.
2008-10-16 17:44:21 -07:00
robk-tahoe
e3bc3148c9 fuse/blackmatch: fix linkage problems with daemonize
the daemonize() function imported from twisted was causing problems when 
run from a frozen (py2app) build.  I simply copied the daemonize function
into this file, and that fixes the problem.

also removed a couple of lines of debugging spam that slipped through.
2008-10-16 09:36:37 -07:00
robk-tahoe
4bc57f19c9 fuse/blackmatch: split into client/server (twisted server)
This implements a client/server split for blackmatch, where the client 
implements the fuse_main bindings and a simple blocking rpc client mechanism.
The server implements the other half of that rpc mechanism, and contains all
the actual logic for interpreting fuse requests in the context of the on disk
cache and requests to the tahoe node.  The server is based on a twisted reactor.

The rpc mechanism implements a simple method dispatch including marshalling,
using json, of basic inert data types, in a flat namespace (no objects).
The client side is written in a blocking idiom, to interface with the threading
model used by the fuse_main bindings, whereas the server side is written for a
twisted reactor-based environment, intended to facilitate implementing more 
sophisticated logic in that paradigm.  The two communicate over a unix domain
socket, allocated within the nodedir.

Command line usage is unchanged; the server is launched automatically by the
client. The server daemonizes itself, to avoid preventing the original parent
process (e.g. 'runtests') from waiting upon the server exiting.

The client keeps open a 'keepalive' connection to the server; upon loss thereof
the server will exit. This addresses the fact that the python-fuse bindings 
provide no notification of exit of the client process upon unmount.

The client thus provides a relatively thin 'shim' proxying requests from the
fuse_main bindings across the rpc to the server process, which handles the 
logic behind each request.  

For the time being, a '--no-split' option is provided to surpress the splitting
into client/server, yielding the prior behaviour.  Once the server logic gets
more complex and more entrenched in a twisted idiom, this might be removed.
The 'runtests' test harness currently tests both modes, as 'impl_c' and 
'impl_c_no_split'
2008-10-16 08:08:46 -07:00
robk-tahoe
c0b2aae0d4 fuse/blackmatch: 'flatten' the fuse api implementation
the previous revision of blackmatch used a file_class to delegate all fuse
api operations on files to a specific per-file class, which is an option
given by the python-fuse bindings.

this is a pre-cursor to the 'split' client/server version, which uses a
simple, moreover flat, rpc mechanism to broker access to methods.
2008-10-16 07:35:47 -07:00
robk-tahoe
6a78000b55 fuse/runtests: disable impl_a/impl_b on mac, as they don't actually work. 2008-10-16 07:32:32 -07:00
robk-tahoe
85d1ec1c44 fuse/runtests: added write_partial_overwrite test
this tests opening a file for update, overwriting a small part of it, and
ensuring that the end result constitutes an overwrite of the original file.
This tests, e.g. the implementation doesn' open a 'fresh' file but does in
fact initialise the file to be uploaded with the contents of any extant
file before applying updates
2008-10-16 07:29:26 -07:00
robk-tahoe
5af8eeeb3d fuse/runtests: added --tests, renamed --suites
changed the --tests option to be --suites, as it takes a prefix, e.g. 'read'
'write' (or 'all', the default) and runs those suites which are applicable to
each implementation being tested.

added a --tests option, which takes a list of tests, e.g. 'read_file_contents'
'write_overlapping_large_writes' and runs all tests specified without regard
to whether the implementation(s) under test are declared to support them.

this is basically to allow a specific test or two to be run, saving time 
during development and debugging by not running the entire suite
2008-10-16 07:28:36 -07:00
robk-tahoe
2bcae75012 fuse/runtests: added 'random scatter' write test
this writes the test file in a randomised order, with randomly sized writes.
also for each 'slice' of the file written, a randomly chosen overlapping
write is also made to the file.  this ensures that the file will be written
in its entirety in a thoroughly random order, with many overlapping writes.
2008-10-03 16:34:36 -07:00
robk-tahoe
0c25628ded fuse/runtests: add overlapping write tests
using both small and large blocksizes for writes, write a 1Mb file to fuse
where every write overlaps another. 

This serves a useful purpose - in manual testing of blackmatch some time ago
most operations e.g. bulk copies, worked fine, but using rsync caused data 
corruption on most files.  it turned out to be that rsync writes in 64K blocks,
but rather than making the last block short, the last block instead overlaps
the preceding (already written) block.  This revealed a problem where cache
files were being opened 'append' rather than 'write' and hence the overlapping
write to the fuse layer caused the overlapping portion of the file to be 
duplicated in cache, leading to oversized and corrupt files being uploaded.
2008-10-03 15:48:33 -07:00
robk-tahoe
50986dc1e0 fuse/runtests: remove write small file test, as it's subsumed by the tiny_file test 2008-10-03 15:39:44 -07:00
robk-tahoe
4a3df3b683 fuse/runtests: added linear write tests for various block sizes
unit tests to test writing contiguous blocks linearly through the file,
for a variety of block sizes;  'tiny_file' is an entire file fitting within
a single io block / write operation.  'linear_{small,large}_writes' test
a 1Mb file written with each write operation containing significantly less
or more, respecitvely, data than fuse will pass into the implementation as
a single operation (which on the mac at least is 64Kib)
2008-10-03 15:35:50 -07:00
robk-tahoe
af9e1f6238 fuse/runtests: add a very simple 'write' test
this performs a very simple write through the fuse layer and confirms that
the file is stored correctly into the tahoe mesh.  ('simple' in the sense
that the entire file body fits trivially in a single write() operation, 
disk block etc)
2008-10-03 10:20:44 -07:00
robk-tahoe
8b9a267920 fuse/runtests: added a --web-open option
similar to the --debug-wait option which causes the test harness to
pause at various stages of the process to facilitate debugging, this
option simplifies that debugging by automatically opening a web browser
to the root dir of that implementation's tests when tests are commenced.

in addition, if --web-open is specfied but --debug-wait is not, the
harness will still pause after running tests but before tearing down
the tahoe grid - this allows all tests to run to completion, but
provide a debugging hook to investigate the end state of the grid's
contents thereafter.
2008-10-03 10:20:26 -07:00
robk-tahoe
236c52bf8b fuse/impl_a: fix a suspected bug in caching
from my examination of the tahoe_fuse ('impl_a') code, it looks like 
the intention is to cache the file contents in memory while it's open,
since it does in fact do that.  however it looks like it also ignored
that cache entirely, and made an individual tahoe webapi GET request
for each and every read() operation regardless of the relative size of
the read block and the file in question.

this changes that to make read() use the data in memory rather than
fetch the data over again.   if there's something more subtle going
on, please let me know.
2008-10-03 10:13:09 -07:00
robk-tahoe
6763df16f1 fuse/impl_c: UNDO --auto-fsid option
rolling back:

Thu Sep 25 14:42:23 BST 2008  robk-tahoe@allmydata.com
  * fuse/impl_c: add --auto-fsid option
  
  this was inspired by reading the fuse docs and discovering the 'fsid' option
  to fuse_main, and was _intended_ to support a sort of 'stability' to the 
  filesystem (specifically derived from the root-uri mounted, whether directly
  or via an alias) to support mac aliases across unmount/remount etc.
  
  some experimentation shows that that doesn't actually work, and that, at
  least for mac aliases in my testing, they're tied to path-to-mountpoint and
  not to the fsid - which seems to have no bearing.  perhaps the 'local' flag
  is causing weirdness therein.
  
  at any rate, I'm recording it simply for posterity, in case it turns out to
  be useful after all somewhere down the road.
  

    M ./contrib/fuse/impl_c/blackmatch.py +13
2008-09-25 06:47:30 -07:00
robk-tahoe
d79001543b fuse/impl_c: add --auto-fsid option
this was inspired by reading the fuse docs and discovering the 'fsid' option
to fuse_main, and was _intended_ to support a sort of 'stability' to the 
filesystem (specifically derived from the root-uri mounted, whether directly
or via an alias) to support mac aliases across unmount/remount etc.

some experimentation shows that that doesn't actually work, and that, at
least for mac aliases in my testing, they're tied to path-to-mountpoint and
not to the fsid - which seems to have no bearing.  perhaps the 'local' flag
is causing weirdness therein.

at any rate, I'm recording it simply for posterity, in case it turns out to
be useful after all somewhere down the road.
2008-09-25 06:42:23 -07:00
robk-tahoe
00aa75d1e8 fuse/impl_c: move mac tahoefuse impl out into contrib/fuse
For a variety of reasons, high amongst them the fact that many people 
interested in fuse support for tahoe seem to have missed its existence,
the existing fuse implementation for tahoe, previously 'mac/tahoefuse.py'
has been renamed and moved.

It was suggested that, even though the mac build depends upon it, that
the mac/tahoefuse implementation be moved into contrib/fuse along with
the other fuse implementations.  The fact that it's not as extensively
covered by unit tests as mainline tahoe was given as corroboration.

In a bid to try and stem the confusion inherent in having tahoe_fuse,
tfuse and tahoefuse jumbled together (not necessarily helped by 
referring to them as impl_a, b and c respectively) I'm hereby renaming
tahoefuse as 'blackmatch'  (black match is, per wikipedia "a type of 
crude fuse" hey, I'm a punny guy)  Maybe one day it'll be promoted to
be 'quickmatch' instead...

Anyway, this patch moves mac/tahoefuse.py out to contrib/fuse/impl_c/
as blackmatch.py, and makes appropriate changes to the mac build process
to transclude blackmatch therein.  this leaves the extant fuse.py and
fuseparts business in mac/ as-is and doesn't attempt to address such
issues in contrib/fuse/impl_c.

it is left as an exercise to the reader (or the reader of a message
to follow) as to how to deal with the 'fuse' python module on the mac.

as of this time, blackmatch should work on both mac and linux, and
passes the four extant tests in runtests.  (fwiw neither impl_a nor
impl_b have I managed to get working on the mac yet)

since blackmatch supports a read-write and caching fuse interface to
tahoe, some write tests obviously need to be added to runtests.
2008-09-24 18:42:14 -07:00
robk-tahoe
0a2f32649f fuse/tests: slew of changes to fuse 'runtests'
This patch makes a significant number of changes to the fuse 'runtests' script
which stem from my efforts to integrate the third fuse implementation into this
framework.  Perhaps not all were necessary to that end, and I beg nejucomo's
forebearance if I got too carried away.

- cleaned up the blank lines; imho blank lines should be empty

- made the unmount command switch based on platform, since macfuse just uses
'umount' not the 'fusermount' command (which doesn't exist)

- made the expected working dir for runtests the contrib/fuse dir, not the 
top-level tahoe source tree - see also discussion of --path-to-tahoe below

- significantly reworked the ImplProcManager class.  rather than subclassing
for each fuse implementation to be tested, the new version is based on 
instantiating objects and providing relevant config info to the constructor.
this was motivated by a desire to eliminate the duplication of similar but
subtly different code between instances, framed by consideration of increasing
the number of platforms and implementations involved. each implementation to
test is thus reduced to the pertinent import and an entry in the 
'implementations' table defining how to handle that implementation. this also
provides a way to specify which sets of tests to run for each implementation,
more on that below.


- significantly reworked the command line options parsing, using twisted.usage;

what used to be a single optional argument is now represented by the 
--test-type option which allows one to choose between running unittests, the
system tests, or both.

the --implementations option allows for a specific (comma-separated) list of
implemenations to be tested, or the default 'all'

the --tests option allows for a specific (comma-separated) list of tests sets
to be run, or the default 'all'.  note that only the intersection of tests
requested on the command line and tests relevant to each implementation will
be run. see below for more on tests sets.

the --path-to-tahoe open allows for the path to the 'tahoe' executable to be
specified. it defaults to '../../bin/tahoe' which is the location of the tahoe
script in the source tree relative to the contrib/fuse dir by default.

the --tmp-dir option controls where temporary directories (and hence 
mountpoints) are created during the test.  this defaults to /tmp - a change
from the previous behaviour of using the system default dir for calls to 
tempfile.mkdtemp(), a behaviour which can be obtained by providing an empty
value, e.g. "--tmp-dir=" 

the --debug-wait flag causes the test runner to pause waiting upon user
input at various stages through the testing, which facilitates debugging e.g.
by allowing the user to open a browser and explore or modify the contents of
the ephemeral grid after it has been instantiated but before tests are run,
or make environmental adjustments before actually triggering fuse mounts etc.
note that the webapi url for the first client node is printed out upon its
startup to facilitate this sort of debugging also.


- the default tmp dir was changed, and made configurable. previously the 
default behaviour of tempfile.mkdtemp() was used.  it turns out that, at least
on the mac, that led to temporary directories to be created in a location
which ultimately led to mountpoint paths longer than could be handled by 
macfuse - specifically mounted filesystems could not be unmounted and would
'leak'. by changing the default location to be rooted at /tmp this leads to
mountpoint paths short enough to be supported without problems.

- tests are now grouped into 'sets' by method name prefix.  all the existing
tests have been moved into the 'read' set, i.e. with method names starting
'test_read_'. this is intended to facilitate the fact that some implementations
are read-only, and some support write, so the applicability of tests will vary
by implementation. the 'implementations' table, which governs the configuration
of the ImplProcManager responsible for a given implementation, provides a list
of 'test' (i.e test set names) which are applicable to that implementation.
note no 'write' tests yet exist, this is merely laying the groundwork.

- the 'expected output' of the tahoe command, which is checked for 'surprising'
output by regex match, can be confused by spurious output from libraries.
specfically, testing on the mac produced a warning message about zope interface
resolution various multiple eggs.  the 'check_tahoe_output()' function now has
a list of 'ignorable_lines' (each a regex) which will be discarded before the
remainder of the output of the tahoe script is matched against expectation.

- cleaned up a typo, and a few spurious imports caught by pyflakes
2008-09-24 11:36:01 -07:00
robk-tahoe
4f04bf99a5 fuse/impl_{a,b}: improve node-url handling
specifically change the expectation of the code to be such that the node-url
(self.url) always includes the trailing slash to be a correctly formed url

moreover read the node-url from the 'node.url' file found in the node 'basedir'
and only if that doesn't exist, then fall back to reading the 'webport' file
from therein and assuming localhost.  This then supports the general tahoe 
pattern that tools needing only a webapi server can be pointed at a directory
containing the node.url file, which can optionally point to another server,
rather than requiring a complete node dir and locally running node instance.
2008-09-24 11:28:54 -07:00
robk-tahoe
97229238b0 fuse/impl_b: tweaks from testing on hardy
from testing on linux (specifically ubuntu hardy) the libfuse dll has a
different name, specifically libfuse.so.2. this patch tries libfuse.so
and then falls back to trying .2 if the former fails.

it also changes the unmount behaviour, to simply return from the handler's
loop_forever() loop upon being unmounted, rather than raising an EOFError,
since none of the client code I looked at actually handled that exception,
but did seem to expect to fall off of main() when loop_forever() returned.
Additionally, from my testing unmount typically led to an OSError from the
fuse fd read, rather than an empty read, as the code seemed to expect.

also removed a spurious import pyflakes quibbled about.
2008-09-24 11:07:38 -07:00
nejucomo
8e9bb72008 fuse: runtests: Create an interface for setup/cleanup of the two implementations...
The impl_b cleanup appears incorrect.  I'm not sure what the proper behavior is.
2008-06-07 00:08:25 -07:00
nejucomo
171b430afb fuse: runtests: Wrap OSError exceptions which are test failures. 2008-06-07 00:07:18 -07:00
nejucomo
609e9b08da fuse: runtests: Move exception classes to top scope. 2008-06-07 00:06:00 -07:00
nejucomo
15fdb572b8 fuse: runtests: Fix typo in summary reporting. 2008-06-07 00:05:07 -07:00
nejucomo
0a94ac5732 fuse: runtests: Make test numbers (and everything in general) 0-indexed for consistency. 2008-06-06 23:19:15 -07:00
nejucomo
f665b10b12 fuse: runtests.py: Fix a typo bug in fusermount output checking. 2008-06-06 23:18:15 -07:00
nejucomo
4d8aac35f4 fuse: runtests.py: Fix bug in polling_operation error that always referred to introducer.furl. 2008-06-06 23:17:19 -07:00
nejucomo
4f94d00abe fuse: impl_b: Support --basedir on commandline. 2008-06-06 23:16:38 -07:00
nejucomo
2fa5785960 fuse: impl_b: Add impl_b to the contrib directory. 2008-06-06 22:22:36 -07:00
nejucomo
e538651947 fuse: Reorganize directory tree and modify runtests.py to run against both implementations...
Currently, fuse impl_b does not support a --basedir argument, and always
uses ~/.tahoe, which makes it incompatible with these system tests.
2008-06-06 22:19:23 -07:00
Zooko O'Whielacronx
c88ea89f14 contrib: add a note about Armin Rigo's fuse implementation 2008-04-28 07:05:44 -07:00
nejucomo
67660232f1 tahoe_fuse: system test: Verify file contents can be properly read. 2008-01-30 03:14:48 -07:00
nejucomo
28552e5967 tahoe_fuse: system test: Populate a testdir with files and empty children directories, then test the fuse interface for proper listings and size metadata. 2008-01-30 02:59:43 -07:00
nejucomo
7240021d1c tahoe_fuse: system test: Create a separate directory for each test and pass the cap and local path to each test. Add two basic sanity tests for empty directories. 2008-01-30 02:57:54 -07:00
nejucomo
9ecfbc62cd tahoe_fuse: system test: Replace repeated attempts at webapi calls with single calls, abstract webapi calls into a single function. 2008-01-30 02:56:25 -07:00
nejucomo
0dbdb0f871 tahoe_fuse: system test: Remove some needless comments. 2008-01-30 02:55:53 -07:00
nejucomo
5e94c07045 tahoe_fuse: system test: Move test summary to end of output. 2008-01-30 02:46:24 -07:00
nejucomo
ab3d399ac7 tahoe_fuse: system tests: Update comments. 2008-01-30 02:45:54 -07:00
nejucomo
98e92e2a48 tahoe_fuse: system test: Make output checking into non-fatal warnings, and make patterns looser. 2008-01-30 01:10:53 -07:00
nejucomo
3aceb6be1e tahoe_fuse.py: system test: Distinguish between TestFailures and unexpected exceptions during testing (and fix a typo). 2008-01-28 22:42:28 -07:00
nejucomo
7421d99f18 tahoe_fuse.py: system test: setup: lexically sort test names, create a TestFailure class, implement an empty directory listing test. 2008-01-28 22:40:47 -07:00
nejucomo
597ab62d12 tahoe_fuse.py: system test: setup: fixed a bug in which the mointpoint was not created before mounting. 2008-01-28 22:39:13 -07:00
nejucomo
7fe264d280 tahoe_fuse.py: system test: Many changes to framework...
The flow control has been de-obfuscated a bit.

Some output changes.

The test framework has quite a few race conditions, but it does a reasonable job of setting up and cleaning up.
2008-01-28 22:27:19 -07:00
nejucomo
b4c566efca Individual tests run after all the setup layers are in place. 2008-01-28 22:25:11 -07:00
nejucomo
de022b2115 tahoe_fuse: cmdline args & system test: Allow nonstandard client basedirs to be specified and update the system tests to use this feature...
The commandline option handling of the version of python-fuse I use is arcane.  This is an ugly hack.
2008-01-20 20:56:27 -07:00