previously, upon opening a file for reading, the open() call would block
while the entire file was retrieved from tahoe into the cache directory.
This change adds a DownloaderWithReadQueue class, and associated plumbing,
such that an open() will return promptly with the download initiated 'in
the background'. Subsequent read() operations will block until enough
data has been downloaded to satisfy that request. This provides a behaviour
similar to streaming, i.e. the client application will be able to read
data from the fuse interface while the remainder of the file is still being
downloaded.
the repr_flags debug/logging function had a list of fields from the os
module that might be passed into an open() call, but it included at
least one which was available on the mac but not on linux. symmetrically
linux has numerous flags which are not present on the mac. the repr_flags
function is now tolerant of flags not being present, and has an expanded
list of flags
simplejson v2 returns strings as either unicode or str, depending upon its
mood. thus the interpretation of the node's json repr of a directory, and
the serialisation of strings in the json based rpc both exploded when built
against simplejson v2. this makes both of these places liberal in their
acceptance of either str or unicode.
humphf. my build runs the fuse stuff fine, but the build from the buildslave
doesn't seem to start up properly. hopefully this will elicit some useful info
adds a couple of functions to unpack 'mode' and 'flags' for open() calls, to
facilitate debugging.
adds a fix to ensure that all tmp files created for writing are opened with
permissions 0600 - one problem I had with testing with the Finder was that
files were being opened write only (0200) and were then failing to upload
to tahoe due to internal permission denied errors.
there remain a variety of problems with finder access which I'm unable to
comprehend at this time. sometimes copies to tahoe will work fine, sometimes
they yield "the finder cannot complete the operation because some data ...
could not be read or written. (Error code -36)" sometimes "You may need to
enter the name and password for an administrator on this computer to change
the item" sometimes "The operation cannot be completed because an item with
the name ... already exists." and sometimes "The operation cannot be completed
because the item ... is locked." What seems to be absent is rhyme or reason.
unix operations (cp, mv) work fine, rsync works fine.
the daemonize() function imported from twisted was causing problems when
run from a frozen (py2app) build. I simply copied the daemonize function
into this file, and that fixes the problem.
also removed a couple of lines of debugging spam that slipped through.
This implements a client/server split for blackmatch, where the client
implements the fuse_main bindings and a simple blocking rpc client mechanism.
The server implements the other half of that rpc mechanism, and contains all
the actual logic for interpreting fuse requests in the context of the on disk
cache and requests to the tahoe node. The server is based on a twisted reactor.
The rpc mechanism implements a simple method dispatch including marshalling,
using json, of basic inert data types, in a flat namespace (no objects).
The client side is written in a blocking idiom, to interface with the threading
model used by the fuse_main bindings, whereas the server side is written for a
twisted reactor-based environment, intended to facilitate implementing more
sophisticated logic in that paradigm. The two communicate over a unix domain
socket, allocated within the nodedir.
Command line usage is unchanged; the server is launched automatically by the
client. The server daemonizes itself, to avoid preventing the original parent
process (e.g. 'runtests') from waiting upon the server exiting.
The client keeps open a 'keepalive' connection to the server; upon loss thereof
the server will exit. This addresses the fact that the python-fuse bindings
provide no notification of exit of the client process upon unmount.
The client thus provides a relatively thin 'shim' proxying requests from the
fuse_main bindings across the rpc to the server process, which handles the
logic behind each request.
For the time being, a '--no-split' option is provided to surpress the splitting
into client/server, yielding the prior behaviour. Once the server logic gets
more complex and more entrenched in a twisted idiom, this might be removed.
The 'runtests' test harness currently tests both modes, as 'impl_c' and
'impl_c_no_split'
the previous revision of blackmatch used a file_class to delegate all fuse
api operations on files to a specific per-file class, which is an option
given by the python-fuse bindings.
this is a pre-cursor to the 'split' client/server version, which uses a
simple, moreover flat, rpc mechanism to broker access to methods.
rolling back:
Thu Sep 25 14:42:23 BST 2008 robk-tahoe@allmydata.com
* fuse/impl_c: add --auto-fsid option
this was inspired by reading the fuse docs and discovering the 'fsid' option
to fuse_main, and was _intended_ to support a sort of 'stability' to the
filesystem (specifically derived from the root-uri mounted, whether directly
or via an alias) to support mac aliases across unmount/remount etc.
some experimentation shows that that doesn't actually work, and that, at
least for mac aliases in my testing, they're tied to path-to-mountpoint and
not to the fsid - which seems to have no bearing. perhaps the 'local' flag
is causing weirdness therein.
at any rate, I'm recording it simply for posterity, in case it turns out to
be useful after all somewhere down the road.
M ./contrib/fuse/impl_c/blackmatch.py +13
this was inspired by reading the fuse docs and discovering the 'fsid' option
to fuse_main, and was _intended_ to support a sort of 'stability' to the
filesystem (specifically derived from the root-uri mounted, whether directly
or via an alias) to support mac aliases across unmount/remount etc.
some experimentation shows that that doesn't actually work, and that, at
least for mac aliases in my testing, they're tied to path-to-mountpoint and
not to the fsid - which seems to have no bearing. perhaps the 'local' flag
is causing weirdness therein.
at any rate, I'm recording it simply for posterity, in case it turns out to
be useful after all somewhere down the road.
For a variety of reasons, high amongst them the fact that many people
interested in fuse support for tahoe seem to have missed its existence,
the existing fuse implementation for tahoe, previously 'mac/tahoefuse.py'
has been renamed and moved.
It was suggested that, even though the mac build depends upon it, that
the mac/tahoefuse implementation be moved into contrib/fuse along with
the other fuse implementations. The fact that it's not as extensively
covered by unit tests as mainline tahoe was given as corroboration.
In a bid to try and stem the confusion inherent in having tahoe_fuse,
tfuse and tahoefuse jumbled together (not necessarily helped by
referring to them as impl_a, b and c respectively) I'm hereby renaming
tahoefuse as 'blackmatch' (black match is, per wikipedia "a type of
crude fuse" hey, I'm a punny guy) Maybe one day it'll be promoted to
be 'quickmatch' instead...
Anyway, this patch moves mac/tahoefuse.py out to contrib/fuse/impl_c/
as blackmatch.py, and makes appropriate changes to the mac build process
to transclude blackmatch therein. this leaves the extant fuse.py and
fuseparts business in mac/ as-is and doesn't attempt to address such
issues in contrib/fuse/impl_c.
it is left as an exercise to the reader (or the reader of a message
to follow) as to how to deal with the 'fuse' python module on the mac.
as of this time, blackmatch should work on both mac and linux, and
passes the four extant tests in runtests. (fwiw neither impl_a nor
impl_b have I managed to get working on the mac yet)
since blackmatch supports a read-write and caching fuse interface to
tahoe, some write tests obviously need to be added to runtests.
