I've also set up a new flappserver on source@allmydata.org to receive the
tarballs. We still need to replace the gutsy buildslave (which is where the
tarballs used to be generated+uploaded) and give it the new FURL.
* "cap" means a python instance which encapsulates a filecap/dircap (uri.py)
* "uri" means a string with a "URI:" prefix
* FileNode instances are created with (and retain) a cap instance, and
generate uri strings on demand
* .get_cap/get_readcap/get_verifycap/get_repaircap return cap instances
* .get_uri/get_readonly_uri return uri strings
* add filenode.download_to_filename() for control.py, should find a better way
* use MutableFileNode.init_from_cap, not .init_from_uri
* directory URI instances: use get_filenode_cap, not get_filenode_uri
* update/cleanup bench_dirnode.py to match, add Makefile target to run it
I made this patch by copying [20090305220021-92b7f-89d987c7d05306b5cb03a64f2956a652c10a7296] and changing the name from "intrepid" to "lenny". I haven't tested it.
'setup.py trial'). 'setup.py trial' clobbers the tahoe .egg's PKG-INFO
"Version:" field (resetting it to 0.0.0), possibly because it isn't invoking
the darcsver subcommand that 'setup.py test' does before it runs the 'trial'
subcommand.
This slows down quicktest by another couple of seconds (see #591) and adds
more noise to its output, but without this change, 'make test' and 'make
quicktest' fail on test_runner (which spawns bin/tahoe as a subprocess, and
with a mangled Version:, the setuptools-based entry point script refuses to
recognize our source tree as a usable version of Tahoe).
This is to fix#591 ("make quicktest" could be quicker and less noisy). This means that "make quicktest" won't work unless the user has manually installed Twisted already such that the "trial" executable is on their PATH and the Twisted package is on their PYTHONPATH. This bypasses the behavior of setuptools_trial which builds and checks package dependencies before executing the tests.
For a variety of reasons, high amongst them the fact that many people
interested in fuse support for tahoe seem to have missed its existence,
the existing fuse implementation for tahoe, previously 'mac/tahoefuse.py'
has been renamed and moved.
It was suggested that, even though the mac build depends upon it, that
the mac/tahoefuse implementation be moved into contrib/fuse along with
the other fuse implementations. The fact that it's not as extensively
covered by unit tests as mainline tahoe was given as corroboration.
In a bid to try and stem the confusion inherent in having tahoe_fuse,
tfuse and tahoefuse jumbled together (not necessarily helped by
referring to them as impl_a, b and c respectively) I'm hereby renaming
tahoefuse as 'blackmatch' (black match is, per wikipedia "a type of
crude fuse" hey, I'm a punny guy) Maybe one day it'll be promoted to
be 'quickmatch' instead...
Anyway, this patch moves mac/tahoefuse.py out to contrib/fuse/impl_c/
as blackmatch.py, and makes appropriate changes to the mac build process
to transclude blackmatch therein. this leaves the extant fuse.py and
fuseparts business in mac/ as-is and doesn't attempt to address such
issues in contrib/fuse/impl_c.
it is left as an exercise to the reader (or the reader of a message
to follow) as to how to deal with the 'fuse' python module on the mac.
as of this time, blackmatch should work on both mac and linux, and
passes the four extant tests in runtests. (fwiw neither impl_a nor
impl_b have I managed to get working on the mac yet)
since blackmatch supports a read-write and caching fuse interface to
tahoe, some write tests obviously need to be added to runtests.
The 'make build' target now runs 'setup.py build_tahoe', which figures out
where the target 'supportlib' directory should go, and invokes 'setup.py
develop' with the appropriate arguments.
The 'make test' target now runs 'setup.py trial', which manages sys.path and
runs trial as a subroutine instead of spawning an external process. This
simplifies the case where Twisted was built as a dependent library (and thus
the 'trial' executable is not on PATH).
setup.py now manages sys.path and PYTHONPATH for its internal subcommands, so
the $(PP) prefix was removed from all Makefile targets that invoke setup.py .
For the remaining ones, the 'setup.py -q show_pythonpath' subcommand was
added to compute this prefix with python rather than with fragile
shell/Makefile syntax.
This should fix#506, but it means that if (for some weird reason) Twisted can't be auto-installed and the find_trial.py script doesn't work, the user will get a weird failure message instead of a clean failure message explaining that trial couldn't be found. Oh well.
Chris Galvan is working on a much nicer fix to all these issues -- see #505.
This hopefully works around the problem that Twisted v8.1.0 has a bug when used
with pyOpenSSL v0.7 which bug causes some unit tests to spuriously fail -- see
known_issues.txt r2788:
http://allmydata.org/trac/tahoe/browser/docs/known_issues.txt?rev=2788#L122
Also it matches with the fact that --reactor=poll is required on cygwin.
Remove docs/install-details.html and README.win32 for now (see #282).
Remove checks for pywin32 and pyopenssl in Makefile -- that is (or will be) automated by setuptools.
Remove twisted from setup_requires. This causes the problem in which Nevow doesn't declare its dependency on Twisted (#440) to yield a clear ImportError mentioning Twisted and to fail repeatedly, rather than yielding a weird ImportError and working on the second identical attempt.
Fix Makefile to set PATH so that trial and twistd can be found by "make test" after Twisted was installed into support/ during "make"
Looking over Brian's shoulder the other day, I saw a message that said "Please seedocs/install.html".
Looking at the source code of the Makefile, it seems like it should say
@echo "ERROR: Not all of Tahoe's dependencies are in place. Please see \
docs/install.html for help on installing dependencies."
instead of its current:
@echo "ERROR: Not all of Tahoe's dependencies are in place. Please see\
docs/install.html for help on installing dependencies."
But, I remember changing this more than once in the past, so either there is a different version of make somewhere which interprets trailing backslashes differently, or someone (possibly me) has repeatedly gotten confused about this issue. Anyway, this time around, I'm trying:
@echo "ERROR: Not all of Tahoe's dependencies are in place. Please see docs/install.html for help on installing dependencies."
Even though it is > 80 chars.
It's evil and wrong to call something a "Makefile" when it contains code that can't be interpreted by POSIX make and requires GNU make.
But everyone else is doing it. ;-)
the make mac-upload target now requires an UPLOAD_DEST argument to be given,
which is the rsync destination (including trailing '/') to which the version
stamped directory containing the .dmg should be placed. the account the
build is running as (e.g. 'buildslave') should have ssh access to the account
specified in that dest. one might also consider locking the key down to the
target directory by adding something like
command="rsync --server -vlogDtpr . /home/amduser/public_html/dist/mac-blah/"
to the corresponding authorized_key entry on the target machine.
We have a desire to collect runtime statistics from multiple nodes primarily
for server monitoring purposes. This implements a simple implementation of
such a system, as a skeleton to build more sophistication upon.
Each client now looks for a 'stats_gatherer.furl' config file. If it has
been configured to use a stats gatherer, then it instantiates internally
a StatsProvider. This is a central place for code which wishes to offer
stats up for monitoring to report them to, either by calling
stats_provider.count('stat.name', value) to increment a counter, or by
registering a class as a stats producer with sp.register_producer(obj).
The StatsProvider connects to the StatsGatherer server and provides its
provider upon startup. The StatsGatherer is then responsible for polling
the attached providers periodically to retrieve the data provided.
The provider queries each registered producer when the gatherer queries
the provider. Both the internal 'counters' and the queried 'stats' are
then reported to the gatherer.
This provides a simple gatherer app, (c.f. make stats-gatherer-run)
which prints its furl and listens for incoming connections. Once a
minute, the gatherer polls all connected providers, and writes the
retrieved data into a pickle file.
Also included is a munin plugin which knows how to read the gatherer's
stats.pickle and output data munin can interpret. this plugin,
tahoe-stats.py can be symlinked as multiple different names within
munin's 'plugins' directory, and inspects argv to determine which
data to display, doing a lookup in a table within that file.
It looks in the environment for 'statsfile' to determine the path to
the gatherer's stats.pickle. An example plugins-conf.d file is
provided.
This patch adds support for a mac native build.
At the moment it's a fairly simple .app - i.e. so simple as to be unacceptable
for a shipping product, but ok for testing and experiment at this point.
notably once launched, the app's ui does not respond at all, although its dock
icon does allow it to be force-quit.
this produces a single .app bundle, which when run will look for a node basedir
in ~/.tahoe. If one is not found, one will be created in ~/Library/Application
Support/Allmydata Tahoe, and that will be symlinked to ~/.tahoe
if the basedir is lacking basic config (introducer.furl and root_dir.cap) then
the wx config wizard will be launched to log into an account and to set up
those files.
if a webport file is not found, the default value of 8123 will be written into
it.
once the node has started running, a webbrowser will be opened to the webish
interface at the users root_dir
note that, once configured, the node runs as the main thread of the .app,
no daemonisation is done, twistd is not involved.
the binary itself, from within the .app bundle, i.e.
"Allmydata Tahoe.app/Contents/MacOS/Allmydata Tahoe"
can be used from the command line and functions as the 'tahoe' executable
would in a unix environment, with one exception - when launched with no args
it triggers the default behaviour of running a node, and if necessary config
wizard, as if the user had launched the .app
one other gotcha to be aware of is that symlinking to this binary from some
other place in ones $PATH will most likely not work. when I tried this,
something - wx I believe - exploded, since it seems to use argv[0] to figure
out where necessary libraries reside and fails if argv[0] isn't in the .app
bundle. it's pretty easy to set up a script a la
#!/bin/bash
/Blah/blah/blah/Allmydata\ Tahoe.app/Contents/MacOS/Allmydata\ Tahoe "${@}"
Using pkg_resources.require() like this also apparently allows people to install multiple different versions of packages on their system and tahoe (if pkg_resources is available to it) will import the version of the package that it requires. I haven't tested this feature.
Now we use "./setup.py develop" to ensure that changes to our source code are immediately used without requiring a "make" step. This simplification will hopefully pave the way for easier py2exe and py2app, solving the "Unit tests test the installed version" bug (#145), and perhaps also #164 and #176.
This patch also conditionalizes the use of setuptools_darcs on the absence of a PKG-INFO file, which is part of fixing #263.
since the installer upload got more complex (needing to chmod files before
rsyncing) I promoted it to a makefile target, simplifying the buildbot steps
involved
this resolves problems of py2exe's modulefinder collection of sources from
.zipped egg files, not by using easy_install to reach the --always-unzip
option, but rather with a small tool which unpacks any zipped egg files found
in misc/dependencies. this fixes the py2exe build given rollback of the
easy_install stuff which had broken the unix builds. misc/hatch-eggs.py
performs the honours.
this also includes a misc/sub-ver.py tool which substitutes elements of the
verion number for the current code base (by importing allmydata.__version__
hence make-version should be run first, and the python path carefully managed)
into template files using python's string interpolation of named args from a
dict as the templating syntax. i.e. %(major)d %(minor)d %(point)d %(nano)d
each expand to the individual components of the version number as codified
by the pyutil.version_class.Version class. there is also a %(build)s tag
which expands to the string form of the whole version number. This tool is
used to interpolate the automatically generated version information into the
innosetup source file in a form consistent with innosetup/windows' restrictions
add 'windows-installer' target to top level makefile to build a windows setup.exe package
using innosetup. this assumes innosetup 5 is installed in program files as normal.
this doesn't include any logic to manage version numbers at this point, it's just a
simple experiment to test out building an installer as yet.
because I don't fully understand the former, I suspect it of being implicated in the current buildslave redness, and we require --always-unzip solely for py2exe.
py2exe is unable to handle .eggs which are packaged as zip files
in preference it will pull in other versions of libraries if they
can be found in the environment.
this changes causes .eggs to be built as .egg directories, which
py2exe can handle.
These changes are a work-in-progress -- there are many incomplete and incorrect parts, but the install.html and running.html files
are complete and should work (and they are delightfully concise!). I'm pushing this just to let people see incremental progress
and to solicit feedback. "Testing out" the install.html and running.html files and submitting patches or bug reports would be
quite welcome.
More to come.
misc/make-version.py has a limitation which prevents it from generating version
stamps from our current darcs history. This limitation has been fixed in
pyutil's "darcsver". Rather than copy the fix from there to
misc/make-version.py, I'm making it so that you have to install pyutil if you
want to automatically generate _version.py files from the current darcs
history.
I don't know what will happen if the path ends up with something like
"C:\Programs and Files\Whatever\;" on Windows, and then that gets passed to
cygwin bash. This reminds me of Brian's suggestion to use Python helper
scripts (c.f. misc/find-dep-eggs.py) instead of writing this stuff in the
GNUmake language. And *that* reminds me of the idea of writing the whole damn
thing in Python instead of in GNUmake, i.e. make all of our build tools be
plugins for setuptools instead of being GNUmake targets.
