Zooko O'Whielacronx
0c91a467e8
setup: remove old bundled setuptools-0.6c10dev (our own toothpick of setuptools)
2009-02-02 22:23:04 -07:00
Zooko O'Whielacronx
5c0d937eec
setup: bundle setuptools-0.6c11dev (our own toothpick of setuptools)
...
Hopefully this one fixes the issue with easy_install not searching the sys.path for packages that were requested to be installed, (setuptools #17 ), thus allowing us to do away with the "--site-dirs=" kludges, which are currently breaking some of our buildbots.
2009-02-02 22:19:50 -07:00
Brian Warner
d8d4070e98
misc/dependencies: update setuptools_trial from 0.4 to 0.5
2009-02-02 17:36:57 -07:00
Zooko O'Whielacronx
6a4d5abc1d
setup: add new bundled darcsver-1.2.0
2009-01-29 12:51:15 -07:00
Zooko O'Whielacronx
20d2136e24
setup: remove old bundled darcsver-1.1.8
2009-01-29 12:51:02 -07:00
Zooko O'Whielacronx
16f0fef785
setup: bundle darcsver-1.1.8
2009-01-28 14:03:26 -07:00
Zooko O'Whielacronx
4945dd0b0a
setup: remove old bundled dependency darcsver-1.1.5
2009-01-28 12:20:12 -07:00
Brian Warner
8d6bdb5856
diskwatcher: cache growth results (which take 30s to generate) for 3 minutes, to help munin, which invokes it 6 times in a row every 5 minutes
2009-01-25 17:06:39 -07:00
Zooko O'Whielacronx
0dbc0f1194
setup: add new bundled setuptools_trial-0.4
2009-01-20 17:40:12 -07:00
Zooko O'Whielacronx
8351d9d41e
setup: remove old bundled setuptools_trial-0.2
2009-01-20 17:39:18 -07:00
Zooko O'Whielacronx
4cd02cc12f
setup: bundle setuptools_trial in misc/dependencies/
2008-12-01 11:44:38 -07:00
Brian Warner
6ec352fe92
misc/cpu-watcher.tac: tolerate disk-full errors when writing the pickle, and pickle corruption from earlier disk-full errors
2008-12-05 15:54:12 -07:00
Brian Warner
995a16e3b1
misc/spacetime/diskwatcher.tac: remove dead code
2008-11-19 14:05:52 -07:00
Brian Warner
d224d22cb7
munin/tahoe_disktotal: add a 'disk used' line, since it will always be less than disktotal
2008-11-18 15:44:31 -07:00
Brian Warner
1e024d5e82
munin/tahoe_introstats: add line for distinct-storage-hosts (which counts machines instead of nodes)
2008-11-18 15:32:38 -07:00
Brian Warner
7d68e4397b
munin/tahoe_diskused: new plugin to show total disk space used across the grid
2008-11-18 01:25:25 -07:00
Brian Warner
3d03e5257c
munin/tahoe_disktotal: new plugin to show total disk space (used and unused) in the grid
2008-11-18 00:51:01 -07:00
Brian Warner
0a819e2912
diskwatcher: record total-space (the size of the disk as reported by df) in the db, report it to HTTP clients. This will involve a 50-item-per-second upgrade process when it is first used on old data
2008-11-17 21:45:16 -07:00
Zooko O'Whielacronx
eae5403e5a
shebang: replace "/usr/bin/python" with "/usr/bin/env python"
2008-11-04 18:03:06 -07:00
Zooko O'Whielacronx
24650490ed
misc/fixshebangs.py
2008-11-04 18:01:30 -07:00
Zooko O'Whielacronx
9f299ad55b
setup: include _pkgutil.py in setuptools bootstrap egg so that it will work on Python 2.4
2008-10-25 08:28:39 -07:00
Zooko O'Whielacronx
8f260826b4
setup: remove old bundled setuptools-0.6c9
2008-10-25 06:49:47 -07:00
Zooko O'Whielacronx
3c974763ab
setup: bundle an SVN snapshot of setuptools instead of the most recent stable release of setuptools
...
This SVN snapshot fixes a problem that prevents the setting up of nevow:
http://bugs.python.org/setuptools/issue20
2008-10-25 06:48:37 -07:00
Zooko O'Whielacronx
cd015228d5
setup: bundle setuptools-0.6c9
2008-09-30 13:04:48 -07:00
Zooko O'Whielacronx
8cad310dc8
setup: remove bundled setuptools-0.6c8
2008-09-30 13:03:36 -07:00
Brian Warner
f3d730428f
misc/incident-gatherer: add classify_tahoe.py: a foolscap incident-gatherer classification plugin
2008-10-15 15:09:40 -07:00
Brian Warner
06770f720e
misc/cpu-watcher.tac: use writeaside-and-rename for the history.pickle file
2008-09-30 17:30:53 -07:00
Brian Warner
3aba70bbcf
misc/spacetime: use async polling so we can add a 60-second timeout, add an index to the 'url' Axiom column for 2x speedup
2008-09-30 16:34:48 -07:00
Brian Warner
1a3f154c21
munin/tahoe_doomsday: oops, tolerate 'null' in the timeleft results, to unbreak the 2wk/4wk graphs
2008-09-30 13:20:51 -07:00
Brian Warner
1f9af40fd4
misc/make-canary-files.py: tool to create 'canary files', explained in the docstring
2008-09-24 17:47:16 -07:00
Brian Warner
d4b522bedf
misc/make_umid: change docs, make elisp code easier to grab
2008-09-20 11:39:33 -07:00
Brian Warner
cf27f81a88
misc/check-build.py: ignore the 'Downloading file:..' line that occurs for the setup_requires= -triggered handling of the setuptools egg
2008-09-16 22:17:25 -07:00
Brian Warner
1853020a5f
#249 : add 'test-desert-island', to assert that a tahoe-deps.tar.gz -enabled build does not download anything
2008-09-16 18:37:02 -07:00
Brian Warner
fafdf58e41
#249 : move dependent libs out of misc/dependencies/, get them from tahoe-deps.tar.gz instead
2008-09-16 18:25:45 -07:00
Brian Warner
7da70b8197
misc/xfer-client.py: small foolscap utility to transfer a file to a waiting server
2008-09-08 16:19:03 -07:00
Brian Warner
be11bfe812
munin: add tahoe_diskleft plugin, update spacetime/diskwatcher.tac to support it
2008-08-28 13:32:36 -07:00
Brian Warner
d47d19cbfb
debian: include misc/cpu-watcher.tac in the debian package
2008-08-27 15:30:26 -07:00
Brian Warner
482b3e0939
munin/tahoe_doomsday: change the graph title, 'time predictor' is more accurate than 'space predictor'
2008-08-27 14:30:13 -07:00
Brian Warner
ba6e525952
munin/tahoe_diskusage: clip the graph at zero, to prevent transient negative excursions (such as when a lot of old logfiles are deleted from a storage server's disk) from scaling the graph into unusability
2008-08-27 12:35:43 -07:00
Brian Warner
96a1ec33b8
misc/make_umid: little script and elisp fragment to insert umid= arguments
2008-08-25 18:59:18 -07:00
Brian Warner
743ca91bcd
munin diskusage/doomsday: oops, fix labels, everything was reported in the 1hr column
2008-08-11 13:34:31 -07:00
Brian Warner
8828fe8769
munin/tahoe_overhead: don't emit non-sensicial numbers
2008-08-07 14:40:08 -07:00
Brian Warner
f7ad0d2f6f
munin: add tahoe_overhead plugin, to measure effectiveness of GC and deleting data from inactive accounts
2008-08-07 13:39:25 -07:00
Brian Warner
39db60cc2b
diskwatcher.tac: include total-bytes-used
2008-08-07 13:12:14 -07:00
Brian Warner
1c36a52db5
misc/dependencies: remove the no-longer-useful foolscap-0.2.5 tarball
2008-08-07 11:45:46 -07:00
Brian Warner
f77af25ef7
diskwatcher.tac: don't report negative timeleft
2008-08-07 10:34:33 -07:00
Brian Warner
7502971df6
diskwatcher.tac: reduce the polling rate to once per hour
2008-08-06 23:20:21 -07:00
Brian Warner
e19564df59
misc/spacetime: add munin plugins, add everything to .deb
2008-08-06 23:00:03 -07:00
Brian Warner
7fbfbcbe4d
diskwatcher.tac: hush pyflakes
2008-08-06 22:04:27 -07:00
Brian Warner
730a28aa0d
diskwatcher.tac: add async-GET code, but leave it commented out: urlopen() seems to work better for now
2008-08-06 22:03:27 -07:00
Brian Warner
c90bc3ad71
cpu-watcher.tac: improve error message
2008-08-06 21:38:01 -07:00
Brian Warner
46d0c9c995
disk-watcher: first draft of a daemon to use the HTTP stats interface and its new storage_server.disk_avail feature, to track changes in disk space over time
2008-08-06 21:22:22 -07:00
Brian Warner
4863b17371
misc/cpu-watcher.tac: tolerate missing pidfiles, just skip over that sample
2008-08-06 21:17:05 -07:00
Brian Warner
027408b9d2
debian/control: update dependencies to match _auto_deps: foolscap-0.3.0, pycryptopp-0.5
2008-08-05 18:32:22 -07:00
Brian Warner
315a89f9d8
munin/tahoe_spacetime: show 2wk data even if 4wk data is unavailable
2008-07-28 12:42:33 -07:00
Zooko O'Whielacronx
497aedfab9
simulate_load: put it in friendnet mode and print out a sum of doubled shares
2008-07-21 14:11:06 -07:00
Brian Warner
870fb68620
munin/tahoe_server_latency: add percentile to graph title
2008-07-16 16:08:17 -07:00
Brian Warner
af0648c0f8
munin/serverops: improve graph title
2008-07-16 12:36:42 -07:00
Brian Warner
5b3868ae2e
debian/rules: put munin plugins in /usr/share/PACKAGENAME/munin/, make them +x, remove packagename from rules to make branches easier to manage
2008-07-15 18:47:41 -07:00
Brian Warner
43b54c1abb
sid/control: set python versions to 2.4,2.5 , to match feisty/control, since sid has had 2.5 available forever now
2008-07-15 18:42:38 -07:00
Brian Warner
340b7add4f
debian/changelog: remove all versions but 0.0.1, so debchange can always override it. The 'deb-X' Makefile targets should not be used; run 'make deb-X-head' instead
2008-07-15 15:23:41 -07:00
Brian Warner
390c5c6ca5
munin/tahoe_stats: add server operations per second (allocate/get/writev/readv
2008-07-14 13:25:35 -07:00
Brian Warner
0f231e2e66
munin/tahoe_stats: rearrange config a bit, no functional changes
2008-07-14 13:25:01 -07:00
Brian Warner
6525742b37
munin/tahoe_server_operations: fix comments
2008-07-14 13:18:47 -07:00
Brian Warner
a72dfcc00d
munin: add plugin for storage-server operations per second
2008-07-14 13:18:11 -07:00
Brian Warner
d14d6d0985
copy debian/sid changes to debian/feisty
2008-07-14 12:56:38 -07:00
Brian Warner
7eaead0fcc
debian/sid: add more docs to package, including munin plugins
2008-07-14 12:55:34 -07:00
Brian Warner
b2f062ab82
misc/munin: add server_latency plugin
2008-07-14 12:29:19 -07:00
Zooko O'Whielacronx
0e77fe1e92
misc/simulate_load.py: fix bug in which only K shares were uploaded instead of N, and fix the x axis in place so that successive runs can be compared
2008-07-12 16:54:29 -07:00
Zooko O'Whielacronx
f969c5f80e
misc/simulate_load.py: use a more realistic distribution of file sizes, and automate the formatting of the ascii art so that it fits in 70 columns and the number of files uploaded is properly marked
2008-07-12 15:37:27 -07:00
Zooko O'Whielacronx
b483c4710d
add misc/simulate_load.py in an attempt to understand why permuting the peerlist per each storage index matters
2008-07-12 14:26:22 -07:00
Brian Warner
f303670265
misc/munin: rename plugins to make munin happy: use underscores rather than hyphens, remove .py suffix
2008-07-11 15:21:13 -07:00
Brian Warner
503ce1cbd7
remove obsolete tahoe-sharesperfile munin plugin
2008-07-11 15:17:33 -07:00
Brian Warner
38252b6df1
remove tahoe-dirnodes.py munin plugin, it is way obsolete
2008-07-11 15:14:28 -07:00
Zooko O'Whielacronx
9bd1cdcdd8
setup: remove bundled darcsver-1.1.2.tar
2008-06-21 14:04:26 -07:00
Zooko O'Whielacronx
c151c13b5e
setup: bundle darcsver-1.1.5 into misc/dependencies
...
Once we implement #249 (move bundled dependencies out of revision control history and make them optional) then we can stop doing this. Until then we try not to do it unless we really need to...
2008-06-21 13:56:24 -07:00
Zooko O'Whielacronx
52a5651da2
setup: remove Nevow 0.9.18 tarball from misc/dependencies
...
The 0.9.18 version of Nevow doesn't declare its dependency on Twisted in a machine-readable way ( http://divmod.org/trac/ticket/2629 ). Neither does the current release of Nevow (0.9.31), but hopefully a future release of Nevow in the near future will fix this.
Also, we're going to be managing external dependencies like this in a separate darcs repository in the future instead of checking them into our Tahoe source tree.
2008-06-09 11:41:56 -07:00
Zooko O'Whielacronx
b06ba64214
setup: quote the results of misc/find_trial.py so that they can be passed through a shell even if they contain spaces
2008-06-05 15:59:45 -07:00
Brian Warner
c23f675095
misc/munin/tahoe_spacetime.py: add a munin plugin to estimate how much time remains before you run out of space
2008-05-28 11:33:54 -07:00
Brian Warner
15b373ad9c
misc/delete-old-helper.sh: simple script to delete old+abandoned helper files
2008-05-27 21:17:20 -07:00
Brian Warner
74df0b58b5
munin: add tahoe_cpu_watcher.py, to track the data from misc/cpu-watcher.tac
2008-05-07 13:19:08 -07:00
Brian Warner
79147a3f6a
misc/cpu-watcher*: add some tools to monitor CPU usage of arbitrary processes, like tahoe nodes
2008-05-07 12:34:29 -07:00
Zooko O'Whielacronx
b1583ac6a0
setup: remove bundled pycryptopp-0.3.0.tar
2008-05-06 08:24:46 -07:00
Zooko O'Whielacronx
045e2bcdf4
setup: bundle pycryptopp-0.5.1.tar
...
pycryptopp-0.3.0 incorrectly crypts AES (depending on compiler and version of Crypto++ used)
Very soon now we're going to set up an "ext" repository to hold all tarballs and no longer check them into our trunk source tree.
2008-05-06 08:23:36 -07:00
Brian Warner
773e9abe24
debian: add python-setuptools to the debian install-time dependencies. Should close #382 .
2008-04-24 16:01:04 -07:00
Brian Warner
d2a2b8a4ae
munin/tahoe_estimate_files.py: tool to estimate the total number of slots (mutable and immutable combined) in the grid, from a small sample
2008-04-24 11:28:35 -07:00
robk-tahoe
5f5ed6b08d
munin stats: fix typo in mutable file stats
2008-04-23 15:50:29 -07:00
robk-tahoe
f74042c75e
munin stats: add mutable/uploader traffic graphs
...
this adds munin graphs to present data already published by nodes to
the stats_gatherer, namely mutable files published/retrieved, and
immutable files uploaded, and the bytes thereof
2008-04-23 14:39:41 -07:00
Brian Warner
239c950810
munin/tahoe-stats.py: category is 'counters' not 'stats', use DERIVE
2008-04-23 14:03:23 -07:00
Zooko O'Whielacronx
1e2698713b
setup: remove the misc/hatch-eggs.py script, which we no longer use since [1937]
2008-04-23 13:41:56 -07:00
Zooko O'Whielacronx
ece902a4cf
setup: windows: make sub-ver.py treat None as 0
2008-04-21 06:37:13 -07:00
Brian Warner
5afc26f1d1
misc/sizes.py: update to match current defaults: 128KiB segsize, 3-of-10
2008-04-16 16:29:53 -07:00
Brian Warner
77c3c616ab
misc/boodlegrid.tac: tweak sounds a bit
2008-04-15 11:01:49 -07:00
Brian Warner
500934b72f
helper stats: fix the /helper_status page, the recent conflict merging missed some uses. Added tests, updated the munin plugins to match
2008-04-14 13:18:53 -07:00
robk-tahoe
766deaa9b6
stats_gatherer: reconcile helper stats gathering
...
I'd implemented stats gathering hooks in the helper a while back.
Brian did the same without reference to my changes. This reconciles
those two changes, encompassing all the stats in both changes,
implemented through the stats_provider interface.
this also provide templates for all 10 helper graphs in the
tahoe-stats munin plugin.
2008-04-10 17:25:44 -07:00
robk-tahoe
d8f3521a02
munin: added a series of munin graphs to report upload helper state
2008-03-25 18:30:46 -07:00
Brian Warner
623d82a130
debian: use our own /usr/bin/tahoe, remove runtime dependency on setuptools (since it required egg-aware versions of all dependencies too)
2008-04-10 16:29:59 -07:00
Brian Warner
5cfccbf44f
debian: we now require setuptools at build time, and that or pkg_resources at runtime
2008-04-10 15:43:56 -07:00
Brian Warner
3f61198042
debian: use setuptools-generated support/bin/tahoe instead of bin/tahoe, to match Zooko's change that makes our in-tree bin/tahoe spawn support/bin/tahoe
2008-04-10 14:36:27 -07:00
Brian Warner
d36d731752
oops, change debian dependency on zfec to 1.1, not 1.1.0
2008-04-09 17:45:52 -07:00
Brian Warner
5385ec8fc9
make debian dependencies match _auto_deps.py ones, for foolscap and zfec
2008-04-09 10:23:01 -07:00
Brian Warner
ec77a5dc92
update our misc/dependencies to have foolscap-0.2.5
2008-04-03 16:08:22 -07:00
Brian Warner
fb51a60993
munin plugins: add 'graph_category tahoe'
2008-03-28 16:18:30 -07:00
Brian Warner
5da9f42cf8
add munin/tahoe-rootdir-space
2008-03-28 16:18:09 -07:00
Brian Warner
da6ab9f522
munin/tahoe-introstats.py: put this graph in the 'tahoe' category
2008-03-28 14:51:37 -07:00
Brian Warner
6b416fc28c
helper: add another munin plugin
2008-03-27 16:50:30 -07:00
Brian Warner
2c96a32633
helper: add more stats to webapi, at /helper_status
2008-03-27 16:46:08 -07:00
Brian Warner
1ad847d4b2
boodlegrid.tac: update sound samples
2008-03-26 18:22:07 -07:00
Brian Warner
ccce37bb4a
misc/boodlegrid.tac: tool to monitor a grid through its flogports
2008-03-26 16:09:34 -07:00
Zooko O'Whielacronx
5aa7986497
setup: remove bundled setuptools-0.6c7
2008-03-26 12:13:21 -07:00
Zooko O'Whielacronx
3e4b3f49b7
setup: bundle setuptools-0.6c8, we need a bugfix in it
2008-03-26 12:12:34 -07:00
Zooko O'Whielacronx
39272a1b6d
setup: simplify makefile's path manipulation now that we rely on setup.py develop
2008-03-26 10:00:33 -07:00
zandr
f4d51ea366
tahoe-nodememory.py: change category to Tahoe
2008-03-25 18:42:11 -07:00
zandr
5245140a1c
Copy amd-nodememory munin plugin over to tahoe and point at correct pidfile
2008-03-25 17:50:04 -07:00
Brian Warner
490f815e8f
add a munin plugin to display introducer stats
2008-03-25 13:15:52 -07:00
Zooko O'Whielacronx
ec3f140824
setup: use "-rNN" with NN the "revision number" instead of "-NN" with NN the "build number" in the Windows builds
2008-03-10 20:15:04 -07:00
Zooko O'Whielacronx
d39d1661bd
setup: remove old bundled pyutil-1.3.12
2008-03-10 19:59:30 -07:00
Zooko O'Whielacronx
45886cbb95
setup: remove old bundled darcsver-1.1.1
2008-03-10 19:59:20 -07:00
Zooko O'Whielacronx
7cc756eb45
setup: bundle pyutil-1.3.16.tar
2008-03-10 19:58:18 -07:00
Zooko O'Whielacronx
7bfb3ac047
setup: bundle darcsver-1.1.2.tar
2008-03-10 19:56:47 -07:00
Zooko O'Whielacronx
5108a73c6f
setup: remove old bundled darcsver-1.0.1.tar
2008-03-10 19:23:34 -07:00
Zooko O'Whielacronx
ff129d648d
setup: bundle darcsver-1.1.1.tar
...
I forgot that we're doing uncompressed misc/deps at the moment. We don't think this is a long-term good approach...
2008-03-10 19:21:16 -07:00
Brian Warner
41045fba0f
'tahoe catalog-shares': add SDMF filesize to the output, update misc/find-share-anomalies.py to match
2008-02-13 15:12:06 -07:00
Zooko O'Whielacronx
98f570c3ee
setup: remove older bundled pycryptopp-0.2.10
2008-02-13 15:23:15 -07:00
Zooko O'Whielacronx
fa92672e24
setup: bundle pycryptopp-0.3.0
2008-02-13 14:57:15 -07:00
Zooko O'Whielacronx
2e84230a18
setup: remove older bundled version of setuptools_darcs
2008-02-13 08:03:40 -07:00
Zooko O'Whielacronx
70b573434b
setup: remove another older bundled setuptools
2008-02-13 08:01:40 -07:00
Zooko O'Whielacronx
99f609d55b
setup: remove older bundled setuptools
2008-02-13 07:59:24 -07:00
Zooko O'Whielacronx
9eb2c7f5aa
setup: remove older bundled version of darcsver
2008-02-13 07:57:22 -07:00
Brian Warner
95f27bb8fb
misc/find-share-anomalies.py: tool to analyze 'tahoe catalog-shares' output and look for problems
2008-02-11 19:53:36 -07:00
Brian Warner
8ae6795e91
bundle foolscap-0.2.4 instead of 0.2.3
2008-02-05 15:27:41 -07:00
Brian Warner
7c7375a5ad
bump foolscap dependency to 0.2.4, since we now use log.err
2008-02-05 15:27:14 -07:00
robk-tahoe
3641b7924b
munin stats: don't suppress series with no current data
...
having changed tahoe-stats to not report data series if there was no recent
data recorded for a node, I wound up making it hide the data series. this
change causes it to report all data series for which stats exist in the
'config' phase, so that they show up, but only report actual data if the
stats are recent, so that they show up as missing if the node is not
reporting stats currently
2008-02-01 18:44:57 -07:00
robk-tahoe
b80cfeb186
munin stats: suppress output of data more that 5min old
...
if a node fails to report stats, the natural thing to do in re munin is to
supress the data for that data series. the previous tahoe-stats would output
whatever data was present in the stats_gatherer's stats.pickle, regardless of
how old.
this change means that if the gatherer hasn't received data within the last
5 min, then no data is reported to munin for that node.
2008-01-31 21:04:23 -07:00
robk-tahoe
e910a64e7d
munin stats: change truncated nodeid to 4 char (not 8)
2008-01-31 19:21:17 -07:00
Brian Warner
bf25a041f3
tarballs: stick with setup.py's sdist command, which gives us bz2/gz/zip but not rz/7z/lrz. Closes #298 .
2008-01-31 13:52:48 -07:00
Brian Warner
4a363f0281
add 'tarballs' target, to generate compressed source tarballs
2008-01-30 20:45:14 -07:00
robk-tahoe
7b9f3207d0
stats: add a simple stats gathering system
...
We have a desire to collect runtime statistics from multiple nodes primarily
for server monitoring purposes. This implements a simple implementation of
such a system, as a skeleton to build more sophistication upon.
Each client now looks for a 'stats_gatherer.furl' config file. If it has
been configured to use a stats gatherer, then it instantiates internally
a StatsProvider. This is a central place for code which wishes to offer
stats up for monitoring to report them to, either by calling
stats_provider.count('stat.name', value) to increment a counter, or by
registering a class as a stats producer with sp.register_producer(obj).
The StatsProvider connects to the StatsGatherer server and provides its
provider upon startup. The StatsGatherer is then responsible for polling
the attached providers periodically to retrieve the data provided.
The provider queries each registered producer when the gatherer queries
the provider. Both the internal 'counters' and the queried 'stats' are
then reported to the gatherer.
This provides a simple gatherer app, (c.f. make stats-gatherer-run)
which prints its furl and listens for incoming connections. Once a
minute, the gatherer polls all connected providers, and writes the
retrieved data into a pickle file.
Also included is a munin plugin which knows how to read the gatherer's
stats.pickle and output data munin can interpret. this plugin,
tahoe-stats.py can be symlinked as multiple different names within
munin's 'plugins' directory, and inspects argv to determine which
data to display, doing a lookup in a table within that file.
It looks in the environment for 'statsfile' to determine the path to
the gatherer's stats.pickle. An example plugins-conf.d file is
provided.
2008-01-30 20:11:07 -07:00
Zooko O'Whielacronx
284c1652a9
setup: make find_trial self-contained so that we don't have a bootstrapping problem -- if allmydata can't be imported we still want to be able to run find_trial
2008-01-23 10:04:26 -07:00
Zooko O'Whielacronx
faad785859
setup: include cli.exe in the bootstrap setptools egg so that it will work on Windows (also include gui.exe just in case)
2008-01-22 12:33:55 -07:00
Zooko O'Whielacronx
b1152860cd
setup: bundle setuptools_darcs-1.1.7
...
fixes #263
2008-01-22 11:01:36 -07:00
Zooko O'Whielacronx
451ca7792e
setup: add a setuptools bootstrap egg that works on all versions of Python
...
For versions of Python >= 2.3.
2008-01-22 11:00:12 -07:00
Zooko O'Whielacronx
1f2e3fc912
setup: simplify the setup by removing the "tahoe dependencies" fake project
...
Now we use "./setup.py develop" to ensure that changes to our source code are immediately used without requiring a "make" step. This simplification will hopefully pave the way for easier py2exe and py2app, solving the "Unit tests test the installed version" bug (#145 ), and perhaps also #164 and #176 .
This patch also conditionalizes the use of setuptools_darcs on the absence of a PKG-INFO file, which is part of fixing #263 .
2008-01-22 08:35:38 -07:00
robk-tahoe
3cb08209d2
windows installer build refinements
...
this resolves problems of py2exe's modulefinder collection of sources from
.zipped egg files, not by using easy_install to reach the --always-unzip
option, but rather with a small tool which unpacks any zipped egg files found
in misc/dependencies. this fixes the py2exe build given rollback of the
easy_install stuff which had broken the unix builds. misc/hatch-eggs.py
performs the honours.
this also includes a misc/sub-ver.py tool which substitutes elements of the
verion number for the current code base (by importing allmydata.__version__
hence make-version should be run first, and the python path carefully managed)
into template files using python's string interpolation of named args from a
dict as the templating syntax. i.e. %(major)d %(minor)d %(point)d %(nano)d
each expand to the individual components of the version number as codified
by the pyutil.version_class.Version class. there is also a %(build)s tag
which expands to the string form of the whole version number. This tool is
used to interpolate the automatically generated version information into the
innosetup source file in a form consistent with innosetup/windows' restrictions
2008-01-14 17:53:54 -07:00
Zooko O'Whielacronx
88c93469ff
setup: add darcsver-1.0.1.tar to misc/dependencies/
2008-01-16 14:08:26 -07:00
Brian Warner
16e2b3307a
build-deps-setup.py: import twisted early, to make setuptools on dapper use the right version
2008-01-10 20:15:02 -07:00
robk-tahoe
2585908296
tweaks to build process to support py2exe
...
py2exe is unable to handle .eggs which are packaged as zip files
in preference it will pull in other versions of libraries if they
can be found in the environment.
this changes causes .eggs to be built as .egg directories, which
py2exe can handle.
2008-01-09 19:02:53 -07:00
Zooko O'Whielacronx
80adf18fcc
setup: remove hard import of ez_setup -- we can proceed even if ez_setup can't be imported
2008-01-10 14:01:52 -07:00
Zooko O'Whielacronx
72aa8f5cb8
setup: shebang usr bin env python
2008-01-10 14:01:31 -07:00
Zooko O'Whielacronx
1ab03fe9d2
setup: update in-line doc of get-version.py
2008-01-08 10:51:01 -07:00
Brian Warner
813706a01e
debian: update dependencies to match calc-deps.py, mainly pycryptopp-0.2.8
2008-01-02 15:14:34 -07:00
Zooko O'Whielacronx
b412847485
setup: whoops, fix the use of source-tree-root-dir eggs in our Makefile
2008-01-01 01:53:31 -07:00
Zooko O'Whielacronx
97f2953cec
setup: fix bin/tahoe to include .egg's from the source tree root dir as well
...
This is necessary, as we can't prevent setuptools from respecting any such eggs, therefore we need to respect them in order to maintain consistency. However, we don't normally install any "install_requires" eggs into the source tree root dir.
2008-01-01 01:51:28 -07:00
Zooko O'Whielacronx
9b38eebcaf
setup: finish renaming of docs/testnet to docs/testgrid
2007-12-31 23:36:59 -07:00
Zooko O'Whielacronx
58bedc60a2
setup: add darcsver source tarball to misc/dependencies
...
In the future, we're likely to stop keeping misc/dependencies under revision control, as per
http://allmydata.org/trac/tahoe/ticket/249
2007-12-31 23:26:51 -07:00
Brian Warner
bd187dfc50
upgrade to foolscap-0.2.3
2007-12-24 17:23:27 -07:00
Zooko O'Whielacronx
bfc69178b2
setup: fix bug in bugfix to patch to include .egg's found in CWD
2007-12-22 11:19:52 -07:00
Zooko O'Whielacronx
2ee424b3a3
setup: fix bug in previous patch to include .egg's from CWD
2007-12-22 11:14:27 -07:00
Zooko O'Whielacronx
7fdcbddbbc
setup: we also need to include .egg's in the CWD in our search path, because if we install a 3rd party library into support/, and *it* installs a library that *it* requires, that one will appear in CWD
...
It would be nice to figure out a way to force them to all appear in support/ where they belong.
2007-12-22 11:04:24 -07:00
Zooko O'Whielacronx
27a0d59308
setup: zope.interface bundle .tar instead of .tar.gz because multiple bundled .tar's compress together much more nicely than multiple bundled .tar.gz's do. :-)
2007-12-22 10:50:54 -07:00
Zooko O'Whielacronx
48b13dcab8
setup: upgrade bundle zfec-1.1.tar.gz to zfec-1.3.4.tar
2007-12-22 10:50:39 -07:00
Zooko O'Whielacronx
a9545aa4b0
setup: simplejson bundle .tar instead of .tar.gz because multiple bundled .tar's compress together much more nicely than multiple bundled .tar.gz's do. :-)
2007-12-22 10:50:29 -07:00
Zooko O'Whielacronx
840fb23229
setup: upgrade bundle from setuptools_darcs-1.0.6.tar.gz to setuptools_darcs-1.1.5.tar
2007-12-22 10:50:03 -07:00
Zooko O'Whielacronx
d5bb1ac316
setup: bundle pyutil-1.3.12.tar
2007-12-22 10:49:47 -07:00
Zooko O'Whielacronx
336c482561
setup: pycryptopp: bundle .tar instead of .tar.gz because multiple bundled .tar's compress together much more nicely than multiple bundled .tar.gz's do. :-)
2007-12-22 10:49:35 -07:00
Zooko O'Whielacronx
2add4f5edb
setup: foolscap bundle .tar instead of .tar.gz because multiple bundled .tar's compress together much more nicely than multiple bundled .tar.gz's do. :-)
2007-12-22 10:49:20 -07:00
Zooko O'Whielacronx
fd4880face
setup: bundle argparse-0.8.0.tar
...
It isn't actually needed for Tahoe, only for the command-line tools from pyutil. Later I will make an "extras" category within pyutil to specify in a machine-readable way that argparse is not required for pyutil unless you want the command-line tools.
2007-12-22 10:47:54 -07:00
Zooko O'Whielacronx
70b84e9298
setup: Nevow bundle .tar instead of .tar.gz because multiple bundled .tar's compress together much more nicely than multiple bundled .tar.gz's do. :-)
2007-12-22 10:47:06 -07:00
Zooko O'Whielacronx
c3a265839c
setup: refactor ez_setup.py and setup.py to satisfy the Desert Island scenario, to find and use setuptools egg in-place in misc/dependencies, and make it setup_require pyutil (for darcsver)
2007-12-22 10:44:47 -07:00
Zooko O'Whielacronx
922234ec29
setup: bundle setuptools_darcs plugin in misc/dependencies
2007-12-12 19:20:42 -07:00
Zooko O'Whielacronx
cbd6881a47
setup: remove misc/make-version.py and invoke "darcsver" from the pyutil library
...
misc/make-version.py has a limitation which prevents it from generating version
stamps from our current darcs history. This limitation has been fixed in
pyutil's "darcsver". Rather than copy the fix from there to
misc/make-version.py, I'm making it so that you have to install pyutil if you
want to automatically generate _version.py files from the current darcs
history.
2007-12-20 18:17:55 -07:00
Brian Warner
e8c55fe85d
move to foolscap-0.2.2
2007-12-12 20:21:45 -07:00
Zooko O'Whielacronx
1ac09840a4
a few documentation and naming convention updates
...
Notable: the argument to make REPORTER has been renamed to TRIALARGS.
2007-12-12 19:34:08 -07:00
Brian Warner
48e7a65882
upgrade to foolscap-0.2.1, with a new logging framework
2007-12-10 18:35:08 -07:00
Brian Warner
0618cf792d
build-deps: require setuptools 0.6c4 or later, because older ones don't allow foolscap to use os.random at import time
2007-12-10 20:06:59 -07:00
Brian Warner
725621aded
debian: add Depends: on python-pycryptopp, now that it's been packaged
2007-11-19 13:04:50 -07:00
Brian Warner
c86e803282
logtool: add 'gather' and 'dump' modes
2007-11-18 18:32:04 -07:00
Brian Warner
b502c3fea1
logtool: rename get-logs.py to logtool.py
2007-11-18 18:30:55 -07:00
Brian Warner
b29ce1c30a
logpublisher: implement subscribe/publish for log, add a sample client
2007-11-16 20:07:50 -07:00
Zooko O'Whielacronx
59d6c3c822
decentralized directories: integration and testing
...
* use new decentralized directories everywhere instead of old centralized directories
* provide UI to them through the web server
* provide UI to them through the CLI
* update unit tests to simulate decentralized mutable directories in order to test other components that rely on them
* remove the notion of a "vdrive server" and a client thereof
* remove the notion of a "public vdrive", which was a directory that was centrally published/subscribed automatically by the tahoe node (you can accomplish this manually by making a directory and posting the URL to it on your web site, for example)
* add a notion of "wait_for_numpeers" when you need to publish data to peers, which is how many peers should be attached before you start. The default is 1.
* add __repr__ for filesystem nodes (note: these reprs contain a few bits of the secret key!)
* fix a few bugs where we used to equate "mutable" with "not read-only". Nowadays all directories are mutable, but some might be read-only (to you).
* fix a few bugs where code wasn't aware of the new general-purpose metadata dict the comes with each filesystem edge
* sundry fixes to unit tests to adjust to the new directories, e.g. don't assume that every share on disk belongs to a chk file.
2007-12-03 14:52:42 -07:00
Zooko O'Whielacronx
211bf154d0
setup: pycryptopp: upgrade from v0.2.7 to v0.2.8
2007-11-19 15:55:17 -07:00
Zooko O'Whielacronx
cadc14fbca
setup: pycryptopp: upgrade from v0.2.6 to v0.2.7 (adds AES)
2007-11-19 15:07:11 -07:00
Zooko O'Whielacronx
a82954eced
setup: upgrade bundled pycryptopp from v0.2.5 to v0.2.6
2007-11-09 18:14:36 -07:00
Zooko O'Whielacronx
f0520b4e9e
setup: upgrade the bundled pycryptopp tarball from pycryptopp v0.2.3 to pycryptopp v0.2.5
2007-11-09 13:02:49 -07:00
Zooko O'Whielacronx
494e07eced
setup: upgrade to pycryptopp v0.2.3
2007-11-07 17:54:46 -07:00
Zooko O'Whielacronx
b699c28b67
setup: add misc/dependencies/pycryptopp-0.2.1.tar.gz
2007-11-07 09:59:51 -07:00
Brian Warner
1e8a693512
Makefile: add 'find-trailing-spaces' tool and target
2007-11-05 21:32:08 -07:00
Brian Warner
475a58fcc5
munin: add a sample file for /etc/munin/plugin-conf.d/
2007-10-22 15:08:39 -07:00
Brian Warner
0aba63ff17
munin: oops, fix that trailing-newline fix, the previous one mangled the config
2007-10-18 12:49:22 -07:00
Brian Warner
b339c76560
munin: remove extra blank line from config output
2007-10-18 12:39:26 -07:00
Brian Warner
33a02e19f1
munin: update shareholder plugins to match current storage layout
2007-10-18 12:08:58 -07:00
Zooko O'Whielacronx
bf3f629a68
setup: use the new find_exe module to find trial
2007-10-15 11:52:26 -07:00
Zooko O'Whielacronx
aba57d9a4c
setup: upgrade zfec from 1.0.4 to 1.1
2007-10-11 16:27:52 -07:00
Brian Warner
002472587e
add public testnet .furls to docs/testnet/, and copy into .deb . Closes #157 .
2007-10-11 14:55:23 -07:00
Brian Warner
1c7f52265f
deb: add docs/* to the debian package
2007-10-11 14:37:29 -07:00
Brian Warner
dc573554c8
rename bin/allmydata-tahoe to bin/tahoe. Closes #155 .
2007-10-11 03:38:24 -07:00
Zooko O'Whielacronx
be51a7e2a4
setup: bundle simplejson-1.7.3 in misc/dependencies
2007-10-04 11:23:20 -07:00
Brian Warner
7626a95636
upgrade to foolscap-0.1.7
2007-09-26 18:24:51 -07:00
Brian Warner
3b494ddf37
debian: oops, set debian/changelog version to 0.6.0-1+, to be less than everything the buildbot is creating
2007-09-26 18:15:13 -07:00
Brian Warner
7a4a31a83f
debian: put a version string of '0.6.0+' in the debian changelog
2007-09-26 17:58:42 -07:00
Brian Warner
caa0a95405
debian/sid: copy rules from feisty, the same can be used for both
2007-09-26 17:55:24 -07:00
Brian Warner
d41f0f9492
debian/feisty: use our original bin/allmydata-tahoe instead of setuptools's
...
because the setuptools "entry points" form asserts that there are
setuptools-visible packages like nevow/zope.interface (i.e. they have .egg-info
metadata). Until very recently, most debian systems did not install this
metadata. Instead, we rely upon the usual debian dependency checking as
expressed in debian/control .
2007-09-26 17:52:30 -07:00
Brian Warner
d1e07daeee
make-version.py: when _darcs doesn't exist, make the warning less scary-looking
2007-09-24 12:38:37 -07:00
Zooko O'Whielacronx
8f1b6a3dae
setup: add zope.interface-3.4.0 in place of zope.interface-3.3.0
2007-09-20 13:02:36 -07:00
Zooko O'Whielacronx
d0c2153c34
add zope.interface-3.3.0.tar.gz to misc/dependencies
2007-09-20 10:47:38 -07:00
Zooko O'Whielacronx
3e43889ad4
replace zfec-1.0.3 with zfec-1.0.4 in misc/dependencies
2007-09-20 10:47:21 -07:00
Brian Warner
1545190b69
hush some pyflakes warnings
2007-09-15 15:07:21 -07:00
Brian Warner
1f035a8f1f
setup.py: factor out dependency stuff, add workaround for nevow-0.6.0-on-dapper problem
2007-09-15 15:05:35 -07:00
Brian Warner
d28d7e91f9
include the Nevow-0.9.18 tarball in our distribution
2007-09-15 14:18:36 -07:00
Brian Warner
5cc1dbaac9
setup.py: add Nevow to our dependency list
2007-09-15 14:18:07 -07:00
Brian Warner
5cedf89a67
build-deps-setup.py: provide dependency_links too
2007-09-15 14:04:57 -07:00
Brian Warner
7bb7d76d9d
build-deps-setup.py: use ez_setup here too, for platforms without setuptools
2007-09-15 02:31:40 -07:00
Brian Warner
a4b00a6b2c
Makefile build-deps: use a fake package named 'tahoe-deps', to avoid a double-build of tahoe
2007-09-14 19:23:55 -07:00
Brian Warner
4772bee01b
remove simplejson tarball from our dependencies/ directory
2007-09-13 20:14:08 -07:00
Zooko O'Whielacronx
771485dc94
re-port Makefile to Windows and remove a bunch of no-longer-needed hacks
2007-09-13 18:20:35 -07:00
Zooko O'Whielacronx
5cbeaf84b6
add simplejson into misc/dependencies
2007-09-13 16:42:38 -07:00
Zooko O'Whielacronx
2eec5598de
add zfec tarball to misc/dependencies
2007-09-13 14:51:04 -07:00
Zooko O'Whielacronx
3032cb65df
add setuptools eggs for py2.4 and py2.5 to misc/dependencies
2007-09-13 14:50:43 -07:00
Zooko O'Whielacronx
0ec576aa9f
add foolscap tarball to misc/dependencies
2007-09-13 14:50:23 -07:00
Brian Warner
b75276af00
packaging: add 'build-deps' target, to automatically build and install (locally, in ./support) necessary dependencies. All such installed files are used during tests.
2007-09-12 16:48:45 -07:00
Brian Warner
4bbc423d70
debian: depend upon python-zfec, stop including zfec in the tahoe .deb
2007-08-21 13:54:51 -07:00
Brian Warner
bd49bafb23
munin/tahoe-sharesperfile: don't truncate to an int, emit a float instead
2007-08-23 16:31:15 -07:00
Brian Warner
b7703df6a5
remove PyCrypto, copy AES/SHA256/Util.number into the allmydata/ tree
2007-08-14 13:57:41 -07:00
Brian Warner
a45c4492b5
misc/get-version.py: get the version string from the new location, _version.py
2007-08-16 15:43:43 -07:00
Zooko O'Whielacronx
9ad02dbe80
fix version class to preferred format and correct parsing
2007-08-16 15:38:01 -07:00
Zooko O'Whielacronx
094b687d6e
import version class and make-version script from pyutil -- fixes win32 build, improves error handling, and eliminates unused features
2007-08-16 14:09:30 -07:00
Brian Warner
b25446d514
munin/tahoe-storagespace.py: use /bin/du instead of our python version, since
...
it a) is faster and b) will include filesystem overhead that is a real cost.
2007-08-09 18:26:57 -07:00
Zooko O'Whielacronx
63d7e126ff
misc/sizes.py: rename "block" to "share" and "subblock" to "block"
...
This renaming is already in place in all of our other documentation and code, but misc/sizes.py got overlooked.
2007-08-08 22:31:12 -07:00
Brian Warner
e2e3b19a37
misc/storage-overhead.py: tool to estimate storage-space overhead per filesize
2007-07-16 13:43:31 -07:00
Brian Warner
032dfc5e58
misc/figleaf.excludes: exclude simplejson too, since we don't run their tests
2007-07-12 16:26:25 -07:00
Brian Warner
b2f13d2635
cease producing .debs for dapper, since they'd depend upon a library (simplejson) that isn't packaged for dapper. Feisty++ are fine. Dapper users are encouraged to build from source.
2007-07-11 14:31:04 -07:00
Brian Warner
4a081cec26
debian: make all .debs dependent upon python-simplejson. We must create the dapper package, the others are available upstream
2007-07-11 14:01:56 -07:00
Brian Warner
11792759e6
figleaf.excludes: ignore allmydata.Crypto, since so little of it gets used by allmydata.test
2007-07-07 23:00:09 -07:00
Brian Warner
43c58b8c21
move debian files down into misc/
2007-07-05 14:49:09 -07:00
Brian Warner
cb890cb9e4
munin plugins: fix labels
2007-07-05 14:38:31 -07:00
Brian Warner
8b2f0ef44c
add some munin plugins
2007-07-05 13:38:15 -07:00
Brian Warner
2cf7cfbe4a
misc/count_dirs.py: dev tool to estimate filetree space consumption
2007-06-16 21:55:13 -07:00
Zooko O'Whielacronx
cc87bdf8c3
make-version.py invokes darcs as "darcs" instead of "realdarcs"
...
Some other people might use the official Windows build of darcs, and people who use my cygwin wrapper for darcs will be compatible with this patch as long as they use the latest version of the wrapper.
2007-06-29 14:22:50 -07:00
Brian Warner
36ae9b5dd9
make-version.py: it is sys.platform, not os.platform
2007-06-12 18:06:10 -07:00
robk-org
6f9bc9b647
make make-version.py exec darcs correctly on windows
2007-06-12 17:56:55 -07:00
Zooko O'Whielacronx
f619f9ab30
shebang usr bin env python
2007-06-12 18:55:25 -07:00
Zooko O'Whielacronx
c3cdb81c65
catch EnvironmentError from attempt to invoke darcs using subprocess module
2007-06-11 11:59:57 -07:00
Brian Warner
319516a5a2
makefile: add a test-darcs-boringfile target
2007-05-23 17:51:28 -07:00
Brian Warner
c946a5f40c
make-version.py: remove the strip-until-<changelog> workaround
2007-05-16 12:02:58 -07:00
Brian Warner
b90d4a243e
make-version.py: use 'subprocess' module instead of 'commands', to improve windows compatibility
2007-05-16 11:57:20 -07:00
Brian Warner
985925d70c
make-version.py: accomodate windows by ignoring the junk that appears at the start of 'darcs changes'
2007-05-15 17:10:42 -07:00
Brian Warner
24d4e072dd
do not include foolscap or zfec coverage in the figleaf reports for the moment: they have their own test suites
2007-05-04 17:00:18 -07:00
Brian Warner
3f6b660272
update version-number handling, pull release tags from darcs history
2007-05-03 20:14:07 -07:00
Zooko O'Whielacronx
d19d1058e0
extirpate all references the "queen" and "metatracker"
...
This is a potentially disruptive and potentially ugly change to the code base,
because I renamed the object that serves in both roles from "Queen" to
"IntroducerAndVdrive", which is a bit of an ugly name.
However, I think that clarity is important enough in this release to make this
change. All unit tests pass. I'm now darcs recording this patch in order to
pull it to other machines for more testing.
2007-04-30 09:57:52 -07:00
Brian Warner
1a6da72861
don't include test code itself in the test-coverage numbers
2007-04-16 12:32:45 -07:00
Brian Warner
3687b37c60
misc cleanup: remove old .tac files, move old stuff into misc/
2007-04-06 20:37:38 -07:00
Zooko O'Whielacronx
4b4f5bbcba
change #!/usr/bin/python to #!/usr/bin/env python
...
Note that using "whatever version of python the name 'python' maps to in the current shell environment" is more error-prone that specifying which python you mean, such as by executing "/usr/bin/python setup.py" instead of executing "./setup.py". When you build tahoe (by running "make") it will make a copy of bin/allmydata-tahoe in instdir/bin/allmydata-tahoe with the shebang line rewritten to execute the specific version of python that was used when building instead of to execute "/usr/bin/env python".
However, it seems better that the default for lazy people be "whatever 'python' means currently" instead of "whatever 'python' meant to the manufacturer of your operating system".
2007-03-29 14:01:28 -07:00
Brian Warner
228e17560a
control.py: fix get_memory_usage, add a sample client tool
2007-03-07 20:31:49 -07:00
Brian Warner
f0b7cf4bdf
figleaf.el: add code to auto-enable the annotation mode for all source files
2007-01-18 01:00:11 -07:00
Brian Warner
aefe54574f
figleaf.el: when converting to the elisp-readable format, use our in-tree version of figleaf rather than one found on the system. Also change the keybinding to toggle annotations to C-cC-a, which is easier to type
2007-01-08 21:29:03 -07:00
Brian Warner
322b6738f0
figleaf.el: add some emacs22 comments
2007-01-07 13:10:15 -07:00
Brian Warner
6d1ba16ef3
figleaf.el: announce annotation/deannotation
2007-01-04 21:52:40 -07:00
Brian Warner
0fd032956d
more figleaf emacs work: enable the minor mode, then type C-cA to toggle annotations
2007-01-04 01:25:36 -07:00
Brian Warner
e8ee3365ef
move figleaf2html and figleaf_htmlizer.py into our tree, for easier customization
2007-01-03 22:06:51 -07:00
Brian Warner
e68118736c
add some experimental emacs test-coverage-annotation tools, still in development
2007-01-01 23:48:42 -07:00
