Commit Graph

2702 Commits

Author SHA1 Message Date
Brian Warner
fabdc28c06 deep-stats: add file-size histogram 2008-05-08 16:19:42 -07:00
Brian Warner
a1ea3d9b37 test_dirnode: the size of the directories varies, tolerate it as low as 500 bytes 2008-05-08 13:42:42 -07:00
Brian Warner
e6ae7a2c60 dirnode: refactor deep-stats a bit 2008-05-08 13:33:07 -07:00
Brian Warner
6c00a70dbc dirnode: add a deep_stats(), like deep-size but with more information. webish adds t=deeps-size too. 2008-05-08 13:21:14 -07:00
Brian Warner
462ef2a0ac run a stats provider even if there's no gatherer, since the HTTP /statistics page is then useful. Only run the once-per-second load-monitor if there is a gatherer configured 2008-05-08 11:37:30 -07:00
Brian Warner
3cb361e233 dirnode: use the concurrency limiter in t=manifest and t=deep-size, allow 10 retrievals in parallel 2008-05-07 18:36:37 -07:00
Brian Warner
d76cf6f748 limiter: fix bug that ignores the limit= argument 2008-05-07 18:35:53 -07:00
robk-tahoe
abae0345ad tahoefuse: fix typo which could cause data corruption
a typo in the 'flags2mode' code would wind up passing the O_APPEND
flag into the os open() call, which would cause the file to be opened
in 'strict append' mode, i.e. all writes extend the file, regardless of
calls to seek.

this causes a problem for tahoefuse in that the seek() calls made to
filehandles open through fuse would be ignored when write()s occurred.
this was evidenced by corruption seen when using rsync.  it turns out
that rsync actually makes overlapping writes in some cases, i.e. even
when writing a new fresh file out, it still doesn't write a simple
contiguous span of data, but will make writes overlapping data already
written. this is probably related to the way it manages data blocks 
internally for rolling checksums etc.  at any rate, this bug would 
thus cause rsync in those cases to write a chunk of duplicate data
into the file - leading to tahoe securely and reliably storing the
wrong data.

fixing this, so that non-append file opens do not pass O_APPEND seems
to eliminate this problem.
2008-05-07 16:42:20 -07:00
robk-tahoe
8179daee4a tahoefuse: return bogus but useful data to statfs call
previously tahoefuse returned the fs stat for the filesystem the fuse plugin
was running upon (e.g. '/').  this works ok until you need to copy more to
tahoe than the local machine has free disk space, at which point Finder will
refuse to copy 'too much' data.

this changes it so that tahoe always reports 2TiB used of an 8TiB filesystem
this is entirely bogus, but allows copies of up to 2TiB to be initiated.
2008-05-07 16:40:09 -07:00
Brian Warner
15a9f3d01b test_util: improve ConcurrencyLimiter tests 2008-05-07 17:00:47 -07:00
Brian Warner
c7e441309d add a basic concurrency limiter utility 2008-05-07 16:53:30 -07:00
Brian Warner
6f089a45b2 boringfile: add Twisted .egg files 2008-05-07 16:52:48 -07:00
Brian Warner
1e4b904b1a web status: abbreviate file size in the upload+download summary page 2008-05-07 13:53:58 -07:00
Brian Warner
74df0b58b5 munin: add tahoe_cpu_watcher.py, to track the data from misc/cpu-watcher.tac 2008-05-07 13:19:08 -07:00
Brian Warner
79147a3f6a misc/cpu-watcher*: add some tools to monitor CPU usage of arbitrary processes, like tahoe nodes 2008-05-07 12:34:29 -07:00
Zooko O'Whielacronx
c1184f51e4 docs: fix a few stale comments in code 2008-05-07 08:39:03 -07:00
Zooko O'Whielacronx
59cdfb8f35 setup: tiny fix to syntax in makefile 2008-05-07 05:37:11 -07:00
Brian Warner
e2f56b8e3c configuration.txt: describe helper config 2008-05-06 15:59:06 -07:00
Zooko O'Whielacronx
a01864c713 docs: shorter running.html 2008-05-06 15:29:04 -07:00
Brian Warner
783fbb12e6 docs/helper.txt: explain more about the helper 2008-05-06 13:49:01 -07:00
Zooko O'Whielacronx
0432f5dee6 docs: mention configuration, suggested by ben hyde's question about storage servers 2008-05-06 13:39:35 -07:00
Zooko O'Whielacronx
8a0deaf53f docs: edit to install.html suggested by Brian 2008-05-06 12:31:15 -07:00
Zooko O'Whielacronx
b41dcbd329 setup: trivial formatting change in _auto_deps.py 2008-05-06 12:30:56 -07:00
Zooko O'Whielacronx
a097f45243 setup: remove specific checks for twisted dependency in makefile
Now that the twisted dependency is handled by the automatic dependency mechanism.
2008-05-06 12:09:00 -07:00
Zooko O'Whielacronx
99d5d2335d setup: require pycryptopp >= v0.5 2008-05-06 11:17:47 -07:00
Zooko O'Whielacronx
b1583ac6a0 setup: remove bundled pycryptopp-0.3.0.tar 2008-05-06 08:24:46 -07:00
Zooko O'Whielacronx
045e2bcdf4 setup: bundle pycryptopp-0.5.1.tar
pycryptopp-0.3.0 incorrectly crypts AES (depending on compiler and version of Crypto++ used)
Very soon now we're going to set up an "ext" repository to hold all tarballs and no longer check them into our trunk source tree.
2008-05-06 08:23:36 -07:00
Brian Warner
fc774803ce test_stats.py: improve test coverage 2008-04-30 11:52:31 -07:00
Brian Warner
a5a7ba24ef stats: add tests for CPUUsageMonitor, modify it a bit to facilitate testing 2008-04-30 11:39:13 -07:00
Zooko O'Whielacronx
fc271a0ee9 setup: don't catch ImportError when importing _auto_deps in allmydata/__init__.py
Nowadays pkg_resources is a runtime requirement, and if there is something screwed up in the installation, we want an explicit ImportError exception as early as possible.
2008-04-30 13:22:04 -07:00
Brian Warner
88951548ee test_mutable: update notify_publish() to match new signature 2008-04-29 18:24:57 -07:00
Brian Warner
4b3adb3fcf mutable stats: track mutable bytes published too 2008-04-29 18:20:05 -07:00
Brian Warner
3be921174b stats: add CPU-percentage monitor, with 1min/5min/15min moving-window averages, using time.clock() 2008-04-29 18:12:53 -07:00
Zooko O'Whielacronx
2f3a44f820 docs: remove the redundant (and therefore bit-rotting) parts of mutable-DSA.txt and instead refer to mutable.txt 2008-04-29 15:51:58 -07:00
Zooko O'Whielacronx
f842656ec2 wui: reorganize the welcome.xhtml page
Jake Edge tried Tahoe out and didn't notice the /status page.  Hopefully with this new organization people like he will see that link more easily.  This also addresses drewp's suggestion that the controls appear above the list of servers instead of below.  (I think that was his suggestion.)  I also reordered the controls.
2008-04-29 15:10:14 -07:00
Brian Warner
351a7f4632 offloaded: add 'resumes' counter to stats: how many upload requests are resuming an interrupted upload? 2008-04-28 13:27:55 -07:00
Zooko O'Whielacronx
c88ea89f14 contrib: add a note about Armin Rigo's fuse implementation 2008-04-28 07:05:44 -07:00
ben
c85869fc08 support freebsd 6 2008-04-28 00:41:40 -07:00
Brian Warner
773e9abe24 debian: add python-setuptools to the debian install-time dependencies. Should close #382. 2008-04-24 16:01:04 -07:00
Brian Warner
19a40d7a2d Makefile: add quicktest-figleaf: this is in my edit-test-repeat loop, and I need it to be fast 2008-04-24 11:30:38 -07:00
Brian Warner
d2a2b8a4ae munin/tahoe_estimate_files.py: tool to estimate the total number of slots (mutable and immutable combined) in the grid, from a small sample 2008-04-24 11:28:35 -07:00
Zooko O'Whielacronx
c2621313bb setup: pyOpenSSL is now easy_installable, and pycryptopp now includes Crypto++, so we can remove those two from the Manual Dependencies 2008-04-24 10:29:17 -07:00
robk-tahoe
5f5ed6b08d munin stats: fix typo in mutable file stats 2008-04-23 15:50:29 -07:00
robk-tahoe
f74042c75e munin stats: add mutable/uploader traffic graphs
this adds munin graphs to present data already published by nodes to
the stats_gatherer, namely mutable files published/retrieved, and 
immutable files uploaded, and the bytes thereof
2008-04-23 14:39:41 -07:00
Brian Warner
186492e620 introducer: only record one announcement per (tubid,service) tuple. Fixes #343. 2008-04-23 15:05:39 -07:00
Brian Warner
55dfb697a4 client: don't start the IntroducerClient until the Tub is ready, otherwise we will sometimes connect to the introducer (or other clients) before we've done Tub.setLocation, which loses some information on the introducer status page 2008-04-23 14:52:34 -07:00
Brian Warner
239c950810 munin/tahoe-stats.py: category is 'counters' not 'stats', use DERIVE 2008-04-23 14:03:23 -07:00
Brian Warner
a557d44877 test_mutable: test that all servers refusing our share means a publish fails 2008-04-22 18:53:20 -07:00
Brian Warner
13eb00145e mutable: improve test coverage in Retrieve, when shares change after mapupdate 2008-04-22 17:25:14 -07:00
Brian Warner
acdd98432d hush pyflakes warning about code that got moved in the recent StallMixin refactoring 2008-04-22 17:14:26 -07:00