Commit Graph

3066 Commits

Author SHA1 Message Date
robk-tahoe
0c25628ded fuse/runtests: add overlapping write tests
using both small and large blocksizes for writes, write a 1Mb file to fuse
where every write overlaps another. 

This serves a useful purpose - in manual testing of blackmatch some time ago
most operations e.g. bulk copies, worked fine, but using rsync caused data 
corruption on most files.  it turned out to be that rsync writes in 64K blocks,
but rather than making the last block short, the last block instead overlaps
the preceding (already written) block.  This revealed a problem where cache
files were being opened 'append' rather than 'write' and hence the overlapping
write to the fuse layer caused the overlapping portion of the file to be 
duplicated in cache, leading to oversized and corrupt files being uploaded.
2008-10-03 15:48:33 -07:00
robk-tahoe
50986dc1e0 fuse/runtests: remove write small file test, as it's subsumed by the tiny_file test 2008-10-03 15:39:44 -07:00
robk-tahoe
4a3df3b683 fuse/runtests: added linear write tests for various block sizes
unit tests to test writing contiguous blocks linearly through the file,
for a variety of block sizes;  'tiny_file' is an entire file fitting within
a single io block / write operation.  'linear_{small,large}_writes' test
a 1Mb file written with each write operation containing significantly less
or more, respecitvely, data than fuse will pass into the implementation as
a single operation (which on the mac at least is 64Kib)
2008-10-03 15:35:50 -07:00
robk-tahoe
af9e1f6238 fuse/runtests: add a very simple 'write' test
this performs a very simple write through the fuse layer and confirms that
the file is stored correctly into the tahoe mesh.  ('simple' in the sense
that the entire file body fits trivially in a single write() operation, 
disk block etc)
2008-10-03 10:20:44 -07:00
robk-tahoe
8b9a267920 fuse/runtests: added a --web-open option
similar to the --debug-wait option which causes the test harness to
pause at various stages of the process to facilitate debugging, this
option simplifies that debugging by automatically opening a web browser
to the root dir of that implementation's tests when tests are commenced.

in addition, if --web-open is specfied but --debug-wait is not, the
harness will still pause after running tests but before tearing down
the tahoe grid - this allows all tests to run to completion, but
provide a debugging hook to investigate the end state of the grid's
contents thereafter.
2008-10-03 10:20:26 -07:00
robk-tahoe
236c52bf8b fuse/impl_a: fix a suspected bug in caching
from my examination of the tahoe_fuse ('impl_a') code, it looks like 
the intention is to cache the file contents in memory while it's open,
since it does in fact do that.  however it looks like it also ignored
that cache entirely, and made an individual tahoe webapi GET request
for each and every read() operation regardless of the relative size of
the read block and the file in question.

this changes that to make read() use the data in memory rather than
fetch the data over again.   if there's something more subtle going
on, please let me know.
2008-10-03 10:13:09 -07:00
robk-tahoe
183dd587d5 gui/macapp: slew of code cleanup; unmount filesystems on quit
a handful of code cleanup, renaming and refactoring.  basically consolidating
'application logic' (mount/unmount fs) into the 'MacGuiApp' class (the wx.App)
and cleaning up various scoping things around that.  renamed all references to
'app' to refer more clearly to the 'AppContainer' or to the guiapp.

globally renamed basedir -> nodedir

also made the guiapp keep a note of each filesystem it mounts, and unmount
them upon 'quit' so as to cleanup the user's environment before the tahoe node
vanishes from out underneath the orphaned tahoe fuse processes
2008-09-25 16:32:35 -07:00
robk-tahoe
9b10f46374 gui/macapp: make submenu of aliases for 'webopen'
this changes the 'open webroot' menu item to be a submenu listing all aliases
defined in ~/.tahoe.  Note that the dock menu does not support submenus, so it
only offers a single 'open webroot' option for the default tahoe: alias.

I had trouble with this at first and concluded that the submenus didn't work,
and made it a distinct 'WebUI' menu in it's own right.  on further inspection,
there are still problems but they seem to be something like once the dock menu
has been used, sometimes the app's main menubar menus will cease to function,
and this happens regardless of whether submenus or plain simple menus are used.
I have no idea what the peoblem is, but it's not submenu specific.
2008-09-25 09:39:19 -07:00
Zooko O'Whielacronx
57d9f6beb6 repairer: fix flaw in testutil.flip_one_bit() that Brian pointed out 2008-10-16 12:48:48 -07:00
Brian Warner
f3d730428f misc/incident-gatherer: add classify_tahoe.py: a foolscap incident-gatherer classification plugin 2008-10-15 15:09:40 -07:00
Zooko O'Whielacronx
8a6d1e5da6 repairer: test all different kinds of corruption that can happen to share files on disk 2008-10-14 16:09:20 -07:00
Brian Warner
b0a00e0f33 util/time_format.py: accept space separator, add unit tests 2008-10-13 15:52:58 -07:00
Brian Warner
46657b797f test_storage: use different filenames, poor stupid windows 2008-10-09 19:11:39 -07:00
Brian Warner
fd32d72b20 scripts/debug.py: emit the immutable-share version number, tolerate v2 2008-10-09 18:34:22 -07:00
Brian Warner
c289a9c31b storage.py: improve some precondition() error messages 2008-10-09 18:14:25 -07:00
Brian Warner
7031a69bee storage: introduce v2 immutable shares, with 8-byte offsets fields, to remove two of the three size limitations in #346. This code handles v2 shares but does not generate them. We'll make a release with this v2-tolerance, wait a while, then make a second release that actually generates v2 shares, to avoid compatibility problems. 2008-10-09 18:13:27 -07:00
Brian Warner
a0250e320b debug.py: oops, add missing import for ReadBucketProxy 2008-10-09 17:29:22 -07:00
Brian Warner
288d55825c storage: split WriteBucketProxy and ReadBucketProxy out into immutable/layout.py . No behavioral changes. 2008-10-09 17:08:00 -07:00
Zooko O'Whielacronx
86e22b8add interfaces: loosen a few max-size constraints which would limit us to a mere 1.09 TB maximum file size
These constraints were originally intended to protect against attacks on the
storage server protocol layer which exhaust memory in the peer.  However,
defending against that sort of DoS is hard -- probably it isn't completely
achieved -- and it costs development time to think about it, and it sometimes
imposes limits on legitimate users which we don't necessarily want to impose.
So, for now we forget about limiting the amount of RAM that a foolscap peer can
cause you to start using.
2008-10-09 12:13:57 -07:00
Brian Warner
93725a6935 util/limiter: add a repr 2008-10-07 13:19:45 -07:00
Brian Warner
9d4749d546 dirnode.build_manifest: include node.list in the limiter, that's the most important thing to slow down 2008-10-07 13:19:29 -07:00
Brian Warner
cd236efc68 web/directory: t=manifest output=html: make the caps into clickable hrefs 2008-10-07 13:18:45 -07:00
Brian Warner
f7edbc1536 web/directory: factor out the get_root function 2008-10-07 13:17:42 -07:00
Brian Warner
53c9867c5c web/directory.py: remove unused imports 2008-10-07 12:48:20 -07:00
Brian Warner
2edfe12b88 test_web: deep-size is more variable than I thought, so assert less 2008-10-06 22:11:47 -07:00
Brian Warner
3ffaded809 web: change t=manifest to return a list of (path,read/writecap) tuples, instead of a list of verifycaps. Add output=html,text,json. 2008-10-06 21:36:18 -07:00
Brian Warner
8e6d122ecf web: rewrite t=deep-size in terms of deep-stats, update test to match inclusion of directory sizes 2008-10-06 21:35:39 -07:00
Brian Warner
18a1070e66 ftpd: hush pyflakes 2008-10-06 18:45:13 -07:00
Brian Warner
701e5cd1da ftpd: make sure we're using a patched/fixed Twisted, to avoid confusion later 2008-10-06 18:14:11 -07:00
Brian Warner
05a8360177 ftp: change the twisted hack necessary for async-write-close, to one more agreeable to the twisted-dev folks, add a copy of the necessary patch to docs/ftp.txt 2008-10-06 18:06:05 -07:00
Brian Warner
70ae1bfc8e ftpd: remove debug messages 2008-10-06 16:16:20 -07:00
Brian Warner
bf3d09d430 ftpd: add native_client.php -based HTTP authentication scheme 2008-10-06 16:15:11 -07:00
Brian Warner
daf4af6699 ftpd: add ftp.accounts checker, remove InMemoryPasswordChecker 2008-10-06 15:51:24 -07:00
Brian Warner
d90a3ed7f8 test_system: add test coverage for immutable download.ConsumerAdapter, remove debug messages 2008-10-06 15:50:37 -07:00
Brian Warner
bc237b3956 ftp server: initial implementation. Still needs unit tests, custom Twisted patches. For #512 2008-10-06 12:52:36 -07:00
Brian Warner
934dc6aa77 test_cli.py: remove unused imports 2008-10-06 17:42:04 -07:00
Brian Warner
dbc2337639 CLI: remove 'tahoe admin generate-keypair', since the pycryptopp ecdsa API is about to change incompatibly. We'll undo this once pycryptopp is updated 2008-10-06 17:23:20 -07:00
Zooko O'Whielacronx
dc2a114290 docs: update architecture.txt 's section on the vdrive a.k.a. filesystem layer
Remove some obsolete parts (correct at the time, now incorrect), change terminology to reflect my preference: s/vdrive/filesystem/ and s/dirnode/directory/, and make a few other small changes.
2008-10-06 14:05:00 -07:00
Brian Warner
09341a969a dirnode: fix my remarkably-consistent 'metdadata' typo 2008-10-02 18:08:45 -07:00
Brian Warner
41bacca3f1 interfaces: fix minor typo 2008-10-02 17:52:49 -07:00
Brian Warner
d0bdf9a611 dirnode: add get_child_and_metadata_at_path 2008-10-02 17:52:03 -07:00
Brian Warner
9c505e49c2 stop using 'as' as an identifier: as with 'with', 'as' has become a reserved word in python 2.6 2008-10-02 17:27:49 -07:00
Brian Warner
034364c80b scripts/admin: split up generate_keypair code so that unit tests can use it more easily 2008-10-01 16:52:38 -07:00
Zooko O'Whielacronx
b01e86d00f docs: add some notes about things to do for a Tahoe release on pypi, freshmeat, and launchpad 2008-10-01 14:07:03 -07:00
Brian Warner
06770f720e misc/cpu-watcher.tac: use writeaside-and-rename for the history.pickle file 2008-09-30 17:30:53 -07:00
Brian Warner
3aba70bbcf misc/spacetime: use async polling so we can add a 60-second timeout, add an index to the 'url' Axiom column for 2x speedup 2008-09-30 16:34:48 -07:00
Brian Warner
cd26f58305 #518: replace various BASEDIR/* config files with a single BASEDIR/tahoe.cfg, with backwards-compatibility of course 2008-09-30 16:21:49 -07:00
Brian Warner
935d185696 tolerate simplejson-2.0.0 and newer, which frequently return bytestrings instead of unicode objects. Closes #523 2008-09-30 15:21:06 -07:00
Brian Warner
1a3f154c21 munin/tahoe_doomsday: oops, tolerate 'null' in the timeleft results, to unbreak the 2wk/4wk graphs 2008-09-30 13:20:51 -07:00
Brian Warner
78d58875a8 test_node: improve coverage of advertised_ip_addresses a bit 2008-09-29 23:08:16 -07:00