Brian Warner
85f3107b12
storage: handle simultanous uploads: add a lease for the pre-empted client
2007-09-02 14:57:49 -07:00
Brian Warner
dbcabc1142
debug: 'dump-uri-extension' command becomes 'dump-share', add 'dump-share-leases'. Both display leases.
2007-09-02 14:48:20 -07:00
Brian Warner
0fe1205789
storage: replace sqlite with in-share lease records
2007-09-02 14:47:15 -07:00
Brian Warner
c7288286ae
test_uri.py: remove tiny whitespace
2007-08-28 23:40:03 -07:00
wilcoxjg
33bae16ed2
created DirnodeURI schema
2007-08-28 23:28:26 -07:00
Brian Warner
2a63fe8b01
deletion phase3: add a sqlite database to track renew/cancel-lease secrets, implement renew/cancel_lease (but nobody calls them yet). Also, move the shares from BASEDIR/storage/* down to BASEDIR/storage/shares/*
2007-08-27 23:41:40 -07:00
Brian Warner
850bc9da02
deletion phase2b: create renew/cancel secrets for real.
2007-08-27 19:30:26 -07:00
Brian Warner
56afda11d1
deletion phase2a: improve creation of renew/cancel secrets. Still fake though.
2007-08-27 19:00:18 -07:00
Brian Warner
739ae1ccde
deletion phase1: send renew/cancel-lease secrets, but my_secret is fake, and the StorageServer discards them
2007-08-27 17:28:51 -07:00
Brian Warner
ed525f7478
test_web: improve provisioning.py test coverage a bit by using a live web hit
2007-08-27 17:26:39 -07:00
Brian Warner
5946985c07
add a provisioning utility page which shows necessary storage space and transfer rates for grids of various sizes
2007-08-26 23:44:24 -07:00
Brian Warner
3e1c417f81
webish: modify JSON to match zooko's proposed API changes in #118
2007-08-23 13:00:39 -07:00
Brian Warner
f3353ee5e4
web: replace FILE links with /uri -based ones, to prevent an XSS attack against the secret vdrive URI contained in the current URL
2007-08-22 17:35:01 -07:00
Brian Warner
4de5767c98
web: remove /vdrive/private, replace with a start.html file that points at the /uri/PRIVATE_URI, to prevent XSRF attacks
2007-08-22 14:54:34 -07:00
Brian Warner
75056b6222
test_system: weaken the assertion about the 410 error message when downloading a bogus URI, since some systems emit error messages differently
2007-08-16 17:05:26 -07:00
Brian Warner
6b5e05cc67
test_system.py: verify that we can replace files in place
2007-08-16 17:03:50 -07:00
Brian Warner
fe06b3be8b
dirnode: change the defined behavior of RIVirtualDriveServer.set to allow replace-in-place without raising an exception
2007-08-16 17:03:19 -07:00
Brian Warner
be0ab3b1f1
test_system.py: many (failing) web tests were accidentally bypassed, fix those. Add some PUT tests.
2007-08-16 16:49:40 -07:00
Zooko O'Whielacronx
4ed7f0ac2a
fix small bug in unit tests which caused spurious failures on Windows
2007-08-16 14:14:41 -07:00
Zooko O'Whielacronx
094b687d6e
import version class and make-version script from pyutil -- fixes win32 build, improves error handling, and eliminates unused features
2007-08-16 14:09:30 -07:00
Brian Warner
c1da0c11bc
webish: implement replace= for POST commands
2007-08-15 15:21:38 -07:00
Brian Warner
1752c9e29e
webish: implement replace= for PUT commands
2007-08-15 13:22:23 -07:00
Brian Warner
42dcc3088e
IDirectoryNode: add has_child() method
2007-08-15 13:22:01 -07:00
Brian Warner
244471dcc4
check_memory.py: Disable the 100MB test for now: our buildslave can't currently handle it because the testnet/framework processes uses something like 600M of RSS.
2007-08-15 12:55:11 -07:00
Brian Warner
179d5462c0
webish: look for when_done= in POST fields as well as queryargs. Closes #101 .
...
We need to look in the fields because that's how we build the mkdir/upload
forms. Without this, uploading or creating directories would leave us on a
page that had just a URI, instead of something actually useful to a human.
2007-08-13 17:45:02 -07:00
Brian Warner
6c6c62037d
fix pyflakes warnings from recent b32decode change
2007-08-12 16:33:51 -07:00
Brian Warner
0332e94656
test_system.py: add coverage for get_permuted_peers()
2007-08-12 16:29:34 -07:00
Zooko O'Whielacronx
42f8e57416
don't over-encode the nodeid many times with ascii-encoding
2007-08-12 10:29:38 -07:00
Brian Warner
2f5a27316f
webish: localfile=/localdir= are now disabled by default, a special switch is required to enable them
2007-08-10 18:21:22 -07:00
Brian Warner
998802fd6d
#96 : add flag to enable pushing data to ourselves, defaulting to False
2007-08-09 18:30:24 -07:00
Zooko O'Whielacronx
531109e803
fix test_web refactoring so that the WebMixin class isn't a TestCase
...
Thanks, Brian.
2007-08-10 08:40:02 -07:00
Zooko O'Whielacronx
39f218ab90
refactor test_web so that other tests can use the part of test_web that sets up a simple filesystem
2007-08-09 13:08:42 -07:00
Brian Warner
9ac5c14037
introducer: don't log.err() an initial connection failure, since that flunks tests. Use self.log() instead. Also improve test_client.py to always trigger this case, before it was finishing to quickly to always hit the problem.
2007-08-09 12:53:44 -07:00
Brian Warner
84d41adfcc
check_memory.py: include a single 100MB test
2007-08-09 11:30:33 -07:00
Brian Warner
57df32aac5
now that the buildslave is moved to a new machine, enable the 50MB test
2007-08-09 01:32:52 -07:00
Zooko O'Whielacronx
42d3afffa2
refactor test_uri and add a test of the empty file
2007-07-26 10:48:30 -07:00
Brian Warner
8de93635af
test_web.py: oops, fix syntax error caused by removing comments
2007-07-25 03:21:30 -07:00
Brian Warner
6aff425fd2
test_web.py: remove spurious '# YES' lines, now that all the tests pass
2007-07-24 20:16:21 -07:00
Brian Warner
e3eba2d6c5
remove unused imports: hush pyflakes warnings
2007-07-24 18:13:58 -07:00
Zooko O'Whielacronx
32c90c6468
switch from rfc 3548 base-32 to z-base-32 except for tubids/nodeids
2007-07-24 13:46:06 -07:00
Brian Warner
e6e9ddc588
refactor upload/encode, to split encrypt and encode responsibilities
2007-07-23 19:31:53 -07:00
Brian Warner
9c5ab89afe
truncate storage index to 128 bits, since it's derived from a 128 bit AES key
2007-07-22 19:48:44 -07:00
Brian Warner
81a9904455
CHK: remove the storage index from the URI, deriving it from the key instead
2007-07-21 18:23:15 -07:00
Brian Warner
1d9a58977f
uri: implement URI-processing classes, IFileURI/IDirnodeURI, use internally
2007-07-21 15:40:36 -07:00
Brian Warner
e3a57fca98
upload: finish refactoring, all unit tests pass now
2007-07-19 22:53:29 -07:00
Brian Warner
9af506900b
upload: refactor to enable streaming upload. not all tests pass yet
2007-07-19 18:21:44 -07:00
Brian Warner
a7f04616bf
webish: improve test coverage further
2007-07-17 12:28:29 -07:00
Brian Warner
3197e1d39b
webish: improve test coverage of WebDownloadTarget
2007-07-17 12:16:45 -07:00
Brian Warner
d927d371b7
check_memory.py: test POST-based uploads as well as from-disk ones
2007-07-16 20:17:51 -07:00
Brian Warner
0269a6229b
check_memory.py: fix benign-but-noisy vdrive.furl handling bug
2007-07-16 19:34:52 -07:00
Brian Warner
0fd5dcacf6
check_memory: update it, write stats to a file, disable 50MB test for now
2007-07-16 18:08:55 -07:00
robk-org
920fed7f2a
added unit test to webish's rename function
...
added unit tests to test various permutations of the rename function, and
some sanity checks on the rename-form function.
also added a guard to prevent '/' in from_/to_name in rename calls.
2007-07-16 17:05:01 -07:00
Brian Warner
56dcb814a8
webish: test error cases more thoroughly by looking inside the response text
2007-07-16 12:01:19 -07:00
Brian Warner
f207f4a199
webish.py: disallow slashes in POSTed filenames. Closes #75 .
2007-07-16 11:53:12 -07:00
Brian Warner
225fbfee57
storage.py: add a test for the next_power_of_k fix I made a few hours ago, basically do an upload with a non-power-of-two number of segments
2007-07-13 22:24:06 -07:00
Brian Warner
6e1ad64c9d
webish.py: test that _get_or_create_directories fix I made a few hours ago
2007-07-13 22:19:16 -07:00
Brian Warner
c6f52e379a
rename storageserver.py to just storage.py, since it has both server and client sides now
2007-07-13 17:25:45 -07:00
Brian Warner
4d868e6649
fix dump-uri-extension
2007-07-13 16:58:08 -07:00
Brian Warner
7589a8ee82
storage: we must truncate short segments. Now most tests pass (except uri_extension)
2007-07-13 16:38:25 -07:00
Brian Warner
1f8e407d9c
more #85 work, system test still fails
2007-07-13 15:09:01 -07:00
Brian Warner
cd8648d39b
storage: use one file per share instead of 7 ( #85 ). work-in-progress, tests still fail
2007-07-13 14:04:49 -07:00
Brian Warner
ba7e14a870
fix several methods to handle LIT URIs correctly, rather than assuming that all filenodes are CHK URIs
2007-07-12 16:17:49 -07:00
Brian Warner
5399395c27
allow the introducer to set default encoding parameters. Closes #84 .
...
By writing something like "25 75 100" into a file named 'encoding_parameters'
in the central Introducer's base directory, all clients which use that
introducer will be advised to use 25-out-of-100 encoding for files (i.e.
100 shares will be produced, 25 are required to reconstruct, and the upload
process will be happy if it can find homes for at least 75 shares). The
default values are "3 7 10". For small meshes, the defaults are probably
good, but for larger ones it may be appropriate to increase the number of
shares.
2007-07-12 15:33:30 -07:00
Brian Warner
def63d193e
implement URI:LIT, which stores small (<55B) files inside the URI itself. Fixes #81 .
2007-07-12 13:22:36 -07:00
Brian Warner
4a16d20d85
runner.py: start to refactor into separate files
2007-07-10 18:41:52 -07:00
Brian Warner
13e84526a1
webapi: normalized API: use t=upload or t=download when providing localdir= or localfile=
2007-07-10 13:24:10 -07:00
Brian Warner
dce1dc2730
storage: wrap buckets in a local proxy
...
This will make it easier to change RIBucketWriter in the future to reduce the wire
protocol to just open/write(offset,data)/close, and do all the structuring on the
client end. The ultimate goal is to store each bucket in a single file, to reduce
the considerable filesystem-quantization/inode overhead on the storage servers.
2007-07-08 23:27:46 -07:00
Brian Warner
72fc8c5cb8
web: use real JSON instead of the fake stubs
...
Also include the encoder portion of Bob Ippolito's simplejson-1.7.1 as
allmydata.util.json_encoder . simplejson is distributed under a more liberal
license than Tahoe (looks to be modified BSD), so redistributing it should be ok.
2007-07-08 00:17:11 -07:00
Brian Warner
2d06a66ec1
test_web.test_welcome: give the rooturl a trailing slash, otherwise older versions of twisted complain
2007-07-07 23:22:52 -07:00
Brian Warner
bd8625076a
web: remove t=XML, and other dead code
2007-07-07 22:55:15 -07:00
Brian Warner
159f8bfd50
web: improve test coverage
2007-07-07 22:47:18 -07:00
Brian Warner
e2f99247f2
web: make sure we present read-only directories properly
2007-07-07 22:21:20 -07:00
Brian Warner
62e8528cc6
web: /uri/ must escape slashes, we use bangs for this
2007-07-07 22:06:52 -07:00
Brian Warner
617a85df67
web: remove debug prints
2007-07-07 20:11:30 -07:00
Brian Warner
464f25e5f2
web: more test work, now all tests pass, POST too, only XMLRPC left to implement
2007-07-07 20:06:58 -07:00
Brian Warner
f35c9c6540
test_system.py: change/remove the tests that currently fail due to web changes
2007-07-07 20:06:44 -07:00
Brian Warner
0cd730a7b3
web: more test work, now all tests either pass or are skipped (POST, XMLRPC, and URI/)
2007-07-07 10:34:05 -07:00
Brian Warner
d501984eba
webapi: checkpointing more test progress
2007-07-07 00:16:36 -07:00
Brian Warner
6570253d6b
checkpointing new webapi: not all tests pass yet
2007-07-06 19:43:55 -07:00
Brian Warner
21e12f383d
web: missed a IndexError-to-KeyError conversion
2007-07-06 19:43:03 -07:00
Brian Warner
71c04fc2e7
web: use KeyError (rather than IndexError) to signal a missing child
2007-07-06 19:40:08 -07:00
Brian Warner
9e42dda6a4
add IDirectoryNode.get_child_at_path
2007-07-06 19:38:37 -07:00
Brian Warner
a1c97aa0be
storageserver: include metadata in the sizelimit, at least after the bucket has been closed
2007-07-03 17:38:49 -07:00
Brian Warner
94e6e6160b
activate storage size limits in the client. Closes #34 .
...
To use this, write a number like 10MB or 5Gb or 5000000000 to a file
named 'sizelimit' in the client's base directory. The node will not grant
leases for shares that would take it much beyond this many bytes of
storage. Note that metadata is not included in the allocation count until
a restart, so the actual space consumed may grow beyond the limit if
the node is not restarted very frequently and the amount of metadata is
significant.
2007-07-03 17:27:07 -07:00
Brian Warner
c80ea7d693
storageserver: implement size limits. No code to enable them yet, though
2007-07-03 17:08:02 -07:00
Brian Warner
9ddb929651
fileutil: add du() function
2007-07-03 15:49:45 -07:00
Brian Warner
622acc690a
webish: improve reporting of web download errors that occur early enough
...
If the error occurs before any data has been sent, we can give a sensible
error message (code 500, stack trace, etc). This will cover most of the error
cases. The ones that aren't covered are when we run out of good peers after
successfully decoding the first segment, either because they go away or
because their shares are corrupt.
2007-07-03 13:47:37 -07:00
Brian Warner
f15bb302a1
webish.py: handle errors during download better. Addresses #65 .
...
Previously, exceptions during a web download caused a hang rather than some
kind of exception or error message. This patch improves the situation by
terminating the HTTP download rather than letting it hang forever. The
behavior still isn't ideal, however, because the error can occur too late to
abort the HTTP request cleanly (i.e. with an error code). In fact, the
Content-Type header and response code have already been set by the time any
download errors have been detected, so the browser is committed to displaying
an image or whatever (thus any error message we put into the stream is
unlikely to be displayed in a meaningful way).
2007-07-03 13:18:14 -07:00
Brian Warner
7e32d3c8af
test_util: improve test coverage of allmydata.util.fileutil
2007-07-03 11:15:05 -07:00
Brian Warner
9d47c2524e
test_system: stall for a second while bouncing the client, it might help windows
2007-06-28 19:20:28 -07:00
Brian Warner
efb99078b6
change IVirtualDrive.get_node_at_path to accept either a list or a single slash-separated string
2007-06-28 17:46:14 -07:00
Brian Warner
5854916299
test_system.py: more vdrive coverage
2007-06-28 15:05:33 -07:00
Brian Warner
d6e732b65f
vdrive: add get_node_at_path(pathlist)
2007-06-28 11:00:03 -07:00
Brian Warner
b06c74c2a1
Add the 'vdrive' service, for clients to access the public/private root dirs.
...
These allow client-side code to conveniently retrieve the IDirectoryNode
instances for both the global shared public root directory, and the per-user
private root directory.
2007-06-27 17:11:06 -07:00
Brian Warner
18ab5ce837
dirnode: add build_manifest() and introduce 'refresh capabilities'
2007-06-26 19:41:20 -07:00
Brian Warner
b11fa20191
merge vdrive.py and filetable.py into a single dirnode.py
2007-06-26 17:16:58 -07:00
Brian Warner
78c2376fa3
more runner.py test coverage: don't bypass argv parsing
2007-06-26 16:51:38 -07:00
Brian Warner
912fc5a77e
runner.py: improve test coverage further: implement --quiet with StringIOs
2007-06-26 16:19:18 -07:00
Brian Warner
de24d3cd94
dump_uri_extension: improve test coverage of runner.py
2007-06-26 15:55:00 -07:00
Brian Warner
a505560bbe
improve test coverage of runner.py
2007-06-26 15:36:46 -07:00
