Brian Warner
670933ecee
storage: clean up use of si_s vs si_dir, add test for BadWriterEnabler message, add some logging
2008-01-31 17:48:48 -07:00
Zooko O'Whielacronx
7bf21082f7
remove unused import (thanks, pyflakes)
2008-01-31 17:00:59 -07:00
Zooko O'Whielacronx
79c439d026
storage: make two levels of share directories so as not to exceed certain filesystems's limitations on directory size
...
The filesystem which gets my vote for most undeservedly popular is ext3, and it has a hard limit of 32,000 entries in a directory. Many other filesystems (even ones that I like more than I like ext3) have either hard limits or bad performance consequences or weird edge cases when you get too many entries in a single directory.
This patch makes it so that there is a layer of intermediate directories between the "shares" directory and the actual storage-index directory (the one whose name contains the entire storage index (z-base-32 encoded) and which contains one or more share files named by their share number).
The intermediate directories are named by the first 14 bits of the storage index, which means there are at most 16384 of them. (This also means that the intermediate directory names are not a leading prefix of the storage-index directory names -- to do that would have required us to have intermediate directories limited to either 1024 (2-char), which is too few, or 32768 (3-chars of a full 5 bits each), which would overrun ext3's funny hard limit of 32,000.))
This closes #150 , and please see the "convertshares.py" script attached to #150 to convert your old tahoe-0.7.0 storage/shares directory into a new tahoe-0.8.0 storage/shares directory.
2008-01-31 16:26:28 -07:00
Brian Warner
8063aa8b86
WriteBucketProxy: improve __repr__
2008-01-28 18:53:51 -07:00
Brian Warner
821521cc3e
test_storage: fix pyflakes warnings
2008-01-14 21:26:48 -07:00
Brian Warner
a6ca98ac53
upload: add Encoder.abort(), to abandon the upload in progress. Add some debug hooks to enable unit tests.
2008-01-14 21:22:55 -07:00
Brian Warner
76ee9cccfe
storage: improve logging a bit
2008-01-14 11:58:58 -07:00
Zooko O'Whielacronx
08a64c3a2b
rename "secret" to "lease_secret" and change its size from 16 to 32 bytes
2007-12-17 18:34:11 -07:00
Brian Warner
e08b091d9f
storage: rewrite slot API, now use testv_and_readv_and_writev or readv
2007-11-05 20:17:14 -07:00
Brian Warner
8f21424449
storage: add readv_slots: get data from all shares
2007-11-05 00:37:01 -07:00
Brian Warner
bcf84c1238
storage.py: fix tests, timestamps get updated when leases are renewed
2007-10-31 12:31:33 -07:00
Brian Warner
70e7961088
storage.py: more test coverage, make sure leases survive resizing
2007-10-31 12:07:47 -07:00
Brian Warner
948e6b34dd
storage.py: improve test coverage even more
2007-10-31 01:44:01 -07:00
Brian Warner
4bd739435f
storage.py: more mutable-slot coverage, renewing/cancelling leases
2007-10-31 01:31:56 -07:00
Brian Warner
256ef1bf53
mutable slots: add some test coverage for lease-addition
2007-10-31 00:38:30 -07:00
Brian Warner
68d3d62002
mutable slots: finish up basic coding on server-side containers, add some tests. Remove all caching from MutableShareFile.
2007-10-31 00:10:40 -07:00
Brian Warner
b24c2925e8
checkpointing mutable-file work. Storage layer is 80% in place.
2007-10-30 19:47:36 -07:00
Brian Warner
8451b485a4
storage: fill alreadygot= with all known shares for the given storageindex, not just the ones they asked about
2007-09-17 00:48:40 -07:00
Brian Warner
d628d5f503
storage: remove the leftover incoming/XYZ/ directory when we're done with it
2007-09-15 14:34:04 -07:00
Brian Warner
277e720f7c
storage: add version number to share data. Closes #90 .
2007-09-04 09:00:24 -07:00
Brian Warner
85f3107b12
storage: handle simultanous uploads: add a lease for the pre-empted client
2007-09-02 14:57:49 -07:00
Brian Warner
0fe1205789
storage: replace sqlite with in-share lease records
2007-09-02 14:47:15 -07:00
Brian Warner
2a63fe8b01
deletion phase3: add a sqlite database to track renew/cancel-lease secrets, implement renew/cancel_lease (but nobody calls them yet). Also, move the shares from BASEDIR/storage/* down to BASEDIR/storage/shares/*
2007-08-27 23:41:40 -07:00
Brian Warner
739ae1ccde
deletion phase1: send renew/cancel-lease secrets, but my_secret is fake, and the StorageServer discards them
2007-08-27 17:28:51 -07:00
Brian Warner
c6f52e379a
rename storageserver.py to just storage.py, since it has both server and client sides now
2007-07-13 17:25:45 -07:00
Brian Warner
7589a8ee82
storage: we must truncate short segments. Now most tests pass (except uri_extension)
2007-07-13 16:38:25 -07:00
Brian Warner
1f8e407d9c
more #85 work, system test still fails
2007-07-13 15:09:01 -07:00
Brian Warner
cd8648d39b
storage: use one file per share instead of 7 ( #85 ). work-in-progress, tests still fail
2007-07-13 14:04:49 -07:00
Brian Warner
a1c97aa0be
storageserver: include metadata in the sizelimit, at least after the bucket has been closed
2007-07-03 17:38:49 -07:00
Brian Warner
c80ea7d693
storageserver: implement size limits. No code to enable them yet, though
2007-07-03 17:08:02 -07:00
Brian Warner
91d72bb504
test_storage: test StorageServer code too: allocation, lookup, multiple-writer behavior
2007-04-18 15:42:34 -07:00
Brian Warner
008e418523
storageserver: assert that blocks are written in-order, clean up tests a bit
2007-04-17 20:21:05 -07:00
Brian Warner
e040b85f5d
test_storage: add (failing) test of the BucketWriter/BucketReader implementation
2007-04-17 20:03:44 -07:00
Brian Warner
7cd9ef3bbf
finish making the new encoder/decoder/upload/download work
2007-03-30 16:50:50 -07:00
Brian Warner
78d19c271c
rearrange service startup a bit, now Node.startService() returns a Deferred that fires when the tub is actually ready, and there is also a Node.when_tub_ready() hook. This allows get_local_addresses() to be slow and not break everything. Changed all necessary test cases to accomodate this slow startup.
2007-03-08 15:10:36 -07:00
Brian Warner
78a9e815c5
add simple metadata (a single string) to the storage protocol
2007-01-15 14:01:22 -07:00
Brian Warner
3490378551
move all packages into src/, fix allmydata.Crypto build. Now you must perform a 'setup.py build' before using anything, and you must add the build directory (build/lib.linux-i686-2.4) to your PYTHONPATH before doing anything
2006-12-14 03:39:50 -07:00
