Zooko O'Whielacronx
7b285ebcb1
immutable: remove the last bits of code (only test code or unused code) which did something with plaintext hashes or plaintext hash trees
2008-12-19 08:18:07 -07:00
Zooko O'Whielacronx
82ee44ed5b
debug: pass empty optional arguments to ReadBucketProxy
...
because those arguments are about to become non-optional (for other code than test/debug code)
2008-12-16 17:51:45 -07:00
Brian Warner
6958b7fa90
test_storage.py: more windows-vs-readonly-storage fixes
2008-12-02 19:41:02 -07:00
Brian Warner
cfba882b30
storage: replace sizelimit with reserved_space, make the stats 'disk_avail' number incorporate this reservation
2008-12-01 17:24:21 -07:00
Brian Warner
db37c14ab7
storage: add remote_advise_corrupt_share, for clients to tell storage servers about share corruption that they've discovered. The server logs the report.
2008-10-24 11:52:48 -07:00
Brian Warner
46657b797f
test_storage: use different filenames, poor stupid windows
2008-10-09 19:11:39 -07:00
Brian Warner
7031a69bee
storage: introduce v2 immutable shares, with 8-byte offsets fields, to remove two of the three size limitations in #346 . This code handles v2 shares but does not generate them. We'll make a release with this v2-tolerance, wait a while, then make a second release that actually generates v2 shares, to avoid compatibility problems.
2008-10-09 18:13:27 -07:00
Brian Warner
288d55825c
storage: split WriteBucketProxy and ReadBucketProxy out into immutable/layout.py . No behavioral changes.
2008-10-09 17:08:00 -07:00
Brian Warner
afda2a43e4
storage: remove update_write_enabler method, it won't serve the desired purpose, and I have a better scheme in mind. See #489 for details
2008-07-21 17:28:28 -07:00
Brian Warner
c46f5bc634
storage: rename the latency key names so they sort properly
2008-07-11 21:51:02 -07:00
Brian Warner
60725ed065
storage: add add_lease/update_write_enabler to remote API, revamp lease handling
2008-07-09 18:06:55 -07:00
Brian Warner
dfe235bcba
test_storage: oops, update the test to match the leave-incoming/ change
2008-06-26 11:36:17 -07:00
Brian Warner
c09c342718
test_storage: yet more coverage
2008-06-17 17:44:10 -07:00
Brian Warner
6b55b8b022
test_storage.py: improve test coverage
2008-06-17 17:01:42 -07:00
Brian Warner
2154b5751b
test_storage: add coverage for discard_storage
2008-06-16 17:52:40 -07:00
Brian Warner
75e662cc46
test_storage: add coverage for readonly_storage
2008-06-16 17:52:13 -07:00
Brian Warner
1ce3b77dde
storage: improve stats, make them accessible via webport /statistics
2008-06-16 16:35:59 -07:00
Brian Warner
6b7ff02e36
storage: measure latency-per-operation, calculate mean/median/percentiles
2008-06-16 15:21:55 -07:00
Brian Warner
814922a9a1
storage: ignore shares in incoming/, to make clients use other servers during simultaneous uploads
2008-06-10 11:53:10 -07:00
Brian Warner
670933ecee
storage: clean up use of si_s vs si_dir, add test for BadWriterEnabler message, add some logging
2008-01-31 17:48:48 -07:00
Zooko O'Whielacronx
7bf21082f7
remove unused import (thanks, pyflakes)
2008-01-31 17:00:59 -07:00
Zooko O'Whielacronx
79c439d026
storage: make two levels of share directories so as not to exceed certain filesystems's limitations on directory size
...
The filesystem which gets my vote for most undeservedly popular is ext3, and it has a hard limit of 32,000 entries in a directory. Many other filesystems (even ones that I like more than I like ext3) have either hard limits or bad performance consequences or weird edge cases when you get too many entries in a single directory.
This patch makes it so that there is a layer of intermediate directories between the "shares" directory and the actual storage-index directory (the one whose name contains the entire storage index (z-base-32 encoded) and which contains one or more share files named by their share number).
The intermediate directories are named by the first 14 bits of the storage index, which means there are at most 16384 of them. (This also means that the intermediate directory names are not a leading prefix of the storage-index directory names -- to do that would have required us to have intermediate directories limited to either 1024 (2-char), which is too few, or 32768 (3-chars of a full 5 bits each), which would overrun ext3's funny hard limit of 32,000.))
This closes #150 , and please see the "convertshares.py" script attached to #150 to convert your old tahoe-0.7.0 storage/shares directory into a new tahoe-0.8.0 storage/shares directory.
2008-01-31 16:26:28 -07:00
Brian Warner
8063aa8b86
WriteBucketProxy: improve __repr__
2008-01-28 18:53:51 -07:00
Brian Warner
821521cc3e
test_storage: fix pyflakes warnings
2008-01-14 21:26:48 -07:00
Brian Warner
a6ca98ac53
upload: add Encoder.abort(), to abandon the upload in progress. Add some debug hooks to enable unit tests.
2008-01-14 21:22:55 -07:00
Brian Warner
76ee9cccfe
storage: improve logging a bit
2008-01-14 11:58:58 -07:00
Zooko O'Whielacronx
08a64c3a2b
rename "secret" to "lease_secret" and change its size from 16 to 32 bytes
2007-12-17 18:34:11 -07:00
Brian Warner
e08b091d9f
storage: rewrite slot API, now use testv_and_readv_and_writev or readv
2007-11-05 20:17:14 -07:00
Brian Warner
8f21424449
storage: add readv_slots: get data from all shares
2007-11-05 00:37:01 -07:00
Brian Warner
bcf84c1238
storage.py: fix tests, timestamps get updated when leases are renewed
2007-10-31 12:31:33 -07:00
Brian Warner
70e7961088
storage.py: more test coverage, make sure leases survive resizing
2007-10-31 12:07:47 -07:00
Brian Warner
948e6b34dd
storage.py: improve test coverage even more
2007-10-31 01:44:01 -07:00
Brian Warner
4bd739435f
storage.py: more mutable-slot coverage, renewing/cancelling leases
2007-10-31 01:31:56 -07:00
Brian Warner
256ef1bf53
mutable slots: add some test coverage for lease-addition
2007-10-31 00:38:30 -07:00
Brian Warner
68d3d62002
mutable slots: finish up basic coding on server-side containers, add some tests. Remove all caching from MutableShareFile.
2007-10-31 00:10:40 -07:00
Brian Warner
b24c2925e8
checkpointing mutable-file work. Storage layer is 80% in place.
2007-10-30 19:47:36 -07:00
Brian Warner
8451b485a4
storage: fill alreadygot= with all known shares for the given storageindex, not just the ones they asked about
2007-09-17 00:48:40 -07:00
Brian Warner
d628d5f503
storage: remove the leftover incoming/XYZ/ directory when we're done with it
2007-09-15 14:34:04 -07:00
Brian Warner
277e720f7c
storage: add version number to share data. Closes #90 .
2007-09-04 09:00:24 -07:00
Brian Warner
85f3107b12
storage: handle simultanous uploads: add a lease for the pre-empted client
2007-09-02 14:57:49 -07:00
Brian Warner
0fe1205789
storage: replace sqlite with in-share lease records
2007-09-02 14:47:15 -07:00
Brian Warner
2a63fe8b01
deletion phase3: add a sqlite database to track renew/cancel-lease secrets, implement renew/cancel_lease (but nobody calls them yet). Also, move the shares from BASEDIR/storage/* down to BASEDIR/storage/shares/*
2007-08-27 23:41:40 -07:00
Brian Warner
739ae1ccde
deletion phase1: send renew/cancel-lease secrets, but my_secret is fake, and the StorageServer discards them
2007-08-27 17:28:51 -07:00
Brian Warner
c6f52e379a
rename storageserver.py to just storage.py, since it has both server and client sides now
2007-07-13 17:25:45 -07:00
Brian Warner
7589a8ee82
storage: we must truncate short segments. Now most tests pass (except uri_extension)
2007-07-13 16:38:25 -07:00
Brian Warner
1f8e407d9c
more #85 work, system test still fails
2007-07-13 15:09:01 -07:00
Brian Warner
cd8648d39b
storage: use one file per share instead of 7 ( #85 ). work-in-progress, tests still fail
2007-07-13 14:04:49 -07:00
Brian Warner
a1c97aa0be
storageserver: include metadata in the sizelimit, at least after the bucket has been closed
2007-07-03 17:38:49 -07:00
Brian Warner
c80ea7d693
storageserver: implement size limits. No code to enable them yet, though
2007-07-03 17:08:02 -07:00
Brian Warner
91d72bb504
test_storage: test StorageServer code too: allocation, lookup, multiple-writer behavior
2007-04-18 15:42:34 -07:00
