Brian Warner
3209fd5e09
rearrange encode/upload, add URIs, switch to ReplicatingEncoder
...
Added metadata to the bucket store, which is used to hold the share number
(but the bucket doesn't know that, it just gets a string).
Modified the codec interfaces a bit.
Try to pass around URIs to/from download/upload instead of verifierids.
URI format is still in flux.
Change the current (primitive) file encoder to use a ReplicatingEncoder
because it provides ICodecEncoder. We will be moving to the (less primitive)
file encoder (currently in allmydata.encode_new) eventually, but for now
this change lets us test out PyRS or zooko's upcoming C-based RS codec in
something larger than a single unit test. This primitive file encoder only
uses a single segment, and has no merkle trees.
Also added allmydata.util.deferredutil for a DeferredList wrapper that
errbacks (but only when all component Deferreds have fired) if there were
any errors, which unfortunately is not a behavior available from the standard
DeferredList.
2007-01-15 21:22:22 -07:00
Brian Warner
78a9e815c5
add simple metadata (a single string) to the storage protocol
2007-01-15 14:01:22 -07:00
Brian Warner
e29032bc36
change IEncoder to ICodecEncoder, to match the previous change
2007-01-11 20:57:14 -07:00
Brian Warner
417c17755b
use the word 'codec' for erasure coding, for now. 'encode' is used for file-level segmentation/hashing
2007-01-11 20:51:27 -07:00
Brian Warner
a65608fec1
sendOnly: oops, I keep forgetting that you can't really use f.trap in a lambda, because it returns the failure it trapped
2007-01-09 21:13:05 -07:00
Brian Warner
ceda350892
test_system.py: exercise queen.Roster._lost_node too
2007-01-09 19:40:36 -07:00
Brian Warner
aad0a9dfac
workqueue: more tests
2007-01-09 01:58:50 -07:00
Brian Warner
d61c0a6ef6
workqueue: more tests, coverage now at 63.4%, yay
2007-01-09 01:36:12 -07:00
Brian Warner
b641f6cbc7
workqueue: more improvements, more tests
2007-01-08 22:29:42 -07:00
Brian Warner
b9edb02820
workqueue: start adding tests
2007-01-08 21:29:33 -07:00
Brian Warner
2567a26519
rename the new filetable code to 'filetree', since robk astutely pointed out that 'table' is misleading and implies a flat list of files in a single directory
2007-01-08 16:29:14 -07:00
Brian Warner
5a34753cdb
fix pyflakes warning in debugshell, by providing a dummy value for 'app' that will be overwritten when the manhole connection is established
2007-01-05 21:15:59 -07:00
Brian Warner
9382d61208
use absolute import of 'allmydata.Crypto' rather than a relative import of just 'Crypto', to make it clear that we're using our own form rather than relying upon the system version
2007-01-05 21:12:26 -07:00
Brian Warner
94b1840295
workqueue: fix pyflakes warnings, code is still quite incomplete
2007-01-05 21:11:19 -07:00
Brian Warner
2566452091
test_encode_share.py: fix some pyflakes warnings
2007-01-05 18:53:22 -07:00
Brian Warner
4ba6f47d2c
add some (disabled) encoder benchmarking code
2007-01-05 18:46:03 -07:00
Brian Warner
cf04073e98
change Encoder to use py_ecc, now that it works
2007-01-05 18:12:45 -07:00
Brian Warner
42c0d2e336
disable figleaf tracing during py_ecc, since it takes *forever*, especially on the slow buildslave
2007-01-05 18:12:04 -07:00
Brian Warner
3d3a7a5b8d
encode.py: add some timing comments
2007-01-05 00:48:42 -07:00
Brian Warner
409d92e746
only run a single (short) py_ecc test on slave3, since it is so slow the tests timeout
2007-01-05 00:42:52 -07:00
Brian Warner
e1c6ee9dcf
encoding: fix the last py_ecc problem, tests pass now
2007-01-05 00:06:42 -07:00
Brian Warner
436409fa4d
change encode_new to use IEncoder
2007-01-04 23:51:35 -07:00
Brian Warner
c91d14dca8
fix our use of py_ecc (set log2FieldSize=8 explicitly)
2007-01-04 23:50:21 -07:00
Brian Warner
f31fc06d89
establish IEncoder/IDecoder, create suitable interfaces for both the simple replicating encoder and the py_ecc one, add a (failing) unit test for it
2007-01-04 21:52:51 -07:00
Brian Warner
fa11c3f722
more pyflakes cleanups
2007-01-04 18:06:20 -07:00
Brian Warner
6587645f76
fix pyflakes warnings in figleaf
2007-01-04 18:04:43 -07:00
Zooko O'Whielacronx
5ad6c43622
add padding to segments and convert type from string to list of integers before passing to py_ecc
...
But then I get an error that I don't understand about ECC math...
2007-01-04 17:58:14 -07:00
Brian Warner
7b2268ab39
improve figleaf: fix some line-numbering bugs, add an alphabetically-sorted index HTML page
2007-01-04 01:26:43 -07:00
Brian Warner
8a78065f16
modify figleaf2html to show module names instead of .py filenames, also add a --root argument to restrict coverage to a specific parent directory
2007-01-03 22:23:25 -07:00
Brian Warner
e8ee3365ef
move figleaf2html and figleaf_htmlizer.py into our tree, for easier customization
2007-01-03 22:06:51 -07:00
Brian Warner
f97eec893e
figleaf: move a copy into allmydata.util.figleaf, update Makefile/trial stuff
2007-01-03 21:38:29 -07:00
Brian Warner
e600571f82
checkpoint work-in-progress for WorkQueue, a disk-persistent list of work to be done
2007-01-01 23:47:16 -07:00
Brian Warner
a6678b9a37
filetable_new: import py_ecc, make the tests get a little bit further
2007-01-01 23:44:12 -07:00
Zooko O'Whielacronx
b0315fc549
make encode_new use py_ecc for real live erasure coding
...
(This patch is not tested -- I'm working on a Mac which doesn't have gcc
installed... (gcc is necessary for the crypto module.) I will now attempt to
connect to a better development computer over my mom's staticy, failure-prone,
14.4 K modem...)
2006-12-29 13:50:53 -07:00
Zooko O'Whielacronx
f3f91105bd
import mathutil from pyutil
2006-12-29 13:50:42 -07:00
Zooko O'Whielacronx
dcfddcac41
encode_new: fix a bug in subshare hashes and tidy-up a couple of things
2006-12-29 12:40:10 -07:00
Brian Warner
cf53abab42
more filetable_new tests
2006-12-25 00:56:18 -07:00
Brian Warner
3e085efa7c
filetable_new: fix the test
2006-12-24 12:47:03 -07:00
Brian Warner
70f5c13e26
checkpointing new filetable work.. tests don't pass yet
2006-12-24 12:39:24 -07:00
Brian Warner
56ff0e2b57
filetable: shuffle lines a bit to appease figleaf's confusion
2006-12-14 04:33:57 -07:00
Brian Warner
eb3b9b16af
encode: start to fix a few problems, still a lot of work left to go
2006-12-14 04:31:17 -07:00
Brian Warner
3e5e2ee9fe
make initial simple encode_new test pass
2006-12-14 04:17:01 -07:00
Brian Warner
3490378551
move all packages into src/, fix allmydata.Crypto build. Now you must perform a 'setup.py build' before using anything, and you must add the build directory (build/lib.linux-i686-2.4) to your PYTHONPATH before doing anything
2006-12-14 03:39:50 -07:00
