mirror of
https://github.com/tahoe-lafs/tahoe-lafs.git
synced 2025-04-08 03:14:21 +00:00
pyfec: make README.txt much more detailed and rename some internal variables and add some docstrings
This commit is contained in:
Zooko O'Whielacronx
parent
0556e2dba8
commit
afa4d8a46f
139
pyfec/README.txt
139
pyfec/README.txt
@ -1,63 +1,136 @@
|
||||
This package provides an "erasure code", or "forward error correction code".
|
||||
* Intro
|
||||
|
||||
This package implements an "erasure code", or "forward error correction code".
|
||||
It is licensed under the GNU General Public License (see the COPYING file for
|
||||
details).
|
||||
|
||||
The most widely known example of an erasure code is the RAID-5 algorithm which
|
||||
makes it so that in the event of the loss of any one hard drive, the stored
|
||||
data can be completely recovered. The algorithm in the pyfec package has a
|
||||
similar effect, but instead of recovering from the loss of any one element, it
|
||||
can be parameterized to choose in advance the number of elements whose loss it
|
||||
can recover from.
|
||||
similar effect, but instead of recovering from the loss of only a single
|
||||
element, it can be parameterized to choose in advance the number of elements
|
||||
whose loss it can tolerate.
|
||||
|
||||
This package is largely based on the old "fec" library by Luigi Rizzo et al.,
|
||||
which is a simple, fast, mature, and optimized implementation of erasure
|
||||
coding. The pyfec package makes several changes from the original "fec"
|
||||
package, including addition of the Python API, refactoring of the C API to be
|
||||
faster (for the way that I use it, at least), and a few clean-ups and
|
||||
micro-optimizations of the core code itself.
|
||||
which is a simple, mature, and optimized implementation of erasure coding. The
|
||||
pyfec package makes several changes from the original "fec" package, including
|
||||
addition of the Python API, refactoring of the C API to be faster (for the way
|
||||
that I use it, at least), and a few clean-ups and micro-optimizations of the
|
||||
core code itself.
|
||||
|
||||
|
||||
* Overview
|
||||
|
||||
This package performs two operations, encoding and decoding. Encoding takes
|
||||
some input data and expands its size by producing extra "check blocks".
|
||||
Decoding takes some blocks -- any combination of original blocks of data (also
|
||||
called "primary shares") and check blocks (also called "secondary shares"), and
|
||||
produces the original data.
|
||||
some input data and expands its size by producing extra "check blocks", also
|
||||
called "secondary shares". Decoding takes some data -- any combination of
|
||||
blocks of the original data (called "primary shares") and "secondary shares",
|
||||
and produces the original data.
|
||||
|
||||
The encoding is parameterized by two integers, k and m. m is the total number
|
||||
of shares produced, and k is how many of those shares are necessary to
|
||||
reconstruct the original data. m is required to be at least 1 and at most 255,
|
||||
and k is required to be at least 1 and at most m. (Note that when k == m then
|
||||
there is no point in doing erasure coding.)
|
||||
reconstruct the original data. m is required to be at least 1 and at most 256,
|
||||
and k is required to be at least 1 and at most m.
|
||||
|
||||
(Note that when k == m then there is no point in doing erasure coding -- it
|
||||
degenerates to the equivalent of the Unix "split" utility which simply splits
|
||||
the input into successive segments. Similarly, when k == 1 it degenerates to
|
||||
the equivalent of the unix "cp" utility -- each share is a complete copy of the
|
||||
input data.)
|
||||
|
||||
Note that each "primary share" is a segment of the original data, so its size
|
||||
is 1/k'th of the size of original data, and each "secondary share" is of the
|
||||
same size, so the total space used by all the shares is about m/k times the
|
||||
size of the original data.
|
||||
same size, so the total space used by all the shares is m/k times the size of
|
||||
the original data (plus some padding to fill out the last primary share to be
|
||||
the same size as all the others).
|
||||
|
||||
The decoding step requires as input k of the shares which were produced by the
|
||||
encoding step. The decoding step produces as output the data that was earlier
|
||||
input to the encoding step.
|
||||
|
||||
This package also includes a Python interface. See the Python docstrings for
|
||||
usage details.
|
||||
|
||||
* API
|
||||
|
||||
Each share is associated with "shareid". The shareid of each primary share is
|
||||
its index (starting from zero), so the 0'th share is the first primary share,
|
||||
which is the first few bytes of the file, the 1'st share is the next primary
|
||||
share, which is the next few bytes of the file, and so on. The last primary
|
||||
share has shareid k-1. The shareid of each secondary share is an arbitrary
|
||||
integer between k and 256 inclusive. (When using the Python API, if you don't
|
||||
specify which shareids you want for your secondary shares when invoking
|
||||
encode(), then it will by default provide the shares with ids from k to m-1
|
||||
inclusive.)
|
||||
|
||||
** C API
|
||||
|
||||
fec_encode() takes as input an array of k pointers, where each pointer points
|
||||
to a memory buffer containing the input data (i.e., the i'th buffer contains
|
||||
the i'th primary share). There is also a second parameter which is an array of
|
||||
the shareids of the secondary shares which are to be produced. (Each element
|
||||
in that array is required to be the shareid of a secondary share, i.e. it is
|
||||
required to be >= k and < m.)
|
||||
|
||||
The output from fec_encode() is the requested set of secondary shares which are
|
||||
written into output buffers provided by the caller.
|
||||
|
||||
fec_decode() takes as input an array of k pointers, where each pointer points
|
||||
to a buffer containing a share. There is also a separate input parameter which
|
||||
is an array of shareids, indicating the shareid of each of the shares which is
|
||||
being passed in.
|
||||
|
||||
The output from fec_decode() is the set of primary shares which were missing
|
||||
from the input and had to be reconstructed. These reconstructed shares are
|
||||
written into putput buffers provided by the caller.
|
||||
|
||||
** Python API
|
||||
|
||||
encode() and decode() take as input a sequence of k buffers, where a "sequence"
|
||||
is any object that implements the Python sequence protocol (such as a list or
|
||||
tuple) and a "buffer" is any object that implements the Python buffer protocol
|
||||
(such as a string or array). The contents that are required to be present in
|
||||
these buffers are the same as for the C API.
|
||||
|
||||
encode() also takes a list of desired shareids. Unlike the C API, the Python
|
||||
API accepts shareids of primary shares as well as secondary shares in its list
|
||||
of desired shareids. encode() returns a list of buffer objects which contain
|
||||
the shares requested. For each requested share which is a primary share, the
|
||||
resulting list contains a reference to the apppropriate primary share from the
|
||||
input list. For each requested share which is a secondary share, the list
|
||||
contains a newly created string object containing that share.
|
||||
|
||||
decode() also takes a list of integers indicating the shareids of the shares
|
||||
being passed int. decode() returns a list of buffer objects which contain all
|
||||
of the primary shares of the original data (in order). For each primary share
|
||||
which was present in the input list, then the result list simply contains a
|
||||
reference to the object that was passed in the input list. For each primary
|
||||
share which was not present in the input, the result list contains a newly
|
||||
created string object containing that primary share.
|
||||
|
||||
Beware of a "gotcha" that can result from the combination of mutable data and
|
||||
the fact that the Python API returns references to inputs when possible.
|
||||
|
||||
Returning references to its inputs is efficient since it avoids making an
|
||||
unnecessary copy of the data, but if the object which was passed as input is
|
||||
mutable and if that object is mutated after the call to pyfec returns, then the
|
||||
result from pyfec -- which is just a reference to that same object -- will also
|
||||
be mutated. This subtlety is the price you pay for avoiding data copying. If
|
||||
you don't want to have to worry about this then you can simply use immutable
|
||||
objects (e.g. Python strings) to hold the data that you pass to pyfec.
|
||||
|
||||
|
||||
* Utilities
|
||||
|
||||
See also the filefec.py module which has a utility function for efficiently
|
||||
reading a file and encoding it piece by piece.
|
||||
|
||||
Beware of a "gotcha" that can result from the combination of mutable buffers
|
||||
and the fact that pyfec never makes an unnecessary data copy. That is:
|
||||
whenever one of the shares produced from a call to encode() or decode() has the
|
||||
same contents as one of the shares passed as input, then pyfec will return as
|
||||
output a pointer (in the C API) or a Python reference (in the Python API) to
|
||||
the object which was passed to it as input. This is efficient as it avoids
|
||||
making an unnecessary copy of the data. But if the object which was passed as
|
||||
input is mutable and if that object is mutated after the call to pyfec returns,
|
||||
then the result from pyfec -- which is just a reference to that same object --
|
||||
will also be mutated. This subtlety is the price you pay for avoiding data
|
||||
copying. If you don't want to have to worry about this, then simply use
|
||||
immutable objects (e.g. Python strings) to hold the data that you pass to
|
||||
pyfec.
|
||||
|
||||
Enjoy!
|
||||
|
||||
Zooko Wilcox-O'Hearn
|
||||
2007-01-27
|
||||
San Francisco
|
||||
|
||||
|
||||
On Peter's fancy Intel Mac laptop, it did about 6.21 million bytes per second from file.
|
||||
On my old PowerPC G4 Mac laptop, it did around 1.3 million bytes per second from file.
|
||||
|
||||
|
||||
@ -212,7 +212,7 @@ Encoder_encode(Encoder *self, PyObject *args) {
|
||||
assert (check_share_index == num_check_shares_produced);
|
||||
|
||||
/* Encode any check shares that are needed. */
|
||||
fec_encode_all(self->fec_matrix, incshares, check_shares_produced, c_desired_shares_ids, num_desired_shares, sz);
|
||||
fec_encode(self->fec_matrix, incshares, check_shares_produced, c_desired_shares_ids, num_desired_shares, sz);
|
||||
|
||||
/* Wrap all requested shares up into a Python list of Python strings. */
|
||||
result = PyList_New(num_desired_shares);
|
||||
@ -383,14 +383,14 @@ Decode a list shares into a list of segments.\n\
|
||||
static PyObject *
|
||||
Decoder_decode(Decoder *self, PyObject *args) {
|
||||
PyObject*restrict shares;
|
||||
PyObject*restrict sharenums;
|
||||
PyObject*restrict shareids;
|
||||
PyObject* result = NULL;
|
||||
|
||||
if (!PyArg_ParseTuple(args, "O!O!", &PyList_Type, &shares, &PyList_Type, &sharenums))
|
||||
if (!PyArg_ParseTuple(args, "O!O!", &PyList_Type, &shares, &PyList_Type, &shareids))
|
||||
return NULL;
|
||||
|
||||
const gf*restrict cshares[self->kk];
|
||||
unsigned char csharenums[self->kk];
|
||||
unsigned char cshareids[self->kk];
|
||||
gf*restrict recoveredcstrs[self->kk]; /* self->kk is actually an upper bound -- we probably won't need all of this space. */
|
||||
PyObject*restrict recoveredpystrs[self->kk]; /* self->kk is actually an upper bound -- we probably won't need all of this space. */
|
||||
unsigned i;
|
||||
@ -399,38 +399,38 @@ Decoder_decode(Decoder *self, PyObject *args) {
|
||||
PyObject*restrict fastshares = PySequence_Fast(shares, "First argument was not a sequence.");
|
||||
if (!fastshares)
|
||||
goto err;
|
||||
PyObject*restrict fastsharenums = PySequence_Fast(sharenums, "Second argument was not a sequence.");
|
||||
if (!fastsharenums)
|
||||
PyObject*restrict fastshareids = PySequence_Fast(shareids, "Second argument was not a sequence.");
|
||||
if (!fastshareids)
|
||||
goto err;
|
||||
|
||||
if (PySequence_Fast_GET_SIZE(fastshares) != self->kk) {
|
||||
py_raise_fec_error("Precondition violation: Wrong length -- first argument is required to contain exactly k shares. len(first): %d, k: %d", PySequence_Fast_GET_SIZE(fastshares), self->kk);
|
||||
goto err;
|
||||
}
|
||||
if (PySequence_Fast_GET_SIZE(fastsharenums) != self->kk) {
|
||||
py_raise_fec_error("Precondition violation: Wrong length -- sharenums is required to contain exactly k shares. len(sharenums): %d, k: %d", PySequence_Fast_GET_SIZE(fastsharenums), self->kk);
|
||||
if (PySequence_Fast_GET_SIZE(fastshareids) != self->kk) {
|
||||
py_raise_fec_error("Precondition violation: Wrong length -- shareids is required to contain exactly k shares. len(shareids): %d, k: %d", PySequence_Fast_GET_SIZE(fastshareids), self->kk);
|
||||
goto err;
|
||||
}
|
||||
|
||||
/* Construct a C array of gf*'s of the data and another of C ints of the sharenums. */
|
||||
/* Construct a C array of gf*'s of the data and another of C ints of the shareids. */
|
||||
unsigned needtorecover=0;
|
||||
PyObject** fastsharenumsitems = PySequence_Fast_ITEMS(fastsharenums);
|
||||
if (!fastsharenumsitems)
|
||||
PyObject** fastshareidsitems = PySequence_Fast_ITEMS(fastshareids);
|
||||
if (!fastshareidsitems)
|
||||
goto err;
|
||||
PyObject** fastsharesitems = PySequence_Fast_ITEMS(fastshares);
|
||||
if (!fastsharesitems)
|
||||
goto err;
|
||||
int sz, oldsz = 0;
|
||||
for (i=0; i<self->kk; i++) {
|
||||
if (!PyInt_Check(fastsharenumsitems[i]))
|
||||
if (!PyInt_Check(fastshareidsitems[i]))
|
||||
goto err;
|
||||
long tmpl = PyInt_AsLong(fastsharenumsitems[i]);
|
||||
long tmpl = PyInt_AsLong(fastshareidsitems[i]);
|
||||
if (tmpl < 0 || tmpl >= UCHAR_MAX) {
|
||||
py_raise_fec_error("Precondition violation: Share ids can't be less than zero or greater than 255. %ld\n", tmpl);
|
||||
goto err;
|
||||
}
|
||||
csharenums[i] = (unsigned char)tmpl;
|
||||
if (csharenums[i] >= self->kk)
|
||||
cshareids[i] = (unsigned char)tmpl;
|
||||
if (cshareids[i] >= self->kk)
|
||||
needtorecover+=1;
|
||||
|
||||
if (!PyObject_CheckReadBuffer(fastsharesitems[i])) {
|
||||
@ -448,13 +448,13 @@ Decoder_decode(Decoder *self, PyObject *args) {
|
||||
|
||||
/* move src packets into position */
|
||||
for (i=0; i<self->kk;) {
|
||||
if (csharenums[i] >= self->kk || csharenums[i] == i)
|
||||
if (cshareids[i] >= self->kk || cshareids[i] == i)
|
||||
i++;
|
||||
else {
|
||||
/* put pkt in the right position. */
|
||||
unsigned char c = csharenums[i];
|
||||
unsigned char c = cshareids[i];
|
||||
|
||||
SWAP (csharenums[i], csharenums[c], int);
|
||||
SWAP (cshareids[i], cshareids[c], int);
|
||||
SWAP (cshares[i], cshares[c], gf*);
|
||||
SWAP (fastsharesitems[i], fastsharesitems[c], PyObject*);
|
||||
}
|
||||
@ -471,7 +471,7 @@ Decoder_decode(Decoder *self, PyObject *args) {
|
||||
}
|
||||
|
||||
/* Decode any recovered shares that are needed. */
|
||||
fec_decode_all(self->fec_matrix, cshares, recoveredcstrs, csharenums, sz);
|
||||
fec_decode(self->fec_matrix, cshares, recoveredcstrs, cshareids, sz);
|
||||
|
||||
/* Wrap up both original primary shares and decoded shares into a Python list of Python strings. */
|
||||
unsigned nextrecoveredix=0;
|
||||
@ -479,7 +479,7 @@ Decoder_decode(Decoder *self, PyObject *args) {
|
||||
if (result == NULL)
|
||||
goto err;
|
||||
for (i=0; i<self->kk; i++) {
|
||||
if (csharenums[i] == i) {
|
||||
if (cshareids[i] == i) {
|
||||
/* Original primary share. */
|
||||
Py_INCREF(fastsharesitems[i]);
|
||||
if (PyList_SetItem(result, i, fastsharesitems[i]) == -1) {
|
||||
@ -502,7 +502,7 @@ Decoder_decode(Decoder *self, PyObject *args) {
|
||||
Py_XDECREF(result); result = NULL;
|
||||
cleanup:
|
||||
Py_XDECREF(fastshares); fastshares=NULL;
|
||||
Py_XDECREF(fastsharenums); fastsharenums=NULL;
|
||||
Py_XDECREF(fastshareids); fastshareids=NULL;
|
||||
return result;
|
||||
}
|
||||
|
||||
|
||||
@ -580,7 +580,7 @@ fec_new (unsigned char k, unsigned char n) {
|
||||
}
|
||||
|
||||
void
|
||||
fec_encode_all(const fec_t* code, const gf*restrict const*restrict const src, gf*restrict const*restrict const fecs, const unsigned char*restrict const share_ids, unsigned char num_share_ids, size_t sz) {
|
||||
fec_encode(const fec_t* code, const gf*restrict const*restrict const src, gf*restrict const*restrict const fecs, const unsigned char*restrict const share_ids, unsigned char num_share_ids, size_t sz) {
|
||||
unsigned i, j;
|
||||
unsigned char fecnum;
|
||||
gf* p;
|
||||
@ -601,7 +601,7 @@ fec_encode_all(const fec_t* code, const gf*restrict const*restrict const src, gf
|
||||
#if 0
|
||||
/* By turning the nested loop inside out, we might incur different cache usage and therefore go slower or faster. However in practice I'm not able to detect a difference, since >90% of the time is spent in my Python test script anyway. :-) */
|
||||
void
|
||||
fec_encode_all(const fec_t* code, const gf*restrict const*restrict const src, gf*restrict const*restrict const fecs, const unsigned char*restrict const share_ids, unsigned char num_share_ids, size_t sz) {
|
||||
fec_encode(const fec_t* code, const gf*restrict const*restrict const src, gf*restrict const*restrict const fecs, const unsigned char*restrict const share_ids, unsigned char num_share_ids, size_t sz) {
|
||||
for (unsigned j=0; j < code->k; j++) {
|
||||
unsigned fecs_ix = 0; /* index into the fecs array */
|
||||
for (unsigned i=0; i<num_share_ids; i++) {
|
||||
@ -639,7 +639,7 @@ build_decode_matrix_into_space(const fec_t*restrict const code, const unsigned c
|
||||
}
|
||||
|
||||
void
|
||||
fec_decode_all(const fec_t* code, const gf*restrict const*restrict const inpkts, gf*restrict const*restrict const outpkts, const unsigned char*restrict const index, size_t sz) {
|
||||
fec_decode(const fec_t* code, const gf*restrict const*restrict const inpkts, gf*restrict const*restrict const outpkts, const unsigned char*restrict const index, size_t sz) {
|
||||
gf m_dec[code->k * code->k];
|
||||
build_decode_matrix_into_space(code, index, code->k, m_dec);
|
||||
|
||||
|
||||
@ -85,17 +85,17 @@ fec_t *fec_new (unsigned char k, unsigned char n);
|
||||
/**
|
||||
* @param inpkts the "primary shares" i.e. the chunks of the input data
|
||||
* @param fecs buffers into which the secondary shares will be written
|
||||
* @param share_ids the numbers of the desired shares -- including both primary shares (the id < k) which fec_encode_all() ignores and check shares (the id >= k) which fec_encode_all() will produce and store into the buffers of the fecs parameter
|
||||
* @param share_ids the numbers of the desired shares -- including both primary shares (the id < k) which fec_encode() ignores and check shares (the id >= k) which fec_encode() will produce and store into the buffers of the fecs parameter
|
||||
* @param num_share_ids the length of the share_ids array
|
||||
*/
|
||||
void fec_encode_all(const fec_t* code, const gf*restrict const*restrict const src, gf*restrict const*restrict const fecs, const unsigned char*restrict const share_ids, unsigned char num_share_ids, size_t sz);
|
||||
void fec_encode(const fec_t* code, const gf*restrict const*restrict const src, gf*restrict const*restrict const fecs, const unsigned char*restrict const share_ids, unsigned char num_share_ids, size_t sz);
|
||||
|
||||
/**
|
||||
* @param inpkts an array of packets (size k)
|
||||
* @param outpkts an array of buffers into which the output packets will be written
|
||||
* @param outpkts an array of buffers into which the reconstructed output packets will be written (only packets which are not present in the inpkts input will be reconstructed and written to outpkts)
|
||||
* @param index an array of the shareids of the packets in inpkts
|
||||
* @param sz size of a packet in bytes
|
||||
*/
|
||||
void fec_decode_all(const fec_t* code, const gf*restrict const*restrict const inpkts, gf*restrict const*restrict const outpkts, const unsigned char*restrict const index, size_t sz);
|
||||
void fec_decode(const fec_t* code, const gf*restrict const*restrict const inpkts, gf*restrict const*restrict const outpkts, const unsigned char*restrict const index, size_t sz);
|
||||
|
||||
/* end of file */
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user