ExternalVendorCode/tahoe-lafs
1
0
Fork 0
mirror of https://github.com/tahoe-lafs/tahoe-lafs.git synced 2024-12-28 16:58:53 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity
7,325 Commits 185 Branches 90 Tags 89 MiB
6b16afaa2e
Commit Graph

396 Commits

Author SHA1 Message Date
david-sarah
752897450d fileutil: docstrings for non-obvious usage restrictions on methods of EncryptedTemporaryFile. 2010-07-16 22:46:47 -07:00
david-sarah
05022dca36 Move EncryptedTemporaryFile from SFTP frontend to allmydata.util.fileutil, and make the FTP frontend also use it (fixing #1083). 2010-07-11 14:37:21 -07:00
david-sarah
a8161c915a Correct stringutils->encodingutil patch to be the newer version, rather than the old version that was committed in error. 2010-07-17 18:34:35 -07:00
david-sarah
11077ea74d Rename stringutils to encodingutil, and drop listdir_unicode and open_unicode (since the Python stdlib functions work fine with Unicode paths). Also move some utility functions to fileutil. 2010-07-11 17:30:15 -07:00
david-sarah
b8df25a8d1 stringutils.py: tolerate sys.stdout having no 'encoding' attribute. 2010-06-25 21:08:17 -07:00
david-sarah
7e7644589a stringutils.py: remove unused import. 2010-06-16 20:44:40 -07:00
david-sarah
9f5488b2d1 stringutils.py: don't NFC-normalize the output of listdir_unicode. 2010-06-16 18:55:37 -07:00
david-sarah
e2c7ad1d88 stringutils.py: Add encoding argument to quote_output. Also work around a bug in locale.getpreferredencoding on older Pythons. 2010-06-15 21:20:12 -07:00
david-sarah
ffd1c5aca5 Fix for Unicode-related test failures on Zooko's OS X 10.6 machine. 2010-06-08 22:54:48 -07:00
david-sarah
80a89d3931 stringutils.py, sftpd.py: Portability fixes for Python <= 2.5. 2010-06-08 18:33:02 -07:00
david-sarah
32d9deace3 Back out Windows-specific Unicode argument support for v1.7. 2010-06-08 17:08:03 -07:00
david-sarah
80252debcd Unicode fixes. 2010-06-06 18:02:15 -07:00
Zooko O'Whielacronx
db8a6f3aa6 unicode: make test_cli test a non-ascii argument, and make the fallback term encoding be locale.getpreferredencoding() 2010-06-04 07:12:51 -07:00
Francois Deppierraz
d0ed14e1bb stringutils.py: Unicode helper functions + associated tests 
This file contains a bunch of helper functions which converts
unicode string from and to argv, filenames and stdout.
 2010-05-19 17:41:05 -07:00
david-sarah
e76092e16c Change relative imports to absolute 2010-02-26 01:14:33 -07:00
Kevan Carstensen
e225f573b9 Fix up the behavior of #778, per reviewers' comments 
- Make some important utility functions clearer and more thoroughly 
    documented.
  - Assert in upload.servers_of_happiness that the buckets attributes
    of PeerTrackers passed to it are mutually disjoint.
  - Get rid of some silly non-Pythonisms that I didn't see when I first
    wrote these patches.
  - Make sure that should_add_server returns true when queried about a 
    shnum that it doesn't know about yet.
  - Change Tahoe2PeerSelector.preexisting_shares to map a shareid to a set
    of peerids, alter dependencies to deal with that.
  - Remove upload.should_add_servers, because it is no longer necessary
  - Move upload.shares_of_happiness and upload.shares_by_server to a utility
    file.
  - Change some points in Tahoe2PeerSelector.
  - Compute servers_of_happiness using a bipartite matching algorithm that 
    we know is optimal instead of an ad-hoc greedy algorithm that isn't.
  - Change servers_of_happiness to just take a sharemap as an argument,
    change its callers to merge existing_shares and used_peers before 
    calling it.
  - Change an error message in the encoder to be more appropriate for 
    servers of happiness.
  - Clarify the wording of an error message in immutable/upload.py
  - Refactor a happiness failure message to happinessutil.py, and make
    immutable/upload.py and immutable/encode.py use it.
  - Move the word "only" as far to the right as possible in failure 
    messages.
  - Use a better definition of progress during peer selection.
  - Do read-only peer share detection queries in parallel, not sequentially.
  - Clean up logging semantics; print the query statistics whenever an
    upload is unsuccessful, not just in one case.
 2010-05-13 17:49:17 -07:00
Zooko O'Whielacronx
80ae6f2706 setup: fix error in licensing declaration from Shawn Willden, as noted by the FSF compliance division 2010-03-09 08:37:36 -08:00
david-sarah
c984a09fe7 Fix race conditions and missing callback in allmydata.test.test_cli.Cp.test_copy_using_filecap, add utilities for one-liner reading and writing of files, and fix cases in test_cli where files were not being closed after writing. 2010-02-05 17:37:27 -08:00
Brian Warner
731d15e56f hush pyflakes-0.4.0 warnings: remove trivial unused variables. For #900. 2010-01-14 14:15:29 -08:00
Brian Warner
d888bf3377 Clean up log.err calls, for one of the issues in #889. 
allmydata.util.log.err() either takes a Failure as the first positional
argument, or takes no positional arguments and must be invoked in an
exception handler. Fixed its signature to match both foolscap.logging.log.err
and twisted.python.log.err . Included a brief unit test.
 2010-01-11 17:33:43 -08:00
Brian Warner
bacb6fe5aa tidy up DeadReferenceError handling, ignore them in add_lease calls 
Stop checking separately for ConnectionDone/ConnectionLost, since those have
been folded into DeadReferenceError since foolscap-0.3.1 . Write
rrefutil.trap_deadref() in terms of rrefutil.trap_and_discard() to improve
code coverage.
 2010-01-11 16:07:23 -08:00
Brian Warner
a4a6c02ef8 more #859: avoid deprecation warning for unit tests too, hush pyflakes 
* factor maybe-import-sha logic into util.hashutil
 2009-12-14 16:01:47 -08:00
Zooko O'Whielacronx
4706859967 setup: ignore _darcs in the "test-clean" test and make the "clean" step remove all .egg's in the root dir 2009-12-06 10:48:35 -08:00
Brian Warner
96834da0a2 Simplify immutable download API: use just filenode.read(consumer, offset, size) 
* remove Downloader.download_to_data/download_to_filename/download_to_filehandle
* remove download.Data/FileName/FileHandle targets
* remove filenode.download/download_to_data/download_to_filename methods
* leave Downloader.download (the whole Downloader will go away eventually)
* add util.consumer.MemoryConsumer/download_to_data, for convenience
  (this is mostly used by unit tests, but it gets used by enough non-test
   code to warrant putting it in allmydata.util)
* update tests
* removes about 180 lines of code. Yay negative code days!

Overall plan is to rewrite immutable/download.py and leave filenode.read() as
the sole read-side API.
 2009-12-01 17:53:30 -05:00
Brian Warner
9901a08fd3 Add dirnodes to backupdb and "tahoe backup", closes #606. 
* backups now share dirnodes with any previous backup, in any location,
  so renames and moves are handled very efficiently
* "tahoe backup" no longer bothers reading the previous snapshot
* if you switch grids, you should delete ~/.tahoe/private/backupdb.sqlite,
  to force new uploads of all files and directories
 2009-11-26 15:42:57 -08:00
Brian Warner
41bcc9f39e PipelineError: fix str() on python2.4 . Closes #842. 2009-11-24 13:25:12 -08:00
Brian Warner
ea373de042 move dirnode.CachingDict to dictutil.AuxValueDict, generalize method names, 
improve tests. Let dirnode _pack_children accept either dict or AuxValueDict.
 2009-10-17 11:00:05 -07:00
Zooko O'Whielacronx
00a44c057a docs: a few licensing clarifications requested by Ubuntu 2009-09-26 20:32:26 -07:00
Zooko O'Whielacronx
aaaa633f18 setup: remove a convenience copy of figleaf, to ease inclusion into Ubuntu Karmic Koala 
We need to carefully document the licence of figleaf in order to get Tahoe-LAFS into Ubuntu Karmic Koala.  However, figleaf isn't really a part of Tahoe-LAFS per se -- this is just a "convenience copy" of a development tool.  The quickest way to make Tahoe-LAFS acceptable for Karmic then, is to remove figleaf from the Tahoe-LAFS tarball itself.  People who want to run figleaf on Tahoe-LAFS (as everyone should want) can install figleaf themselves.  I haven't tested this -- there may be incompatibilities between upstream figleaf and the copy that we had here...
 2009-09-23 22:32:15 -07:00
Zooko O'Whielacronx
9f0da4012e docs: Shawn Willden grants permission for his contributions under GPL2+|TGPPL1+ 2009-09-21 09:46:51 -07:00
Zooko O'Whielacronx
603e08c4a9 doc: licensing cleanups 
Use nice utf-8 © instead of "(c)". Remove licensing statements on utility modules that have been assigned to allmydata.com by their original authors. (Nattraverso was not assigned to allmydata.com -- it was LGPL'ed -- but I checked and src/allmydata/util/iputil.py was completely rewritten and doesn't contain any line of code from nattraverso.)  Add notes to misc/debian/copyright about licensing on files that aren't just allmydata.com-licensed.
 2009-09-20 10:16:31 -07:00
Zooko O'Whielacronx
2a63fc9159 docs: update relnotes.txt, relnotes-short.txt, and others documentation bits for v1.5.0 release! 2009-08-01 19:57:10 -07:00
Zooko O'Whielacronx
b76ca4549a iputil: added "openbsd3" => "bsd" and "openbsd4" => "bsd" per tdelaet's report; should fix ticket #768 2009-07-20 06:13:14 -07:00
Zooko O'Whielacronx
7607268a58 trivial: remove obsolete comment text 2009-07-20 06:08:43 -07:00
Zooko O'Whielacronx
2f704ed001 dirnode: finish renaming "iv" to "salt" in the code and the hash tag 2009-07-12 17:13:20 -07:00
Zooko O'Whielacronx
786ed012b3 directories: make the IV for the writecaps in directory entries be computed from the secure hash of the writecap itself 
This makes encoding of directory entries deterministic, and it is also a tad faster on Macbook Pro than getting a random IV with os.urandom(16).
 2009-07-04 19:48:15 -07:00
Zooko O'Whielacronx
fcedca966e tests: raise the timeout on pollmixin from 100s to 1000s -- it looks like it may have triggered too eagerly on Zandr's 266 MHz armel 2009-07-08 21:47:29 -07:00
Zooko O'Whielacronx
efafcfb91a directories: keep track of your position as you decode netstring after netstring from an input buffer instead of copying the trailing part 
This makes decoding linear in the number of netstrings instead of O(N^2).
 2009-07-04 19:51:09 -07:00
Brian Warner
c6ae255847 remove trailing whitespace 2009-06-29 13:03:58 -07:00
Brian Warner
dddc2d0378 remove trailing whitespace from some util classes 2009-06-29 13:03:41 -07:00
midnightmagic
96cda877ff Basically just a trivial platform detection patch for NetBSD. 2009-06-17 16:36:47 -07:00
Brian Warner
aa23ff9180 rrefutil: add trap_deafref utility, to make the callRemote-plus-ignore-DeadReferenceError-plug-log-other-errors pattern easier 2009-06-22 19:08:26 -07:00
Brian Warner
699510c8f1 PollMixin: snoop trial's error observer to halt the test early if an error is seen. This turns a lot of timeouts into fast failures. 2009-06-22 19:07:31 -07:00
Zooko O'Whielacronx
cc2953e663 util: hooray! A clean implementation of this simple utility! Black Dew pointed out that the inverse of time.gmtime() is hidden in the "calendar" module. 2009-06-13 09:01:12 -07:00
Zooko O'Whielacronx
45928315f6 util: Brian's horrible hack to figure out how much localtime and utctime differ. Now we'll see if it works on Windows. 2009-06-12 13:45:56 -07:00
Zooko O'Whielacronx
4a30c5899c util: oops, time.tzset() doesn't work on Windows -- hopefully the new "London" unit test passes on Windows when we skip tzset() on platforms that don't have it 2009-06-11 17:09:20 -07:00
Zooko O'Whielacronx
8978cb0738 util: fix time_format.iso_utc_time_to_seconds() so that it works even in London 2009-06-11 15:11:29 -07:00
Brian Warner
1863aee0aa switch to using RemoteException instead of 'wrapped' RemoteReferences. Should fix #653, the rref-EQ problem 2009-05-21 17:46:32 -07:00
Brian Warner
c9803d5217 switch all foolscap imports to use foolscap.api or foolscap.logging 2009-05-21 17:38:23 -07:00
Brian Warner
e76c6b606f util/pipeline.py: new utility class to manage size-limited work pipelines, for #392 2009-05-18 16:43:26 -07:00
Zooko O'Whielacronx
9729753692 dirnode: add 'tahoe'/'linkcrtime' and 'tahoe'/'linkmotime' to take the place of what 'mtime'/'ctime' originally did, and make the 'tahoe' subdict be unwritable through the set_children API 
Also add extensive documentation in docs/frontends/webapi.txt about the behaviors of these values.  See ticket #628.
 2009-04-11 15:52:05 -07:00
Zooko O'Whielacronx
3ac5f427ba trivial: source code metadata 2009-04-03 16:33:15 -07:00
Zooko O'Whielacronx
b12a7f9ee8 leases, time_format: modify time stamping in lease description 
* emit lease expiry date in ISO-8601'ish format as well as Brian's format
 * rename iso_utc_time_to_localseconds() to iso_utc_time_to_seconds()
 * add iso_utc_date()
 * simplify the body of iso_utc_time_to_seconds()
 2009-04-03 15:59:04 -07:00
Brian Warner
01e2032669 hashutil: add constant-time comparison function, to avoid timing attacks when python's short-circuiting data-dependent == operator is used to, say, check a write-enabler 2009-03-22 20:20:55 -07:00
Brian Warner
66fe8ca24c parse_date: insist that it returns an int 2009-03-19 11:44:43 -07:00
Brian Warner
babcf632da util/time_format: new routine to parse dates like 2009-03-18, switch expirer to use it. I'd prefer to use 18-Mar-2009, but it is surprisingly non-trivial to build a parser that will take UTC dates instead of local dates 2009-03-18 17:58:14 -07:00
Brian Warner
406fdba61f add utility function to parse durations, for lease-expiration config 2009-03-17 00:01:17 -07:00
Brian Warner
8c3013c4f7 rrefutil: add check_remote utility function 2009-02-27 00:59:57 -07:00
Brian Warner
1b3e635936 rrefutil: add trap_remote utility and friends 2009-02-27 00:55:24 -07:00
Brian Warner
89f041ac83 fileutil: add move_into_place(), to perform the standard unix trick of atomically replacing a file, with a fallback for windows 2009-02-18 23:13:10 -07:00
Brian Warner
191fd8556a tests: fix no_network framework to work with upload/download and checker 2009-02-16 17:19:47 -07:00
Brian Warner
e1380b132b util/statistics: add tests, fix mean_repair_cost 2009-02-15 16:23:26 -07:00
Shawn Willden
51ab76875b More lossmodel work, on repair. 2009-01-15 20:56:48 -07:00
Shawn Willden
21832280da Statistics module 
Added a statistics module for calculating various facets of
share survival statistics.
 2009-01-13 20:12:35 -07:00
Brian Warner
4aee51ff6f figleaf_htmlizer: fix order of summary counters 2009-02-13 09:57:53 -07:00
Brian Warner
aee319e236 figleaf_htmlizer: oops, re-ignore files that aren't under root, like code in auto-built eggs 2009-02-13 00:00:22 -07:00
Brian Warner
30724593fc figleaf_htmlizer: emit stats to stdout, so buildbot can see it 2009-02-12 15:10:20 -07:00
Brian Warner
781c271b82 figleaf_htmlizer: render changes in coverage relative to a previous test run using --old-coverage 2009-02-12 15:04:12 -07:00
Brian Warner
125bf09528 figleaf_htmlizer: more rearranging, behavior should still be unchanged 2009-02-11 20:05:15 -07:00
Brian Warner
e02ecec1ad figleaf_htmlizer: break it up into subfunctions, behavior should still be unchanged 2009-02-11 19:56:07 -07:00
Brian Warner
6bde1b0345 figleaf_htmlizer: rewrite in class form, behavior should be the same as before 2009-02-11 19:40:50 -07:00
Brian Warner
6db747b1f0 figleaf_htmlizer: rewrite with twisted.python.usage, remove logging: should behave the same as before 2009-02-11 19:16:43 -07:00
Brian Warner
f3ed579e74 figleaf_htmlizer: expand tabs, fix to 4-space indents. No functional changes. 2009-02-11 19:05:42 -07:00
Zooko O'Whielacronx
9247dc70e5 trivial: whitespace 2009-01-18 10:54:58 -07:00
Zooko O'Whielacronx
e449052a17 util: dictutil: add DictOfSets.union(key, values) and DictOfSets.update(otherdictofsets) 2009-01-12 10:55:39 -07:00
Zooko O'Whielacronx
c01cfc0035 util: log: allow empty msgs (because downloader is using the "format" alternative with no "msg" argument) 2009-01-07 11:54:11 -07:00
Zooko O'Whielacronx
2122ba78b3 util: deferredutil: undo my recent patch to use our own implementation of gatherResults 
It seems to cause lots of failures on some builders.
 2009-01-07 11:00:05 -07:00
Zooko O'Whielacronx
79535e4f5b util: deferredutil: implement our own gatherResults instead of using Twisted's 
Because we want to maintain backwards compatibility to Twisted 2.4.0.
 2009-01-07 10:32:07 -07:00
Zooko O'Whielacronx
78306cf7fa trivial: M-x whitespace-cleanup 2009-01-07 10:25:28 -07:00
Zooko O'Whielacronx
8e92dd1231 trivial: whitespace cleanup 2009-01-06 11:20:58 -07:00
Zooko O'Whielacronx
d4a5f9131b util: base32: require str-not-unicode inputs -- effectively rolls back [3306] and [3307] 2009-01-06 10:41:22 -07:00
Zooko O'Whielacronx
5d5e89d96d util: add gatherResults which is a deferred-list-like thing that doesn't wrap failures in a FirstError 2009-01-04 10:52:02 -07:00
Zooko O'Whielacronx
b8904e023c trivial: remove unused import -- thanks, pyflakes 2008-12-31 15:25:56 -07:00
Zooko O'Whielacronx
c00d20361f rrefutil: generically wrap any errback from callRemote() in a ServerFailure instance 
This facilitates client code to easily catch ServerFailures without also catching exceptions arising from client-side code.
See also:
http://foolscap.lothar.com/trac/ticket/105 # make it easy to distinguish server-side failures/exceptions from client-side
 2008-12-31 14:28:30 -07:00
Zooko O'Whielacronx
ce5effbedf util/base32: the identity trans table needn't have any contents -- we are using string.translate solely to delete known chars 2008-12-22 17:48:08 -07:00
Zooko O'Whielacronx
3a26a4bd58 util/base32: allow unicode inputs to a2b() or could_be_base32_encoded(), and encode them with utf-8 before processing them 2008-12-22 17:47:13 -07:00
Zooko O'Whielacronx
dde2376c4e util/base32: loosen the precondition forbidding unicode and requiring str -- now it requires either unicode or str 
Hopefully this will make it so that tests pass with François Deppierraz's patch to fix the tahoe cli's handling of unicode argument.
 2008-12-22 16:22:37 -07:00
Zooko O'Whielacronx
f1fbd4feae dirnode: don't check MAC on entries in dirnodes 
In an ancient version of directories, we needed a MAC on each entry.  In modern times, the entire dirnode comes with a digital signature, so the MAC on each entry is redundant.
With this patch, we no longer check those MACs when reading directories, but we still produce them so that older readers will accept directories that we write.
 2008-12-21 17:35:18 -07:00
Zooko O'Whielacronx
2b2c2a566b util: logging: refactor some common logging behavior into mixins 2008-12-16 17:38:07 -07:00
Zooko O'Whielacronx
5fb44a3e79 pyutil: assertutil: copy in simplified assertutil from pyutil 2008-12-16 17:37:45 -07:00
Zooko O'Whielacronx
e29e4c04d9 pyutil: assertutil: simplify handling of exception during formatting of precondition message, and reduce dependency to just the Python Standard Library's logging module 2008-12-10 07:10:57 -07:00
Brian Warner
39a41f1d85 util/abbreviate: add abbreviated-size parser 2008-12-01 17:24:12 -07:00
Brian Warner
bf06492a90 #538: fetch version and attach to the rref. Make IntroducerClient demand v1 support. 2008-11-21 20:07:27 -07:00
Brian Warner
7391f63fc1 util/abbreviate: little utility to abbreviate seconds and bytes 2008-11-18 20:11:42 -07:00
Brian Warner
fc7cd23bd1 limiter.py: fix stack blowout by inserting an eventual-send between _done and maybe_start_task. This was causing failures during a 'tahoe manifest' of a large set of directories 2008-11-14 21:11:44 -07:00
Zooko O'Whielacronx
9c35ca76a4 util: copy in pyutil.fileutil.ReopenableNamedTemporaryFile 2008-11-04 17:47:15 -07:00
Zooko O'Whielacronx
cef24792e7 util: copy in nummedobj from pyutil 2008-11-04 13:55:50 -07:00
Zooko O'Whielacronx
fb85a6280f util: copy in dictutil from pyutil 2008-11-04 13:53:27 -07:00
Brian Warner
c205a54965 util/cachedir.py: add a cache-directory manager class, which expires+deletes unused files after a while 2008-10-30 13:01:20 -07:00
Brian Warner
b73c380cdb move testutil into test/common_util.py, since it doesn't count as 'code under test' for our pyflakes numbers 2008-10-28 21:28:31 -07:00
Brian Warner
294e3fb682 util: move PollMixin to a separate file (pollmixin.py), so testutil can be moved into test/ 2008-10-28 21:15:48 -07:00
Brian Warner
2d4e895096 iputil.py: avoid a DNS lookup at startup (which may timeout tests when run on a partially-offline host) by using 198.41.0.4 instead of A.ROOT-SERVERS.NET 2008-10-28 13:36:46 -07:00
Zooko O'Whielacronx
57d9f6beb6 repairer: fix flaw in testutil.flip_one_bit() that Brian pointed out 2008-10-16 12:48:48 -07:00
Zooko O'Whielacronx
8a6d1e5da6 repairer: test all different kinds of corruption that can happen to share files on disk 2008-10-14 16:09:20 -07:00
Brian Warner
b0a00e0f33 util/time_format.py: accept space separator, add unit tests 2008-10-13 15:52:58 -07:00
Brian Warner
93725a6935 util/limiter: add a repr 2008-10-07 13:19:45 -07:00
Brian Warner
1e3500ae5f testutil.PollMixin: set default timeout (to 100s), emit a more helpful error when the timeout is hit 2008-09-29 22:23:09 -07:00
Brian Warner
98c8e25709 netstring: add required_trailer= argument 2008-09-26 09:57:54 -07:00
Brian Warner
e8cf581e3f move netstring() and split_netstring() into a separate util.netstring module 2008-09-25 21:38:24 -07:00
Brian Warner
0a59991ce6 various: use util.log.err instead of twisted.log.err, so we get both Incidents and trial-test-flunking 2008-09-20 10:35:45 -07:00
Brian Warner
8854d0c1b5 util.log: send log.err to Twisted too, so that Trial tests are flunked 2008-09-20 10:34:27 -07:00
Brian Warner
bab799297f testutil.PollMixin: use a custom exception (and convert it) to avoid the ugly 'stash' cycle 2008-09-02 20:32:51 -07:00
Zooko O'Whielacronx
9d14341c7d test: add testutil.flip_one_bit which flips a randomly chosen bit of the input string 2008-07-28 16:42:17 -07:00
Brian Warner
dba0ad3d28 test_mutable.py: add more tests of post-mapupdate corruption, to support #474 testing 2008-06-24 11:08:10 -07:00
Brian Warner
fe44b8fb4b util.dictutil: move DictOfSets out to a separate utility module 2008-05-21 09:43:49 -07:00
Brian Warner
1121df6664 testutil.shouldFail: mention the 'which' string in substring failures too 2008-05-16 16:08:38 -07:00
Brian Warner
d76cf6f748 limiter: fix bug that ignores the limit= argument 2008-05-07 18:35:53 -07:00
Brian Warner
c7e441309d add a basic concurrency limiter utility 2008-05-07 16:53:30 -07:00
ben
c85869fc08 support freebsd 6 2008-04-28 00:41:40 -07:00
Brian Warner
81ab1ec785 testutil: factor stall() out into a common location 2008-04-22 16:47:15 -07:00
Brian Warner
b6b3b6b1bd testutil: minor reformatting 2008-04-17 19:56:45 -07:00
Zooko O'Whielacronx
bdda08df8c copy pyutil.time_format into src/allmydata/util 
time_format() provides a good format for logging timestamps
 2007-08-14 09:33:49 -07:00
Zooko O'Whielacronx
fc3bd0c987 use added secret to protect convergent encryption 
Now upload or encode methods take a required argument named "convergence" which can be either None, indicating no convergent encryption at all, or a string, which is the "added secret" to be mixed in to the content hash key.  If you want traditional convergent encryption behavior, set the added secret to be the empty string.

This patch also renames "content hash key" to "convergent encryption" in a argument names and variable names.  (A different and larger renaming is needed in order to clarify that Tahoe supports immutable files which are not encrypted content-hash-key a.k.a. convergent encryption.)

This patch also changes a few unit tests to use non-convergent encryption, because it doesn't matter for what they are testing and non-convergent encryption is slightly faster.
 2008-03-24 09:46:06 -07:00
Zooko O'Whielacronx
b2f5ac9a0a Merge patch which switches to SHA-256d with patch that adds punctuation and capitalization to the comment about the hash value. 2008-02-15 13:16:43 -07:00
Brian Warner
72874390fe BIG COMPATIBILITY BREAK: update hash tags, switch to SHA-256d everywhere 2008-02-14 19:58:01 -07:00
Zooko O'Whielacronx
fc0d637523 docs: update install and usage docs, improve cli "usage" output, make new example directories, add unit test that fails code which prints out sentences that don't end with punctuation marks 2008-02-15 13:11:02 -07:00
robk-tahoe
ca41693162 idlib: make failures much clearer when encountering unicode 
while investigating fuse related stuff, I spent quite a while staring at
very cryptic explosions I got from idlib.  it turns out that unicode
objects and str objects have .translate() methods with differing signatures.
to save anyone else the headache, this makes it very clear if you accidentally
try to pass a unicode object in to a2b() etc.
 2008-02-14 17:23:07 -07:00
Brian Warner
09bfa3bea2 remove bencode, yay 2008-02-14 18:06:41 -07:00
Zooko O'Whielacronx
7c6de95bc6 switch from base62 to base32 for storage indices, switch from z-base-32 to rfc 3548 base-32 for everything, separate out base32 encoding from idlib 2008-02-14 19:27:47 -07:00
Zooko O'Whielacronx
e89edf8803 fix a few unused imports and suchlike, discovered by pyflakes 2008-02-13 07:38:08 -07:00
Zooko O'Whielacronx
3f8df27063 use base62 encoding for storage indexes, on disk and in verifier caps, and in logging and diagnostic tools 
base62 encoding fits more information into alphanumeric chars while avoiding the troublesome non-alphanumeric chars of base64 encoding.  In particular, this allows us to work around the ext3 "32,000 entries in a directory" limit while retaining the convenient property that the intermediate directory names are leading prefixes of the storage index file names.
 2008-02-12 20:48:37 -07:00
Brian Warner
da9610e90a change encryption-key hash to include encoding parameters. This is a minor compatibility break: CHK files encoded (with convergence) before and after this will have different keys and ciphertexts. Also switched to SHA-256d for both the data-to-key hash and the key-to-storageindex hash 2008-02-06 19:50:47 -07:00
Brian Warner
9f9a458249 hashutil: add tagged_hash_256d and tagged_hasher_256d 2008-02-06 19:36:43 -07:00
Brian Warner
3a5ba35215 PollMixin: add timeout= argument, rewrite to avoid tail-recursion problems 2008-02-04 20:35:07 -07:00
Zooko O'Whielacronx
1d1628e525 rename storage_index_chk_hash() to storage_index_hash() and add TODO about how our use of it now includes keys that are not CHKs 2008-02-01 12:27:37 -07:00
Brian Warner
a1b155725f log.py: update log.err() to take advantage of current foolscap's log.err 2008-01-30 18:47:23 -07:00
robk-tahoe
5085c35002 cleanup mac and windows build code 
this moves some of the code common to both windows and mac builds into the
allmydata module hierarchy, and cleans up the windows and mac build directories
to import the code from there.
 2008-01-23 21:06:41 -07:00
Brian Warner
f664dcb489 testutil.py: hush the new (more strict) pyflakes 2008-01-14 18:27:55 -07:00
Brian Warner
c24a1bbaaa iputil.py: hush the new (more strict) pyflakes 2008-01-14 18:27:43 -07:00
robk-tahoe
9ed5533760 added tweaked sibpath implementation 
use of twisted.python.util.sibpath to find files relative to modules doesn't
work when those modules are bundled into a library by py2exe.  this provides
an alternative implementation (in allmydata.util.sibpath) which checks for
the existence of the file, and if it is not found, attempts to find it relative
to sys.executable instead.
 2008-01-10 15:23:41 -07:00
Brian Warner
d9d1697996 util.log: add levels like UNUSUAL 2008-01-07 17:32:45 -07:00
Zooko O'Whielacronx
46db7cf1dc add regexes to idlib to match certain kinds of base32-encoded values 2008-01-03 16:44:46 -07:00
Zooko O'Whielacronx
c066903566 fix idlib.could_be_base32_encoded_l() to accept *only* valid strings 2008-01-03 11:43:11 -07:00
Zooko O'Whielacronx
1ac11301a7 setup: finish switching from Tahoe's versions of autoversioning tools to pyutil's versions 2007-12-21 14:42:38 -07:00
Zooko O'Whielacronx
08a64c3a2b rename "secret" to "lease_secret" and change its size from 16 to 32 bytes 2007-12-17 18:34:11 -07:00
Zooko O'Whielacronx
8c65bdcf9d put all private state in $BASEDIR/private 
fixes #219

The only part of #219 that this doesn't include is the part about 
logpublisher, which has been moved out of tahoe into foolscap.
 2007-12-17 16:39:54 -07:00
Brian Warner
e52e14cc66 remove tests of logging functionality that's been subsumed by foolscap logging 2007-12-12 20:23:53 -07:00
Zooko O'Whielacronx
1ac09840a4 a few documentation and naming convention updates 
Notable: the argument to make REPORTER has been renamed to TRIALARGS.
 2007-12-12 19:34:08 -07:00
Zooko O'Whielacronx
192477e4a2 move increase_rlimits() into iputil and make it a no-op on Windows 2007-12-07 08:03:43 -07:00
Brian Warner
0f5ef5184d test_dirnode.py: obtain full coverage of dirnode.py 2007-12-04 14:32:04 -07:00
Brian Warner
0bf5a762a9 use AES from pycryptopp instead of pycrypto, also truncate the keys slightly differently 2007-12-03 17:27:46 -07:00
Brian Warner
cc1612aee5 hierarchical logging: add numbered messages and parent= args 2007-11-19 18:23:18 -07:00
Zooko O'Whielacronx
59d6c3c822 decentralized directories: integration and testing 
* use new decentralized directories everywhere instead of old centralized directories
 * provide UI to them through the web server
 * provide UI to them through the CLI
 * update unit tests to simulate decentralized mutable directories in order to test other components that rely on them
 * remove the notion of a "vdrive server" and a client thereof
 * remove the notion of a "public vdrive", which was a directory that was centrally published/subscribed automatically by the tahoe node (you can accomplish this manually by making a directory and posting the URL to it on your web site, for example)
 * add a notion of "wait_for_numpeers" when you need to publish data to peers, which is how many peers should be attached before you start.  The default is 1.
 * add __repr__ for filesystem nodes (note: these reprs contain a few bits of the secret key!)
 * fix a few bugs where we used to equate "mutable" with "not read-only".  Nowadays all directories are mutable, but some might be read-only (to you).
 * fix a few bugs where code wasn't aware of the new general-purpose metadata dict the comes with each filesystem edge
 * sundry fixes to unit tests to adjust to the new directories, e.g. don't assume that every share on disk belongs to a chk file.
 2007-12-03 14:52:42 -07:00
Zooko O'Whielacronx
3c7f96400d hashutil.py: switch from pycrypto to pycryptopp SHA256 2007-11-09 14:40:13 -07:00
Brian Warner
e7ad7f8791 mutable: grab encprivkey when necessary during publish, fix test_mutable 2007-11-08 02:46:27 -07:00
Zooko O'Whielacronx
f60dc4adfb UNDO: hashutil: replace pycrypto's SHA256 with pycryptopp's SHA256 2007-11-07 18:02:39 -07:00
Zooko O'Whielacronx
11412da436 hashutil: replace pycrypto's SHA256 with pycryptopp's SHA256 2007-11-07 18:02:39 -07:00
Brian Warner
5739126d11 mutable: storage_index is always 16 bytes 2007-11-06 18:54:34 -07:00
Brian Warner
c4f7412f1c stabilize on 20-byte nodeids everywhere, printed with foolscap's base32 2007-11-06 18:49:59 -07:00
Brian Warner
59632c6812 mutable: use proper enable/renew/cancel secrets 2007-11-05 21:51:08 -07:00
Brian Warner
78c45c82d1 mutable: implement filenode share-packing, still pretty rough 2007-11-02 20:51:39 -07:00
Brian Warner
57373c9889 bencode.py: reencode as UTF-8 2007-11-01 15:42:57 -07:00
Brian Warner
4423bc7510 trailing-whitespace eradication, no functional changes 2007-11-01 15:34:35 -07:00
Brian Warner
c63b43bdbc trailing-whitespace eradication, no functional changes 2007-11-01 15:34:31 -07:00
Brian Warner
f714c5915a trailing-whitespace eradication, no functional changes 2007-11-01 15:34:28 -07:00
Brian Warner
f425ee3600 trailing-whitespace eradication, no functional changes 2007-11-01 15:34:21 -07:00
Brian Warner
6849647856 trailing-whitespace eradication, no functional changes 2007-11-01 15:34:18 -07:00
Brian Warner
ef546da72b trailing-whitespace eradication, no functional changes 2007-11-01 15:29:31 -07:00
Brian Warner
8d000aa1ec trailing-whitespace eradication, no functional changes 2007-11-01 15:29:28 -07:00
Brian Warner
8390d83322 trailing-whitespace eradication, no functional changes 2007-11-01 15:29:17 -07:00
Brian Warner
fb59627faa trailing-whitespace eradication, no functional changes 2007-11-01 15:25:20 -07:00
Brian Warner
5e974ede20 trailing-whitespace eradication, no functional changes 2007-11-01 15:25:16 -07:00
Brian Warner
164da05ead trailing-whitespace eradication, no functional changes 2007-11-01 15:25:12 -07:00
Brian Warner
284b3e77e0 trailing-whitespace eradication, no functional changes 2007-11-01 15:25:09 -07:00
Brian Warner
1d8a4cdfe7 mutable: first pass at dirnodes, filenodes, new URIs. Some test coverage. 
The URI typenames need revision, and only a few dirnode methods are
implemented. Filenodes are non-functional, but URI/key-management is in
place. There are a lot of classes with names like "NewDirectoryNode" that
will need to be rename once we decide what (if any) backwards compatibility
want to retain.
 2007-11-01 15:15:29 -07:00
Zooko O'Whielacronx
5b476d73ee setup: generalize the kludge of finding an executable (i.e. trial or twistd) when there might be only a .py script version of it available 2007-10-15 10:25:04 -07:00
Zooko O'Whielacronx
af0edec753 filter out "0.0.0.0" from detected IP addresses 2007-10-13 00:38:16 -07:00
Zooko O'Whielacronx
8fd98624a3 don't try to use bindann 
It causes a mysterious misbehavior in Python import which causes the previous patch to fail (the patch to not run trial tests if dependencies can't be imported)
 2007-09-13 19:14:46 -07:00
Brian Warner
56afda11d1 deletion phase2a: improve creation of renew/cancel secrets. Still fake though. 2007-08-27 19:00:18 -07:00
Brian Warner
739ae1ccde deletion phase1: send renew/cancel-lease secrets, but my_secret is fake, and the StorageServer discards them 2007-08-27 17:28:51 -07:00
Zooko O'Whielacronx
9ad02dbe80 fix version class to preferred format and correct parsing 2007-08-16 15:38:01 -07:00
Zooko O'Whielacronx
5a1d4aa8b7 version_class.py: if you don't have pkg_resources for comparing version numbers, use distutils.version.LooseVersion 2007-08-16 16:16:41 -07:00
Zooko O'Whielacronx
a431811011 in-line doc edit: s/pyutil/allmydata.util/ 2007-08-16 14:10:48 -07:00
Zooko O'Whielacronx
094b687d6e import version class and make-version script from pyutil -- fixes win32 build, improves error handling, and eliminates unused features 2007-08-16 14:09:30 -07:00
Zooko O'Whielacronx
9b08691f51 fix iputil so that it doesn't launch dozens of processes when you give it a full path and so that it tries executables in preference order and stops as soon as one gives out a dotted-quad string 2007-08-09 10:56:47 -07:00
Brian Warner
e3eba2d6c5 remove unused imports: hush pyflakes warnings 2007-07-24 18:13:58 -07:00
Zooko O'Whielacronx
32c90c6468 switch from rfc 3548 base-32 to z-base-32 except for tubids/nodeids 2007-07-24 13:46:06 -07:00
Brian Warner
9c5ab89afe truncate storage index to 128 bits, since it's derived from a 128 bit AES key 2007-07-22 19:48:44 -07:00
Brian Warner
9af506900b upload: refactor to enable streaming upload. not all tests pass yet 2007-07-19 18:21:44 -07:00
Brian Warner
ea3c3e6780 import simplejson directly, and remove our local copies of the component .py files from util/json* 2007-07-10 15:49:32 -07:00
Brian Warner
ad038497db import the decoder portion of simplejson-1.7.1 too 2007-07-10 13:12:40 -07:00
Brian Warner
72fc8c5cb8 web: use real JSON instead of the fake stubs 
Also include the encoder portion of Bob Ippolito's simplejson-1.7.1 as
allmydata.util.json_encoder . simplejson is distributed under a more liberal
license than Tahoe (looks to be modified BSD), so redistributing it should be ok.
 2007-07-08 00:17:11 -07:00
Brian Warner
9ddb929651 fileutil: add du() function 2007-07-03 15:49:45 -07:00
Brian Warner
bc2603c818 vdrive: protect dirnode contents with an HMAC 2007-06-26 12:36:21 -07:00
Brian Warner
fb02488a8e vdrive: switch to URI:DIR and URI:DIR-RO, providing transitive readonlyness 2007-06-25 13:23:51 -07:00
Brian Warner
956d5ae256 rename fileid/verifierid to plaintext_hash/crypttext_hash 2007-06-09 20:46:04 -07:00
Brian Warner
c9ef291c02 rename thingA to 'uri extension' 2007-06-08 15:59:16 -07:00
Brian Warner
c049941529 move almost all hashing to SHA256, consolidate into hashutil.py 
The only SHA-1 hash that remains is used in the permutation of nodeids,
where we need to decide if we care about performance or long-term security.
I suspect that we could use a much weaker hash (and faster) hash for
this purpose. In the long run, we'll be doing thousands of such hashes
for each file uploaded or downloaded (one per known peer).
 2007-06-07 21:47:21 -07:00
Brian Warner
2db17df833 iputil.get_local_ip_for: tolerate running on a disconnected host 2007-06-07 19:23:33 -07:00
Brian Warner
ea78b4b605 check_memory: getting closer, now we have memusage numbers for uploads of 10kB and 10MB files 2007-05-29 17:39:39 -07:00