Brian Warner
0eb6b324a4
#538 : add remote_get_version() to four main Referenceable objects: Introducer Service, Storage Server, Helper, CHK Upload Helper. Remove unused storage-server get_versions().
2008-11-21 17:43:52 -07:00
Brian Warner
288d55825c
storage: split WriteBucketProxy and ReadBucketProxy out into immutable/layout.py . No behavioral changes.
2008-10-09 17:08:00 -07:00
Brian Warner
0a59991ce6
various: use util.log.err instead of twisted.log.err, so we get both Incidents and trial-test-flunking
2008-09-20 10:35:45 -07:00
Brian Warner
a94af879ff
logging: add 'unique-message-ids' (or 'umids') to each WEIRD-or-higher log.msg call, to make it easier to correlate log message with source code
2008-08-25 18:57:59 -07:00
Brian Warner
7394607141
move encode/upload/download/checker.py into a new immutable/ directory. No behavior changes expected.
2008-07-16 13:14:39 -07:00
Brian Warner
351a7f4632
offloaded: add 'resumes' counter to stats: how many upload requests are resuming an interrupted upload?
2008-04-28 13:27:55 -07:00
Brian Warner
e1838ba217
mutable WIP: clean up status handling, shrink the code a lot, improve test coverage
2008-04-17 13:02:22 -07:00
Brian Warner
a0fa21e442
helper: log SI in ciphertext-read-failed message
2008-04-14 19:41:23 -07:00
Brian Warner
a719b5060f
helper: add SI to logged progress messages
2008-04-14 19:26:53 -07:00
Brian Warner
a2dd4d14a6
helper status: include percentage fetched+pushed, add helper-uploads to the upload/download list
2008-04-14 18:36:27 -07:00
Brian Warner
8bea663fb2
offloaded: add fetched-percentage to the log message
2008-04-14 14:16:38 -07:00
Brian Warner
500934b72f
helper stats: fix the /helper_status page, the recent conflict merging missed some uses. Added tests, updated the munin plugins to match
2008-04-14 13:18:53 -07:00
robk-tahoe
a32275771a
stats_gatherer: fix typo in helper stats gathering.
2008-04-10 17:41:42 -07:00
robk-tahoe
766deaa9b6
stats_gatherer: reconcile helper stats gathering
...
I'd implemented stats gathering hooks in the helper a while back.
Brian did the same without reference to my changes. This reconciles
those two changes, encompassing all the stats in both changes,
implemented through the stats_provider interface.
this also provide templates for all 10 helper graphs in the
tahoe-stats munin plugin.
2008-04-10 17:25:44 -07:00
robk-tahoe
f47ebd0814
conflict reconciliation (part 1, stats gathering in helper)
2008-03-27 17:25:16 -07:00
robk-tahoe
5446ea5d67
stats: added stats reporting to the upload helper
...
adds a stats_producer for the upload helper, which provides a series of counters
to the stats gatherer, under the name 'chk_upload_helper'.
it examines both the 'incoming' directory, and the 'encoding' dir, providing
inc_count inc_size inc_size_old enc_count enc_size enc_size_old, respectively
the number of files in each dir, the total size thereof, and the aggregate
size of all files older than 48hrs
2008-03-25 18:19:08 -07:00
Brian Warner
2c96a32633
helper: add more stats to webapi, at /helper_status
2008-03-27 16:46:08 -07:00
Brian Warner
36f5c025a6
helper: add stats for the gatherer, show some on the webish welcome page
2008-03-27 15:55:32 -07:00
Brian Warner
3a8bd1b772
webish: add more share information to upload status, including assisted uploads
2008-03-05 19:51:51 -07:00
Brian Warner
2b49605c51
webish: add 'download results', with some basic timing information
2008-03-03 19:19:21 -07:00
Brian Warner
d96f90e1fb
log more peerinfo in download/upload/checker problems
2008-02-26 17:33:14 -07:00
Zooko O'Whielacronx
3f8df27063
use base62 encoding for storage indexes, on disk and in verifier caps, and in logging and diagnostic tools
...
base62 encoding fits more information into alphanumeric chars while avoiding the troublesome non-alphanumeric chars of base64 encoding. In particular, this allows us to work around the ext3 "32,000 entries in a directory" limit while retaining the convenient property that the intermediate directory names are leading prefixes of the storage index file names.
2008-02-12 20:48:37 -07:00
Brian Warner
d0ce8694c1
add upload-status objects, to track upload progress
2008-02-12 15:36:05 -07:00
Brian Warner
75e4332462
offloaded: oops, need more tricks to make the unit tests pass
2008-02-06 17:51:11 -07:00
Brian Warner
a2cace9cfb
helper: return full uri-extension data to the client, so it can compare encoding parameters
2008-02-06 17:30:58 -07:00
Brian Warner
652d50d7e0
UploadResults: add more helper timing stats (ciphertext fetch times)
2008-02-06 02:12:35 -07:00
Brian Warner
124fb5ecdf
add upload-results timing info for helper uploads. This changes the Helper protocol, and introduces a compatibility break
2008-02-06 01:52:25 -07:00
Brian Warner
daecca6589
big introducer refactoring: separate publish+subscribe. Addresses #271 .
2008-02-05 13:05:13 -07:00
Brian Warner
6b414074a9
offloaded.py: fix logging a bit
2008-01-31 13:45:01 -07:00
Brian Warner
81eeafc574
upload-helper: avoid duplicate uploads: check the grid to see if the file already exists
2008-01-30 18:49:02 -07:00
Brian Warner
e14dcb92e2
offloaded: when uploading a file that failed to upload before, ask for the last byte of ciphertext, so the reader is prepared to give us the plaintext hashes
2008-01-28 19:05:43 -07:00
Brian Warner
69a0b5cc00
CHK upload helper: don't let one failed upload prevent us from trying again
2008-01-28 12:58:13 -07:00
Brian Warner
46fe024612
offloaded uploader: don't use a huge amount of memory when skipping over previously-uploaded data
2008-01-24 17:25:33 -07:00
robk-tahoe
d1de1f180a
offloaded: reinstate fix for windows tests
...
in a discussion the other day, brian had asked me to try removing this fix, since
it leads to double-closing the reader. since on my windows box, the test failures
I'd experienced were related to the ConnectionLost exception problem, and this
close didn't see to make a difference to test results, I agreed.
turns out that the buildbot's environment does fail without this fix, even with
the exception fix, as I'd kind of expected.
it makes sense, because the reader (specifically the file handle) must be closed
before it can be unlinked. at any rate, I'm reinstating this, in order to fix the
windows build
2008-01-21 15:25:15 -07:00
robk-tahoe
7b990cc9af
UNDO: offloaded: close reader before removing its file
...
unlinking a file before closing it is not portable. it works on unix, but fails
since an open file holds a lock on windows.
this closes the reader before trying to unlink the encoding file within the
CHKUploadHelper.
2008-01-17 17:36:28 -07:00
robk-tahoe
46154beace
offloaded: close reader before removing its file
...
unlinking a file before closing it is not portable. it works on unix, but fails
since an open file holds a lock on windows.
this closes the reader before trying to unlink the encoding file within the
CHKUploadHelper.
2008-01-17 17:36:28 -07:00
Brian Warner
e9307d3fda
offloaded: close the local filehandle after encoding is done, otherwise windows fails
2008-01-17 01:52:33 -07:00
Brian Warner
0e3b218e7d
offloaded.py: delete encoding tempfile when upload is complete
2008-01-17 01:15:54 -07:00
Brian Warner
41cbd1e3a7
offloaded.py: when resuming, append new data to incoming file, rather than overwrite it.
2008-01-17 01:15:32 -07:00
Brian Warner
8386b0226b
offloaded.py: remove dead/redundant code
2008-01-17 01:14:45 -07:00
Brian Warner
c597e67c2b
offloaded: improve logging across the board
2008-01-17 01:11:35 -07:00
Brian Warner
51321944f0
megapatch: overhaul encoding_parameters handling: now it comes from the Uploadable, or the Client. Removed options= too. Also move helper towards resumability.
2008-01-16 03:03:35 -07:00
Brian Warner
f0430ccc48
offloaded: cleanup to handle multiple simultaneous uploaders gracefully
2008-01-14 22:20:03 -07:00
Brian Warner
168a8c3b73
offloaded: improve logging, pass through options, get ready for testing interrupted uploads. test_system: add (disabled) interrupted-upload test
2008-01-14 21:24:26 -07:00
Brian Warner
7ac2b94aba
remove wait_for_numpeers and the when_enough_peers call in mutable.Publish
2008-01-14 14:55:59 -07:00
Brian Warner
964edadf44
offloaded: add a system test, make it pass. files are now being uploaded through the helper.
2008-01-11 05:42:55 -07:00
Brian Warner
08f9c3f2ab
offloaded.py: hush pyflakes
2008-01-11 05:05:14 -07:00
Brian Warner
6ac01fde4c
offloaded: more test coverage on client side, change interfaces a bit
2008-01-11 04:53:37 -07:00
Brian Warner
e825406fc2
offloaded: move interfaces to interfaces.py, start implementing backend
2008-01-09 21:25:47 -07:00
Brian Warner
2ad84eeed8
offloaded: create a Helper if 'run_helper' is non-empty
2008-01-09 20:25:05 -07:00
