Brian Warner
9281a0aedc
webish: reduce POST memory footprint by overriding http.Request
...
The original twisted.web.http.Request class has a requestReceived method
that parses the form body (in the request's .content filehandle) using
the stdlib cgi.parse_multipart() function. parse_multipart() consumes a
lot of memory when handling large file uploads, because it holds the
arguments entirely in RAM. Nevow's subclass of Request uses cgi.FieldStorage
instead, which is much more memory-efficient.
This patch uses a local subclass of Request and a modified copy of the
requestReceived method. It disables the cgi.parse_multipart parsing and
instead relies upon Nevow's use of FieldStorage. Our code must look for
form elements (which come from the body of the POST request) in req.fields,
rather than assuming that they will be copied into req.args (which now
only contains the query arguments that appear in the URL).
As a result, memory usage is no longer inflated by the size of the file
being uploaded in a POST upload request. Note that cgi.FieldStorage uses
temporary files (tempfile.TemporaryFile) to hold the data.
This closes #29 .
2007-08-10 17:25:33 -07:00
Brian Warner
b6ff38e4c7
webish.POSTHandler: fix typo that meant we didn't look for 'name' in req.fields
2007-08-10 17:25:28 -07:00
Brian Warner
54b6900526
webapi.txt: minor clarifications and examples
2007-08-10 15:52:27 -07:00
Brian Warner
3d42856eb1
boringfile: ignore the setuptools .egg that ez_setup.py likes to download/build/install during compilation
2007-08-09 19:45:18 -07:00
Brian Warner
17d776fd07
Makefile: add 'upload-self' pass to check-memory
2007-08-10 01:41:34 -07:00
Brian Warner
998802fd6d
#96 : add flag to enable pushing data to ourselves, defaulting to False
2007-08-09 18:30:24 -07:00
Brian Warner
8b14ad1673
encode.py: log a percentage complete as well as the raw byte counts
2007-08-09 18:28:45 -07:00
Brian Warner
08ee80176a
Encoder.__repr__: mention the file being encoded
2007-08-09 18:26:56 -07:00
Brian Warner
684966d103
encode.py: add a reactor turn barrier between segments, to allow Deferreds to retire and free their arguments, all in the name of reducing memory footprint
2007-08-09 18:26:17 -07:00
Zooko O'Whielacronx
c0c6417e95
webapi.txt: put back the manifest feature
...
So that we can compare versions of webapi.txt with and without this documentation, side by side.
2007-08-10 12:58:33 -07:00
Zooko O'Whielacronx
ba2e9aa4cb
webapi.txt: put back the localfile feature
...
So that we can compare versions webapi.txt with and without this documentation side by side.
2007-08-10 12:52:37 -07:00
Zooko O'Whielacronx
193f603e74
webapi.txt: add URI-based GET variants
2007-08-10 12:33:29 -07:00
Zooko O'Whielacronx
89d2e1ac57
webapi.txt: some editing, and remove the localfile feature and the manifest feature
...
My motivation to remove these features is as per:
http://allmydata.org/pipermail/tahoe-dev/2007-August/000067.html
However, I haven't heard back from Brian yet, so I'm actually going to put them back in the next patch so that I can compare the two versions of webapi.txt side by side.
2007-08-10 12:24:13 -07:00
Zooko O'Whielacronx
887240e7a3
webapi.txt: further refactoring and add a section explaining TOCTTOU bugs and how to avoid them by using URIs
2007-08-10 12:04:30 -07:00
Zooko O'Whielacronx
e68a0e07de
webapi.txt: further refactoring and editing to clarify the fact that you don't know whether a thing is a file or a directory before you fetch it
2007-08-10 10:19:27 -07:00
Zooko O'Whielacronx
8b0807812b
webapi.txt: add "?t=file" flag and reorganize doc to discourage people from thinking that they know before hand the file-or-dir type of the thing that they are naming
2007-08-10 09:43:52 -07:00
Zooko O'Whielacronx
531109e803
fix test_web refactoring so that the WebMixin class isn't a TestCase
...
Thanks, Brian.
2007-08-10 08:40:02 -07:00
wilcoxjg
e13783cd6a
If do_stop attempts to stop a nonexistent process the OSError exception is caught (by errno) and handled tersely.
2007-08-09 08:57:45 -07:00
Zooko O'Whielacronx
39f218ab90
refactor test_web so that other tests can use the part of test_web that sets up a simple filesystem
2007-08-09 13:08:42 -07:00
Brian Warner
9ac5c14037
introducer: don't log.err() an initial connection failure, since that flunks tests. Use self.log() instead. Also improve test_client.py to always trigger this case, before it was finishing to quickly to always hit the problem.
2007-08-09 12:53:44 -07:00
Brian Warner
84d41adfcc
check_memory.py: include a single 100MB test
2007-08-09 11:30:33 -07:00
Brian Warner
b25446d514
munin/tahoe-storagespace.py: use /bin/du instead of our python version, since
...
it a) is faster and b) will include filesystem overhead that is a real cost.
2007-08-09 18:26:57 -07:00
Brian Warner
1bb44df8ba
webapi.txt: update rfc reference
2007-08-09 11:24:35 -07:00
Zooko O'Whielacronx
6d905265a1
zfec: required version of setuptools is 0.6c6 on cygwin, 0.6a9 on other platforms
2007-08-09 10:29:03 -07:00
Zooko O'Whielacronx
5bbbdd1e91
zfec: default setuptools version is now 0.6c6
2007-08-09 10:28:38 -07:00
Zooko O'Whielacronx
d1c40a7ab5
zfec: add benchmark utility
2007-08-09 10:19:13 -07:00
Zooko O'Whielacronx
279561b61b
zfec: add to README.txt how to run the unit tests
2007-07-20 15:51:24 -07:00
Zooko O'Whielacronx
31a1e03194
zfec: fix link to zfec-dev mailing list
2007-07-20 13:57:47 -07:00
Zooko O'Whielacronx
a1fbfc9e53
zfec: fix README.txt to explain how to use setuptools with GNU stow better
...
Also remove requirement of setuptools >= v0.6c3. I *think* that using setuptools in the modern way relieves this requirement, but I haven't tested it on dapper yet.
2007-07-20 13:57:01 -07:00
Zooko O'Whielacronx
df3fc9b5ed
zfec: clarify licensing requirements (transitive grace period)
2007-07-20 13:55:46 -07:00
Zooko O'Whielacronx
8522689641
zfec: shebang usr bin env python in ez_setup.py
2007-06-04 04:45:17 -07:00
Zooko O'Whielacronx
5643ef81a1
zfec: update doc strings in fileutil
2007-06-04 04:44:44 -07:00
Zooko O'Whielacronx
9b08691f51
fix iputil so that it doesn't launch dozens of processes when you give it a full path and so that it tries executables in preference order and stops as soon as one gives out a dotted-quad string
2007-08-09 10:56:47 -07:00
wilcoxjg
e274f4350c
in --> across
2007-08-08 21:17:54 -07:00
wilcoxjg
1cc2040d18
cleaning grammar
2007-08-08 21:11:54 -07:00
wilcoxjg
db073791b9
/twistd.log -> /logs/twistd.log
2007-08-08 20:53:28 -07:00
Zooko O'Whielacronx
63d7e126ff
misc/sizes.py: rename "block" to "share" and "subblock" to "block"
...
This renaming is already in place in all of our other documentation and code, but misc/sizes.py got overlooked.
2007-08-08 22:31:12 -07:00
Zooko O'Whielacronx
fedab34f83
architecture.txt: small edits
2007-08-08 22:31:05 -07:00
Zooko O'Whielacronx
524c9f1fc2
startstop_node now works if the logs directory already exists
...
Thanks to Arno and Arch for pointing this out.
2007-08-08 21:43:48 -07:00
Zooko O'Whielacronx
57800e84b4
README: formatting (line-wrapping) of arch_o_median's edits to README
2007-08-08 20:47:42 -07:00
wilcoxjg
d7c842c4a9
changed README to inslude simplejson in "The Python Way" install
2007-08-08 20:43:40 -07:00
wilcoxjg
4490d0b723
changed MakeFile to install simplejson
2007-08-08 20:41:26 -07:00
wilcoxjg
29ab6c6467
edited README "The Python Way" installs to /usr/bin
2007-08-08 20:32:09 -07:00
Zooko O'Whielacronx
ac74e6d4c9
README: clarify/edit build, install, and test instructions
2007-08-08 20:44:58 -07:00
Zooko O'Whielacronx
8c2ba22ad5
clarify licence
2007-08-08 20:44:42 -07:00
Zooko O'Whielacronx
42b919f79e
README: point people to allmydata.org if they are looking for precompiled packages
2007-08-08 20:44:03 -07:00
Zooko O'Whielacronx
7add1b7efc
README: edit to arch_o_median's contribution re: installing the Python way
2007-08-08 20:32:45 -07:00
Zooko O'Whielacronx
f599a4433e
README: add note about what "darcs get" will do and how to run "darcs pull"
2007-08-08 20:28:32 -07:00
wilcoxjg
c5af444399
changed README install "Python Way" section
2007-08-08 20:00:35 -07:00
Brian Warner
57df32aac5
now that the buildslave is moved to a new machine, enable the 50MB test
2007-08-09 01:32:52 -07:00
