Commit Graph

6362 Commits

Author SHA1 Message Date
Brian Warner
62d847663c Revert "test/common: add a slight stall in tearDown"
This reverts commit bb7184163e.

We changed test_runner.BinTahoe.run_bintahoe since this commit landed:
the new version can no longer cause the test to be skipped late (we've
gotten rid of the bin/tahoe script entirely, so it's no longer possible
for us to miss it). Hence I think we don't need this unsightly stall any
longer.
2016-04-26 14:18:40 -07:00
Brian Warner
17cd487dc8 bump Twisted dependency to 16.1.0
closes ticket:2781
2016-04-26 13:56:08 -07:00
Brian Warner
e9e12653f8 SystemTestMixin: don't auto-detect IP addresses
The tubs created for test_system should always use 127.0.0.1 .

refs ticket:2491
2016-04-26 13:55:59 -07:00
Brian Warner
6dfcd52b06 iputil.py: copy allocate_tcp_port() from foolscap
refs ticket:2491
2016-04-26 13:55:51 -07:00
Brian Warner
5d6fe5abdf Merge PR#269: add when-connected-enough hook
This makes it possible for automated-upload tools (like drop-upload and
magic-folders) to be told when there are "enough" servers connected for
uploads to be successful. This should help prevent the pathological case
where the tools attempt to upload files immediately after node
startup (or before the user turns on their wifi connection), and the
node stores all the shares on itself.

This new notification is single-shot and edge-triggered: when it fires,
you know that, at some point in the past, the node *was* connected to at
least $threshold servers. However you might have lost several
connections since then. The user might turn off wifi after this fires,
causing all connections to be dropped.

In the long run, this API will change: clients will receive continuous
notifications about servers coming and going, and tools like
magic-folder should refrain from uploading during periods of
insufficient connection. It might be as simple as checking the size of
the connected server list when a periodic timer goes off, or something
more responsive like an edge-triggered "upload as soon as you can"
observer.
2016-04-26 12:42:20 -07:00
meejah
b834b71dac Change API to a listener-style, with helper 2016-04-26 13:40:02 -06:00
meejah
55898941da Get rid of the 'connected_threshold' argument, too
- use 'when_connected_to', which takes a threshold
2016-04-26 12:26:40 -06:00
meejah
4c619ac98d fix StorageFarmBroker constructors after connected_d gone 2016-04-26 12:26:40 -06:00
meejah
b65a8fe142 Switch to when_connected_enough()
- instead of passing in a Deferred(), we use an observer
- fix up the tests
- TODO: fix magic-folder
2016-04-26 12:26:40 -06:00
meejah
be2576f15d test covering StorageBroker's connected_threshold code 2016-04-26 12:26:39 -06:00
Daira Hopwood
a56a3adaae Teach StorageFarmBroker to fire a deferred when a connection threshold is reached. refs #1449
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-04-26 12:26:39 -06:00
Brian Warner
84a1064b87 setup.py: depend on 'mock' when using [test] extra
I think this is useful enough that we should have it available when
running tests.

refs ticket:2777
2016-04-26 11:21:59 -07:00
Brian Warner
d2ab4c593b initial tahoe.cfg: explain encoding params better
Meejah pointed out that new users might think the encoding parameters
are fixed, something you must pick correctly when you first set up the
node, and then are never allowed to change again, which is kind of
anxiety-inducing. This updates the comment to explain that the encoding
is stored in each filecap, and the tahoe.cfg values are only used for
newly-uploaded files.
2016-04-26 11:18:44 -07:00
Brian Warner
d05da27ae8 hush pyflakes 2016-04-26 10:43:26 -07:00
meejah
6910309e8f use Resource instead of Page for web-status 2016-04-26 10:33:59 -06:00
meejah
01b09f3bac Only accept 'token' in POST fields (stop using get_arg()) 2016-04-25 15:42:54 -07:00
Brian Warner
afb7718f89 Merge PR #29 from zooko:1903-deprecate-synonymous-file-and-update-docs
refs ticket:1903
2016-04-22 19:25:31 -07:00
Brian Warner
c582a21e99 webapi.rst: fixups
Re-indent the blocks for consistency, improve the explanation of
?filename=foo.jpg to match it's new location, use new-style reference
for urls-and-utf8 footnote.
2016-04-22 19:23:49 -07:00
Zooko O'Whielacronx
3e7346100f update webapi.rst's View/Download File docs
• mark "/file/" as a synonym for "/named/" to be deprecated (fixes #1903)
• move the options common to all three forms to the bottom and dedent them
• name the protocol/format as "LAFS" and the implementation/client "Tahoe"
• reflow (with fill-column 77)
2016-04-22 19:17:22 -07:00
Brian Warner
f665d0690e test_util: remove obsolete flushLoggedErrors check
We now depend on a much newer version of Twisted than the one that
lacked TestCase.flushLoggedErrors.
2016-04-22 18:48:58 -07:00
Brian Warner
cffa682ddc test_util: fix skip-on-32bit-platform behavior
I set up a raspberry pi buildslave (which, on the "raspbian jesse"
image, uses a 32-bit python, and perhaps a 32-bit kernel too). It fails
test_util.TimeFormat.test_format_time_y2038 with a ValueError inside the
call to time.gmtime(). The test was looking for the equality check to
fail instead. I think catching ValueError is the more-correct way to
detect a system with a 32-bit time type.
2016-04-22 18:45:29 -07:00
Daira Hopwood
5186e68f6b Add ignore_count to deferredutil 2016-04-19 15:16:13 -06:00
Daira Hopwood
1c9a3b4b01 Simplify run_bintahoe and make it work post-pippification.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-04-19 17:47:00 +01:00
Daira Hopwood
ee8bddbb1a Allow tahoe to be run using 'python -m allmydata.scripts.runner'.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-04-19 17:22:34 +01:00
Brian Warner
04f1a3c38d Merge branch 'pr259' from meejah:add-tox
refs ticket:2779
2016-04-13 00:03:07 -07:00
meejah
fd00f9b56c tox.ini missing from MANIFEST 2016-04-12 19:46:49 -06:00
meejah
6e17328eb5 teach abbreviate_time to understand timedelta
original review wanted to do this instead of depending on "humanize"
2016-04-12 18:42:53 -06:00
Brian Warner
c97e8520f4 tox.ini: use --editable to work around old-pip bug
One of the buildslaves (Ubuntu wily 15.10) has a very old pip-1.5.6,
which doesn't know how to "pip install" a filepath+extra (like
".[test]") unless --editable is also used.

It's convenient to have --editable set anyways (so you can do subsequent
narrow testing without re-running tox, by running ".tox/py27/bin/trial
TESTCASE" or use .tox/py27/bin/activate), so changing the dependency
from ".[test]" to "--editable=.[test]" is the easiest way to work around
that older buildslave. (I could also have upgraded the buildslave to use
a newer pip, but 15.10 is pretty recent and other people will probably
hit this too, so this way it's fixed for everybody).

refs ticket:2776
2016-04-12 14:28:02 -07:00
Brian Warner
23f871a409 add [test] "extra" to install testing-only dependencies
closes ticket:2776
2016-04-12 12:32:46 -07:00
Brian Warner
359c233d26 Merge PR #256 from meejah/2774.status-api-only.0-part2 2016-04-12 12:31:30 -07:00
meejah
308bb8c41f match get_or_create_private_config API (closes ticket:2775) 2016-04-12 12:58:42 -06:00
meejah
4a25573e2d Add simple auth-token to get JSON data 2016-04-12 12:58:27 -06:00
Brian Warner
47b921855a Merge PR #255 from meejah/2774.status-api-only.0-part1
(rebased to current master, added a few fixups)
2016-04-12 01:00:59 -07:00
Brian Warner
d385bb7a21 hush pyflakes 2016-04-12 00:39:03 -07:00
Brian Warner
04a9c51e3a download_to_data: fix comment 2016-04-12 00:38:08 -07:00
meejah
db517e8edd Progress API
- IProgress interface + PercentProgress implementation
 - progress= args to many upload/download APIs
 - ultimately for magic-folder
2016-04-12 00:30:50 -07:00
Brian Warner
bb7184163e test/common: add a slight stall in tearDown
With the new Foolscap-0.11.0 (which changed the way connections are
established), I'm seeing DirtyReactorErrors getting thrown by
allmydata.test.test_system.SystemTest.test_filesystem_with_cli_in_subprocess
, on a host that has three IP addresses (one is 127.0.0.1, two is wifi,
three is a VPN). The test itself is getting skipped because bin/tahoe
isn't in the expected place, but by that point, the nodes have already
been launched and have established connections over one of the three
hints (probably 127.0.0.1). The test terminates so quickly that the
connections to the other two addresses have not finished being
abandoned. The extra stall seems to give Foolscap enough time to reap
the cancelled connections and makes the DRT go away.

I think an offline test, or maybe one with a single external IP address,
wouldn't hit this case.

Arbitrary stalls are never very satisfactory, of course. Usually there
is some threshold delay value, below which it fails reliably, above
which it works on my own machine (for now). This one is weird: the
threshold seems to be below the resolution of the system clock. Stalling
for one nanosecond was enough to fix the problem, but using a simple
fireEventually() didn't work.
2016-04-07 20:59:42 -10:00
Brian Warner
8279d919f3 remove "manhole" (ssh-accessible REPL)
This little-used debugging feature allowed you to SSH or Telnet "into" a
Tahoe node, and get an interactive Read-Eval-Print-Loop (REPL) that
executed inside the context of the running process. The SSH
authentication code used a deprecated feature of Twisted, this code had
no unit-test coverage, and I haven't personally used it in at least 6
years (despite writing it in the first place). Time to go.

Also experiment with a Twisted-style "topfiles/" directory of NEWS
fragments. The idea is that we require all user-visible changes to
include a file or two (named as $TICKETNUM.$TYPE), and then run a script
to generate NEWS during the release process, instead of having a human
scan the commit logs and summarize the changes long after they landed.

Closes ticket:2367
2016-04-06 13:40:06 -10:00
Brian Warner
a8161028d6 tox upcoming-deprecations: emit foolscap/twisted versions
Note that Twisted doesn't do anything like Versioneer, so this will
generally show e.g. "Twisted-16.1.0" for the entire interval between
16.1.0 and 16.2.0.
2016-04-06 13:37:56 -10:00
Brian Warner
b81c6a7208 run-deprecations: supress duplicates
I thought 'warnings' or twisted.python.deprecate was supposed to do this
already, but it's clearly not working.
2016-04-06 13:37:52 -10:00
Brian Warner
9d20de3db9 improve run-deprecations script
Rewrote in Twisted, which lets us read/scan/print all log lines in
realtime. The output is now correctly interleaved (as well as
maintaining the stdout-vs-stderr of each message). The renamed
--warnings= logfile records all relevant lines from *both* stdout and
stderr (i.e. any that includes "DeprecationWarning"), which handles a
change (perhaps in recent Twisteds?) that emits these warnings on stdout
instead of stderr.
2016-04-06 11:07:06 -10:00
Brian Warner
d5e1b21a8a tox: add "upcoming-deprecations" environment
This runs trial against the latest (git) Twisted and Foolscap, to give
us an early warning about anything we're currently doing that will
become deprecated in their next releases.
2016-04-06 09:24:15 -10:00
Brian Warner
c781fc188b run-deprecations: flush stdout before starting trial
This should get the notes printed in the right order.
2016-04-06 09:23:48 -10:00
Brian Warner
14c513db8a tox -e deprecations: add helper script to make it work
Tox doesn't run shell pipelines, and is unhappy about running tools that
it didn't install itself (including non-python things like "make"). So
this adds misc/build_helpers/run-deprecations.py, a python script that
runs a given command (i.e. trial) and writes stdout into a separate file
where the buildbot can see it, and counts the "DeprecationWarning" lines
from the file to decide the returncode.

This ought to improve the status display on the buildbot "test
deprecations" step.
2016-04-06 08:29:48 -10:00
Brian Warner
ce548687f8 iputil.py: avoid DirtyReactorError when running tests offline
The udpprot.transport.connect() fails if we don't have a network
connection, but the port is still listening, so trial gives us a
DirtyReactorError. The fix is a "finally:" which does
port.stopListening() even in this case.

Closes ticket:2769
2016-04-03 19:26:57 -10:00
Brian Warner
431d762872 desert-island: new approach, works better 2016-03-31 17:12:08 -07:00
Brian Warner
dfc09745a0 add 'tox -e docs', to check+render .rst files 2016-03-31 15:59:49 -07:00
Brian Warner
2b5633eedb relnotes, docs: post-release fixups 2016-03-30 17:27:44 -07:00
Brian Warner
04a3e7993f NEWS: release 1.11.0 2016-03-30 16:09:28 -07:00
Brian Warner
14d1250bec CREDITS: update 2016-03-30 16:09:12 -07:00