Commit Graph

6560 Commits

Author SHA1 Message Date
Brian Warner
cfb939aa99 improve test_client_cache 2016-05-11 16:48:05 -07:00
Brian Warner
ecec58b339 test_introducer: factor out _load_cache 2016-05-11 15:03:32 -07:00
Brian Warner
1197b15106 introducer_client: split out _deliver_announcements 2016-05-11 12:54:11 -07:00
Brian Warner
b49b409c32 Merge branch 'pr278': write (but don't read) YAML cache
refs ticket:2759
2016-05-10 13:02:00 -07:00
Brian Warner
155ca03174 test_introducer: cleanups
* Use tempfile for cache to avoid collisions
* Fix pyflakes complaints
* Remove test_client_cache_2, which exercises unsigned announcements.
  These are scheduled to be removed soon (see ticket:2784) and don't
  need to be tested.
2016-05-10 12:14:36 -07:00
Brian Warner
0c2dd8b86b IntroducerClient: small changes to cache handling
* don't remove the cache at connection establishment, we can just wait
  for the first announcement to truncate the cache
* save announcements before notifying subscribers, so they can safely
  read it right away
* remove unused self._got_announcement_cb
2016-05-10 12:14:04 -07:00
David Stainton
ae2b82a1f0 Add IntroducerClient write-only yaml cache file
this change also includes unit tests
2016-05-10 20:19:35 +00:00
Brian Warner
e3aef2b966 travis.yml: use tox, with 'coverage' environment 2016-05-09 14:40:48 -07:00
Brian Warner
608719df91 tox: add "coverage" environment
Run with "tox -e coverage". Uses a new helper
module (allmydata.test.run_trial) to let us import+execute trial without
knowing exactly where the "trial" binary lives, which helps with using
"coverage run" under tox.
2016-05-09 14:32:21 -07:00
Brian Warner
1e1e86fe35 tox.ini: skip default install
We use "--deps = --editable=.[test]" to achieve three goals:

* make tahoe and it's dependencies available for tests
* use --editable, which is faster and allows "coverage run" to get the
  source filenames right
* use the [test] extra, which includes "mock"

Tox's default install command does the first, but doesn't use
--editable, so when the "deps" stage comes around, there's already a
non-editable install in place. It seems to get the [test] extra right,
but it doesn't wind up with an editable install.

So we disable the default install command and rely on the "deps" clause
instead.
2016-05-09 13:56:06 -07:00
Brian Warner
f57d1e9d3d Merge branch '2773-stats'
This changes "tahoe create-stats-gatherer" to take --hostname,
--location, and --port, according to (refs ticket:2773).
2016-05-04 18:37:23 -07:00
Brian Warner
93bb3e995a stats-gatherer: add --hostname/--location/--port
Updates docs, tests, explains how to update an old gatherer.
2016-05-04 17:58:45 -07:00
Brian Warner
d1d988410b remove "key-generator" node type and client support
closes ticket:2783
2016-05-04 16:53:04 -07:00
Brian Warner
c715e0d839 Merge pull request #265 from meejah/fileutil-improvements 2016-05-04 08:04:51 -07:00
meejah
fa418a7809 appease windows, and skip test 2016-05-04 07:54:42 -07:00
Daira Hopwood
dd02f31a3b Fix missing import 2016-05-04 07:54:42 -07:00
Daira Hopwood
242f04818e Fix test to use arguments with absolute unicode paths 2016-05-04 07:54:42 -07:00
David Stainton
3524729fb1 Add make_dirs_with_absolute_mode to fileutils 2016-05-04 07:54:42 -07:00
Daira Hopwood
50747e2ca0 Strip any long path marker in the input to flush_volume.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-05-04 07:54:42 -07:00
Daira Hopwood
d07c45bdf4 Improve all of the Windows-specific error reporting.
Also make the Windows function declarations more readable and consistent.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-05-04 07:54:42 -07:00
Daira Hopwood
b517e200aa replace_file should allow the replaced file not to exist on Windows.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-05-04 07:54:42 -07:00
Daira Hopwood
7015f5d826 Fix fileutil tests.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-05-04 07:54:42 -07:00
Daira Hopwood
07389571c2 Add long_path=False option to abspath_expanduser_unicode.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-05-04 07:54:42 -07:00
Daira Hopwood
62dc2f7ac1 Re-order some imports in fileutil
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-05-04 07:54:42 -07:00
Daira Hopwood
9bfcb72400 Windows fileutil improvements
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-05-04 07:54:42 -07:00
Brian Warner
9402b40da5 Merge branch 'storage_broker_tub.0' 2016-05-03 16:12:12 -07:00
Brian Warner
bbeb397a00 storage_client: remove unnecessary startService upcall 2016-05-03 15:48:08 -07:00
Brian Warner
76f6d2d21a create Tub inside NativeStorageServer, not broker
This makes IServer instances responsible for their own network
connections, which will help when we add HTTP-based servers in the
future. The StorageFarmBroker should not care about how the IServer uses
the network, it just provides the announcement (and local config).
2016-05-03 15:09:13 -07:00
Brian Warner
e147aa3da2 test: use patch() to control Tub creation
This reduces the dependency on the internals of storage_client.py
2016-05-03 14:49:31 -07:00
Brian Warner
e301ac88ea make test slightly easier to follow 2016-05-03 14:03:17 -07:00
Brian Warner
f5291b9366 document Tub-per-server change
refs ticket:2759
2016-05-03 11:54:26 -07:00
Brian Warner
b9edccbeaa set the same options on all Tubs 2016-05-03 11:38:20 -07:00
meejah
aea0abf54a fix one utest 2016-05-02 18:09:53 -06:00
David Stainton
6061b6fc3c WIP Storage broker client creates one tub per server
i was unable to get all the unit tests working;
this is my work in progress.
2016-05-02 15:23:07 +00:00
Brian Warner
22c1031d19 test_introducer: stop using tcp:0
This fixes some of the upcoming-deprecation warnings against Foolscap
(>=0.11.0). There are still a bunch related to the key-generator and the
stats gatherer.
2016-04-28 11:01:40 -07:00
Brian Warner
5a5ba643e6 use twisted.web.static, not nevow.static, for public_html/
This avoids a privacy leak when the web.static= directory is configured
but doesn't exist (which is almost always, since we set `web.static =
public_html` in the default config file, but nothing automatically
creates it). The nevow.static.File class tries to os.stat() the
directory before doing anything else, which causes an exception, which
renders the traceback to the HTTP client as a 500 Internal Server Error,
and the traceback includes the full path of the missing public_html
directory, which reveals the node's basedir.

Plain twisted.web.static.File doesn't do this check, and a missing
web.static directory just results in a plain old 404.

Closes ticket:1720.
2016-04-28 00:42:59 -07:00
Brian Warner
bde22ad1f7 docs are now at readthedocs.io
The badge image URL remains the same, but the docs themselves are now
hosted on tahoe-lafs.readthedocs.io rather than
tahoe-lafs.readthedocs.org .
2016-04-27 23:12:18 -07:00
Brian Warner
c9047b1510 Merge branch PR#242: switch stats-gatherer to JSON 2016-04-27 17:40:04 -07:00
Brian Warner
ea473cdc76 remove pickle-based stats output altogether
* update munin plugin to read JSON, not pickle
* update docs
2016-04-27 17:27:33 -07:00
Corbin Simpson
e1dba4abeb stats: Dump a JSON file next to the pickle file.
Extremely useful for interoperating with non-Python (e.g. Monte) tooling.
2016-04-27 14:45:37 -07:00
Brian Warner
e187fba58e Merge branch '2491-sync'
Change the Node/Client startup process to set the Tub location during
`__init__`, rather than after the reactor starts up. This simplifies the
init process considerably, because components (like StorageServer and
IntroducerClient) can do their tub.registerReferences() immediately.

This addresses most of the runtime changes called for in ticket:2491,
but not the node-creation changes.
2016-04-27 14:29:22 -07:00
Brian Warner
25b64041cc update config docs, add NEWS snippet 2016-04-26 22:26:31 -07:00
Brian Warner
96c625920c Teach tub to start synchronously even with ip autodetect
* remove when_tub_ready() from all code
* synchronous-ify all node/client/introducer startup code

refs ticket:2491
2016-04-26 21:54:45 -07:00
Brian Warner
ffc2f48cfe compute tub location before creating Tub
This can be done synchronously because we now know the port number
earlier. This still uses get_local_addresses_sync() (not _async) to do
automatic IP-address detection if the config file didn't set
tub.location or used the special word "AUTO" in it.

The new implementation slightly changes the mapping from tub.location to
the assigned location string. The old code removed all instances of
"AUTO" from the location and then extended the hints with the local
ones (so "hint1:AUTO:hint2" turns into "hint1:hint2:auto1:auto2"). The
new code exactly replaces each "AUTO" with the local hints (so that
example turns into "hint1:auto1:auto2:hint2", and a silly
"hint1:AUTO:AUTO" would turn into "hint1:auto1:auto2:auto1:auto2"). This
is unlikely to affect anybody.
2016-04-26 18:39:47 -07:00
Brian Warner
bdf1f8460d iputil.py: expose get_local_addresses_sync 2016-04-26 18:39:47 -07:00
Brian Warner
cf5b02b487 clean up tub.port processing, reject tcp:0
This is the first step towards making node startup be synchronous: the
tub.port is entirely determined (including any TCP port allocation that
might be necessary) before creating the Tub, so the portnumber part of
FURLs can be determined earlier.
2016-04-26 18:39:47 -07:00
Brian Warner
7637d70f1b test_introducer.ClientSeqnums: disable storage
This test was depending upon the storage announcement happening *after*
startup, but the upcoming synchronous-Tub-startup change will modify the
ordering. Fix it in both cases by disabling storage in the client being
tested.
2016-04-26 18:39:47 -07:00
Brian Warner
1f6c04518e Client: init_helper() must run after init_client()
This has worked so far because everything waited for the Tub to be
ready. We'll soon be making Tub setup synchronous, so we won't have to
wait anymore, so the order will matter.
2016-04-26 18:35:03 -07:00
Brian Warner
62d847663c Revert "test/common: add a slight stall in tearDown"
This reverts commit bb7184163e.

We changed test_runner.BinTahoe.run_bintahoe since this commit landed:
the new version can no longer cause the test to be skipped late (we've
gotten rid of the bin/tahoe script entirely, so it's no longer possible
for us to miss it). Hence I think we don't need this unsightly stall any
longer.
2016-04-26 14:18:40 -07:00
Brian Warner
17cd487dc8 bump Twisted dependency to 16.1.0
closes ticket:2781
2016-04-26 13:56:08 -07:00