Commit Graph

6398 Commits

Author SHA1 Message Date
Brian Warner
d1d988410b remove "key-generator" node type and client support
closes ticket:2783
2016-05-04 16:53:04 -07:00
Brian Warner
c715e0d839 Merge pull request #265 from meejah/fileutil-improvements 2016-05-04 08:04:51 -07:00
meejah
fa418a7809 appease windows, and skip test 2016-05-04 07:54:42 -07:00
Daira Hopwood
dd02f31a3b Fix missing import 2016-05-04 07:54:42 -07:00
Daira Hopwood
242f04818e Fix test to use arguments with absolute unicode paths 2016-05-04 07:54:42 -07:00
David Stainton
3524729fb1 Add make_dirs_with_absolute_mode to fileutils 2016-05-04 07:54:42 -07:00
Daira Hopwood
50747e2ca0 Strip any long path marker in the input to flush_volume.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-05-04 07:54:42 -07:00
Daira Hopwood
d07c45bdf4 Improve all of the Windows-specific error reporting.
Also make the Windows function declarations more readable and consistent.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-05-04 07:54:42 -07:00
Daira Hopwood
b517e200aa replace_file should allow the replaced file not to exist on Windows.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-05-04 07:54:42 -07:00
Daira Hopwood
7015f5d826 Fix fileutil tests.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-05-04 07:54:42 -07:00
Daira Hopwood
07389571c2 Add long_path=False option to abspath_expanduser_unicode.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-05-04 07:54:42 -07:00
Daira Hopwood
62dc2f7ac1 Re-order some imports in fileutil
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-05-04 07:54:42 -07:00
Daira Hopwood
9bfcb72400 Windows fileutil improvements
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-05-04 07:54:42 -07:00
Brian Warner
9402b40da5 Merge branch 'storage_broker_tub.0' 2016-05-03 16:12:12 -07:00
Brian Warner
bbeb397a00 storage_client: remove unnecessary startService upcall 2016-05-03 15:48:08 -07:00
Brian Warner
76f6d2d21a create Tub inside NativeStorageServer, not broker
This makes IServer instances responsible for their own network
connections, which will help when we add HTTP-based servers in the
future. The StorageFarmBroker should not care about how the IServer uses
the network, it just provides the announcement (and local config).
2016-05-03 15:09:13 -07:00
Brian Warner
e147aa3da2 test: use patch() to control Tub creation
This reduces the dependency on the internals of storage_client.py
2016-05-03 14:49:31 -07:00
Brian Warner
e301ac88ea make test slightly easier to follow 2016-05-03 14:03:17 -07:00
Brian Warner
f5291b9366 document Tub-per-server change
refs ticket:2759
2016-05-03 11:54:26 -07:00
Brian Warner
b9edccbeaa set the same options on all Tubs 2016-05-03 11:38:20 -07:00
meejah
aea0abf54a fix one utest 2016-05-02 18:09:53 -06:00
David Stainton
6061b6fc3c WIP Storage broker client creates one tub per server
i was unable to get all the unit tests working;
this is my work in progress.
2016-05-02 15:23:07 +00:00
Brian Warner
22c1031d19 test_introducer: stop using tcp:0
This fixes some of the upcoming-deprecation warnings against Foolscap
(>=0.11.0). There are still a bunch related to the key-generator and the
stats gatherer.
2016-04-28 11:01:40 -07:00
Brian Warner
5a5ba643e6 use twisted.web.static, not nevow.static, for public_html/
This avoids a privacy leak when the web.static= directory is configured
but doesn't exist (which is almost always, since we set `web.static =
public_html` in the default config file, but nothing automatically
creates it). The nevow.static.File class tries to os.stat() the
directory before doing anything else, which causes an exception, which
renders the traceback to the HTTP client as a 500 Internal Server Error,
and the traceback includes the full path of the missing public_html
directory, which reveals the node's basedir.

Plain twisted.web.static.File doesn't do this check, and a missing
web.static directory just results in a plain old 404.

Closes ticket:1720.
2016-04-28 00:42:59 -07:00
Brian Warner
bde22ad1f7 docs are now at readthedocs.io
The badge image URL remains the same, but the docs themselves are now
hosted on tahoe-lafs.readthedocs.io rather than
tahoe-lafs.readthedocs.org .
2016-04-27 23:12:18 -07:00
Brian Warner
c9047b1510 Merge branch PR#242: switch stats-gatherer to JSON 2016-04-27 17:40:04 -07:00
Brian Warner
ea473cdc76 remove pickle-based stats output altogether
* update munin plugin to read JSON, not pickle
* update docs
2016-04-27 17:27:33 -07:00
Corbin Simpson
e1dba4abeb stats: Dump a JSON file next to the pickle file.
Extremely useful for interoperating with non-Python (e.g. Monte) tooling.
2016-04-27 14:45:37 -07:00
Brian Warner
e187fba58e Merge branch '2491-sync'
Change the Node/Client startup process to set the Tub location during
`__init__`, rather than after the reactor starts up. This simplifies the
init process considerably, because components (like StorageServer and
IntroducerClient) can do their tub.registerReferences() immediately.

This addresses most of the runtime changes called for in ticket:2491,
but not the node-creation changes.
2016-04-27 14:29:22 -07:00
Brian Warner
25b64041cc update config docs, add NEWS snippet 2016-04-26 22:26:31 -07:00
Brian Warner
96c625920c Teach tub to start synchronously even with ip autodetect
* remove when_tub_ready() from all code
* synchronous-ify all node/client/introducer startup code

refs ticket:2491
2016-04-26 21:54:45 -07:00
Brian Warner
ffc2f48cfe compute tub location before creating Tub
This can be done synchronously because we now know the port number
earlier. This still uses get_local_addresses_sync() (not _async) to do
automatic IP-address detection if the config file didn't set
tub.location or used the special word "AUTO" in it.

The new implementation slightly changes the mapping from tub.location to
the assigned location string. The old code removed all instances of
"AUTO" from the location and then extended the hints with the local
ones (so "hint1:AUTO:hint2" turns into "hint1:hint2:auto1:auto2"). The
new code exactly replaces each "AUTO" with the local hints (so that
example turns into "hint1:auto1:auto2:hint2", and a silly
"hint1:AUTO:AUTO" would turn into "hint1:auto1:auto2:auto1:auto2"). This
is unlikely to affect anybody.
2016-04-26 18:39:47 -07:00
Brian Warner
bdf1f8460d iputil.py: expose get_local_addresses_sync 2016-04-26 18:39:47 -07:00
Brian Warner
cf5b02b487 clean up tub.port processing, reject tcp:0
This is the first step towards making node startup be synchronous: the
tub.port is entirely determined (including any TCP port allocation that
might be necessary) before creating the Tub, so the portnumber part of
FURLs can be determined earlier.
2016-04-26 18:39:47 -07:00
Brian Warner
7637d70f1b test_introducer.ClientSeqnums: disable storage
This test was depending upon the storage announcement happening *after*
startup, but the upcoming synchronous-Tub-startup change will modify the
ordering. Fix it in both cases by disabling storage in the client being
tested.
2016-04-26 18:39:47 -07:00
Brian Warner
1f6c04518e Client: init_helper() must run after init_client()
This has worked so far because everything waited for the Tub to be
ready. We'll soon be making Tub setup synchronous, so we won't have to
wait anymore, so the order will matter.
2016-04-26 18:35:03 -07:00
Brian Warner
62d847663c Revert "test/common: add a slight stall in tearDown"
This reverts commit bb7184163e.

We changed test_runner.BinTahoe.run_bintahoe since this commit landed:
the new version can no longer cause the test to be skipped late (we've
gotten rid of the bin/tahoe script entirely, so it's no longer possible
for us to miss it). Hence I think we don't need this unsightly stall any
longer.
2016-04-26 14:18:40 -07:00
Brian Warner
17cd487dc8 bump Twisted dependency to 16.1.0
closes ticket:2781
2016-04-26 13:56:08 -07:00
Brian Warner
e9e12653f8 SystemTestMixin: don't auto-detect IP addresses
The tubs created for test_system should always use 127.0.0.1 .

refs ticket:2491
2016-04-26 13:55:59 -07:00
Brian Warner
6dfcd52b06 iputil.py: copy allocate_tcp_port() from foolscap
refs ticket:2491
2016-04-26 13:55:51 -07:00
Brian Warner
5d6fe5abdf Merge PR#269: add when-connected-enough hook
This makes it possible for automated-upload tools (like drop-upload and
magic-folders) to be told when there are "enough" servers connected for
uploads to be successful. This should help prevent the pathological case
where the tools attempt to upload files immediately after node
startup (or before the user turns on their wifi connection), and the
node stores all the shares on itself.

This new notification is single-shot and edge-triggered: when it fires,
you know that, at some point in the past, the node *was* connected to at
least $threshold servers. However you might have lost several
connections since then. The user might turn off wifi after this fires,
causing all connections to be dropped.

In the long run, this API will change: clients will receive continuous
notifications about servers coming and going, and tools like
magic-folder should refrain from uploading during periods of
insufficient connection. It might be as simple as checking the size of
the connected server list when a periodic timer goes off, or something
more responsive like an edge-triggered "upload as soon as you can"
observer.
2016-04-26 12:42:20 -07:00
meejah
b834b71dac Change API to a listener-style, with helper 2016-04-26 13:40:02 -06:00
meejah
55898941da Get rid of the 'connected_threshold' argument, too
- use 'when_connected_to', which takes a threshold
2016-04-26 12:26:40 -06:00
meejah
4c619ac98d fix StorageFarmBroker constructors after connected_d gone 2016-04-26 12:26:40 -06:00
meejah
b65a8fe142 Switch to when_connected_enough()
- instead of passing in a Deferred(), we use an observer
- fix up the tests
- TODO: fix magic-folder
2016-04-26 12:26:40 -06:00
meejah
be2576f15d test covering StorageBroker's connected_threshold code 2016-04-26 12:26:39 -06:00
Daira Hopwood
a56a3adaae Teach StorageFarmBroker to fire a deferred when a connection threshold is reached. refs #1449
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-04-26 12:26:39 -06:00
Brian Warner
84a1064b87 setup.py: depend on 'mock' when using [test] extra
I think this is useful enough that we should have it available when
running tests.

refs ticket:2777
2016-04-26 11:21:59 -07:00
Brian Warner
d2ab4c593b initial tahoe.cfg: explain encoding params better
Meejah pointed out that new users might think the encoding parameters
are fixed, something you must pick correctly when you first set up the
node, and then are never allowed to change again, which is kind of
anxiety-inducing. This updates the comment to explain that the encoding
is stored in each filecap, and the tahoe.cfg values are only used for
newly-uploaded files.
2016-04-26 11:18:44 -07:00
Brian Warner
d05da27ae8 hush pyflakes 2016-04-26 10:43:26 -07:00