parse_cli() got added during the async-CLI-dispatch work
assertRaises/assertFailure have been in Twisted for a while, but I only
learned about them recently. Over time I'm looking forward to changing
all tahoe tests to use them (and getting rid of ShouldFailMixin/etc).
In addition, CLI functions are allowed to use sys.exit() instead of
always needing to return the exit code as an integer.
runner.py now knows about the blocking httplib calls in scripts/cli and
scripts/magic_folder, and uses deferToThread() to invoke them. Those
functions cannot return a Deferred: when rewrite them to use twisted.web
or treq, we'll remove this deferToThread call.
Option parsing was split out to a separate function for testing. We now
use twisted.internet.task.react() to start the reactor, which required
changing the way runner.py is tested.
closes ticket:2826
The main part of CLITestMixin.do_cli() was split into a standalone
function named run_cli(), leaving do_cli() as a method which includes a
nodedir in the arguments (for use by GridTestMixin tests which do a lot
of CLI operations against one of their client nodes, for which adding
the extra --nodedir argument would be ugly).
These are obsolete. Tests are run with 'tox', or by running 'trial
allmydata' from a populated virtualenv. A populated virtualenv is also
the right way to get a repl: just run 'python'.
refs ticket:2735
The google image chart API has been deprecated since 2012, sending the
URL to google leaks server IDs and the client's IP address (especially
important when the client is otherwise behind Tor), and the X-axis has
no units anyways.
refs ticket:1942 , which is both about removing the URL-based chart, and
eventually replacing it with a browser-rendered d3.js-based one
So "tahoe create-node --hide-ip" causes "reveal-IP-address = false" to
get written into tahoe.cfg . This also changes the default tahoe.cfg to
include "reveal-IP-address = true", for clarity.
refs ticket:1010
(instead of using a copy). Foolscap-0.12.3 fixes a problem with
allocate_tcp_port() that was causing intermittent test failures. I think
it makes more sense to use Foolscap's copy (and fixes) than to keep
re-copying it into Tahoe each time it changes.
If/when we manage to stop depending upon foolscap for server RPC, we can
re-copy this back into tahoe's source tree.
refs ticket:2795
We now use::
tub.port = disabled
tub.location = disabled
instead of using an empty value (but the key still being present, since
if the key is missing entirely, that means "be automatic").
closes ticket:2816
This was triggered when the initial Introducer connection failed, so the
node read the introducer_cache.yaml from disk. That always returns
unicode strings, and the StorageFarmBroker insisted that it's
server-IDs (aka "key_s") were bytestrings.
The tests were extended to exercise the code that loads from disk and
delivers to the StorageFarmBroker, and more preconditions were put in
place to catch this sort of thing earlier next time.
closes ticket:2817
This adds a safety flag named `[node] reveal-IP-address`, for which the
default value is True. When this is set to False, any configuration that
might reveal the node's IP address (to servers, or the external network)
will cause a PrivacyError to be raised at startup, terminating the node
before it gets a chance to betray the user's privacy. It also adds docs
and tests.
refs ticket:1010
This only catches txtorcon not being installed (which should be fixed by
doing `pip install tahoe-lafs[tor]`). It doesn't notice that the Tor
daemon is not running (which we can't detect during startup, only
afterwards, when it's harder to notify the user), in which case Tor
connections (and all connections when "tcp = tor" is enabled) will just
fail silently.
This introduces a py.test-based integration suite (currently just
containing magic-folder end-to-end tests). Also adds a tox environment
("integration") to run them.
The test setup is:
- a "flogtool gather" instance
- an Introducer
- five Storage nodes
- Alice and Bob client nodes
- Alice and Bob have paired magic-folders
Updated config docs. Added errors if we're not listening but were told
to enable storage, helper, or if we're the Introducer server.
closes ticket:2816
"gifts" are third-party references (where Alice sends Bob a reference to
Carol, and Carol lives in a different Tub than either Alice or Bob).
Tahoe doesn't currently use these, and they make it possible for e.g. a
server to provoke a client into making a connection to an IP address of
the server's choosing. If the client isn't configured otherwise, this
would use raw TCP (revealing the client's IP address) even if the
connection to the server used Tor.
A connection policy of `tcp: tor` would block this, but it seems like a
good idea to disable gifts entirely.
closes ticket:2810
Foolscap has limitations that prevent us from accepting anything but a
TCP endpoint, but that will change in the future, so make the tahoe.cfg
syntax accept an endpoint, but then reject non-TCP ones. See the ticket
for details: refs ticket:2813.
This depends upon the new `foolscap.connections.tor.socks_port(host,
port)` API in foolscap-0.12.2, so it bumps the dependency to that (the
previous commit depended upon 0.12.1, but I hadn't gotten around to
updating the dep before now).
Note that many of the Foolscap handler-creation functions are still
stubbed out, so Tahoe won't be able to honor the full range of config
syntax until foolscap support is complete.
YAML, like JSON, is all-unicode. StorageFarmBroker.set_static_servers()
is defined to take an all-unicode dictionary (the "storage:" key from
private/servers.yaml), so the server_id keys it gets will be unicode.
NativeStorageServer is defined to accept server_ids which are bytes (at
least it is now). The tests were only passing bytes into
set_static_servers(), whereas a real launch passed unicode in, causing a
problem when NativeStorageServer tried to base32.a2b() the pubkey and
choked on the unicode it received.
This fixes set_static_servers() to convert the server_id to bytes, and
changes NativeStorageServer to assert that it gets bytes. It also fixes
the test to match real usage more closely.
The node now attempts to create Tor/I2P connection handlers (if the
right libraries are available), and will use them for tor/i2p FURL hints
by default. For now it only creates default handlers: there is not yet
any code to interpret the `[tor]`/`[i2p]` sections of tahoe.cfg which
would let you override that process.
The node also parses the `[connections]` section, allowing `tcp: tor` to
use Tor for all outbound TCP connections. It defaults to `tcp: tcp`, of
course.
Static storage-server connections will now honor the `connections:`
overrides in `servers.yaml`, allowing specific servers to use TCP where
they would normally be restricted to Tor.
refs ticket:2788
refs ticket:517
This adds Node._create_tub(), which knows how to make a Tub with all the
right options and connection handlers that were specified in
tahoe.cfg (the connection handlers are disabled for now, but they'll get
implemented soon).
The new Node.create_main_tub() calls it. This main Tub is used:
* to connect to the Introducer
* to host the Helper (if enabled)
* to host the Storage Server (if enabled)
Node._create_tub() is also passed into the StorageFarmBroker, which
passes it into each NativeStorageServer, to create the (separate) Tub
for each server connection. _create_tub knows about the options, and
NativeStorageServer can override the connection handlers. This way we
don't need to pass tub options or default handlers into Client,
StorageFarmBroker, or NativeStorageServer.
A number of tests create NativeStorageServer objects: these were updated
to match the new arguments. test_storage_client was simplified because
we no longer need to mock out the Tub() constructor.
A minimally-defined static server only specifies server_id,
anonymous-storage-FURL, and permutation-seed-base32. But the WUI Welcome
page wouldn't render (it raised an exception) without also defining
nickname and version. This allows those values to be missing.
This follows the latest comments in ticket:2788, moving the static
server definitions from "connections.yaml" to "servers.yaml". It removes
the "connections" and "introducers" blocks from that file, leaving it
responsible for just static servers (I think connections and introducers
can be configured from tahoe.cfg).
This feeds all the static server specs to the StorageFarmBroker in a
single call, rather than delivering them as simulated introducer
announcements. It cleans up the way handlers are specified too (the
handler dictionary is ignored, but that will change soon).
They're the same thing, but knowing that is the responsibility of the
caller, not NativeStorageServer. Try to normalize on "server_id" as the
spelling. Remove support for missing key_s, now that we require V2
introductions.
This is a change I've wanted to make for many years, because when we get
to HTTP-based servers, we won't have tubids for them. What held me back
was that there's code all over the place that uses the serverid for
various purposes, so I wasn't sure it was safe. I did a big push a few
years ago to use IServer instances instead of serverids in most
places (in #1363), and to split out the values that actually depend upon
tubid into separate accessors (like get_lease_seed and
get_foolscap_write_enabler_seed), which I think took care of all the
important uses.
There are a number of places that use get_serverid() as dictionary key
to track shares (Checker results, mutable servermap). I believe these
are happy to use pubkeys instead of tubids: the only thing they do with
get_serverid() is to compare it to other values obtained from
get_serverid(). A few places in the WUI used serverid to compute display
values: these were fixed.
The main trouble was the Helper: it returns a HelperUploadResults (a
Copyable) with a share->server mapping that's keyed by whatever the
Helper's get_serverid() returns. If the uploader and the helper are on
different sides of this change, the Helper could return values that the
uploader won't recognize. This is cosmetic: that mapping is only used to
display the upload results on the "Recent and Active Operations" page.
I've added code to StorageFarmBroker.get_stub_server() to fall back to
tubids when looking up a server, so this should still work correctly
when the uploader is new and the Helper is old. If the Helper is new and
the uploader is old, the upload results will show unusual server ids.
refs ticket:1363
This re-factors the magic-folder tests to abstract
the whole "do a file operation" so we can properly
send fake (or wait for real) inotify events to the
uploader/downloader. This speeds up the tests quite
a bit and makes test_alice_bob reasonable again (at
about 1.5s instead of over 30s).
We no longer need the complexity of choosing the application name at
runtime. This removes the setup.py code which populates the _appname.py
file, and the code in __init__.py which reads it. It does not yet remove
the tests which compare the output of e.g. `tahoe --version` against
`allmydata.__appname__`, which I think could be removed, but that's more
invasive than I want to do right now.
closes ticket:2754
This doesn't reveal very much information, but does tell
you if magic-folder is currently working and if not it will
indicate when the last attempt to do a remote scan was.
Improve error-handling for directories if you ask for JSON from
the /uri endpoint, but an error occurs (you get a proper HTTP
status code and a valid JSON object).
For 'tahoe magic-folder status' e now retrieve *all* the remote data
required in the CLI before doing anything else so that errors can be
shown immediately. Use the improved JSON endpoints to print better
errors.
This keeps re-trying the initial magic-folder scan and alerts
the user (via logs only :/) until it succeeds at least once.
After this happens and the node has started up, it will continue
to re-try if enough storage servers go away later such that the
remote collection can't be retrieved.
Previously, this file importing "allmydata.immutable" but assuming that
"allmydata.immutable.upload" was available, which only worked if some
other file had imported upload.py . This didn't affect running the
entire test suite (something imported upload.py before anything else
needed it), but caused errors when running specific tests like
test_repairer.py .
I can't currently test this (my OS-X laptop can't run those tests), but
based on how much time test_magic_folder takes on the buildbots, I
expect oneshare=True to help considerably.
This saves more time (as measured on my laptop):
* test_sftp: 17.7s -> 13s
* test_dirnode: 26.5s -> 20s
* test_ftp, test_configutil, test_web show negligible speedups
As before, some tests care about the number of shares, generally ones
which delete or corrupt shares and then expect to see the errors get
noticed or fixed. Those tests continue to use k=3/N=10.
Most of the CLI tests don't care about the actual shares. Configuring
the test client to use k=N=1 reduces the runtime from 180s to 90s on my
laptop.
A few tests *do* care, like test_check (which delete some shares, then
assert that 'tahoe check' shows the damage). These still use k=3/N=10.
Many of the test cases would exercise two copies of each file: one with
k=3/N=10, and a second with k=127/N=255 (255 being the maximum supported
by zfec).
Large number of shares increases the overhead of the testing apparatus,
which is pushing those shares to lots of local servers.
I don't think the "max_shares" case is necessary, and it takes forever.
Because of it, "mutable.Update" was consuming 15% of the total test
runtime, and a third of that was just a single
function (test_replace_locations_max_shares, now deleted). On a
Raspberry Pi 3 (our "slow computer" benchmark), including branch
coverage, this one class took 42 minutes to complete, and requires
disabling a bunch of timeouts to finish at all.
The total number of shares in a file ("N") affects one thing: the
width (and thus height) of the share hash tree. This should be exercised
in test_hashtree.
The number of required shares ("k") affects one thing: the segment size
must be a multiple of k. I don't think we need to exercise this, but if
so, it could be exercised by a few small values for k, rather than 127.
Removing the max_shares cases saves 82% of the mutable.update
runtime (on top of the previous three-segment fix), reducing it from 64s
to 11.3s on my laptop.
* uses @inlineCallbacks to turn the _lazy_tail recursion into
a "real" looking loop;
* remove the need for "immediate" vs delayed iteration of said loop;
* make it easier for the unit-tests to control the behavior of the
uploader/downloader;
* consolidates (some) setup/teardown code into the setUp and tearDown
hooks provided by unittest so unit-tests aren't doing that themselves
* re-factors some of the unit-tests to use an @inlineCallbacks style
so they're easier to follow and debug
This doesn't tackle the "how to know when our inotify events have arrived"
problem the unit-tests still have, nor does it eliminate the myriad bits
of state that get added to tests via all the MixIns.
Adds:
- a JSON endpoint
- CLI to display information
- QueuedItem + IQueuedItem for uploader/downloader
- IProgress interface + PercentProgress implementation
- progress= args to many upload/download APIs
This gives the integration-style CLI-based tests a chance
to set the delay to 0 before the first 3-second delayed
call is queued to _lazy_tail in the Downloader
1. Split alice/bob clocks to avoid races conditions
in the tests
2. Wrap ._notify so we can advance the clock after inotify
calls in the RealTest (since it takes >0ms to do the "real" notifies)
The yaml.SafeLoader.add_constructor() should probably only be done once,
and moving this all into a module gives us an opportunity to test it
directly.
Without these, clients with a non-empty connections.yaml would crash as
they start up. It's safe to say we need some tests for this :-).
pyflakes catches all of these, but it got accidentally disabled
recently, so travis wasn't running it. I'll fix that in the next commit.
This also removes the tahoe.cfg keys that would have configured the
control-port. And it deletes the logport.furl file before asking the Tub
to re-create it, because we're now using an ephemeral Tub (so we're not
persisting the private key, so the tubid will change each time).
closes ticket:2794
The old copy had a bug which occasionally returns a port that was
actually in use, causing intermittent test failures (when large numbers
of ports were allocated). I finally figured out how to fix it in
Foolscap, so this is just a copy of the updated function.
closes ticket:2795
Historical note: V2 introducers have been around for three years
now (released in 1.10.0), so it's time to drop v1. This branch removes a
lot of fallback code, and tests which exercised it. refs ticket:2784
This patch removes some now-unused code: v1-related support functions on
the client, "stub-client" handlers, and v1-tolerant remote methods on
the server. The unit tests have been cleaned up a bit too, now that
there are fewer cases to exercise.
* use yaml.safe_load and yaml.safe_dump
* configure SafeLoader to return unicode consistently, not str
* log+ignore bad cache, instead of throwing error, since we're already
in the log+ignore chain from connect_failed()
* use a local exception type, instead of one from storage_client.py
* delegate delivery to self._deliver_announcements
Using yaml.safe_dump gives us:
- ann:
my-version: tahoe-lafs/1.11.0.post96.dev0
nickname: node-4
instead of:
- ann:
!!python/unicode 'my-version': !!python/unicode 'tahoe-lafs/1.11.0.post96.dev0'
!!python/unicode 'nickname': !!python/unicode 'node-4'
We want SafeLoader to consistently return unicode instead of sometimes
plain strings (for ASCII-safe values) and sometimes unicode
(for everything else). The data we write into the cache was all unicode
to start with (it came from a JSON parser), so it seems better to get
back unicode too.
* Use tempfile for cache to avoid collisions
* Fix pyflakes complaints
* Remove test_client_cache_2, which exercises unsigned announcements.
These are scheduled to be removed soon (see ticket:2784) and don't
need to be tested.
* don't remove the cache at connection establishment, we can just wait
for the first announcement to truncate the cache
* save announcements before notifying subscribers, so they can safely
read it right away
* remove unused self._got_announcement_cb
Run with "tox -e coverage". Uses a new helper
module (allmydata.test.run_trial) to let us import+execute trial without
knowing exactly where the "trial" binary lives, which helps with using
"coverage run" under tox.
This makes IServer instances responsible for their own network
connections, which will help when we add HTTP-based servers in the
future. The StorageFarmBroker should not care about how the IServer uses
the network, it just provides the announcement (and local config).
This fixes some of the upcoming-deprecation warnings against Foolscap
(>=0.11.0). There are still a bunch related to the key-generator and the
stats gatherer.
This avoids a privacy leak when the web.static= directory is configured
but doesn't exist (which is almost always, since we set `web.static =
public_html` in the default config file, but nothing automatically
creates it). The nevow.static.File class tries to os.stat() the
directory before doing anything else, which causes an exception, which
renders the traceback to the HTTP client as a 500 Internal Server Error,
and the traceback includes the full path of the missing public_html
directory, which reveals the node's basedir.
Plain twisted.web.static.File doesn't do this check, and a missing
web.static directory just results in a plain old 404.
Closes ticket:1720.
This can be done synchronously because we now know the port number
earlier. This still uses get_local_addresses_sync() (not _async) to do
automatic IP-address detection if the config file didn't set
tub.location or used the special word "AUTO" in it.
The new implementation slightly changes the mapping from tub.location to
the assigned location string. The old code removed all instances of
"AUTO" from the location and then extended the hints with the local
ones (so "hint1:AUTO:hint2" turns into "hint1:hint2:auto1:auto2"). The
new code exactly replaces each "AUTO" with the local hints (so that
example turns into "hint1:auto1:auto2:hint2", and a silly
"hint1:AUTO:AUTO" would turn into "hint1:auto1:auto2:auto1:auto2"). This
is unlikely to affect anybody.
This is the first step towards making node startup be synchronous: the
tub.port is entirely determined (including any TCP port allocation that
might be necessary) before creating the Tub, so the portnumber part of
FURLs can be determined earlier.
This test was depending upon the storage announcement happening *after*
startup, but the upcoming synchronous-Tub-startup change will modify the
ordering. Fix it in both cases by disabling storage in the client being
tested.
This has worked so far because everything waited for the Tub to be
ready. We'll soon be making Tub setup synchronous, so we won't have to
wait anymore, so the order will matter.
This reverts commit bb7184163e.
We changed test_runner.BinTahoe.run_bintahoe since this commit landed:
the new version can no longer cause the test to be skipped late (we've
gotten rid of the bin/tahoe script entirely, so it's no longer possible
for us to miss it). Hence I think we don't need this unsightly stall any
longer.
Meejah pointed out that new users might think the encoding parameters
are fixed, something you must pick correctly when you first set up the
node, and then are never allowed to change again, which is kind of
anxiety-inducing. This updates the comment to explain that the encoding
is stored in each filecap, and the tahoe.cfg values are only used for
newly-uploaded files.
I set up a raspberry pi buildslave (which, on the "raspbian jesse"
image, uses a 32-bit python, and perhaps a 32-bit kernel too). It fails
test_util.TimeFormat.test_format_time_y2038 with a ValueError inside the
call to time.gmtime(). The test was looking for the equality check to
fail instead. I think catching ValueError is the more-correct way to
detect a system with a 32-bit time type.
