Commit Graph

260 Commits

Author SHA1 Message Date
Ross Patterson
447881a0e0 feat(py3): Cleanup redundant string cast
I confirmed that `add_version(...)` itself calls `str(...)` on the argument that
`things_version` is passed in to under both the Python 2.7 and Python 3.6 version of the
library so this is unnecessary here.

This results in an empty diff in py3 tests output.
2020-10-04 15:57:01 -07:00
Itamar Turner-Trauring
7d8320b843 Python 3 fixes. 2020-09-16 11:13:23 -04:00
Jason R. Coombs
d74d7e733c Merge branch 'master' into 3394.py38-tests 2020-09-09 21:42:32 -04:00
Itamar Turner-Trauring
c3494f1356 Enough changes to make allmydata.test.test_storage run on Python 3.
Still lots of failures, of course.
2020-08-27 15:19:49 -04:00
Itamar Turner-Trauring
a758f32edf Try to make test_storage import on Python 3. 2020-08-26 10:53:02 -04:00
Jason R. Coombs
8b553d2045 Add Python 3 compatibility for code paths relevant to 'tahoe --version'. 2020-08-21 19:55:34 -04:00
Jean-Paul Calderone
a47463e032 Pass _Config instead of a smaller dict to get_client_resource 2019-08-19 11:21:03 -04:00
Jean-Paul Calderone
21bf7fc25c Merge remote-tracking branch 'origin/master' into integration/storage-economics 2019-08-16 15:39:31 -04:00
Jean-Paul Calderone
13409a2449 Move the remaining package checking machinery to its own module. 2019-08-13 15:11:01 -04:00
Jean-Paul Calderone
6068b6c1b2 don't reach through the tahoe-lafs config object 2019-08-03 06:19:01 -04:00
Jean-Paul Calderone
756c21c251 actually provide validating client-config-from-string function 2019-08-03 06:19:01 -04:00
Jean-Paul Calderone
fb4c5cf91f Allow for dynamic configuration validation rules 2019-08-03 05:34:21 -04:00
Jean-Paul Calderone
edba0747a3 Use listenOnUnused instead of allocate_tcp_port in create_log_tub 2019-04-16 11:32:27 -04:00
Jean-Paul Calderone
6ea1684995 use listenOnUnused instead of allocate_tcp_port in create_control_tub 2019-04-16 11:26:52 -04:00
heartsucker
129ef22185
updated instances of octal literals to use the format 0o123 for python3 compatibility 2019-03-29 11:08:44 +01:00
meejah
2b5d3be3c8 convert XXX comment to ticket 2018-09-10 21:58:28 -06:00
meejah
9a8ef9512e clarify comments 2018-09-10 21:58:28 -06:00
meejah
f488b79c71 self._portnumfile -> config.portnum_fname 2018-09-10 21:58:28 -06:00
meejah
a5287add80 clarify comments 2018-09-10 21:58:27 -06:00
meejah
714b0887dc more docstrings 2018-09-10 21:58:27 -06:00
meejah
767f4ddd5d reorder args to match other method 2018-09-10 21:58:27 -06:00
meejah
045af64c2b dead code 2018-09-10 21:58:27 -06:00
meejah
edc50f655b get rid of is_tub_listening 2018-09-10 21:58:27 -06:00
meejah
955d7abfa3 move validation code to parser-helper 2018-09-10 21:58:27 -06:00
meejah
9e34d15b90 simplify _tub_portlocation helper 2018-09-10 21:58:27 -06:00
meejah
053b494054 comment 2018-09-10 21:58:27 -06:00
meejah
2d044e1324 remove unused method 2018-09-10 21:58:27 -06:00
meejah
08e0c3b7e2 get rid of 'add_service' (just an alias to setServiceParent anyway) 2018-09-10 21:58:27 -06:00
meejah
7685fb34cd pull 'basedir' entirely into _Config
Put all config-related methods into _Config; change
code to ask config for paths instead of using basedir;
add some better docstrings
2018-09-10 21:58:27 -06:00
meejah
26007f363b pull 'StorageFarmBroker' out of __init__
This means also pulling out introducer-clients and some
related utility methods
2018-09-10 21:58:27 -06:00
meejah
739aaa3ef9 put create() methods in i2p_, tor_provider
Also Provider -> _Provider, improve docs and update tests
2018-09-10 21:58:27 -06:00
meejah
903d4afaa4 handle None for providers 2018-09-10 21:58:27 -06:00
meejah
793827f8a6 put providers back in service tree 2018-09-10 21:58:27 -06:00
meejah
c2946cc2e3 remove unused code 2018-09-10 21:58:27 -06:00
meejah
f0c3db0e5a pyflakes 2018-09-10 21:58:27 -06:00
meejah
0e51bb183e fix more things 2018-09-10 21:58:27 -06:00
meejah
279bd814fc remember info for init_client_storage_broker temporarily 2018-09-10 21:58:27 -06:00
meejah
43d857a0bd more refactor + fix some node tests 2018-09-10 21:58:27 -06:00
meejah
71484b4a12 upgrade create_introducer 2018-09-10 21:58:27 -06:00
meejah
d1a83e9be0 more tests work 2018-09-10 21:58:27 -06:00
meejah
329ef1256a refactor create_client to be async (works to run, some unit-test fails still) 2018-09-10 21:58:27 -06:00
meejah
0f22b9bad0 fixups after rebase 2018-08-25 02:23:58 -06:00
meejah
536ccf8b6d better 'file not found' handling 2018-08-25 02:10:44 -06:00
meejah
c0772cdd5f improve docstring 2018-08-25 02:10:44 -06:00
meejah
37d4b59a39 modern syntax 2018-08-25 02:10:44 -06:00
meejah
a432fc35da docstring improvements 2018-08-25 02:10:44 -06:00
meejah
d544284f92 introduce create_node_dir 2018-08-25 02:10:44 -06:00
meejah
f7f3c54f93 dead code 2018-08-25 02:10:44 -06:00
meejah
86a9ce5793 create _NoNetworkClient using same code as _Client 2018-08-25 02:10:44 -06:00
meejah
22e2d0a417 re-expand path because user input 2018-08-25 02:10:44 -06:00
meejah
f68a0ab74c remove debug 2018-08-25 02:10:44 -06:00
meejah
79756c088e split client, introducer READMEs 2018-08-25 02:10:44 -06:00
meejah
990f23d5c7 _Config does this for us 2018-08-25 02:10:44 -06:00
meejah
38063037c1 add documentation 2018-08-25 02:10:44 -06:00
meejah
f37ab3b12d get_app_version -> global function 2018-08-25 02:10:44 -06:00
meejah
35810a5692 pull 'basedir' entirely into _Config
Put all config-related methods into _Config; change
code to ask config for paths instead of using basedir;
add some better docstrings
2018-08-25 02:10:44 -06:00
meejah
3723945296 Actually validate configs when loading them
An indenting problem meant the validators weren't being
called, which revealed some follow-on errors.
2018-08-17 15:53:59 -06:00
tpltnt
a41d827f79 _None() as new style class 2018-03-02 17:17:30 +01:00
tpltnt
1967cd7357 removed unnecessary after return 2018-03-02 16:45:11 +01:00
tpltnt
1a14d5218b added some docstrings 2018-03-02 16:45:11 +01:00
tpltnt
3f99eceb36 minor whitespace fixes 2018-03-02 16:45:11 +01:00
tpltnt
0461cc0148 only one module import per line 2018-03-02 16:45:11 +01:00
tpltnt
f7bbc9dffe made mkstemp() call compatible with mktemp() 2018-03-02 16:45:11 +01:00
tpltnt
710ba84a02 switched to secure mkstemp() 2018-02-27 18:35:08 +01:00
Brian Warner
da4e7dcfbe hush a bunch of not-really-problems caught by the lgtm.com static analyzer
This is all minor stuff: unreachable debug code (that should be commented-out
instead of in an 'if False:' block), unnecessary 'pass' and 'global'
statements, redundantly-initialized variables. No behavior changes. Nothing
here was actually broken, it just looked suspicious to the static analysis at
https://lgtm.com/projects/g/tahoe-lafs/tahoe-lafs/alerts/?mode=list .
2017-12-31 00:19:41 +01:00
meejah
1b6f477549 Pull "config" handling out of Node and hide node-derivitives
This includes:

 - refactor to create_client and _Client
 - refactor to _IntroducerNode and create_introducer
2017-11-20 12:57:20 -07:00
Brian Warner
d713e0e57d address review feedback: docstrings, better test class name 2017-11-03 00:24:46 -07:00
Brian Warner
097abb42fa tahoe.cfg: add tub.port=listen:i2p (and/or listen:tor)
This delegates the construction of the server Endpoint object to the i2p/tor
Provider, which can use the i2p/tor section of the config file to add options
which would be awkward to express as text in an endpoint descriptor string.

refs ticket:2889 (but note this merely makes room for a function to be
written that can process I2CP options, it does not actually handle such
options, so it does not close this ticket yet)
2017-11-03 00:24:46 -07:00
str4d
6071c2b6f8
Implement i2p_provider and --listen=i2p
Closes ticket:2838
2016-10-23 20:30:01 -05:00
Brian Warner
89eb86c5a4 node.py: use tor_provider at startup
We use it for two things: to create the foolscap connection handler, and
to possibly start an .onion listener at startup.

This also updates node._common_config_sections to accept the new tor
settings written by create-node/create-introducer.
2016-10-09 01:21:51 -04:00
David Stainton
b3b80d376b Use a TorProvider to get a Tor handler 2016-10-09 00:30:33 -04:00
Brian Warner
0e3225ac27 set up logging before init_connections 2016-10-09 00:11:17 -04:00
David Stainton
e6d7895865 Support comma separated endpoint list in tub.port
fixes ticket #867
2016-09-20 10:20:28 -07:00
str4d
e82e2c31e2 Update error message to match implementation and docs
Closes ticket:2824
2016-09-20 10:02:05 -07:00
Brian Warner
a638a97806 implement connections:tcp=disabled
This enables an I2P-only node, which disables TCP entirely (instead of
mapping TCP to Tor, which was the only other option that
reveal-IP-address=False would allow).

closes ticket:2824
2016-09-14 16:27:12 -07:00
Brian Warner
aa162f5a90 close more filehandles with context managers 2016-09-13 22:17:22 -07:00
meejah
2732c37941 Check for unknown config options
The list of valid sections + config-items came from
grep'ing the source for `.get_config`
2016-09-06 15:05:46 -06:00
Brian Warner
0116c965fa private-mode: reject legacy host:port locations 2016-09-02 09:25:26 -07:00
Brian Warner
076b3895dc config: change syntax of no-listen mode
We now use::

  tub.port = disabled
  tub.location = disabled

instead of using an empty value (but the key still being present, since
if the key is missing entirely, that means "be automatic").

closes ticket:2816
2016-09-01 21:26:48 -07:00
David Stainton
1307c7262d Use the new Foolscap Tor handler, pass an endpoint for SOCKS connectivity
closes ticket:2813
2016-09-02 03:17:45 +00:00
Brian Warner
a03f68b787 tahoe.cfg: fix spelling of 'enabled' for tor/i2p
We should use "enabled = true", instead of "enable = true", since
the rest of tahoe.cfg uses "enabled".

refs ticket:2788
2016-08-31 15:23:47 -07:00
Brian Warner
d47fc0fd27 config: add reveal-IP-address=False
This adds a safety flag named `[node] reveal-IP-address`, for which the
default value is True. When this is set to False, any configuration that
might reveal the node's IP address (to servers, or the external network)
will cause a PrivacyError to be raised at startup, terminating the node
before it gets a chance to betray the user's privacy. It also adds docs
and tests.

refs ticket:1010
2016-08-31 02:44:27 -07:00
Brian Warner
325028c967 error if tcp=tor is requested but tor is unimportable
This only catches txtorcon not being installed (which should be fixed by
doing `pip install tahoe-lafs[tor]`). It doesn't notice that the Tor
daemon is not running (which we can't detect during startup, only
afterwards, when it's harder to notify the user), in which case Tor
connections (and all connections when "tcp = tor" is enabled) will just
fail silently.
2016-08-31 01:50:13 -07:00
meejah
72f17afa76 Move check_magicfolder_smoke.py to proper integration tests
This introduces a py.test-based integration suite (currently just
containing magic-folder end-to-end tests). Also adds a tox environment
("integration") to run them.

The test setup is:

 - a "flogtool gather" instance
 - an Introducer
 - five Storage nodes
 - Alice and Bob client nodes
 - Alice and Bob have paired magic-folders
2016-08-30 20:47:47 -06:00
Brian Warner
fea8b627dd if tub.port is empty, don't listen
Updated config docs. Added errors if we're not listening but were told
to enable storage, helper, or if we're the Introducer server.

closes ticket:2816
2016-08-29 22:28:55 -07:00
Brian Warner
2d358f2e35 Disable foolscap "gifts"
"gifts" are third-party references (where Alice sends Bob a reference to
Carol, and Carol lives in a different Tub than either Alice or Bob).
Tahoe doesn't currently use these, and they make it possible for e.g. a
server to provoke a client into making a connection to an IP address of
the server's choosing. If the client isn't configured otherwise, this
would use raw TCP (revealing the client's IP address) even if the
connection to the server used Tor.

A connection policy of `tcp: tor` would block this, but it seems like a
good idea to disable gifts entirely.

closes ticket:2810
2016-08-28 18:15:30 -07:00
Brian Warner
a099b9237d tor: socks.port is now a (restricted) endpoint string
Foolscap has limitations that prevent us from accepting anything but a
TCP endpoint, but that will change in the future, so make the tahoe.cfg
syntax accept an endpoint, but then reject non-TCP ones. See the ticket
for details: refs ticket:2813.

This depends upon the new `foolscap.connections.tor.socks_port(host,
port)` API in foolscap-0.12.2, so it bumps the dependency to that (the
previous commit depended upon 0.12.1, but I hadn't gotten around to
updating the dep before now).
2016-08-28 16:30:31 -07:00
Brian Warner
bc079a71eb implement+test I2P setup
Note that many of the Foolscap handler-creation functions are still
stubbed out, so Tahoe won't be able to honor the full range of config
syntax until foolscap support is complete.
2016-08-28 03:16:43 -07:00
Brian Warner
3cc8daf5bd implement full range of Tor handlers 2016-08-28 02:42:47 -07:00
Brian Warner
73d5376b6a Implement basic connection handlers (tor/i2p)
The node now attempts to create Tor/I2P connection handlers (if the
right libraries are available), and will use them for tor/i2p FURL hints
by default. For now it only creates default handlers: there is not yet
any code to interpret the `[tor]`/`[i2p]` sections of tahoe.cfg which
would let you override that process.

The node also parses the `[connections]` section, allowing `tcp: tor` to
use Tor for all outbound TCP connections. It defaults to `tcp: tcp`, of
course.

Static storage-server connections will now honor the `connections:`
overrides in `servers.yaml`, allowing specific servers to use TCP where
they would normally be restricted to Tor.

refs ticket:2788
refs ticket:517
2016-08-28 02:41:44 -07:00
Brian Warner
cac99569e9 factor out "Tub maker"
This adds Node._create_tub(), which knows how to make a Tub with all the
right options and connection handlers that were specified in
tahoe.cfg (the connection handlers are disabled for now, but they'll get
implemented soon).

The new Node.create_main_tub() calls it. This main Tub is used:

* to connect to the Introducer
* to host the Helper (if enabled)
* to host the Storage Server (if enabled)

Node._create_tub() is also passed into the StorageFarmBroker, which
passes it into each NativeStorageServer, to create the (separate) Tub
for each server connection. _create_tub knows about the options, and
NativeStorageServer can override the connection handlers. This way we
don't need to pass tub options or default handlers into Client,
StorageFarmBroker, or NativeStorageServer.

A number of tests create NativeStorageServer objects: these were updated
to match the new arguments. test_storage_client was simplified because
we no longer need to mock out the Tub() constructor.
2016-08-27 17:36:55 -07:00
David Stainton
61eb839843 Add tub_handlers arg to NativeStorageServer constructor
Here we also define tub_handlers as a node attribute and pass it all the way
down to the NativeStorageServer via the Client and StorageFarmBroker
2016-08-22 09:15:25 -07:00
Brian Warner
af1e2a1e2f node.py: advise twistd.log readers about flogtool 2016-08-09 15:27:33 -04:00
Brian Warner
7d692d0afd Split control and log to two separate tubs, remove control knobs.
This also removes the tahoe.cfg keys that would have configured the
control-port. And it deletes the logport.furl file before asking the Tub
to re-create it, because we're now using an ephemeral Tub (so we're not
persisting the private key, so the tubid will change each time).

closes ticket:2794
2016-07-06 21:09:35 -07:00
David Stainton
e5ffbdbcdf Move control/logging to a separate Tub 2016-07-06 21:09:35 -07:00
Brian Warner
b9edccbeaa set the same options on all Tubs 2016-05-03 11:38:20 -07:00
Brian Warner
96c625920c Teach tub to start synchronously even with ip autodetect
* remove when_tub_ready() from all code
* synchronous-ify all node/client/introducer startup code

refs ticket:2491
2016-04-26 21:54:45 -07:00
Brian Warner
ffc2f48cfe compute tub location before creating Tub
This can be done synchronously because we now know the port number
earlier. This still uses get_local_addresses_sync() (not _async) to do
automatic IP-address detection if the config file didn't set
tub.location or used the special word "AUTO" in it.

The new implementation slightly changes the mapping from tub.location to
the assigned location string. The old code removed all instances of
"AUTO" from the location and then extended the hints with the local
ones (so "hint1:AUTO:hint2" turns into "hint1:hint2:auto1:auto2"). The
new code exactly replaces each "AUTO" with the local hints (so that
example turns into "hint1:auto1:auto2:hint2", and a silly
"hint1:AUTO:AUTO" would turn into "hint1:auto1:auto2:auto1:auto2"). This
is unlikely to affect anybody.
2016-04-26 18:39:47 -07:00
Brian Warner
cf5b02b487 clean up tub.port processing, reject tcp:0
This is the first step towards making node startup be synchronous: the
tub.port is entirely determined (including any TCP port allocation that
might be necessary) before creating the Tub, so the portnumber part of
FURLs can be determined earlier.
2016-04-26 18:39:47 -07:00
meejah
308bb8c41f match get_or_create_private_config API (closes ticket:2775) 2016-04-12 12:58:42 -06:00