Commit Graph

451 Commits

Author SHA1 Message Date
Jean-Paul Calderone
9940beaae1 Thread a _Config all the way down 2019-08-19 16:09:26 -04:00
Jean-Paul Calderone
a47463e032 Pass _Config instead of a smaller dict to get_client_resource 2019-08-19 11:21:03 -04:00
Jean-Paul Calderone
21d735ece9 whitespace more conforming to PEP8 2019-08-03 06:39:31 -04:00
Jean-Paul Calderone
d69de15664 implement the feature
improve the test slightly, too, to verify the configuration supplied to the
plugin is as expected.
2019-08-03 06:39:31 -04:00
Jean-Paul Calderone
9842447a07 Don't start or announce anonymous access if config says not to 2019-08-03 06:39:31 -04:00
Jean-Paul Calderone
853cf62530 Allow [storage]anonymous through the validator
And provide a helpful accessor for reading it
2019-08-03 06:39:31 -04:00
Jean-Paul Calderone
311afa8a75 Test & fix supplying plugin configuration 2019-08-03 06:39:31 -04:00
Jean-Paul Calderone
b5a2c70a4a create a StorageClientConfig object
Make it easier to pass more storage configuration down into StorageFarmBroker
and beyond
2019-08-03 06:39:31 -04:00
Jean-Paul Calderone
6b7e0dd700 add a test for the negative case 2019-08-03 06:39:31 -04:00
Jean-Paul Calderone
895cf37a84 docstrings 2019-08-03 06:19:01 -04:00
Jean-Paul Calderone
53861e2a0f Change the shape of the storage announcement(s)
Instead of generating a sequence of announcements like:

    - anonymous storage server announcement
    - plugin 1 storage server announcement
    - ...
    - plugin N storage server announcement

The client now generates a single announcement like:

    - anonymous storage server details
    - storage-options
      - plugin 1 storage server details
      - ...
      - plugin N storage server details
2019-08-03 06:19:01 -04:00
Jean-Paul Calderone
9c240b61ac Persist the furl ourselves rather than relying on Foolscap
Going via our config abstraction here will let us change how config is
persisted more easily, later.
2019-08-03 06:19:01 -04:00
Jean-Paul Calderone
6068b6c1b2 don't reach through the tahoe-lafs config object 2019-08-03 06:19:01 -04:00
Jean-Paul Calderone
d2e16df5cc link to a ticket about implementing better missing-plugin behavior 2019-08-03 06:19:01 -04:00
Jean-Paul Calderone
251eda0b80 rename introducer_factory parameter to be private 2019-08-03 06:19:01 -04:00
Jean-Paul Calderone
58db131787 remove unused imports 2019-08-03 06:19:01 -04:00
Jean-Paul Calderone
fd9ae24149 fix indentation 2019-08-03 06:19:01 -04:00
Jean-Paul Calderone
756c21c251 actually provide validating client-config-from-string function 2019-08-03 06:19:01 -04:00
Jean-Paul Calderone
6cf48f7d4f Separate async initialization from _Client.__init__ 2019-08-03 06:19:01 -04:00
Jean-Paul Calderone
a6959d111c Log init_storage and its result (particularly failures) 2019-08-03 06:19:01 -04:00
Jean-Paul Calderone
7919cf205e Test the actual interface
get_storage_server is supposed to return a Deferred
2019-08-03 06:19:01 -04:00
Jean-Paul Calderone
a45e2bebfe Allow the new plugins item in the [storage] section 2019-08-03 06:19:01 -04:00
Jean-Paul Calderone
f606beb065 Test and support plugins without any configuration 2019-08-03 06:19:01 -04:00
Jean-Paul Calderone
e2982c0129 Support multiple plugins 2019-08-03 06:19:01 -04:00
Jean-Paul Calderone
646cd452b9 Add tests for announcements for plugins
And a basic implementation
2019-08-03 06:19:01 -04:00
Jean-Paul Calderone
212f96dfe7 Baseline tests for anonymous storage server announcements 2019-08-03 05:34:21 -04:00
Jean-Paul Calderone
fb4c5cf91f Allow for dynamic configuration validation rules 2019-08-03 05:34:21 -04:00
meejah
5644f421fe get rid of ed25519.bytes_from_* methods 2019-07-08 12:46:22 -06:00
meejah
47633fb01e unused import 2019-06-26 14:55:35 -06:00
meejah
dd55accec8 Merge branch 'master' into 3031-replace-pycryptopp 2019-06-26 14:42:43 -06:00
meejah
47ccdb0177 refactor ed25519 helpers to functional style
eliminates the wrapper classes and uses some more-explicit
names throughout (e.g "sk" -> "signing_key")
2019-06-13 22:19:28 -06:00
meejah
df4671f90e refactor away from pycryptopp "helper" classes 2019-06-13 22:19:28 -06:00
heartsucker
8063d93c6d replaced pytcryptopp rsa with our own wrapper 2019-06-13 22:19:28 -06:00
heartsucker
9e31bfe2f4 update code/test to use new ed25512 module 2019-06-13 22:19:28 -06:00
heartsucker
3a5a0fb572 replaced uses of pycryptopp's ed25519 with our own 2019-06-13 22:18:55 -06:00
Jean-Paul Calderone
b604d08463 Add a test for the success case 2019-06-12 16:47:25 -04:00
meejah
f0e3b69f90 switch around how we do config (avoid space-separated filenames) 2019-05-30 16:01:46 -06:00
meejah
2cf36cf884 flake8 2019-05-30 16:01:46 -06:00
meejah
c88409afc5 add a note 2019-05-30 16:01:46 -06:00
meejah
4f481bbb4c remove FIXME 2019-05-30 16:01:45 -06:00
meejah
e453db879a make options work, config API usage 2019-05-30 16:01:45 -06:00
meejah
82a74e8986 updates and a bunch of debugging for cert/grid-manager handling 2019-05-30 16:01:45 -06:00
meejah
36c471fdef don't need our pubkey for anything 2019-05-30 16:01:45 -06:00
meejah
e5f608f80e fixes to Options use 2019-05-30 16:01:45 -06:00
meejah
2031b723b8 various fixes (flake8, errors) 2019-05-30 16:01:45 -06:00
meejah
ee3e1cbcf2 wip; updates to grid-manager impl 2019-05-30 16:01:45 -06:00
meejah
64eb9d7c30 hack in grid-manager announcements to storage-servers 2019-05-30 16:01:45 -06:00
tpltnt
c8f11dc2d3 ported old-style classes to new-style 2019-05-26 08:28:18 +02:00
advanced4
ef127f113d
update client.py to use DEFAULT_MAX_SEGMENT_SIZE 2019-05-08 16:04:57 -04:00
Jean-Paul Calderone
53b40ef0e0 don't double start; don't try to handle double start. 2019-03-28 16:10:36 -04:00
meejah
1ad2174df9 windows yaml.safe_load returns None on unreadable files.. 2018-09-11 11:14:41 -06:00
meejah
3d35723a59 windows yamlutil.safe_load returns None when files are unreadable 2018-09-10 21:58:28 -06:00
meejah
692128ad1d better defaults handling 2018-09-10 21:58:28 -06:00
meejah
f6b62ca192 import 2018-09-10 21:58:28 -06:00
meejah
a344bb24dc immutable default 2018-09-10 21:58:27 -06:00
meejah
dee8b72206 dead code 2018-09-10 21:58:27 -06:00
meejah
9aae4bd459 only handle ENOENT 2018-09-10 21:58:27 -06:00
meejah
a5287add80 clarify comments 2018-09-10 21:58:27 -06:00
meejah
efce7b1f6a use Failure for all errors from async methods 2018-09-10 21:58:27 -06:00
meejah
714b0887dc more docstrings 2018-09-10 21:58:27 -06:00
meejah
c22d7c632e docstring improvements 2018-09-10 21:58:27 -06:00
meejah
3974af6660 redundant imports 2018-09-10 21:58:27 -06:00
meejah
edc50f655b get rid of is_tub_listening 2018-09-10 21:58:27 -06:00
meejah
a82aa4ba2c code cleanup, docs 2018-09-10 21:58:27 -06:00
meejah
e74d2a7d01 get rid of redundant 'introducer_clients' var 2018-09-10 21:58:27 -06:00
meejah
c84e50baf0 post-rebase fixup: _client_factory and docs 2018-09-10 21:58:27 -06:00
meejah
17fa32633c post-rebase fixups; needs review 2018-09-10 21:58:27 -06:00
meejah
08e0c3b7e2 get rid of 'add_service' (just an alias to setServiceParent anyway) 2018-09-10 21:58:27 -06:00
meejah
7685fb34cd pull 'basedir' entirely into _Config
Put all config-related methods into _Config; change
code to ask config for paths instead of using basedir;
add some better docstrings
2018-09-10 21:58:27 -06:00
meejah
f80e61fef6 put setServiceParent calls in create_* 2018-09-10 21:58:27 -06:00
meejah
26007f363b pull 'StorageFarmBroker' out of __init__
This means also pulling out introducer-clients and some
related utility methods
2018-09-10 21:58:27 -06:00
meejah
739aaa3ef9 put create() methods in i2p_, tor_provider
Also Provider -> _Provider, improve docs and update tests
2018-09-10 21:58:27 -06:00
meejah
f0c3db0e5a pyflakes 2018-09-10 21:58:27 -06:00
meejah
279bd814fc remember info for init_client_storage_broker temporarily 2018-09-10 21:58:27 -06:00
meejah
43d857a0bd more refactor + fix some node tests 2018-09-10 21:58:27 -06:00
meejah
71484b4a12 upgrade create_introducer 2018-09-10 21:58:27 -06:00
meejah
4f2d45626c no more reveal_ip 2018-09-10 21:58:27 -06:00
meejah
d1a83e9be0 more tests work 2018-09-10 21:58:27 -06:00
meejah
329ef1256a refactor create_client to be async (works to run, some unit-test fails still) 2018-09-10 21:58:27 -06:00
meejah
41cfd8fb16 change imports; introducer client.read_config 2018-09-10 21:58:27 -06:00
meejah
eab56276e5 introduce create_node_dir 2018-09-10 21:58:27 -06:00
meejah
91517bbad0 unused import 2018-08-25 02:12:02 -06:00
meejah
1b30e9edfc consistently use 'config' not 'self.config' in __init__ 2018-08-25 02:10:44 -06:00
meejah
a432fc35da docstring improvements 2018-08-25 02:10:44 -06:00
meejah
5000787c18 flake8 2018-08-25 02:10:44 -06:00
meejah
2937c729e3 change imports; introducer client.read_config 2018-08-25 02:10:44 -06:00
meejah
d544284f92 introduce create_node_dir 2018-08-25 02:10:44 -06:00
meejah
86a9ce5793 create _NoNetworkClient using same code as _Client 2018-08-25 02:10:44 -06:00
meejah
79756c088e split client, introducer READMEs 2018-08-25 02:10:44 -06:00
meejah
f37ab3b12d get_app_version -> global function 2018-08-25 02:10:44 -06:00
meejah
35810a5692 pull 'basedir' entirely into _Config
Put all config-related methods into _Config; change
code to ask config for paths instead of using basedir;
add some better docstrings
2018-08-25 02:10:44 -06:00
meejah
3723945296 Actually validate configs when loading them
An indenting problem meant the validators weren't being
called, which revealed some follow-on errors.
2018-08-17 15:53:59 -06:00
Jean-Paul Calderone
e1c469e3b6 make sure we pass the client node 2018-04-23 11:41:29 -04:00
Jean-Paul Calderone
cfa33332a5 Add missing information/import 2018-04-23 11:09:24 -04:00
Jean-Paul Calderone
8d104dab1c Move the complicated MagicFolder constructor
All that complexity can be part of MagicFolder itself.
2018-04-23 10:59:33 -04:00
Jean-Paul Calderone
ac6269dd2d Only read magic-folder config from config reader
Also, fix the umask feature which was completely broken previously due
to failure to parse the umask string into an integer.
2018-04-23 10:41:48 -04:00
Jean-Paul Calderone
06a1ada624 Remove double-encoding of magic-folder params 2018-04-20 14:43:34 -04:00
Jean-Paul Calderone
1f92879788 improve test coverage 2018-03-19 14:21:28 -04:00
Ruben Pollan
136de7d7f7 Add storage_dir config field
On the [storage] section of the tahoe.cfg now there is a field
'storage_dir' where the path to the storage folder can be configured.
2018-03-19 11:20:04 -04:00
meejah
672475cb2b Multiple magic-folders
This moves all magic-folder configs to a single YAML
file. We load legacy config fine and don't mess with
legacy config unless you use a magic-folder command that
changes the config.

Increase test coverage
2017-12-05 10:34:48 -07:00
meejah
1b6f477549 Pull "config" handling out of Node and hide node-derivitives
This includes:

 - refactor to create_client and _Client
 - refactor to _IntroducerNode and create_introducer
2017-11-20 12:57:20 -07:00
meejah
05f48c3601 Various cleanups, fixes and improvements
Squashed all commits that were meejah's between
30d68fb499f300a393fa0ced5980229f4bb6efda
and
33c268ed3a8c63a809f4403e307ecc13d848b1ab
On the branch meejah:1382.markberger-rewrite-rebase.6 as
per review
2017-06-05 16:31:41 -06:00
meejah
49ae2c5eb7 use @implementer instead of implements
Also, derive some more things from 'object'
2017-02-27 11:01:30 -07:00
Brian Warner
64e344b03a introducer.yaml: ascii-fy furls before use
This fixes the foolscap-rejects-unicode-furls problem which prevented a
correctly-formatted introducers.yaml from working.

closes ticket:2862
2017-01-08 22:59:51 -08:00
Brian Warner
77fd41b66e update WUI welcome page with new connection-status info
This shows current-connection info, and provides per-hint status details in a
tooltip.

The "Connection" section no longer shows seconds-since-loss when the server
was not connected (previously it showed seconds-since-connect when connected,
and flipped to seconds-since-loss when disconnected). We already have the
"Last RX" column, which is arguably more meaningful (and I can't think of a
good case when these would differ), so we don't really need
seconds-since-loss, and the new ConnectionStatus doesn't track it anyways.

So now the "Connection" timestamp for non-connected servers is just
"N/A" (both the main text and the tooltip). The "Introducers" section was
changed the same way.

This moves the per-server connection timestamp out of the nickname/serverid
box and over into the Connection box. It also right-floats all timestamps,
regardless of which box they're in, which makes them share the box with
connection_status more politely.

Internally, this adds code to create ConnectionStatus objects when necessary.
2016-12-08 16:24:32 -08:00
Brian Warner
6879622894 create-node: avoid introducer.furl=None
Previously, "tahoe create-node" without an --introducer= argument would
result in the literal string "None" being written into tahoe.cfg:

 [client]
 introducer.furl = None

We were using config.get("introducer",""), but that didn't suffice because
the key was actually present: it just had a value of None, which then got
stringified into "None" when writing out tahoe.cfg.

This briefly caused test/cli/test_create to fail, as the startup code tried
to parse "None" as a FURL. This only happened against a development version
of Foolscap which accidentally became sensitive to unparseable FURLs in
started Reconnectors. I fixed that in the final foolscap-0.12.5 release, so
we shouldn't hit this bug, but I wanted to fix it properly in the tahoe-side
source.
2016-12-08 15:37:49 -08:00
meejah
0636d44cc1 add poll-interval config for magic-folder 2016-10-25 12:38:57 -07:00
meejah
ddaa29ce39 refactor/white-space MagicFolder creation 2016-10-25 12:38:57 -07:00
David Stainton
4f717ba808 Add a multi-introducer implementation
this is based the previous work of warner, killyourtv and leif
 fixes ticket #68
2016-09-13 00:15:52 +00:00
Brian Warner
74dc7194b6 validate_config: improve tests
closes ticket:2809
2016-09-06 19:23:51 -07:00
meejah
2732c37941 Check for unknown config options
The list of valid sections + config-items came from
grep'ing the source for `.get_config`
2016-09-06 15:05:46 -06:00
Brian Warner
fea8b627dd if tub.port is empty, don't listen
Updated config docs. Added errors if we're not listening but were told
to enable storage, helper, or if we're the Introducer server.

closes ticket:2816
2016-08-29 22:28:55 -07:00
Brian Warner
cac99569e9 factor out "Tub maker"
This adds Node._create_tub(), which knows how to make a Tub with all the
right options and connection handlers that were specified in
tahoe.cfg (the connection handlers are disabled for now, but they'll get
implemented soon).

The new Node.create_main_tub() calls it. This main Tub is used:

* to connect to the Introducer
* to host the Helper (if enabled)
* to host the Storage Server (if enabled)

Node._create_tub() is also passed into the StorageFarmBroker, which
passes it into each NativeStorageServer, to create the (separate) Tub
for each server connection. _create_tub knows about the options, and
NativeStorageServer can override the connection handlers. This way we
don't need to pass tub options or default handlers into Client,
StorageFarmBroker, or NativeStorageServer.

A number of tests create NativeStorageServer objects: these were updated
to match the new arguments. test_storage_client was simplified because
we no longer need to mock out the Tub() constructor.
2016-08-27 17:36:55 -07:00
Brian Warner
d75b9f822a Improve loading of static servers
This follows the latest comments in ticket:2788, moving the static
server definitions from "connections.yaml" to "servers.yaml". It removes
the "connections" and "introducers" blocks from that file, leaving it
responsible for just static servers (I think connections and introducers
can be configured from tahoe.cfg).

This feeds all the static server specs to the StorageFarmBroker in a
single call, rather than delivering them as simulated introducer
announcements. It cleans up the way handlers are specified too (the
handler dictionary is ignored, but that will change soon).
2016-08-26 18:05:03 -07:00
Brian Warner
a970828b94 dicts don't have .haskey()
Apparently we don't have test coverage for this yet.
2016-08-26 13:56:50 -07:00
David Stainton
140c9a92cd Make StorageFarmBroker use specified or default handler 2016-08-26 11:17:56 +00:00
David Stainton
61eb839843 Add tub_handlers arg to NativeStorageServer constructor
Here we also define tub_handlers as a node attribute and pass it all the way
down to the NativeStorageServer via the Client and StorageFarmBroker
2016-08-22 09:15:25 -07:00
meejah
8a33fc612a Add magic-folder indicator to main welcome page
This doesn't reveal very much information, but does tell
you if magic-folder is currently working and if not it will
indicate when the last attempt to do a remote scan was.
2016-08-09 18:07:00 -06:00
Brian Warner
1b64ab5e85 simplify when_connected_enough()
This seems happier as a method on StorageBroker, rather than a
completely separate helper class.
2016-07-21 17:23:22 -07:00
meejah
58f9641179 Correct magic-folder use of StorageFarmBroker
When landing the "connected enough" changes on master,
we changed the API; this fixes up usage of it in magic-folder.
2016-07-21 12:35:59 -07:00
meejah
86abe56d91 Flesh out "tahoe magic-folder status" command
Adds:

 - a JSON endpoint
 - CLI to display information
 - QueuedItem + IQueuedItem for uploader/downloader
 - IProgress interface + PercentProgress implementation
 - progress= args to many upload/download APIs
2016-07-21 12:35:59 -07:00
David Stainton
f2a7978b29 Add download.umask config option with default of 077 2016-07-21 12:35:59 -07:00
meejah
4ed2fb725a Fix call to ready() 2016-07-21 12:35:59 -07:00
Daira Hopwood
e03a1e3361 WIP.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-07-21 12:35:59 -07:00
Daira Hopwood
3170cab0fc Prepare to move drop_upload.py to magic_folder.py.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-07-21 12:35:59 -07:00
Daira Hopwood
976c89d2e4 Rename upload_ready_d to connected_enough_d.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-07-21 12:35:59 -07:00
Daira Hopwood
61b3a9cef2 Add magic folder db.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-07-21 12:35:59 -07:00
Daira Hopwood
7b4067850f Unicode path fixes for drop-upload.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-07-21 12:35:59 -07:00
Brian Warner
b6dedc6223 clean up got_static_announcement() call 2016-07-19 17:31:43 -07:00
Brian Warner
2c5f7ed425 factor out yamlutil.py
The yaml.SafeLoader.add_constructor() should probably only be done once,
and moving this all into a module gives us an opportunity to test it
directly.
2016-07-19 17:31:41 -07:00
Brian Warner
9c1e22209f client.py: add missing import, fix wrong name
Without these, clients with a non-empty connections.yaml would crash as
they start up. It's safe to say we need some tests for this :-).

pyflakes catches all of these, but it got accidentally disabled
recently, so travis wasn't running it. I'll fix that in the next commit.
2016-07-19 17:30:45 -07:00
David Stainton
9063106f3a Change 'servers' sections of connections.yaml to support new design in #2788 2016-07-12 17:44:21 +01:00
David Stainton
a67e7c0db9 Remove old self.testing reference and conditional 2016-07-12 17:44:21 +01:00
David Stainton
e4d93a745c Load static server configuration from connections.yaml 2016-07-12 17:44:21 +01:00
David Stainton
e5ffbdbcdf Move control/logging to a separate Tub 2016-07-06 21:09:35 -07:00
David Stainton
ae2b82a1f0 Add IntroducerClient write-only yaml cache file
this change also includes unit tests
2016-05-10 20:19:35 +00:00
Brian Warner
d1d988410b remove "key-generator" node type and client support
closes ticket:2783
2016-05-04 16:53:04 -07:00
Brian Warner
b9edccbeaa set the same options on all Tubs 2016-05-03 11:38:20 -07:00
David Stainton
6061b6fc3c WIP Storage broker client creates one tub per server
i was unable to get all the unit tests working;
this is my work in progress.
2016-05-02 15:23:07 +00:00
Brian Warner
96c625920c Teach tub to start synchronously even with ip autodetect
* remove when_tub_ready() from all code
* synchronous-ify all node/client/introducer startup code

refs ticket:2491
2016-04-26 21:54:45 -07:00
Brian Warner
1f6c04518e Client: init_helper() must run after init_client()
This has worked so far because everything waited for the Tub to be
ready. We'll soon be making Tub setup synchronous, so we won't have to
wait anymore, so the order will matter.
2016-04-26 18:35:03 -07:00
meejah
b834b71dac Change API to a listener-style, with helper 2016-04-26 13:40:02 -06:00
meejah
55898941da Get rid of the 'connected_threshold' argument, too
- use 'when_connected_to', which takes a threshold
2016-04-26 12:26:40 -06:00
meejah
4c619ac98d fix StorageFarmBroker constructors after connected_d gone 2016-04-26 12:26:40 -06:00
Daira Hopwood
a56a3adaae Teach StorageFarmBroker to fire a deferred when a connection threshold is reached. refs #1449
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-04-26 12:26:39 -06:00
meejah
4a25573e2d Add simple auth-token to get JSON data 2016-04-12 12:58:27 -06:00
Leif Ryge
96eaca6a6d new feature: preferred storage servers
this includes a squash merge of dca1de6856 which
was previously seen in pull request #128, as well as daira's suggested changes
from pull request #204.
2015-12-01 19:35:41 +00:00
Daira Hopwood
b6be693cbe Add a test, add missing imports. refs #2388
This tests ftpd, but not sftpd. Doing this sort of test on sftpd
requires the creation of a valid pubkey/privkey file pair, which is more
work than I want to do right now.

init_ftp/init_sftp were changed to interpret the configured
accounts.file as relative to the node's basedir, with
abspath_expanduser_unicode(accountfile, base=self.basedir).
This would happen naturally in a real node, since it os.chdir()s
to the basedir before doing anything. But tests don't do that.

Author: Brian Warner <warner@lothar.com>
Author: Daira Hopwood <daira@jacaranda.org>
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2015-03-24 17:10:00 +00:00
Daira Hopwood
3066039f0f Ensure that path parameters to SFTPServer and FTPServer constructors are unicode. refs #2388
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2015-03-24 16:46:40 +00:00
Daira Hopwood
14f783086f Change uses of os.path.expanduser and os.path.abspath. refs #2235
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2015-01-30 00:50:18 +00:00