Commit Graph

253 Commits

Author SHA1 Message Date
Ruben Pollan
136de7d7f7 Add storage_dir config field
On the [storage] section of the tahoe.cfg now there is a field
'storage_dir' where the path to the storage folder can be configured.
2018-03-19 11:20:04 -04:00
meejah
672475cb2b Multiple magic-folders
This moves all magic-folder configs to a single YAML
file. We load legacy config fine and don't mess with
legacy config unless you use a magic-folder command that
changes the config.

Increase test coverage
2017-12-05 10:34:48 -07:00
meejah
1b6f477549 Pull "config" handling out of Node and hide node-derivitives
This includes:

 - refactor to create_client and _Client
 - refactor to _IntroducerNode and create_introducer
2017-11-20 12:57:20 -07:00
meejah
05f48c3601 Various cleanups, fixes and improvements
Squashed all commits that were meejah's between
30d68fb499f300a393fa0ced5980229f4bb6efda
and
33c268ed3a8c63a809f4403e307ecc13d848b1ab
On the branch meejah:1382.markberger-rewrite-rebase.6 as
per review
2017-06-05 16:31:41 -06:00
meejah
49ae2c5eb7 use @implementer instead of implements
Also, derive some more things from 'object'
2017-02-27 11:01:30 -07:00
Brian Warner
64e344b03a introducer.yaml: ascii-fy furls before use
This fixes the foolscap-rejects-unicode-furls problem which prevented a
correctly-formatted introducers.yaml from working.

closes ticket:2862
2017-01-08 22:59:51 -08:00
Brian Warner
77fd41b66e update WUI welcome page with new connection-status info
This shows current-connection info, and provides per-hint status details in a
tooltip.

The "Connection" section no longer shows seconds-since-loss when the server
was not connected (previously it showed seconds-since-connect when connected,
and flipped to seconds-since-loss when disconnected). We already have the
"Last RX" column, which is arguably more meaningful (and I can't think of a
good case when these would differ), so we don't really need
seconds-since-loss, and the new ConnectionStatus doesn't track it anyways.

So now the "Connection" timestamp for non-connected servers is just
"N/A" (both the main text and the tooltip). The "Introducers" section was
changed the same way.

This moves the per-server connection timestamp out of the nickname/serverid
box and over into the Connection box. It also right-floats all timestamps,
regardless of which box they're in, which makes them share the box with
connection_status more politely.

Internally, this adds code to create ConnectionStatus objects when necessary.
2016-12-08 16:24:32 -08:00
Brian Warner
6879622894 create-node: avoid introducer.furl=None
Previously, "tahoe create-node" without an --introducer= argument would
result in the literal string "None" being written into tahoe.cfg:

 [client]
 introducer.furl = None

We were using config.get("introducer",""), but that didn't suffice because
the key was actually present: it just had a value of None, which then got
stringified into "None" when writing out tahoe.cfg.

This briefly caused test/cli/test_create to fail, as the startup code tried
to parse "None" as a FURL. This only happened against a development version
of Foolscap which accidentally became sensitive to unparseable FURLs in
started Reconnectors. I fixed that in the final foolscap-0.12.5 release, so
we shouldn't hit this bug, but I wanted to fix it properly in the tahoe-side
source.
2016-12-08 15:37:49 -08:00
meejah
0636d44cc1 add poll-interval config for magic-folder 2016-10-25 12:38:57 -07:00
meejah
ddaa29ce39 refactor/white-space MagicFolder creation 2016-10-25 12:38:57 -07:00
David Stainton
4f717ba808 Add a multi-introducer implementation
this is based the previous work of warner, killyourtv and leif
 fixes ticket #68
2016-09-13 00:15:52 +00:00
Brian Warner
74dc7194b6 validate_config: improve tests
closes ticket:2809
2016-09-06 19:23:51 -07:00
meejah
2732c37941 Check for unknown config options
The list of valid sections + config-items came from
grep'ing the source for `.get_config`
2016-09-06 15:05:46 -06:00
Brian Warner
fea8b627dd if tub.port is empty, don't listen
Updated config docs. Added errors if we're not listening but were told
to enable storage, helper, or if we're the Introducer server.

closes ticket:2816
2016-08-29 22:28:55 -07:00
Brian Warner
cac99569e9 factor out "Tub maker"
This adds Node._create_tub(), which knows how to make a Tub with all the
right options and connection handlers that were specified in
tahoe.cfg (the connection handlers are disabled for now, but they'll get
implemented soon).

The new Node.create_main_tub() calls it. This main Tub is used:

* to connect to the Introducer
* to host the Helper (if enabled)
* to host the Storage Server (if enabled)

Node._create_tub() is also passed into the StorageFarmBroker, which
passes it into each NativeStorageServer, to create the (separate) Tub
for each server connection. _create_tub knows about the options, and
NativeStorageServer can override the connection handlers. This way we
don't need to pass tub options or default handlers into Client,
StorageFarmBroker, or NativeStorageServer.

A number of tests create NativeStorageServer objects: these were updated
to match the new arguments. test_storage_client was simplified because
we no longer need to mock out the Tub() constructor.
2016-08-27 17:36:55 -07:00
Brian Warner
d75b9f822a Improve loading of static servers
This follows the latest comments in ticket:2788, moving the static
server definitions from "connections.yaml" to "servers.yaml". It removes
the "connections" and "introducers" blocks from that file, leaving it
responsible for just static servers (I think connections and introducers
can be configured from tahoe.cfg).

This feeds all the static server specs to the StorageFarmBroker in a
single call, rather than delivering them as simulated introducer
announcements. It cleans up the way handlers are specified too (the
handler dictionary is ignored, but that will change soon).
2016-08-26 18:05:03 -07:00
Brian Warner
a970828b94 dicts don't have .haskey()
Apparently we don't have test coverage for this yet.
2016-08-26 13:56:50 -07:00
David Stainton
140c9a92cd Make StorageFarmBroker use specified or default handler 2016-08-26 11:17:56 +00:00
David Stainton
61eb839843 Add tub_handlers arg to NativeStorageServer constructor
Here we also define tub_handlers as a node attribute and pass it all the way
down to the NativeStorageServer via the Client and StorageFarmBroker
2016-08-22 09:15:25 -07:00
meejah
8a33fc612a Add magic-folder indicator to main welcome page
This doesn't reveal very much information, but does tell
you if magic-folder is currently working and if not it will
indicate when the last attempt to do a remote scan was.
2016-08-09 18:07:00 -06:00
Brian Warner
1b64ab5e85 simplify when_connected_enough()
This seems happier as a method on StorageBroker, rather than a
completely separate helper class.
2016-07-21 17:23:22 -07:00
meejah
58f9641179 Correct magic-folder use of StorageFarmBroker
When landing the "connected enough" changes on master,
we changed the API; this fixes up usage of it in magic-folder.
2016-07-21 12:35:59 -07:00
meejah
86abe56d91 Flesh out "tahoe magic-folder status" command
Adds:

 - a JSON endpoint
 - CLI to display information
 - QueuedItem + IQueuedItem for uploader/downloader
 - IProgress interface + PercentProgress implementation
 - progress= args to many upload/download APIs
2016-07-21 12:35:59 -07:00
David Stainton
f2a7978b29 Add download.umask config option with default of 077 2016-07-21 12:35:59 -07:00
meejah
4ed2fb725a Fix call to ready() 2016-07-21 12:35:59 -07:00
Daira Hopwood
e03a1e3361 WIP.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-07-21 12:35:59 -07:00
Daira Hopwood
3170cab0fc Prepare to move drop_upload.py to magic_folder.py.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-07-21 12:35:59 -07:00
Daira Hopwood
976c89d2e4 Rename upload_ready_d to connected_enough_d.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-07-21 12:35:59 -07:00
Daira Hopwood
61b3a9cef2 Add magic folder db.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-07-21 12:35:59 -07:00
Daira Hopwood
7b4067850f Unicode path fixes for drop-upload.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-07-21 12:35:59 -07:00
Brian Warner
b6dedc6223 clean up got_static_announcement() call 2016-07-19 17:31:43 -07:00
Brian Warner
2c5f7ed425 factor out yamlutil.py
The yaml.SafeLoader.add_constructor() should probably only be done once,
and moving this all into a module gives us an opportunity to test it
directly.
2016-07-19 17:31:41 -07:00
Brian Warner
9c1e22209f client.py: add missing import, fix wrong name
Without these, clients with a non-empty connections.yaml would crash as
they start up. It's safe to say we need some tests for this :-).

pyflakes catches all of these, but it got accidentally disabled
recently, so travis wasn't running it. I'll fix that in the next commit.
2016-07-19 17:30:45 -07:00
David Stainton
9063106f3a Change 'servers' sections of connections.yaml to support new design in #2788 2016-07-12 17:44:21 +01:00
David Stainton
a67e7c0db9 Remove old self.testing reference and conditional 2016-07-12 17:44:21 +01:00
David Stainton
e4d93a745c Load static server configuration from connections.yaml 2016-07-12 17:44:21 +01:00
David Stainton
e5ffbdbcdf Move control/logging to a separate Tub 2016-07-06 21:09:35 -07:00
David Stainton
ae2b82a1f0 Add IntroducerClient write-only yaml cache file
this change also includes unit tests
2016-05-10 20:19:35 +00:00
Brian Warner
d1d988410b remove "key-generator" node type and client support
closes ticket:2783
2016-05-04 16:53:04 -07:00
Brian Warner
b9edccbeaa set the same options on all Tubs 2016-05-03 11:38:20 -07:00
David Stainton
6061b6fc3c WIP Storage broker client creates one tub per server
i was unable to get all the unit tests working;
this is my work in progress.
2016-05-02 15:23:07 +00:00
Brian Warner
96c625920c Teach tub to start synchronously even with ip autodetect
* remove when_tub_ready() from all code
* synchronous-ify all node/client/introducer startup code

refs ticket:2491
2016-04-26 21:54:45 -07:00
Brian Warner
1f6c04518e Client: init_helper() must run after init_client()
This has worked so far because everything waited for the Tub to be
ready. We'll soon be making Tub setup synchronous, so we won't have to
wait anymore, so the order will matter.
2016-04-26 18:35:03 -07:00
meejah
b834b71dac Change API to a listener-style, with helper 2016-04-26 13:40:02 -06:00
meejah
55898941da Get rid of the 'connected_threshold' argument, too
- use 'when_connected_to', which takes a threshold
2016-04-26 12:26:40 -06:00
meejah
4c619ac98d fix StorageFarmBroker constructors after connected_d gone 2016-04-26 12:26:40 -06:00
Daira Hopwood
a56a3adaae Teach StorageFarmBroker to fire a deferred when a connection threshold is reached. refs #1449
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-04-26 12:26:39 -06:00
meejah
4a25573e2d Add simple auth-token to get JSON data 2016-04-12 12:58:27 -06:00
Leif Ryge
96eaca6a6d new feature: preferred storage servers
this includes a squash merge of dca1de6856 which
was previously seen in pull request #128, as well as daira's suggested changes
from pull request #204.
2015-12-01 19:35:41 +00:00
Daira Hopwood
b6be693cbe Add a test, add missing imports. refs #2388
This tests ftpd, but not sftpd. Doing this sort of test on sftpd
requires the creation of a valid pubkey/privkey file pair, which is more
work than I want to do right now.

init_ftp/init_sftp were changed to interpret the configured
accounts.file as relative to the node's basedir, with
abspath_expanduser_unicode(accountfile, base=self.basedir).
This would happen naturally in a real node, since it os.chdir()s
to the basedir before doing anything. But tests don't do that.

Author: Brian Warner <warner@lothar.com>
Author: Daira Hopwood <daira@jacaranda.org>
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2015-03-24 17:10:00 +00:00