Commit Graph

81 Commits

Author SHA1 Message Date
Jean-Paul Calderone
d4b5de2e08
Merge pull request #653 from tahoe-lafs/3248.pass-config-to-get_storage_client
Pass the full _Config to IFoolscapStoragePlugin.get_storage_client

Fixes: ticket:3248
2019-08-23 09:04:23 -04:00
Jean-Paul Calderone
25c476aaa8
Merge pull request #654 from tahoe-lafs/3250.synchronous-get_storage_client
Synchronous IFoolscapStoragePlugin.get_storage_client

Fixes: ticket:3250
2019-08-23 08:44:52 -04:00
Jean-Paul Calderone
e62d2a5a27 In reality we cannot handle a Deferred here
... yet?  This code is invoked from an `__init__` where async is always
tricky.  Maybe we can invert the relationship someday.
2019-08-20 09:28:05 -04:00
Jean-Paul Calderone
9940beaae1 Thread a _Config all the way down 2019-08-19 16:09:26 -04:00
Jean-Paul Calderone
a47463e032 Pass _Config instead of a smaller dict to get_client_resource 2019-08-19 11:21:03 -04:00
Jean-Paul Calderone
21d735ece9 whitespace more conforming to PEP8 2019-08-03 06:39:31 -04:00
Jean-Paul Calderone
d69de15664 implement the feature
improve the test slightly, too, to verify the configuration supplied to the
plugin is as expected.
2019-08-03 06:39:31 -04:00
Jean-Paul Calderone
375f917607 Be explicit that we expect to be operating on bytes here 2019-08-03 06:39:31 -04:00
Jean-Paul Calderone
311afa8a75 Test & fix supplying plugin configuration 2019-08-03 06:39:31 -04:00
Jean-Paul Calderone
57160f65c6 Pass get_rref in to get_storage_client
plugins don't otherwise have a way to talk to the server.
2019-08-03 06:39:31 -04:00
Jean-Paul Calderone
2616c66a49 Fix confusion between IStorageServer and the thing above it
IStorageServer is what uses a connection.  You need a thing above it to _get_
a connection.
2019-08-03 06:39:31 -04:00
Jean-Paul Calderone
9743a1ab4e docstring 2019-08-03 06:39:31 -04:00
Jean-Paul Calderone
bee3ee8ff1 docstrings 2019-08-03 06:39:31 -04:00
Jean-Paul Calderone
11418a9f87 Fix test_add_rref users by making them supply coherent values
... of the right type
2019-08-03 06:39:31 -04:00
Jean-Paul Calderone
166c5ab53f there is only a NativeStorageServer
though it is used by the client
2019-08-03 06:39:31 -04:00
Jean-Paul Calderone
48b8bd6eb0 pass config down and use it to make the client plugin 2019-08-03 06:39:31 -04:00
Jean-Paul Calderone
3c3ebc368a note about some logging we should do 2019-08-03 06:39:31 -04:00
Jean-Paul Calderone
2e0e9f0cad remove duplication of NativeStorageServer instantiation 2019-08-03 06:39:31 -04:00
Jean-Paul Calderone
bbd1c706e4 teach StorageConfigClient to load storage client plugins 2019-08-03 06:39:31 -04:00
Jean-Paul Calderone
b5a2c70a4a create a StorageClientConfig object
Make it easier to pass more storage configuration down into StorageFarmBroker
and beyond
2019-08-03 06:39:31 -04:00
Jean-Paul Calderone
87b37a7e27 be more data-type-y 2019-08-03 05:34:21 -04:00
Jean-Paul Calderone
1c6433b43b Factor details of the storage announcement out of NativeStorageClient
A separate object can be responsible for the details of each kind of announcement.
2019-08-03 05:34:21 -04:00
Jean-Paul Calderone
e0a31aebf5 Sort the static storage servers for deterministic tests 2019-06-21 08:38:57 -04:00
Jean-Paul Calderone
f6ad8fa56b Make the new test pass by catching and logging 2019-06-13 09:08:42 -04:00
Jean-Paul Calderone
ad29e627cc refer to the interface 2019-06-11 16:32:29 -04:00
Jean-Paul Calderone
d32d020b05 Remove another test use of get_rref 2019-05-31 16:09:29 -04:00
Jean-Paul Calderone
46a2065357 Offer a pass-through IStorageServer 2019-05-31 13:40:51 -04:00
Jean-Paul Calderone
8f4c994bea Make NativeStorageServer.rref private 2019-05-31 11:09:20 -04:00
meejah
49ae2c5eb7 use @implementer instead of implements
Also, derive some more things from 'object'
2017-02-27 11:01:30 -07:00
Brian Warner
38935bbca6 remove old unused connection-status APIs 2016-12-08 16:25:00 -08:00
Brian Warner
77fd41b66e update WUI welcome page with new connection-status info
This shows current-connection info, and provides per-hint status details in a
tooltip.

The "Connection" section no longer shows seconds-since-loss when the server
was not connected (previously it showed seconds-since-connect when connected,
and flipped to seconds-since-loss when disconnected). We already have the
"Last RX" column, which is arguably more meaningful (and I can't think of a
good case when these would differ), so we don't really need
seconds-since-loss, and the new ConnectionStatus doesn't track it anyways.

So now the "Connection" timestamp for non-connected servers is just
"N/A" (both the main text and the tooltip). The "Introducers" section was
changed the same way.

This moves the per-server connection timestamp out of the nickname/serverid
box and over into the Connection box. It also right-floats all timestamps,
regardless of which box they're in, which makes them share the box with
connection_status more politely.

Internally, this adds code to create ConnectionStatus objects when necessary.
2016-12-08 16:24:32 -08:00
Brian Warner
5ef04ef59e add hashutil.permute_server_hash
which uses SHA1 to combine the file's storage index (known as "peer
selection index" in this context) and each server's "server permutation
seed". This is the only thing in tahoe that uses SHA1.

With this change, we stop importing sha1 from anywhere else.
2016-09-26 20:42:42 -07:00
David Stainton
f88ae3861a Make the grid status display the location hints
Instead of displaying what appears to be the remote peer address
we display the list of connection hints.

closes ticket:2818
2016-09-02 11:09:16 -07:00
Brian Warner
c56a0a4ba9 fix unicode handling in server_id from YAML
YAML, like JSON, is all-unicode. StorageFarmBroker.set_static_servers()
is defined to take an all-unicode dictionary (the "storage:" key from
private/servers.yaml), so the server_id keys it gets will be unicode.
NativeStorageServer is defined to accept server_ids which are bytes (at
least it is now). The tests were only passing bytes into
set_static_servers(), whereas a real launch passed unicode in, causing a
problem when NativeStorageServer tried to base32.a2b() the pubkey and
choked on the unicode it received.

This fixes set_static_servers() to convert the server_id to bytes, and
changes NativeStorageServer to assert that it gets bytes. It also fixes
the test to match real usage more closely.
2016-08-28 02:42:51 -07:00
Brian Warner
cac99569e9 factor out "Tub maker"
This adds Node._create_tub(), which knows how to make a Tub with all the
right options and connection handlers that were specified in
tahoe.cfg (the connection handlers are disabled for now, but they'll get
implemented soon).

The new Node.create_main_tub() calls it. This main Tub is used:

* to connect to the Introducer
* to host the Helper (if enabled)
* to host the Storage Server (if enabled)

Node._create_tub() is also passed into the StorageFarmBroker, which
passes it into each NativeStorageServer, to create the (separate) Tub
for each server connection. _create_tub knows about the options, and
NativeStorageServer can override the connection handlers. This way we
don't need to pass tub options or default handlers into Client,
StorageFarmBroker, or NativeStorageServer.

A number of tests create NativeStorageServer objects: these were updated
to match the new arguments. test_storage_client was simplified because
we no longer need to mock out the Tub() constructor.
2016-08-27 17:36:55 -07:00
Brian Warner
8cf53d2d12 derive permutation seed from pubkey/server_id 2016-08-27 13:08:18 -07:00
Brian Warner
663e39593b static servers: tolerate missing nickname/versions
A minimally-defined static server only specifies server_id,
anonymous-storage-FURL, and permutation-seed-base32. But the WUI Welcome
page wouldn't render (it raised an exception) without also defining
nickname and version. This allows those values to be missing.
2016-08-27 11:34:32 -07:00
Brian Warner
d75b9f822a Improve loading of static servers
This follows the latest comments in ticket:2788, moving the static
server definitions from "connections.yaml" to "servers.yaml". It removes
the "connections" and "introducers" blocks from that file, leaving it
responsible for just static servers (I think connections and introducers
can be configured from tahoe.cfg).

This feeds all the static server specs to the StorageFarmBroker in a
single call, rather than delivering them as simulated introducer
announcements. It cleans up the way handlers are specified too (the
handler dictionary is ignored, but that will change soon).
2016-08-26 18:05:03 -07:00
Brian Warner
f23660e178 NativeStorageServer: create with server_id, not key_s
They're the same thing, but knowing that is the responsibility of the
caller, not NativeStorageServer. Try to normalize on "server_id" as the
spelling. Remove support for missing key_s, now that we require V2
introductions.
2016-08-26 18:04:31 -07:00
Brian Warner
54f974d44c make IServer.get_serverid() use pubkey, not tubid
This is a change I've wanted to make for many years, because when we get
to HTTP-based servers, we won't have tubids for them. What held me back
was that there's code all over the place that uses the serverid for
various purposes, so I wasn't sure it was safe. I did a big push a few
years ago to use IServer instances instead of serverids in most
places (in #1363), and to split out the values that actually depend upon
tubid into separate accessors (like get_lease_seed and
get_foolscap_write_enabler_seed), which I think took care of all the
important uses.

There are a number of places that use get_serverid() as dictionary key
to track shares (Checker results, mutable servermap). I believe these
are happy to use pubkeys instead of tubids: the only thing they do with
get_serverid() is to compare it to other values obtained from
get_serverid(). A few places in the WUI used serverid to compute display
values: these were fixed.

The main trouble was the Helper: it returns a HelperUploadResults (a
Copyable) with a share->server mapping that's keyed by whatever the
Helper's get_serverid() returns. If the uploader and the helper are on
different sides of this change, the Helper could return values that the
uploader won't recognize. This is cosmetic: that mapping is only used to
display the upload results on the "Recent and Active Operations" page.
I've added code to StorageFarmBroker.get_stub_server() to fall back to
tubids when looking up a server, so this should still work correctly
when the uploader is new and the Helper is old. If the Helper is new and
the uploader is old, the upload results will show unusual server ids.

refs ticket:1363
2016-08-26 14:29:50 -07:00
David Stainton
140c9a92cd Make StorageFarmBroker use specified or default handler 2016-08-26 11:17:56 +00:00
David Stainton
de61cd260c Add got_static_announcement and unit test 2016-08-24 21:11:58 +00:00
David Stainton
61eb839843 Add tub_handlers arg to NativeStorageServer constructor
Here we also define tub_handlers as a node attribute and pass it all the way
down to the NativeStorageServer via the Client and StorageFarmBroker
2016-08-22 09:15:25 -07:00
Brian Warner
1b64ab5e85 simplify when_connected_enough()
This seems happier as a method on StorageBroker, rather than a
completely separate helper class.
2016-07-21 17:23:22 -07:00
Brian Warner
bbeb397a00 storage_client: remove unnecessary startService upcall 2016-05-03 15:48:08 -07:00
Brian Warner
76f6d2d21a create Tub inside NativeStorageServer, not broker
This makes IServer instances responsible for their own network
connections, which will help when we add HTTP-based servers in the
future. The StorageFarmBroker should not care about how the IServer uses
the network, it just provides the announcement (and local config).
2016-05-03 15:09:13 -07:00
Brian Warner
b9edccbeaa set the same options on all Tubs 2016-05-03 11:38:20 -07:00
David Stainton
6061b6fc3c WIP Storage broker client creates one tub per server
i was unable to get all the unit tests working;
this is my work in progress.
2016-05-02 15:23:07 +00:00
meejah
b834b71dac Change API to a listener-style, with helper 2016-04-26 13:40:02 -06:00
meejah
55898941da Get rid of the 'connected_threshold' argument, too
- use 'when_connected_to', which takes a threshold
2016-04-26 12:26:40 -06:00