Commit Graph

68 Commits

Author SHA1 Message Date
Jean-Paul Calderone
11418a9f87 Fix test_add_rref users by making them supply coherent values
... of the right type
2019-08-03 06:39:31 -04:00
Jean-Paul Calderone
166c5ab53f there is only a NativeStorageServer
though it is used by the client
2019-08-03 06:39:31 -04:00
Jean-Paul Calderone
48b8bd6eb0 pass config down and use it to make the client plugin 2019-08-03 06:39:31 -04:00
Jean-Paul Calderone
3c3ebc368a note about some logging we should do 2019-08-03 06:39:31 -04:00
Jean-Paul Calderone
2e0e9f0cad remove duplication of NativeStorageServer instantiation 2019-08-03 06:39:31 -04:00
Jean-Paul Calderone
bbd1c706e4 teach StorageConfigClient to load storage client plugins 2019-08-03 06:39:31 -04:00
Jean-Paul Calderone
b5a2c70a4a create a StorageClientConfig object
Make it easier to pass more storage configuration down into StorageFarmBroker
and beyond
2019-08-03 06:39:31 -04:00
Jean-Paul Calderone
87b37a7e27 be more data-type-y 2019-08-03 05:34:21 -04:00
Jean-Paul Calderone
1c6433b43b Factor details of the storage announcement out of NativeStorageClient
A separate object can be responsible for the details of each kind of announcement.
2019-08-03 05:34:21 -04:00
Jean-Paul Calderone
e0a31aebf5 Sort the static storage servers for deterministic tests 2019-06-21 08:38:57 -04:00
Jean-Paul Calderone
f6ad8fa56b Make the new test pass by catching and logging 2019-06-13 09:08:42 -04:00
Jean-Paul Calderone
ad29e627cc refer to the interface 2019-06-11 16:32:29 -04:00
Jean-Paul Calderone
d32d020b05 Remove another test use of get_rref 2019-05-31 16:09:29 -04:00
Jean-Paul Calderone
46a2065357 Offer a pass-through IStorageServer 2019-05-31 13:40:51 -04:00
Jean-Paul Calderone
8f4c994bea Make NativeStorageServer.rref private 2019-05-31 11:09:20 -04:00
meejah
49ae2c5eb7 use @implementer instead of implements
Also, derive some more things from 'object'
2017-02-27 11:01:30 -07:00
Brian Warner
38935bbca6 remove old unused connection-status APIs 2016-12-08 16:25:00 -08:00
Brian Warner
77fd41b66e update WUI welcome page with new connection-status info
This shows current-connection info, and provides per-hint status details in a
tooltip.

The "Connection" section no longer shows seconds-since-loss when the server
was not connected (previously it showed seconds-since-connect when connected,
and flipped to seconds-since-loss when disconnected). We already have the
"Last RX" column, which is arguably more meaningful (and I can't think of a
good case when these would differ), so we don't really need
seconds-since-loss, and the new ConnectionStatus doesn't track it anyways.

So now the "Connection" timestamp for non-connected servers is just
"N/A" (both the main text and the tooltip). The "Introducers" section was
changed the same way.

This moves the per-server connection timestamp out of the nickname/serverid
box and over into the Connection box. It also right-floats all timestamps,
regardless of which box they're in, which makes them share the box with
connection_status more politely.

Internally, this adds code to create ConnectionStatus objects when necessary.
2016-12-08 16:24:32 -08:00
Brian Warner
5ef04ef59e add hashutil.permute_server_hash
which uses SHA1 to combine the file's storage index (known as "peer
selection index" in this context) and each server's "server permutation
seed". This is the only thing in tahoe that uses SHA1.

With this change, we stop importing sha1 from anywhere else.
2016-09-26 20:42:42 -07:00
David Stainton
f88ae3861a Make the grid status display the location hints
Instead of displaying what appears to be the remote peer address
we display the list of connection hints.

closes ticket:2818
2016-09-02 11:09:16 -07:00
Brian Warner
c56a0a4ba9 fix unicode handling in server_id from YAML
YAML, like JSON, is all-unicode. StorageFarmBroker.set_static_servers()
is defined to take an all-unicode dictionary (the "storage:" key from
private/servers.yaml), so the server_id keys it gets will be unicode.
NativeStorageServer is defined to accept server_ids which are bytes (at
least it is now). The tests were only passing bytes into
set_static_servers(), whereas a real launch passed unicode in, causing a
problem when NativeStorageServer tried to base32.a2b() the pubkey and
choked on the unicode it received.

This fixes set_static_servers() to convert the server_id to bytes, and
changes NativeStorageServer to assert that it gets bytes. It also fixes
the test to match real usage more closely.
2016-08-28 02:42:51 -07:00
Brian Warner
cac99569e9 factor out "Tub maker"
This adds Node._create_tub(), which knows how to make a Tub with all the
right options and connection handlers that were specified in
tahoe.cfg (the connection handlers are disabled for now, but they'll get
implemented soon).

The new Node.create_main_tub() calls it. This main Tub is used:

* to connect to the Introducer
* to host the Helper (if enabled)
* to host the Storage Server (if enabled)

Node._create_tub() is also passed into the StorageFarmBroker, which
passes it into each NativeStorageServer, to create the (separate) Tub
for each server connection. _create_tub knows about the options, and
NativeStorageServer can override the connection handlers. This way we
don't need to pass tub options or default handlers into Client,
StorageFarmBroker, or NativeStorageServer.

A number of tests create NativeStorageServer objects: these were updated
to match the new arguments. test_storage_client was simplified because
we no longer need to mock out the Tub() constructor.
2016-08-27 17:36:55 -07:00
Brian Warner
8cf53d2d12 derive permutation seed from pubkey/server_id 2016-08-27 13:08:18 -07:00
Brian Warner
663e39593b static servers: tolerate missing nickname/versions
A minimally-defined static server only specifies server_id,
anonymous-storage-FURL, and permutation-seed-base32. But the WUI Welcome
page wouldn't render (it raised an exception) without also defining
nickname and version. This allows those values to be missing.
2016-08-27 11:34:32 -07:00
Brian Warner
d75b9f822a Improve loading of static servers
This follows the latest comments in ticket:2788, moving the static
server definitions from "connections.yaml" to "servers.yaml". It removes
the "connections" and "introducers" blocks from that file, leaving it
responsible for just static servers (I think connections and introducers
can be configured from tahoe.cfg).

This feeds all the static server specs to the StorageFarmBroker in a
single call, rather than delivering them as simulated introducer
announcements. It cleans up the way handlers are specified too (the
handler dictionary is ignored, but that will change soon).
2016-08-26 18:05:03 -07:00
Brian Warner
f23660e178 NativeStorageServer: create with server_id, not key_s
They're the same thing, but knowing that is the responsibility of the
caller, not NativeStorageServer. Try to normalize on "server_id" as the
spelling. Remove support for missing key_s, now that we require V2
introductions.
2016-08-26 18:04:31 -07:00
Brian Warner
54f974d44c make IServer.get_serverid() use pubkey, not tubid
This is a change I've wanted to make for many years, because when we get
to HTTP-based servers, we won't have tubids for them. What held me back
was that there's code all over the place that uses the serverid for
various purposes, so I wasn't sure it was safe. I did a big push a few
years ago to use IServer instances instead of serverids in most
places (in #1363), and to split out the values that actually depend upon
tubid into separate accessors (like get_lease_seed and
get_foolscap_write_enabler_seed), which I think took care of all the
important uses.

There are a number of places that use get_serverid() as dictionary key
to track shares (Checker results, mutable servermap). I believe these
are happy to use pubkeys instead of tubids: the only thing they do with
get_serverid() is to compare it to other values obtained from
get_serverid(). A few places in the WUI used serverid to compute display
values: these were fixed.

The main trouble was the Helper: it returns a HelperUploadResults (a
Copyable) with a share->server mapping that's keyed by whatever the
Helper's get_serverid() returns. If the uploader and the helper are on
different sides of this change, the Helper could return values that the
uploader won't recognize. This is cosmetic: that mapping is only used to
display the upload results on the "Recent and Active Operations" page.
I've added code to StorageFarmBroker.get_stub_server() to fall back to
tubids when looking up a server, so this should still work correctly
when the uploader is new and the Helper is old. If the Helper is new and
the uploader is old, the upload results will show unusual server ids.

refs ticket:1363
2016-08-26 14:29:50 -07:00
David Stainton
140c9a92cd Make StorageFarmBroker use specified or default handler 2016-08-26 11:17:56 +00:00
David Stainton
de61cd260c Add got_static_announcement and unit test 2016-08-24 21:11:58 +00:00
David Stainton
61eb839843 Add tub_handlers arg to NativeStorageServer constructor
Here we also define tub_handlers as a node attribute and pass it all the way
down to the NativeStorageServer via the Client and StorageFarmBroker
2016-08-22 09:15:25 -07:00
Brian Warner
1b64ab5e85 simplify when_connected_enough()
This seems happier as a method on StorageBroker, rather than a
completely separate helper class.
2016-07-21 17:23:22 -07:00
Brian Warner
bbeb397a00 storage_client: remove unnecessary startService upcall 2016-05-03 15:48:08 -07:00
Brian Warner
76f6d2d21a create Tub inside NativeStorageServer, not broker
This makes IServer instances responsible for their own network
connections, which will help when we add HTTP-based servers in the
future. The StorageFarmBroker should not care about how the IServer uses
the network, it just provides the announcement (and local config).
2016-05-03 15:09:13 -07:00
Brian Warner
b9edccbeaa set the same options on all Tubs 2016-05-03 11:38:20 -07:00
David Stainton
6061b6fc3c WIP Storage broker client creates one tub per server
i was unable to get all the unit tests working;
this is my work in progress.
2016-05-02 15:23:07 +00:00
meejah
b834b71dac Change API to a listener-style, with helper 2016-04-26 13:40:02 -06:00
meejah
55898941da Get rid of the 'connected_threshold' argument, too
- use 'when_connected_to', which takes a threshold
2016-04-26 12:26:40 -06:00
meejah
b65a8fe142 Switch to when_connected_enough()
- instead of passing in a Deferred(), we use an observer
- fix up the tests
- TODO: fix magic-folder
2016-04-26 12:26:40 -06:00
Daira Hopwood
a56a3adaae Teach StorageFarmBroker to fire a deferred when a connection threshold is reached. refs #1449
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-04-26 12:26:39 -06:00
Leif Ryge
5b33f1fdf3 wui: improved columns in welcome page server list
As discussed at https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1973 and in
previous pull request #129.

 - replace lengthy timestamps with human-readable deltas (eg 1h 2m 3s)
 - replace "announced" column with "Last RX" column
 - remove service column (it always said the same thing, "storage")
 - fix colspan on 'You are not presently connected' message

Previous versions, some with github comments: 3fe9053134 , 486dbfc7bd , and c89ea62580, 9fabb92486, bbd8b42a25

Unlike previous attempts, the tests on this one should pass in any timezone.
(But like current master, will fail with Nevow >=0.12...)

Thanks to an anonymous contributor who wrote some of the tests.
2016-01-23 14:28:09 +00:00
Leif Ryge
c86388b79c revert previous commit to fix attribution (vanity)
This reverts the previous commit (modulo the one line that daira added) so that
my contributions will be properly attributed by git blame etc.
2016-01-23 14:27:11 +00:00
Daira Hopwood
a2d724aab7 wui: improved columns in welcome page server list
As discussed at https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1973 and in
previous pull request #129.

 - replace lengthy timestamps with human-readable deltas (eg 1h 2m 3s)
 - replace "announced" column with "Last RX" column
 - remove service column (it always said the same thing, "storage")
 - fix colspan on 'You are not presently connected' message

Previous versions, some with github comments: 3fe9053134 , 486dbfc7bd , and c89ea62580, 9fabb92486, bbd8b42a25

Unlike previous attempts, the tests on this one should pass in any timezone.
(But like current master, will fail with Nevow >=0.12...)

Thanks to an anonymous contributor who wrote some of the tests.
2016-01-15 20:02:19 +00:00
Leif Ryge
96eaca6a6d new feature: preferred storage servers
this includes a squash merge of dca1de6856 which
was previously seen in pull request #128, as well as daira's suggested changes
from pull request #204.
2015-12-01 19:35:41 +00:00
Leif Ryge
b9e0d19944 add available-space to VERSION_DEFAULTS
as per warner's suggestion in https://github.com/warner/tahoe-lafs/pull/4
2014-11-23 06:03:00 +00:00
Leif Ryge
335c2ed06a add "Available" column to welcome page (#648)
add get_available_space() to NativeStorageServer

It uses a new 'available-space' key in the server's v1 version dict, or falls
back to 'maximum-immutable-share-size' (which presently always has the same
value but could have a different meaning in the future).

This is a squash merge of 9773555bb87fab71145ad7a0e84785a4e92d11f7
2014-11-23 06:03:00 +00:00
Leif Ryge
bca538ac64 rm NativeStorageServer's unused min_shares argument 2014-11-22 04:08:45 +00:00
David-Sarah Hopwood
68b7f9e979 Correct off-by-one in maximum-immutable-share-size for old servers. fixes #1781
Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>
2013-03-23 03:10:50 +00:00
David-Sarah Hopwood
0d4a826f0e Allow welcome and storage status pages to display a non-ASCII node nickname. fixes #1298
Signed-off-by: David-Sarah Hopwood <davidsarah@jacaranda.org>
2012-12-27 05:08:53 +00:00
david-sarah
33e2d2962e Change the maximum mutable share size to 69105 TB, and add a maximum-mutable-share-size field to the version announcement. Includes a test. refs #1778 2012-06-22 15:43:54 +00:00
david-sarah
1b07d30761 After a server disconnects, make the IServer retain the dead RemoteReference, and continue to return it to anyone who calls get_rref(). This removes the need for callers to guard against receiving a None (as long as the server was connected at least once, which is always the case for servers returned by get_servers_for_psi(), which is how all upload/download code gets servers). Includes test. fixes #1636 2012-06-15 01:48:55 +00:00