Improve error-handling for directories if you ask for JSON from
the /uri endpoint, but an error occurs (you get a proper HTTP
status code and a valid JSON object).
For 'tahoe magic-folder status' e now retrieve *all* the remote data
required in the CLI before doing anything else so that errors can be
shown immediately. Use the improved JSON endpoints to print better
errors.
Adds:
- a JSON endpoint
- CLI to display information
- QueuedItem + IQueuedItem for uploader/downloader
- IProgress interface + PercentProgress implementation
- progress= args to many upload/download APIs
Historical note: V2 introducers have been around for three years
now (released in 1.10.0), so it's time to drop v1. This branch removes a
lot of fallback code, and tests which exercised it. refs ticket:2784
This patch removes some now-unused code: v1-related support functions on
the client, "stub-client" handlers, and v1-tolerant remote methods on
the server. The unit tests have been cleaned up a bit too, now that
there are fewer cases to exercise.
As discussed at https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1973 and in
previous pull request #129.
- replace lengthy timestamps with human-readable deltas (eg 1h 2m 3s)
- replace "announced" column with "Last RX" column
- remove service column (it always said the same thing, "storage")
- fix colspan on 'You are not presently connected' message
Previous versions, some with github comments: 3fe9053134 , 486dbfc7bd , and c89ea62580, 9fabb92486, bbd8b42a25
Unlike previous attempts, the tests on this one should pass in any timezone.
(But like current master, will fail with Nevow >=0.12...)
Thanks to an anonymous contributor who wrote some of the tests.
As discussed at https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1973 and in
previous pull request #129.
- replace lengthy timestamps with human-readable deltas (eg 1h 2m 3s)
- replace "announced" column with "Last RX" column
- remove service column (it always said the same thing, "storage")
- fix colspan on 'You are not presently connected' message
Previous versions, some with github comments: 3fe9053134 , 486dbfc7bd , and c89ea62580, 9fabb92486, bbd8b42a25
Unlike previous attempts, the tests on this one should pass in any timezone.
(But like current master, will fail with Nevow >=0.12...)
Thanks to an anonymous contributor who wrote some of the tests.
A long time ago, the introducer's status web page would show the
advertised IP addresses for all published services, by parsing their
FURL's connection hints. This hasn't worked since about 12-Aug-2014 when
foolscap-0.6.5 changed the internal format of these hints (the column
has been empty this whole time).
This removes the "Advertised IPs" column from the Service Announcements
table. Instead, the service's full connection hints (not just the IP
address) is displayed in a tooltip/popup on the "Announced" timestamp
column.
The code that pulls these connection hints is now tolerant of all three
foolscap styles:
* foolscap<=0.6.4 : tuples of ("ipv4",host,port)
* 0.6.5 .. 0.8.0 : tuples of ("tcp",host,port)
* foolscap>=0.9.0 : strings
fixes ticket:2510
The machine-parseable JSON output for the introducer status web page
used to include a key named "announcement_distinct_hosts", which counted
the number of distinct IP addresses advertised by all connected storage
servers. This hasn't worked since Aug-2014 when foolscap-0.6.5 change
the internal hints format.
This removes that field.
A long time ago, the introducer's status web page would show the
advertised IP addresses for all subscribers, by parsing their
RemoteReference's FURL's connection hints. This hasn't worked since
about 12-Aug-2014 when foolscap-0.6.5 changed the internal format of
these hints.
This removes the feature: we no longer attempt to show advertised IP
addresses of subscribed clients. It also removes the code that looked
inside foolscap internals for this information.
twisted.web.http.Request.setHeader() really wants a "bytes" object, but
we've been passing integers like len(body). Twisted-12.3 started to
complain about this (with a DeprecationWarning), but the warning is
usually silenced because py2.7 disables deprecations by default.
This fixes Tahoe's misbehavior, but others remain (in Nevow, at least).
I plan to set up some tooling to run tests with
PYTHONWARNINGS=default::DeprecationWarning and collect others. We won't
be able to fix the ones that occur outside of Tahoe, but at least we
should be able to fix our own.
refs ticket:2312
This replaces the status display which was only distinct by color which is a usability issue for color-blind users. This commit includes test coverage by way of pattern matching on rendered templates. The PNG icons are conversions of original SVG source which I've included and placed in the public domain.
add get_available_space() to NativeStorageServer
It uses a new 'available-space' key in the server's v1 version dict, or falls
back to 'maximum-immutable-share-size' (which presently always has the same
value but could have a different meaning in the future).
This is a squash merge of 9773555bb87fab71145ad7a0e84785a4e92d11f7
Solution was very simple to implement, no content disposition header was
necessary.
Tested with both Firefox and Chrome, using binary image file stored in
folder, as well as with text data using LIT cap.
Add a tooltip to explain what SDMF means. Cannot find a definition for MDMF; I presume "Medium" but at the risk of being wrong, I don't want to just blindly make that suggested change.
This is an initial conversion of the directory pages from the old style
to the new style which is based on Twitter Bootstrap.
Still some remaining work to be done. You can see a screenshot here:
http://i.imgur.com/MPEngGx.png
This should hopefully satisfy the Debian requirement to include original
sources. The old minified files for d3 and jquery were 63k and 91k
respectively, while the new unminified files are 133k and 293k.
Nevow automatically HTML-escapes strings passed in stan without a raw marker.
Written by MK_FG. fixes#1143
Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>
Improve the column headers to make it clear that this list shows Tub
IDs. (we can't show pubkey-based serverids because clients don't give
those to us: only servers provide pubkeys). This should be the only
place in the whole webapi that shows TubIDs for modern (V2-introducer)
nodes.
This makes it easy to distinguish between old V1-Introducer
nodes (identified by their Foolscap TubID) and new V2 nodes (identified
by their ed25519 pubkey).
This fixes a few places where we used to display a tubid even if we had
a pubkey, making it hard to visually correlate servers in two different
displays. It also cleans up the way we pass serverids to the JS-based
download timeline.
The "introweb" subscribed-clients list still shows tubids.
DeepResultsBase also has a get_corrupt_shares(), and it is populated
from CheckResults.get_corrupt_shares(). It has been updated too, along
with get_remaining_corrupt_shares().
Remove temporary get_new_corrupt_shares() and
get_new_incompatible_shares().
Complete the getter-based transformation, by hiding ".uri" and updating
callers to use get_uri(). Also don't set a dummy self._uri, leave it
undefined until someone calls set_uri().
This hides attributes with e.g. _sharemap, and creates getters like
get_sharemap() to access them, for every field except .uri . This will
make it easier to modify the internal representation of .sharemap
without requiring callers to adjust quite yet.
".uri" has so many users that it seemed better to update it in a
subsequent patch.
This measured how long the Helper took to do a filecheck before asking
for ciphertext. The "Contacting Helper" report includes both
existence_check and the client-helper RTT.
For non-overlapping uploads, it was being returned correctly. But when
multiple upload requests overlapped, and the file was not already in the
grid, the filecheck would only run once, and its existence_check time
would be reported for all uploaders (even if they didn't have to wait
for that time). Cleaning that up proved too difficult: the only correct
place to report this time is from the initial remote_upload_chk() call,
but the return value of that is too constrained to accomodate it in the
needs-upload case.
So I'm removing it altogether. Eventually I plan to add a proper
events/times field and record more data, including this check, in a form
that can be drawn on a nice zoomable timeline view.
Old clients talking to a new Helper (which doesn't supply the value)
will tolerate the loss (they'll just display an empty field on the web
view).
t=info contains randomly-generated ophandles, and t=rename-form contains the
name of the child being renamed, so neither is eligible for a
short-circuiting ETag. Enhanced test_web to exercise this. Had to improve
FakeCHKFileNode slightly to let it participate. Refs #443.
When client does a conditional GET/HEAD with If-none-match:, if the condition
fails (ie, the client's ETag matches the file's) then we can short-circuit
the whole process and immediately return an empty body.
Like immutable files, the ETag is based on the storage index. However, since
a directory is a special interpretation of a file, it is distinguished from
the file by prepending "DIR:" onto the start of the ETag, and adding
-representation on the end (where -representation is the ?t= argument, json,
info, etc).
It also checks the return of setETag and avoids generating a representation
if the client already has it.
test_web.py: use shouldFail2(), safer than old shouldFail()
directory.py: forbid slashes in from_name=, return BAD_REQUEST instead of
GONE when trying to move into a non-directory
The move webapi function now takes a target_type argument which lets it
know whether the target is a subdirectory name or URI. This is an
improvement over the old system in which the move handler tried to guess
whether the target was a name or a URI. Also fixed a little docs
copypaste problem and tweaked some line wrapping.
This adds "move file" capability to the web UI's directory display. The
support and test framework is heavily based on the similar "rename file"
feature. Unit tests and documentation are included. Multiple in-progress
versions of this patch may be found in ticket 1579. This version
includes arbitrary URI target support and is compatible with the change
from tahoe_css to tahoe.css.
'serverid' is the pubkey (for V2 clients), falling back to the tubid (for V1
clients). This also required cleaning up the way the index is created for the
old V1 introducer.
This significantly cleans up the IntroducerServer web-status renderers.
Instead of poking around in the introducer's internals, now the web-status
renderers get clean AnnouncementDescriptor and SubscriberDescriptor
objects. They are still somewhat foolscap-centric, but will provide a clean
abstraction boundary for future improvements.
The specific #1721 bug was that old (V1) subscribers were handled by
wrapping their RemoteReference in a special WrapV1SubscriberInV2Interface
object, but the web-status display was trying to peek inside the object to
learn what host+port it was associated with, and the wrapper did not proxy
those extra attributes.
A test was added to test_introducer to make sure the introweb page renders
properly and at least contains the nicknames of both the V1 and V2 clients.
This avoids the name collision between the actual results
objects (defined in allmydata.check_results) and the code that renders
these objects into HTML (defined in allmydata.web.check_results). Only
the web-side objects were renamed.
This still leaves immutable-publish results incorrectly using tubids instead
of serverids. That will need some more work, since it might change the Helper
interface.
This introduces new client and server halves to the Introducer (renaming the
old one with a _V1 suffix). Both have fallbacks to accomodate talking to a
different version: the publishing client switches on whether the server's
.get_version() advertises V2 support, the server switches on which
subscription method was invoked by the subscribing client.
The V2 protocol sends a three-tuple of (serialized announcement dictionary,
signature, pubkey) for each announcement. The V2 server dispatches messages
to subscribers according to the service-name, and throws errors for invalid
signatures, but does not otherwise examine the messages. The V2 receiver's
subscription callback will receive a (serverid, ann_dict) pair. The
'serverid' will be equal to the pubkey if all of the following are true:
the originating client is V2, and was told a privkey to use
the announcement went through a V2 server
the signature is valid
If not, 'serverid' will be equal to the tubid portion of the announced FURL,
as was the case for V1 receivers.
Servers will create a keypair if one does not exist yet, stored in
private/server.privkey .
The signed announcement dictionary puts the server FURL in a key named
"anonymous-storage-FURL", which anticipates upcoming Accounting-related
changes in the server advertisements. It also provides a key named
"permutation-seed-base32" to tell clients what permutation seed to use. This
is computed at startup, using tubid if there are existing shares, otherwise
the pubkey, to retain share-order compatibility for existing servers.
The "#section" declaration (which matches id="section") should have been
".section" (which matches class="section").
The welcome page has a feature that I actually liked: the little "This
Client" sidebar sits just to the right of the start of the Controls block.
Fixing .section broke that (the clear:both introduces a gap, forcing the
Controls block to start strictly below the bottom of the This Client block).
So I also removed class="section" from the Controls block to allow them to
share the horizontal space again.
We're slowly moving away from Nevow, and marcusw's previous patch removed
uses of the formless CSS file, so now we can stop testing that nevow can find
that file, and remove the lingering unused "import formless" call.
They were probably meant to be links to webform_css, but we aren't really
using Nevow's form-generation code anyways, so they can just be removed.
Thanks to 'marcusw' for the catch.
* use d3.js v2.4.6
* add a "toggle misc events" button, to get hash/bitmap-checking details
* only draw data that's on screen, for speed
* add fragment-arg to fetch timeline data.json from somewhere else
* fix tahoe.cfg control of default mutable type
* tolerate arbitrary case in [client]mutable.format value
* small docs improvements
* use get_mutable_type() as a format-is-mutable predicate
* tighten up error message
* fix CLI commands (put, mkdir) to send format=, not mutable-type=
* fix tests
* test_cli: fix tests that observe t=json output, don't ignore failures in
'tahoe put'
* fix handling of version= to make it easier to use the default
* interpret ?mutable=true&format=MDMF as MDMF, not SDMF
- Learn how to create MDMF files and directories through the
mutable-type argument.
- Operate with the interface changes associated with MDMF and #993.
- Learn how to do partial updates of mutable files.
