ExternalVendorCode/tahoe-lafs
1
0
Fork 0
mirror of https://github.com/tahoe-lafs/tahoe-lafs.git synced 2025-02-21 10:01:54 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity

Merge remote-tracking branch 'origin/master' into 3927.release

Browse Source
This commit is contained in:
Jean-Paul Calderone 2022-10-12 15:18:26 -04:00
commit 62563a9b58
16 changed files with 428 additions and 151 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

73
docs/proposed/http-storage-node-protocol.rst
View File

@ -395,8 +395,8 @@ Encoding
General
~~~~~~~
``GET /v1/version``
!!!!!!!!!!!!!!!!!!!
``GET /storage/v1/version``
!!!!!!!!!!!!!!!!!!!!!!!!!!!
Retrieve information about the version of the storage server.
Information is returned as an encoded mapping.
@ -409,14 +409,13 @@ For example::
"tolerates-immutable-read-overrun": true,
"delete-mutable-shares-with-zero-length-writev": true,
"fills-holes-with-zero-bytes": true,
"prevents-read-past-end-of-share-data": true,
"gbs-anonymous-storage-url": "pb://...#v=1"
"prevents-read-past-end-of-share-data": true
},
"application-version": "1.13.0"
}
``PUT /v1/lease/:storage_index``
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
``PUT /storage/v1/lease/:storage_index``
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Either renew or create a new lease on the bucket addressed by ``storage_index``.
@ -468,8 +467,8 @@ Immutable
Writing
~~~~~~~
``POST /v1/immutable/:storage_index``
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
``POST /storage/v1/immutable/:storage_index``
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Initialize an immutable storage index with some buckets.
The buckets may have share data written to them once.
@ -504,7 +503,7 @@ Handling repeat calls:
Discussion
``````````
We considered making this ``POST /v1/immutable`` instead.
We considered making this ``POST /storage/v1/immutable`` instead.
The motivation was to keep *storage index* out of the request URL.
Request URLs have an elevated chance of being logged by something.
We were concerned that having the *storage index* logged may increase some risks.
@ -539,8 +538,8 @@ Rejected designs for upload secrets:
it must contain randomness.
Randomness means there is no need to have a secret per share, since adding share-specific content to randomness doesn't actually make the secret any better.
``PATCH /v1/immutable/:storage_index/:share_number``
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
``PATCH /storage/v1/immutable/:storage_index/:share_number``
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Write data for the indicated share.
The share number must belong to the storage index.
@ -580,8 +579,8 @@ Responses:
the response is ``CONFLICT``.
At this point the only thing to do is abort the upload and start from scratch (see below).
``PUT /v1/immutable/:storage_index/:share_number/abort``
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
``PUT /storage/v1/immutable/:storage_index/:share_number/abort``
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
This cancels an *in-progress* upload.
@ -616,8 +615,8 @@ From RFC 7231::
PATCH method defined in [RFC5789]).
``POST /v1/immutable/:storage_index/:share_number/corrupt``
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
``POST /storage/v1/immutable/:storage_index/:share_number/corrupt``
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Advise the server the data read from the indicated share was corrupt. The
request body includes an human-meaningful text string with details about the
@ -635,8 +634,8 @@ couldn't be found.
Reading
~~~~~~~
``GET /v1/immutable/:storage_index/shares``
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
``GET /storage/v1/immutable/:storage_index/shares``
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Retrieve a list (semantically, a set) indicating all shares available for the
indicated storage index. For example::
@ -645,8 +644,8 @@ indicated storage index. For example::
An unknown storage index results in an empty list.
``GET /v1/immutable/:storage_index/:share_number``
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
``GET /storage/v1/immutable/:storage_index/:share_number``
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Read a contiguous sequence of bytes from one share in one bucket.
The response body is the raw share data (i.e., ``application/octet-stream``).
@ -686,8 +685,8 @@ Mutable
Writing
~~~~~~~
``POST /v1/mutable/:storage_index/read-test-write``
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
``POST /storage/v1/mutable/:storage_index/read-test-write``
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
General purpose read-test-and-write operation for mutable storage indexes.
A mutable storage index is also called a "slot"
@ -742,18 +741,18 @@ As a result, if there is no data at all, an empty bytestring is returned no matt
Reading
~~~~~~~
``GET /v1/mutable/:storage_index/shares``
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
``GET /storage/v1/mutable/:storage_index/shares``
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Retrieve a set indicating all shares available for the indicated storage index.
For example (this is shown as list, since it will be list for JSON, but will be set for CBOR)::
[1, 5]
``GET /v1/mutable/:storage_index/:share_number``
``GET /storage/v1/mutable/:storage_index/:share_number``
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Read data from the indicated mutable shares, just like ``GET /v1/immutable/:storage_index``
Read data from the indicated mutable shares, just like ``GET /storage/v1/immutable/:storage_index``
The ``Range`` header may be used to request exactly one ``bytes`` range, in which case the response code will be 206 (partial content).
Interpretation and response behavior is as specified in RFC 7233 § 4.1.
@ -765,8 +764,8 @@ The resulting ``Content-Range`` header will be consistent with the returned data
If the response to a query is an empty range, the ``NO CONTENT`` (204) response code will be used.
``POST /v1/mutable/:storage_index/:share_number/corrupt``
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
``POST /storage/v1/mutable/:storage_index/:share_number/corrupt``
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Advise the server the data read from the indicated share was corrupt.
Just like the immutable version.
@ -779,7 +778,7 @@ Immutable Data
1. Create a bucket for storage index ``AAAAAAAAAAAAAAAA`` to hold two immutable shares, discovering that share ``1`` was already uploaded::
POST /v1/immutable/AAAAAAAAAAAAAAAA
POST /storage/v1/immutable/AAAAAAAAAAAAAAAA
Authorization: Tahoe-LAFS nurl-swissnum
X-Tahoe-Authorization: lease-renew-secret efgh
X-Tahoe-Authorization: lease-cancel-secret jjkl
@ -792,7 +791,7 @@ Immutable Data
#. Upload the content for immutable share ``7``::
PATCH /v1/immutable/AAAAAAAAAAAAAAAA/7
PATCH /storage/v1/immutable/AAAAAAAAAAAAAAAA/7
Authorization: Tahoe-LAFS nurl-swissnum
Content-Range: bytes 0-15/48
X-Tahoe-Authorization: upload-secret xyzf
@ -800,7 +799,7 @@ Immutable Data
200 OK
PATCH /v1/immutable/AAAAAAAAAAAAAAAA/7
PATCH /storage/v1/immutable/AAAAAAAAAAAAAAAA/7
Authorization: Tahoe-LAFS nurl-swissnum
Content-Range: bytes 16-31/48
X-Tahoe-Authorization: upload-secret xyzf
@ -808,7 +807,7 @@ Immutable Data
200 OK
PATCH /v1/immutable/AAAAAAAAAAAAAAAA/7
PATCH /storage/v1/immutable/AAAAAAAAAAAAAAAA/7
Authorization: Tahoe-LAFS nurl-swissnum
Content-Range: bytes 32-47/48
X-Tahoe-Authorization: upload-secret xyzf
@ -818,7 +817,7 @@ Immutable Data
#. Download the content of the previously uploaded immutable share ``7``::
GET /v1/immutable/AAAAAAAAAAAAAAAA?share=7
GET /storage/v1/immutable/AAAAAAAAAAAAAAAA?share=7
Authorization: Tahoe-LAFS nurl-swissnum
Range: bytes=0-47
@ -827,7 +826,7 @@ Immutable Data
#. Renew the lease on all immutable shares in bucket ``AAAAAAAAAAAAAAAA``::
PUT /v1/lease/AAAAAAAAAAAAAAAA
PUT /storage/v1/lease/AAAAAAAAAAAAAAAA
Authorization: Tahoe-LAFS nurl-swissnum
X-Tahoe-Authorization: lease-cancel-secret jjkl
X-Tahoe-Authorization: lease-renew-secret efgh
@ -842,7 +841,7 @@ The special test vector of size 1 but empty bytes will only pass
if there is no existing share,
otherwise it will read a byte which won't match `b""`::
POST /v1/mutable/BBBBBBBBBBBBBBBB/read-test-write
POST /storage/v1/mutable/BBBBBBBBBBBBBBBB/read-test-write
Authorization: Tahoe-LAFS nurl-swissnum
X-Tahoe-Authorization: write-enabler abcd
X-Tahoe-Authorization: lease-cancel-secret efgh
@ -874,7 +873,7 @@ otherwise it will read a byte which won't match `b""`::
#. Safely rewrite the contents of a known version of mutable share number ``3`` (or fail)::
POST /v1/mutable/BBBBBBBBBBBBBBBB/read-test-write
POST /storage/v1/mutable/BBBBBBBBBBBBBBBB/read-test-write
Authorization: Tahoe-LAFS nurl-swissnum
X-Tahoe-Authorization: write-enabler abcd
X-Tahoe-Authorization: lease-cancel-secret efgh
@ -906,14 +905,14 @@ otherwise it will read a byte which won't match `b""`::
#. Download the contents of share number ``3``::
GET /v1/mutable/BBBBBBBBBBBBBBBB?share=3&offset=0&size=10
GET /storage/v1/mutable/BBBBBBBBBBBBBBBB?share=3&offset=0&size=10
Authorization: Tahoe-LAFS nurl-swissnum
<complete 16 bytes of previously uploaded data>
#. Renew the lease on previously uploaded mutable share in slot ``BBBBBBBBBBBBBBBB``::
PUT /v1/lease/BBBBBBBBBBBBBBBB
PUT /storage/v1/lease/BBBBBBBBBBBBBBBB
Authorization: Tahoe-LAFS nurl-swissnum
X-Tahoe-Authorization: lease-cancel-secret efgh
X-Tahoe-Authorization: lease-renew-secret ijkl

38
docs/specifications/url.rst
View File

@ -47,27 +47,27 @@ This can be considered to expand to "**N**\ ew URLs" or "Authe\ **N**\ ticating
The anticipated use for a **NURL** will still be to establish a TLS connection to a peer.
The protocol run over that TLS connection could be Foolscap though it is more likely to be an HTTP-based protocol (such as GBS).
Unlike fURLs, only a single net-loc is included, for consistency with other forms of URLs.
As a result, multiple NURLs may be available for a single server.
Syntax
------
The EBNF for a NURL is as follows::
nurl = scheme, hash, "@", net-loc-list, "/", swiss-number, [ version1 ]
scheme = "pb://"
nurl = tcp-nurl | tor-nurl | i2p-nurl
tcp-nurl = "pb://", hash, "@", tcp-loc, "/", swiss-number, [ version1 ]
tor-nurl = "pb+tor://", hash, "@", tcp-loc, "/", swiss-number, [ version1 ]
i2p-nurl = "pb+i2p://", hash, "@", i2p-loc, "/", swiss-number, [ version1 ]
hash = unreserved
net-loc-list = net-loc, [ { ",", net-loc } ]
net-loc = tcp-loc | tor-loc | i2p-loc
tcp-loc = [ "tcp:" ], hostname, [ ":" port ]
tor-loc = "tor:", hostname, [ ":" port ]
i2p-loc = "i2p:", i2p-addr, [ ":" port ]
i2p-addr = { unreserved }, ".i2p"
tcp-loc = hostname, [ ":" port ]
hostname = domain | IPv4address | IPv6address
i2p-loc = i2p-addr, [ ":" port ]
i2p-addr = { unreserved }, ".i2p"
swiss-number = segment
version1 = "#v=1"
@ -87,11 +87,13 @@ These differences are separated into distinct versions.
Version 0
---------
A Foolscap fURL is considered the canonical definition of a version 0 NURL.
In theory, a Foolscap fURL with a single netloc is considered the canonical definition of a version 0 NURL.
Notably,
the hash component is defined as the base32-encoded SHA1 hash of the DER form of an x509v3 certificate.
A version 0 NURL is identified by the absence of the ``v=1`` fragment.
In practice, real world fURLs may have more than one netloc, so lack of version fragment will likely just involve dispatching the fURL to a different parser.
Examples
~~~~~~~~
@ -103,11 +105,8 @@ Version 1
The hash component of a version 1 NURL differs in three ways from the prior version.
1. The hash function used is SHA3-224 instead of SHA1.
The security of SHA1 `continues to be eroded`_.
Contrariwise SHA3 is currently the most recent addition to the SHA family by NIST.
The 224 bit instance is chosen to keep the output short and because it offers greater collision resistance than SHA1 was thought to offer even at its inception
(prior to security research showing actual collision resistance is lower).
1. The hash function used is SHA-256, to match RFC 7469.
The security of SHA1 `continues to be eroded`_; Latacora `SHA-2`_.
2. The hash is computed over the certificate's SPKI instead of the whole certificate.
This allows certificate re-generation so long as the public key remains the same.
This is useful to allow contact information to be updated or extension of validity period.
@ -122,7 +121,7 @@ The hash component of a version 1 NURL differs in three ways from the prior vers
*all* certificate fields should be considered within the context of the relationship identified by the SPKI hash.
3. The hash is encoded using urlsafe-base64 (without padding) instead of base32.
This provides a more compact representation and minimizes the usability impacts of switching from a 160 bit hash to a 224 bit hash.
This provides a more compact representation and minimizes the usability impacts of switching from a 160 bit hash to a 256 bit hash.
A version 1 NURL is identified by the presence of the ``v=1`` fragment.
Though the length of the hash string (38 bytes) could also be used to differentiate it from a version 0 NURL,
@ -140,7 +139,8 @@ Examples
* ``pb://azEu8vlRpnEeYm0DySQDeNY3Z2iJXHC_bsbaAw@localhost:47877/64i4aokv4ej#v=1``
.. _`continues to be eroded`: https://en.wikipedia.org/wiki/SHA-1#Cryptanalysis_and_validation
.. _`explored by the web community`: https://www.imperialviolet.org/2011/05/04/pinning.html
.. _`SHA-2`: https://latacora.micro.blog/2018/04/03/cryptographic-right-answers.html
.. _`explored by the web community`: https://www.rfc-editor.org/rfc/rfc7469
.. _Foolscap: https://github.com/warner/foolscap
.. [1] ``foolscap.furl.decode_furl`` is taken as the canonical definition of the syntax of a fURL.

4
integration/test_i2p.py
View File

@ -55,7 +55,7 @@ def i2p_network(reactor, temp_dir, request):
proto,
which("docker"),
(
"docker", "run", "-p", "7656:7656", "purplei2p/i2pd",
"docker", "run", "-p", "7656:7656", "purplei2p/i2pd:release-2.43.0",
# Bad URL for reseeds, so it can't talk to other routers.
"--reseed.urls", "http://localhost:1/",
),
@ -63,7 +63,7 @@ def i2p_network(reactor, temp_dir, request):
def cleanup():
try:
proto.transport.signalProcess("KILL")
proto.transport.signalProcess("INT")
util.block_with_timeout(proto.exited, reactor)
except ProcessExitedAlready:
pass

1
newsfragments/3902.feature Normal file
View File

@ -0,0 +1 @@
The new HTTPS-based storage server is now enabled transparently on the same port as the Foolscap server. This will not have any user-facing impact until the HTTPS storage protocol is supported in clients as well.

0
newsfragments/3904.minor Normal file
View File

0
newsfragments/3928.minor Normal file
View File

3
setup.py
View File

@ -386,6 +386,9 @@ setup(name="tahoe-lafs", # also set in __init__.py
],
"test": [
"flake8",
# On Python 3.7, importlib_metadata v5 breaks flake8.
# https://github.com/python/importlib_metadata/issues/407
"importlib_metadata<5; python_version < '3.8'",
# Pin a specific pyflakes so we don't have different folks
# disagreeing on what is or is not a lint issue. We can bump
# this version from time to time, but we will do it

28
src/allmydata/client.py
View File

@ -1,17 +1,9 @@
"""
Ported to Python 3.
"""
from __future__ import absolute_import
from __future__ import division
from __future__ import print_function
from __future__ import unicode_literals
from future.utils import PY2
if PY2:
from future.builtins import filter, map, zip, ascii, chr, hex, input, next, oct, open, pow, round, super, bytes, dict, list, object, range, max, min # noqa: F401
# Don't use future str to prevent leaking future's newbytes into foolscap, which they break.
from past.builtins import unicode as str
from __future__ import annotations
from typing import Optional
import os, stat, time, weakref
from base64 import urlsafe_b64encode
from functools import partial
@ -591,6 +583,10 @@ def anonymous_storage_enabled(config):
@implementer(IStatsProducer)
class _Client(node.Node, pollmixin.PollMixin):
"""
This class should be refactored; see
https://tahoe-lafs.org/trac/tahoe-lafs/ticket/3931
"""
STOREDIR = 'storage'
NODETYPE = "client"
@ -658,6 +654,14 @@ class _Client(node.Node, pollmixin.PollMixin):
if webport:
self.init_web(webport) # strports string
# TODO this may be the wrong location for now? but as temporary measure
# it allows us to get NURLs for testing in test_istorageserver.py. This
# will eventually get fixed one way or another in
# https://tahoe-lafs.org/trac/tahoe-lafs/ticket/3901. See also
# https://tahoe-lafs.org/trac/tahoe-lafs/ticket/3931 for the bigger
# picture issue.
self.storage_nurls : Optional[set] = None
def init_stats_provider(self):
self.stats_provider = StatsProvider(self)
self.stats_provider.setServiceParent(self)
@ -818,6 +822,10 @@ class _Client(node.Node, pollmixin.PollMixin):
if anonymous_storage_enabled(self.config):
furl_file = self.config.get_private_path("storage.furl").encode(get_filesystem_encoding())
furl = self.tub.registerReference(FoolscapStorageServer(ss), furlFile=furl_file)
(_, _, swissnum) = decode_furl(furl)
self.storage_nurls = self.tub.negotiationClass.add_storage_server(
ss, swissnum.encode("ascii")
)
announcement["anonymous-storage-FURL"] = furl
enabled_storage_servers = self._enable_storage_servers(

7
src/allmydata/node.py
View File

@ -55,6 +55,8 @@ from allmydata.util.yamlutil import (
from . import (
__full_version__,
)
from .protocol_switch import create_tub_with_https_support
def _common_valid_config():
return configutil.ValidConfiguration({
@ -706,7 +708,10 @@ def create_tub(tub_options, default_connection_handlers, foolscap_connection_han
:param dict tub_options: every key-value pair in here will be set in
the new Tub via `Tub.setOption`
"""
tub = Tub(**kwargs)
# We listen simulataneously for both Foolscap and HTTPS on the same port,
# so we have to create a special Foolscap Tub for that to work:
tub = create_tub_with_https_support(**kwargs)
for (name, value) in list(tub_options.items()):
tub.setOption(name, value)
handlers = default_connection_handlers.copy()

210
src/allmydata/protocol_switch.py Normal file
View File

@ -0,0 +1,210 @@
"""
Support for listening with both HTTPS and Foolscap on the same port.
The goal is to make the transition from Foolscap to HTTPS-based protocols as
simple as possible, with no extra configuration needed. Listening on the same
port means a user upgrading Tahoe-LAFS will automatically get HTTPS working
with no additional changes.
Use ``create_tub_with_https_support()`` creates a new ``Tub`` that has its
``negotiationClass`` modified to be a new subclass tied to that specific
``Tub`` instance. Calling ``tub.negotiationClass.add_storage_server(...)``
then adds relevant information for a storage server once it becomes available
later in the configuration process.
"""
from __future__ import annotations
from itertools import chain
from twisted.internet.protocol import Protocol
from twisted.internet.interfaces import IDelayedCall
from twisted.internet.ssl import CertificateOptions
from twisted.web.server import Site
from twisted.protocols.tls import TLSMemoryBIOFactory
from twisted.internet import reactor
from hyperlink import DecodedURL
from foolscap.negotiate import Negotiation
from foolscap.api import Tub
from .storage.http_server import HTTPServer, build_nurl
from .storage.server import StorageServer
class _PretendToBeNegotiation(type):
"""
Metaclass that allows ``_FoolscapOrHttps`` to pretend to be a
``Negotiation`` instance, since Foolscap does some checks like
``assert isinstance(protocol, tub.negotiationClass)`` in its internals,
and sometimes that ``protocol`` is a ``_FoolscapOrHttps`` instance, but
sometimes it's a ``Negotiation`` instance.
"""
def __instancecheck__(self, instance):
return issubclass(instance.__class__, self) or isinstance(instance, Negotiation)
class _FoolscapOrHttps(Protocol, metaclass=_PretendToBeNegotiation):
"""
Based on initial query, decide whether we're talking Foolscap or HTTP.
Additionally, pretends to be a ``foolscap.negotiate.Negotiation`` instance,
since these are created by Foolscap's ``Tub``, by setting this to be the
tub's ``negotiationClass``.
Do not instantiate directly, use ``create_tub_with_https_support(...)``
instead. The way this class works is that a new subclass is created for a
specific ``Tub`` instance.
"""
# These are class attributes; they will be set by
# create_tub_with_https_support() and add_storage_server().
# The Twisted HTTPS protocol factory wrapping the storage server HTTP API:
https_factory: TLSMemoryBIOFactory
# The tub that created us:
tub: Tub
@classmethod
def add_storage_server(
cls, storage_server: StorageServer, swissnum: bytes
) -> set[DecodedURL]:
"""
Update a ``_FoolscapOrHttps`` subclass for a specific ``Tub`` instance
with the class attributes it requires for a specific storage server.
Returns the resulting NURLs.
"""
# We need to be a subclass:
assert cls != _FoolscapOrHttps
# The tub instance must already be set:
assert hasattr(cls, "tub")
assert isinstance(cls.tub, Tub)
# Tub.myCertificate is a twisted.internet.ssl.PrivateCertificate
# instance.
certificate_options = CertificateOptions(
privateKey=cls.tub.myCertificate.privateKey.original,
certificate=cls.tub.myCertificate.original,
)
http_storage_server = HTTPServer(storage_server, swissnum)
cls.https_factory = TLSMemoryBIOFactory(
certificate_options,
False,
Site(http_storage_server.get_resource()),
)
storage_nurls = set()
# Individual hints can be in the form
# "tcp:host:port,tcp:host:port,tcp:host:port".
for location_hint in chain.from_iterable(
hints.split(",") for hints in cls.tub.locationHints
):
if location_hint.startswith("tcp:"):
_, hostname, port = location_hint.split(":")
port = int(port)
storage_nurls.add(
build_nurl(
hostname,
port,
str(swissnum, "ascii"),
cls.tub.myCertificate.original.to_cryptography(),
)
)
# TODO this is probably where we'll have to support Tor and I2P?
# See https://tahoe-lafs.org/trac/tahoe-lafs/ticket/3888#comment:9
# for discussion (there will be separate tickets added for those at
# some point.)
return storage_nurls
def __init__(self, *args, **kwargs):
self._foolscap: Negotiation = Negotiation(*args, **kwargs)
def __setattr__(self, name, value):
if name in {"_foolscap", "_buffer", "transport", "__class__", "_timeout"}:
object.__setattr__(self, name, value)
else:
setattr(self._foolscap, name, value)
def __getattr__(self, name):
return getattr(self._foolscap, name)
def _convert_to_negotiation(self):
"""
Convert self to a ``Negotiation`` instance.
"""
self.__class__ = Negotiation # type: ignore
self.__dict__ = self._foolscap.__dict__
def initClient(self, *args, **kwargs):
# After creation, a Negotiation instance either has initClient() or
# initServer() called. Since this is a client, we're never going to do
# HTTP, so we can immediately become a Negotiation instance.
assert not hasattr(self, "_buffer")
self._convert_to_negotiation()
return self.initClient(*args, **kwargs)
def connectionMade(self):
self._buffer: bytes = b""
self._timeout: IDelayedCall = reactor.callLater(
30, self.transport.abortConnection
)
def connectionLost(self, reason):
if self._timeout.active():
self._timeout.cancel()
def dataReceived(self, data: bytes) -> None:
"""Handle incoming data.
Once we've decided which protocol we are, update self.__class__, at
which point all methods will be called on the new class.
"""
self._buffer += data
if len(self._buffer) < 8:
return
# Check if it looks like a Foolscap request. If so, it can handle this
# and later data, otherwise assume HTTPS.
self._timeout.cancel()
if self._buffer.startswith(b"GET /id/"):
# We're a Foolscap Negotiation server protocol instance:
transport = self.transport
buf = self._buffer
self._convert_to_negotiation()
self.makeConnection(transport)
self.dataReceived(buf)
return
else:
# We're a HTTPS protocol instance, serving the storage protocol:
assert self.transport is not None
protocol = self.https_factory.buildProtocol(self.transport.getPeer())
protocol.makeConnection(self.transport)
protocol.dataReceived(self._buffer)
# Update the factory so it knows we're transforming to a new
# protocol object (we'll do that next)
value = self.https_factory.protocols.pop(protocol)
self.https_factory.protocols[self] = value
# Transform self into the TLS protocol 🪄
self.__class__ = protocol.__class__
self.__dict__ = protocol.__dict__
def create_tub_with_https_support(**kwargs) -> Tub:
"""
Create a new Tub that also supports HTTPS.
This involves creating a new protocol switch class for the specific ``Tub``
instance.
"""
the_tub = Tub(**kwargs)
class FoolscapOrHttpForTub(_FoolscapOrHttps):
tub = the_tub
the_tub.negotiationClass = FoolscapOrHttpForTub # type: ignore
return the_tub

28
src/allmydata/storage/http_client.py
View File

@ -392,7 +392,7 @@ class StorageClientGeneral(object):
"""
Return the version metadata for the server.
"""
url = self._client.relative_url("/v1/version")
url = self._client.relative_url("/storage/v1/version")
response = yield self._client.request("GET", url)
decoded_response = yield _decode_cbor(response, _SCHEMAS["get_version"])
returnValue(decoded_response)
@ -408,7 +408,7 @@ class StorageClientGeneral(object):
Otherwise a new lease is added.
"""
url = self._client.relative_url(
"/v1/lease/{}".format(_encode_si(storage_index))
"/storage/v1/lease/{}".format(_encode_si(storage_index))
)
response = yield self._client.request(
"PUT",
@ -457,7 +457,9 @@ def read_share_chunk(
always provided by the current callers.
"""
url = client.relative_url(
"/v1/{}/{}/{}".format(share_type, _encode_si(storage_index), share_number)
"/storage/v1/{}/{}/{}".format(
share_type, _encode_si(storage_index), share_number
)
)
response = yield client.request(
"GET",
@ -518,7 +520,7 @@ async def advise_corrupt_share(
):
assert isinstance(reason, str)
url = client.relative_url(
"/v1/{}/{}/{}/corrupt".format(
"/storage/v1/{}/{}/{}/corrupt".format(
share_type, _encode_si(storage_index), share_number
)
)
@ -563,7 +565,9 @@ class StorageClientImmutables(object):
Result fires when creating the storage index succeeded, if creating the
storage index failed the result will fire with an exception.
"""
url = self._client.relative_url("/v1/immutable/" + _encode_si(storage_index))
url = self._client.relative_url(
"/storage/v1/immutable/" + _encode_si(storage_index)
)
message = {"share-numbers": share_numbers, "allocated-size": allocated_size}
response = yield self._client.request(
@ -588,7 +592,9 @@ class StorageClientImmutables(object):
) -> Deferred[None]:
"""Abort the upload."""
url = self._client.relative_url(
"/v1/immutable/{}/{}/abort".format(_encode_si(storage_index), share_number)
"/storage/v1/immutable/{}/{}/abort".format(
_encode_si(storage_index), share_number
)
)
response = yield self._client.request(
"PUT",
@ -620,7 +626,9 @@ class StorageClientImmutables(object):
been uploaded.
"""
url = self._client.relative_url(
"/v1/immutable/{}/{}".format(_encode_si(storage_index), share_number)
"/storage/v1/immutable/{}/{}".format(
_encode_si(storage_index), share_number
)
)
response = yield self._client.request(
"PATCH",
@ -668,7 +676,7 @@ class StorageClientImmutables(object):
Return the set of shares for a given storage index.
"""
url = self._client.relative_url(
"/v1/immutable/{}/shares".format(_encode_si(storage_index))
"/storage/v1/immutable/{}/shares".format(_encode_si(storage_index))
)
response = yield self._client.request(
"GET",
@ -774,7 +782,7 @@ class StorageClientMutables:
are done and if they are valid the writes are done.
"""
url = self._client.relative_url(
"/v1/mutable/{}/read-test-write".format(_encode_si(storage_index))
"/storage/v1/mutable/{}/read-test-write".format(_encode_si(storage_index))
)
message = {
"test-write-vectors": {
@ -817,7 +825,7 @@ class StorageClientMutables:
List the share numbers for a given storage index.
"""
url = self._client.relative_url(
"/v1/mutable/{}/shares".format(_encode_si(storage_index))
"/storage/v1/mutable/{}/shares".format(_encode_si(storage_index))
)
response = await self._client.request("GET", url)
if response.code == http.OK:

84
src/allmydata/storage/http_server.py
View File

@ -4,12 +4,13 @@ HTTP server for storage.
from __future__ import annotations
from typing import Dict, List, Set, Tuple, Any, Callable, Union
from typing import Dict, List, Set, Tuple, Any, Callable, Union, cast
from functools import wraps
from base64 import b64decode
import binascii
from tempfile import TemporaryFile
from cryptography.x509 import Certificate as CryptoCertificate
from zope.interface import implementer
from klein import Klein
from twisted.web import http
@ -18,6 +19,7 @@ from twisted.internet.interfaces import (
IStreamServerEndpoint,
IPullProducer,
)
from twisted.internet.address import IPv4Address, IPv6Address
from twisted.internet.defer import Deferred
from twisted.internet.ssl import CertificateOptions, Certificate, PrivateCertificate
from twisted.web.server import Site, Request
@ -193,7 +195,12 @@ class UploadsInProgress(object):
def remove_write_bucket(self, bucket: BucketWriter):
"""Stop tracking the given ``BucketWriter``."""
storage_index, share_number = self._bucketwriters.pop(bucket)
try:
storage_index, share_number = self._bucketwriters.pop(bucket)
except KeyError:
# This is probably a BucketWriter created by Foolscap, so just
# ignore it.
return
uploads_index = self._uploads[storage_index]
uploads_index.shares.pop(share_number)
uploads_index.upload_secrets.pop(share_number)
@ -545,7 +552,7 @@ class HTTPServer(object):
##### Generic APIs #####
@_authorized_route(_app, set(), "/v1/version", methods=["GET"])
@_authorized_route(_app, set(), "/storage/v1/version", methods=["GET"])
def version(self, request, authorization):
"""Return version information."""
return self._send_encoded(request, self._storage_server.get_version())
@ -555,7 +562,7 @@ class HTTPServer(object):
@_authorized_route(
_app,
{Secrets.LEASE_RENEW, Secrets.LEASE_CANCEL, Secrets.UPLOAD},
"/v1/immutable/<storage_index:storage_index>",
"/storage/v1/immutable/<storage_index:storage_index>",
methods=["POST"],
)
def allocate_buckets(self, request, authorization, storage_index):
@ -591,7 +598,7 @@ class HTTPServer(object):
@_authorized_route(
_app,
{Secrets.UPLOAD},
"/v1/immutable/<storage_index:storage_index>/<int(signed=False):share_number>/abort",
"/storage/v1/immutable/<storage_index:storage_index>/<int(signed=False):share_number>/abort",
methods=["PUT"],
)
def abort_share_upload(self, request, authorization, storage_index, share_number):
@ -622,7 +629,7 @@ class HTTPServer(object):
@_authorized_route(
_app,
{Secrets.UPLOAD},
"/v1/immutable/<storage_index:storage_index>/<int(signed=False):share_number>",
"/storage/v1/immutable/<storage_index:storage_index>/<int(signed=False):share_number>",
methods=["PATCH"],
)
def write_share_data(self, request, authorization, storage_index, share_number):
@ -665,7 +672,7 @@ class HTTPServer(object):
@_authorized_route(
_app,
set(),
"/v1/immutable/<storage_index:storage_index>/shares",
"/storage/v1/immutable/<storage_index:storage_index>/shares",
methods=["GET"],
)
def list_shares(self, request, authorization, storage_index):
@ -678,7 +685,7 @@ class HTTPServer(object):
@_authorized_route(
_app,
set(),
"/v1/immutable/<storage_index:storage_index>/<int(signed=False):share_number>",
"/storage/v1/immutable/<storage_index:storage_index>/<int(signed=False):share_number>",
methods=["GET"],
)
def read_share_chunk(self, request, authorization, storage_index, share_number):
@ -694,7 +701,7 @@ class HTTPServer(object):
@_authorized_route(
_app,
{Secrets.LEASE_RENEW, Secrets.LEASE_CANCEL},
"/v1/lease/<storage_index:storage_index>",
"/storage/v1/lease/<storage_index:storage_index>",
methods=["PUT"],
)
def add_or_renew_lease(self, request, authorization, storage_index):
@ -715,7 +722,7 @@ class HTTPServer(object):
@_authorized_route(
_app,
set(),
"/v1/immutable/<storage_index:storage_index>/<int(signed=False):share_number>/corrupt",
"/storage/v1/immutable/<storage_index:storage_index>/<int(signed=False):share_number>/corrupt",
methods=["POST"],
)
def advise_corrupt_share_immutable(
@ -736,7 +743,7 @@ class HTTPServer(object):
@_authorized_route(
_app,
{Secrets.LEASE_RENEW, Secrets.LEASE_CANCEL, Secrets.WRITE_ENABLER},
"/v1/mutable/<storage_index:storage_index>/read-test-write",
"/storage/v1/mutable/<storage_index:storage_index>/read-test-write",
methods=["POST"],
)
def mutable_read_test_write(self, request, authorization, storage_index):
@ -771,7 +778,7 @@ class HTTPServer(object):
@_authorized_route(
_app,
set(),
"/v1/mutable/<storage_index:storage_index>/<int(signed=False):share_number>",
"/storage/v1/mutable/<storage_index:storage_index>/<int(signed=False):share_number>",
methods=["GET"],
)
def read_mutable_chunk(self, request, authorization, storage_index, share_number):
@ -795,7 +802,10 @@ class HTTPServer(object):
return read_range(request, read_data, share_length)
@_authorized_route(
_app, set(), "/v1/mutable/<storage_index:storage_index>/shares", methods=["GET"]
_app,
set(),
"/storage/v1/mutable/<storage_index:storage_index>/shares",
methods=["GET"],
)
def enumerate_mutable_shares(self, request, authorization, storage_index):
"""List mutable shares for a storage index."""
@ -805,7 +815,7 @@ class HTTPServer(object):
@_authorized_route(
_app,
set(),
"/v1/mutable/<storage_index:storage_index>/<int(signed=False):share_number>/corrupt",
"/storage/v1/mutable/<storage_index:storage_index>/<int(signed=False):share_number>/corrupt",
methods=["POST"],
)
def advise_corrupt_share_mutable(
@ -859,6 +869,29 @@ class _TLSEndpointWrapper(object):
)
def build_nurl(
hostname: str, port: int, swissnum: str, certificate: CryptoCertificate
) -> DecodedURL:
"""
Construct a HTTPS NURL, given the hostname, port, server swissnum, and x509
certificate for the server. Clients can then connect to the server using
this NURL.
"""
return DecodedURL().replace(
fragment="v=1", # how we know this NURL is HTTP-based (i.e. not Foolscap)
host=hostname,
port=port,
path=(swissnum,),
userinfo=(
str(
get_spki_hash(certificate),
"ascii",
),
),
scheme="pb",
)
def listen_tls(
server: HTTPServer,
hostname: str,
@ -878,22 +911,15 @@ def listen_tls(
"""
endpoint = _TLSEndpointWrapper.from_paths(endpoint, private_key_path, cert_path)
def build_nurl(listening_port: IListeningPort) -> DecodedURL:
nurl = DecodedURL().replace(
fragment="v=1", # how we know this NURL is HTTP-based (i.e. not Foolscap)
host=hostname,
port=listening_port.getHost().port,
path=(str(server._swissnum, "ascii"),),
userinfo=(
str(
get_spki_hash(load_pem_x509_certificate(cert_path.getContent())),
"ascii",
),
),
scheme="pb",
def get_nurl(listening_port: IListeningPort) -> DecodedURL:
address = cast(Union[IPv4Address, IPv6Address], listening_port.getHost())
return build_nurl(
hostname,
address.port,
str(server._swissnum, "ascii"),
load_pem_x509_certificate(cert_path.getContent()),
)
return nurl
return endpoint.listen(Site(server.get_resource())).addCallback(
lambda listening_port: (build_nurl(listening_port), listening_port)
lambda listening_port: (get_nurl(listening_port), listening_port)
)

48
src/allmydata/test/test_istorageserver.py
View File

@ -18,21 +18,14 @@ from unittest import SkipTest
from twisted.internet.defer import inlineCallbacks, returnValue, succeed
from twisted.internet.task import Clock
from twisted.internet import reactor
from twisted.internet.endpoints import serverFromString
from twisted.python.filepath import FilePath
from foolscap.api import Referenceable, RemoteException
from allmydata.interfaces import IStorageServer # really, IStorageClient
# A better name for this would be IStorageClient...
from allmydata.interfaces import IStorageServer
from .common_system import SystemTestMixin
from .common import AsyncTestCase, SameProcessStreamEndpointAssigner
from .certs import (
generate_certificate,
generate_private_key,
private_key_to_file,
cert_to_file,
)
from .common import AsyncTestCase
from allmydata.storage.server import StorageServer # not a IStorageServer!!
from allmydata.storage.http_server import HTTPServer, listen_tls
from allmydata.storage.http_client import StorageClient
from allmydata.storage_client import _HTTPStorageServer
@ -1084,40 +1077,17 @@ class _FoolscapMixin(_SharedMixin):
class _HTTPMixin(_SharedMixin):
"""Run tests on the HTTP version of ``IStorageServer``."""
def setUp(self):
self._port_assigner = SameProcessStreamEndpointAssigner()
self._port_assigner.setUp()
self.addCleanup(self._port_assigner.tearDown)
return _SharedMixin.setUp(self)
@inlineCallbacks
def _get_istorage_server(self):
swissnum = b"1234"
http_storage_server = HTTPServer(self.server, swissnum)
# Listen on randomly assigned port, using self-signed cert:
private_key = generate_private_key()
certificate = generate_certificate(private_key)
_, endpoint_string = self._port_assigner.assign(reactor)
nurl, listening_port = yield listen_tls(
http_storage_server,
"127.0.0.1",
serverFromString(reactor, endpoint_string),
private_key_to_file(FilePath(self.mktemp()), private_key),
cert_to_file(FilePath(self.mktemp()), certificate),
)
self.addCleanup(listening_port.stopListening)
nurl = list(self.clients[0].storage_nurls)[0]
# Create HTTP client with non-persistent connections, so we don't leak
# state across tests:
returnValue(
_HTTPStorageServer.from_http_client(
StorageClient.from_nurl(nurl, reactor, persistent=False)
)
client: IStorageServer = _HTTPStorageServer.from_http_client(
StorageClient.from_nurl(nurl, reactor, persistent=False)
)
self.assertTrue(IStorageServer.providedBy(client))
# Eventually should also:
# self.assertTrue(IStorageServer.providedBy(client))
return succeed(client)
class FoolscapSharedAPIsTests(

43
src/allmydata/test/test_protocol_switch.py Normal file
View File

@ -0,0 +1,43 @@
"""
Unit tests for ``allmydata.protocol_switch``.
By its nature, most of the testing needs to be end-to-end; essentially any test
that uses real Foolscap (``test_system.py``, integration tests) ensures
Foolscap still works. ``test_istorageserver.py`` tests the HTTP support.
"""
from foolscap.negotiate import Negotiation
from .common import TestCase
from ..protocol_switch import _PretendToBeNegotiation
class UtilityTests(TestCase):
"""Tests for utilities in the protocol switch code."""
def test_metaclass(self):
"""
A class that has the ``_PretendToBeNegotiation`` metaclass will support
``isinstance()``'s normal semantics on its own instances, but will also
indicate that ``Negotiation`` instances are its instances.
"""
class Parent(metaclass=_PretendToBeNegotiation):
pass
class Child(Parent):
pass
class Other:
pass
p = Parent()
self.assertIsInstance(p, Parent)
self.assertIsInstance(Negotiation(), Parent)
self.assertNotIsInstance(Other(), Parent)
c = Child()
self.assertIsInstance(c, Child)
self.assertIsInstance(c, Parent)
self.assertIsInstance(Negotiation(), Child)
self.assertNotIsInstance(Other(), Child)

8
src/allmydata/test/test_storage_http.py
View File

@ -255,7 +255,7 @@ class TestApp(object):
else:
return "BAD: {}".format(authorization)
@_authorized_route(_app, set(), "/v1/version", methods=["GET"])
@_authorized_route(_app, set(), "/storage/v1/version", methods=["GET"])
def bad_version(self, request, authorization):
"""Return version result that violates the expected schema."""
request.setHeader("content-type", CBOR_MIME_TYPE)
@ -534,7 +534,7 @@ class GenericHTTPAPITests(SyncTestCase):
lease_secret = urandom(32)
storage_index = urandom(16)
url = self.http.client.relative_url(
"/v1/immutable/" + _encode_si(storage_index)
"/storage/v1/immutable/" + _encode_si(storage_index)
)
message = {"bad-message": "missing expected keys"}
@ -1418,7 +1418,7 @@ class SharedImmutableMutableTestsMixin:
self.http.client.request(
"GET",
self.http.client.relative_url(
"/v1/{}/{}/1".format(self.KIND, _encode_si(storage_index))
"/storage/v1/{}/{}/1".format(self.KIND, _encode_si(storage_index))
),
)
)
@ -1441,7 +1441,7 @@ class SharedImmutableMutableTestsMixin:
self.http.client.request(
"GET",
self.http.client.relative_url(
"/v1/{}/{}/1".format(self.KIND, _encode_si(storage_index))
"/storage/v1/{}/{}/1".format(self.KIND, _encode_si(storage_index))
),
headers=headers,
)

4
src/allmydata/test/test_storage_https.py
View File

@ -198,6 +198,10 @@ class PinningHTTPSValidation(AsyncTestCase):
response = await self.request(url, certificate)
self.assertEqual(await response.content(), b"YOYODYNE")
# We keep getting TLSMemoryBIOProtocol being left around, so try harder
# to wait for it to finish.
await deferLater(reactor, 0.001)
# A potential attack to test is a private key that doesn't match the
# certificate... but OpenSSL (quite rightly) won't let you listen with that
# so I don't know how to test that! See