Merge remote-tracking branch 'origin/master' into 3927.release

docs/proposed/http-storage-node-protocol.rst

@@ -395,8 +395,8 @@ Encoding
 General
 ~~~~~~~
 
-``GET /v1/version``
+``GET /storage/v1/version``
-!!!!!!!!!!!!!!!!!!!
+!!!!!!!!!!!!!!!!!!!!!!!!!!!
 
 Retrieve information about the version of the storage server.
 Information is returned as an encoded mapping.
@@ -409,14 +409,13 @@ For example::
       "tolerates-immutable-read-overrun": true,
       "delete-mutable-shares-with-zero-length-writev": true,
       "fills-holes-with-zero-bytes": true,
-      "prevents-read-past-end-of-share-data": true,
+      "prevents-read-past-end-of-share-data": true
-      "gbs-anonymous-storage-url": "pb://...#v=1"
       },
     "application-version": "1.13.0"
     }
 
-``PUT /v1/lease/:storage_index``
+``PUT /storage/v1/lease/:storage_index``
-!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
+!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
 
 Either renew or create a new lease on the bucket addressed by ``storage_index``.
 
@@ -468,8 +467,8 @@ Immutable
 Writing
 ~~~~~~~
 
-``POST /v1/immutable/:storage_index``
+``POST /storage/v1/immutable/:storage_index``
-!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
+!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
 
 Initialize an immutable storage index with some buckets.
 The buckets may have share data written to them once.
@@ -504,7 +503,7 @@ Handling repeat calls:
 Discussion
 ``````````
 
-We considered making this ``POST /v1/immutable`` instead.
+We considered making this ``POST /storage/v1/immutable`` instead.
 The motivation was to keep *storage index* out of the request URL.
 Request URLs have an elevated chance of being logged by something.
 We were concerned that having the *storage index* logged may increase some risks.
@@ -539,8 +538,8 @@ Rejected designs for upload secrets:
   it must contain randomness.
   Randomness means there is no need to have a secret per share, since adding share-specific content to randomness doesn't actually make the secret any better.
 
-``PATCH /v1/immutable/:storage_index/:share_number``
+``PATCH /storage/v1/immutable/:storage_index/:share_number``
-!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
+!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
 
 Write data for the indicated share.
 The share number must belong to the storage index.
@@ -580,8 +579,8 @@ Responses:
   the response is ``CONFLICT``.
   At this point the only thing to do is abort the upload and start from scratch (see below).
 
-``PUT /v1/immutable/:storage_index/:share_number/abort``
+``PUT /storage/v1/immutable/:storage_index/:share_number/abort``
-!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
+!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
 
 This cancels an *in-progress* upload.
 
@@ -616,8 +615,8 @@ From RFC 7231::
    PATCH method defined in [RFC5789]).
 
 
-``POST /v1/immutable/:storage_index/:share_number/corrupt``
+``POST /storage/v1/immutable/:storage_index/:share_number/corrupt``
-!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
+!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
 
 Advise the server the data read from the indicated share was corrupt. The
 request body includes an human-meaningful text string with details about the
@@ -635,8 +634,8 @@ couldn't be found.
 Reading
 ~~~~~~~
 
-``GET /v1/immutable/:storage_index/shares``
+``GET /storage/v1/immutable/:storage_index/shares``
-!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
+!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
 
 Retrieve a list (semantically, a set) indicating all shares available for the
 indicated storage index. For example::
@@ -645,8 +644,8 @@ indicated storage index. For example::
 
 An unknown storage index results in an empty list.
 
-``GET /v1/immutable/:storage_index/:share_number``
+``GET /storage/v1/immutable/:storage_index/:share_number``
-!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
+!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
 
 Read a contiguous sequence of bytes from one share in one bucket.
 The response body is the raw share data (i.e., ``application/octet-stream``).
@@ -686,8 +685,8 @@ Mutable
 Writing
 ~~~~~~~
 
-``POST /v1/mutable/:storage_index/read-test-write``
+``POST /storage/v1/mutable/:storage_index/read-test-write``
-!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
+!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
 
 General purpose read-test-and-write operation for mutable storage indexes.
 A mutable storage index is also called a "slot"
@@ -742,18 +741,18 @@ As a result, if there is no data at all, an empty bytestring is returned no matt
 Reading
 ~~~~~~~
 
-``GET /v1/mutable/:storage_index/shares``
+``GET /storage/v1/mutable/:storage_index/shares``
-!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
+!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
 
 Retrieve a set indicating all shares available for the indicated storage index.
 For example (this is shown as list, since it will be list for JSON, but will be set for CBOR)::
 
   [1, 5]
 
-``GET /v1/mutable/:storage_index/:share_number``
+``GET /storage/v1/mutable/:storage_index/:share_number``
 !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
 
-Read data from the indicated mutable shares, just like ``GET /v1/immutable/:storage_index``
+Read data from the indicated mutable shares, just like ``GET /storage/v1/immutable/:storage_index``
 
 The ``Range`` header may be used to request exactly one ``bytes`` range, in which case the response code will be 206 (partial content).
 Interpretation and response behavior is as specified in RFC 7233 § 4.1.
@@ -765,8 +764,8 @@ The resulting ``Content-Range`` header will be consistent with the returned data
 If the response to a query is an empty range, the ``NO CONTENT`` (204) response code will be used.
 
 
-``POST /v1/mutable/:storage_index/:share_number/corrupt``
+``POST /storage/v1/mutable/:storage_index/:share_number/corrupt``
-!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
+!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
 
 Advise the server the data read from the indicated share was corrupt.
 Just like the immutable version.
@@ -779,7 +778,7 @@ Immutable Data
 
 1. Create a bucket for storage index ``AAAAAAAAAAAAAAAA`` to hold two immutable shares, discovering that share ``1`` was already uploaded::
 
-     POST /v1/immutable/AAAAAAAAAAAAAAAA
+     POST /storage/v1/immutable/AAAAAAAAAAAAAAAA
      Authorization: Tahoe-LAFS nurl-swissnum
      X-Tahoe-Authorization: lease-renew-secret efgh
      X-Tahoe-Authorization: lease-cancel-secret jjkl
@@ -792,7 +791,7 @@ Immutable Data
 
 #. Upload the content for immutable share ``7``::
 
-     PATCH /v1/immutable/AAAAAAAAAAAAAAAA/7
+     PATCH /storage/v1/immutable/AAAAAAAAAAAAAAAA/7
      Authorization: Tahoe-LAFS nurl-swissnum
      Content-Range: bytes 0-15/48
      X-Tahoe-Authorization: upload-secret xyzf
@@ -800,7 +799,7 @@ Immutable Data
 
      200 OK
 
-     PATCH /v1/immutable/AAAAAAAAAAAAAAAA/7
+     PATCH /storage/v1/immutable/AAAAAAAAAAAAAAAA/7
      Authorization: Tahoe-LAFS nurl-swissnum
      Content-Range: bytes 16-31/48
      X-Tahoe-Authorization: upload-secret xyzf
@@ -808,7 +807,7 @@ Immutable Data
 
      200 OK
 
-     PATCH /v1/immutable/AAAAAAAAAAAAAAAA/7
+     PATCH /storage/v1/immutable/AAAAAAAAAAAAAAAA/7
      Authorization: Tahoe-LAFS nurl-swissnum
      Content-Range: bytes 32-47/48
      X-Tahoe-Authorization: upload-secret xyzf
@@ -818,7 +817,7 @@ Immutable Data
 
 #. Download the content of the previously uploaded immutable share ``7``::
 
-     GET /v1/immutable/AAAAAAAAAAAAAAAA?share=7
+     GET /storage/v1/immutable/AAAAAAAAAAAAAAAA?share=7
      Authorization: Tahoe-LAFS nurl-swissnum
      Range: bytes=0-47
 
@@ -827,7 +826,7 @@ Immutable Data
 
 #. Renew the lease on all immutable shares in bucket ``AAAAAAAAAAAAAAAA``::
 
-     PUT /v1/lease/AAAAAAAAAAAAAAAA
+     PUT /storage/v1/lease/AAAAAAAAAAAAAAAA
      Authorization: Tahoe-LAFS nurl-swissnum
      X-Tahoe-Authorization: lease-cancel-secret jjkl
      X-Tahoe-Authorization: lease-renew-secret efgh
@@ -842,7 +841,7 @@ The special test vector of size 1 but empty bytes will only pass
 if there is no existing share,
 otherwise it will read a byte which won't match `b""`::
 
-     POST /v1/mutable/BBBBBBBBBBBBBBBB/read-test-write
+     POST /storage/v1/mutable/BBBBBBBBBBBBBBBB/read-test-write
      Authorization: Tahoe-LAFS nurl-swissnum
      X-Tahoe-Authorization: write-enabler abcd
      X-Tahoe-Authorization: lease-cancel-secret efgh
@@ -874,7 +873,7 @@ otherwise it will read a byte which won't match `b""`::
 
 #. Safely rewrite the contents of a known version of mutable share number ``3`` (or fail)::
 
-     POST /v1/mutable/BBBBBBBBBBBBBBBB/read-test-write
+     POST /storage/v1/mutable/BBBBBBBBBBBBBBBB/read-test-write
      Authorization: Tahoe-LAFS nurl-swissnum
      X-Tahoe-Authorization: write-enabler abcd
      X-Tahoe-Authorization: lease-cancel-secret efgh
@@ -906,14 +905,14 @@ otherwise it will read a byte which won't match `b""`::
 
 #. Download the contents of share number ``3``::
 
-     GET /v1/mutable/BBBBBBBBBBBBBBBB?share=3&offset=0&size=10
+     GET /storage/v1/mutable/BBBBBBBBBBBBBBBB?share=3&offset=0&size=10
      Authorization: Tahoe-LAFS nurl-swissnum
 
      <complete 16 bytes of previously uploaded data>
 
 #. Renew the lease on previously uploaded mutable share in slot ``BBBBBBBBBBBBBBBB``::
 
-     PUT /v1/lease/BBBBBBBBBBBBBBBB
+     PUT /storage/v1/lease/BBBBBBBBBBBBBBBB
      Authorization: Tahoe-LAFS nurl-swissnum
      X-Tahoe-Authorization: lease-cancel-secret efgh
      X-Tahoe-Authorization: lease-renew-secret ijkl

docs/specifications/url.rst

@@ -47,27 +47,27 @@ This can be considered to expand to "**N**\ ew URLs" or "Authe\ **N**\ ticating
 The anticipated use for a **NURL** will still be to establish a TLS connection to a peer.
 The protocol run over that TLS connection could be Foolscap though it is more likely to be an HTTP-based protocol (such as GBS).
 
+Unlike fURLs, only a single net-loc is included, for consistency with other forms of URLs.
+As a result, multiple NURLs may be available for a single server.
+
 Syntax
 ------
 
 The EBNF for a NURL is as follows::
 
-  nurl         = scheme, hash, "@", net-loc-list, "/", swiss-number, [ version1 ]
+  nurl         = tcp-nurl | tor-nurl | i2p-nurl
-
+  tcp-nurl     = "pb://", hash, "@", tcp-loc, "/", swiss-number, [ version1 ]
-  scheme       = "pb://"
+  tor-nurl     = "pb+tor://", hash, "@", tcp-loc, "/", swiss-number, [ version1 ]
+  i2p-nurl     = "pb+i2p://", hash, "@", i2p-loc, "/", swiss-number, [ version1 ]
 
   hash         = unreserved
 
-  net-loc-list = net-loc, [ { ",", net-loc } ]
+  tcp-loc      = hostname, [ ":" port ]
-  net-loc      = tcp-loc | tor-loc | i2p-loc
-
-  tcp-loc      = [ "tcp:" ], hostname, [ ":" port ]
-  tor-loc      = "tor:", hostname, [ ":" port ]
-  i2p-loc      = "i2p:", i2p-addr, [ ":" port ]
-
-  i2p-addr     = { unreserved }, ".i2p"
   hostname     = domain | IPv4address | IPv6address
 
+  i2p-loc      = i2p-addr, [ ":" port ]
+  i2p-addr     = { unreserved }, ".i2p"
+
   swiss-number = segment
 
   version1     = "#v=1"
@@ -87,11 +87,13 @@ These differences are separated into distinct versions.
 Version 0
 ---------
 
-A Foolscap fURL is considered the canonical definition of a version 0 NURL.
+In theory, a Foolscap fURL with a single netloc is considered the canonical definition of a version 0 NURL.
 Notably,
 the hash component is defined as the base32-encoded SHA1 hash of the DER form of an x509v3 certificate.
 A version 0 NURL is identified by the absence of the ``v=1`` fragment.
 
+In practice, real world fURLs may have more than one netloc, so lack of version fragment will likely just involve dispatching the fURL to a different parser.
+
 Examples
 ~~~~~~~~
 
@@ -103,11 +105,8 @@ Version 1
 
 The hash component of a version 1 NURL differs in three ways from the prior version.
 
-1. The hash function used is SHA3-224 instead of SHA1.
+1. The hash function used is SHA-256, to match RFC 7469.
-   The security of SHA1 `continues to be eroded`_.
+   The security of SHA1 `continues to be eroded`_; Latacora `SHA-2`_.
-   Contrariwise SHA3 is currently the most recent addition to the SHA family by NIST.
-   The 224 bit instance is chosen to keep the output short and because it offers greater collision resistance than SHA1 was thought to offer even at its inception
-   (prior to security research showing actual collision resistance is lower).
 2. The hash is computed over the certificate's SPKI instead of the whole certificate.
    This allows certificate re-generation so long as the public key remains the same.
    This is useful to allow contact information to be updated or extension of validity period.
@@ -122,7 +121,7 @@ The hash component of a version 1 NURL differs in three ways from the prior vers
    *all* certificate fields should be considered within the context of the relationship identified by the SPKI hash.
 
 3. The hash is encoded using urlsafe-base64 (without padding) instead of base32.
-   This provides a more compact representation and minimizes the usability impacts of switching from a 160 bit hash to a 224 bit hash.
+   This provides a more compact representation and minimizes the usability impacts of switching from a 160 bit hash to a 256 bit hash.
 
 A version 1 NURL is identified by the presence of the ``v=1`` fragment.
 Though the length of the hash string (38 bytes) could also be used to differentiate it from a version 0 NURL,
@@ -140,7 +139,8 @@ Examples
 * ``pb://azEu8vlRpnEeYm0DySQDeNY3Z2iJXHC_bsbaAw@localhost:47877/64i4aokv4ej#v=1``
 
 .. _`continues to be eroded`: https://en.wikipedia.org/wiki/SHA-1#Cryptanalysis_and_validation
-.. _`explored by the web community`: https://www.imperialviolet.org/2011/05/04/pinning.html
+.. _`SHA-2`: https://latacora.micro.blog/2018/04/03/cryptographic-right-answers.html
+.. _`explored by the web community`: https://www.rfc-editor.org/rfc/rfc7469
 .. _Foolscap: https://github.com/warner/foolscap
 
 .. [1] ``foolscap.furl.decode_furl`` is taken as the canonical definition of the syntax of a fURL.

integration/test_i2p.py

@@ -55,7 +55,7 @@ def i2p_network(reactor, temp_dir, request):
         proto,
         which("docker"),
         (
-            "docker", "run", "-p", "7656:7656", "purplei2p/i2pd",
+            "docker", "run", "-p", "7656:7656", "purplei2p/i2pd:release-2.43.0",
             # Bad URL for reseeds, so it can't talk to other routers.
             "--reseed.urls", "http://localhost:1/",
         ),
@@ -63,7 +63,7 @@ def i2p_network(reactor, temp_dir, request):
 
     def cleanup():
         try:
-            proto.transport.signalProcess("KILL")
+            proto.transport.signalProcess("INT")
             util.block_with_timeout(proto.exited, reactor)
         except ProcessExitedAlready:
             pass

newsfragments/3902.feature

@@ -0,0 +1 @@
+The new HTTPS-based storage server is now enabled transparently on the same port as the Foolscap server. This will not have any user-facing impact until the HTTPS storage protocol is supported in clients as well.

setup.py

@@ -386,6 +386,9 @@ setup(name="tahoe-lafs", # also set in __init__.py
           ],
           "test": [
               "flake8",
+              # On Python 3.7, importlib_metadata v5 breaks flake8.
+              # https://github.com/python/importlib_metadata/issues/407
+              "importlib_metadata<5; python_version < '3.8'",
               # Pin a specific pyflakes so we don't have different folks
               # disagreeing on what is or is not a lint issue.  We can bump
               # this version from time to time, but we will do it

src/allmydata/client.py

@@ -1,17 +1,9 @@
 """
 Ported to Python 3.
 """
-from __future__ import absolute_import
+from __future__ import annotations
-from __future__ import division
-from __future__ import print_function
-from __future__ import unicode_literals
-
-from future.utils import PY2
-if PY2:
-    from future.builtins import filter, map, zip, ascii, chr, hex, input, next, oct, open, pow, round, super, bytes, dict, list, object, range, max, min  # noqa: F401
-    # Don't use future str to prevent leaking future's newbytes into foolscap, which they break.
-    from past.builtins import unicode as str
 
+from typing import Optional
 import os, stat, time, weakref
 from base64 import urlsafe_b64encode
 from functools import partial
@@ -591,6 +583,10 @@ def anonymous_storage_enabled(config):
 
 @implementer(IStatsProducer)
 class _Client(node.Node, pollmixin.PollMixin):
+    """
+    This class should be refactored; see
+    https://tahoe-lafs.org/trac/tahoe-lafs/ticket/3931
+    """
 
     STOREDIR = 'storage'
     NODETYPE = "client"
@@ -658,6 +654,14 @@ class _Client(node.Node, pollmixin.PollMixin):
         if webport:
             self.init_web(webport) # strports string
 
+        # TODO this may be the wrong location for now? but as temporary measure
+        # it allows us to get NURLs for testing in test_istorageserver.py. This
+        # will eventually get fixed one way or another in
+        # https://tahoe-lafs.org/trac/tahoe-lafs/ticket/3901. See also
+        # https://tahoe-lafs.org/trac/tahoe-lafs/ticket/3931 for the bigger
+        # picture issue.
+        self.storage_nurls : Optional[set] = None
+
     def init_stats_provider(self):
         self.stats_provider = StatsProvider(self)
         self.stats_provider.setServiceParent(self)
@@ -818,6 +822,10 @@ class _Client(node.Node, pollmixin.PollMixin):
         if anonymous_storage_enabled(self.config):
             furl_file = self.config.get_private_path("storage.furl").encode(get_filesystem_encoding())
             furl = self.tub.registerReference(FoolscapStorageServer(ss), furlFile=furl_file)
+            (_, _, swissnum) = decode_furl(furl)
+            self.storage_nurls = self.tub.negotiationClass.add_storage_server(
+                ss, swissnum.encode("ascii")
+            )
             announcement["anonymous-storage-FURL"] = furl
 
         enabled_storage_servers = self._enable_storage_servers(

src/allmydata/node.py

@@ -55,6 +55,8 @@ from allmydata.util.yamlutil import (
 from . import (
     __full_version__,
 )
+from .protocol_switch import create_tub_with_https_support
+
 
 def _common_valid_config():
     return configutil.ValidConfiguration({
@@ -706,7 +708,10 @@ def create_tub(tub_options, default_connection_handlers, foolscap_connection_han
     :param dict tub_options: every key-value pair in here will be set in
         the new Tub via `Tub.setOption`
     """
-    tub = Tub(**kwargs)
+    # We listen simulataneously for both Foolscap and HTTPS on the same port,
+    # so we have to create a special Foolscap Tub for that to work:
+    tub = create_tub_with_https_support(**kwargs)
+
     for (name, value) in list(tub_options.items()):
         tub.setOption(name, value)
     handlers = default_connection_handlers.copy()

src/allmydata/protocol_switch.py

@@ -0,0 +1,210 @@
+"""
+Support for listening with both HTTPS and Foolscap on the same port.
+
+The goal is to make the transition from Foolscap to HTTPS-based protocols as
+simple as possible, with no extra configuration needed.  Listening on the same
+port means a user upgrading Tahoe-LAFS will automatically get HTTPS working
+with no additional changes.
+
+Use ``create_tub_with_https_support()`` creates a new ``Tub`` that has its
+``negotiationClass`` modified to be a new subclass tied to that specific
+``Tub`` instance.  Calling ``tub.negotiationClass.add_storage_server(...)``
+then adds relevant information for a storage server once it becomes available
+later in the configuration process.
+"""
+
+from __future__ import annotations
+
+from itertools import chain
+
+from twisted.internet.protocol import Protocol
+from twisted.internet.interfaces import IDelayedCall
+from twisted.internet.ssl import CertificateOptions
+from twisted.web.server import Site
+from twisted.protocols.tls import TLSMemoryBIOFactory
+from twisted.internet import reactor
+
+from hyperlink import DecodedURL
+from foolscap.negotiate import Negotiation
+from foolscap.api import Tub
+
+from .storage.http_server import HTTPServer, build_nurl
+from .storage.server import StorageServer
+
+
+class _PretendToBeNegotiation(type):
+    """
+    Metaclass that allows ``_FoolscapOrHttps`` to pretend to be a
+    ``Negotiation`` instance, since Foolscap does some checks like
+    ``assert isinstance(protocol, tub.negotiationClass)`` in its internals,
+    and sometimes that ``protocol`` is a ``_FoolscapOrHttps`` instance, but
+    sometimes it's a ``Negotiation`` instance.
+    """
+
+    def __instancecheck__(self, instance):
+        return issubclass(instance.__class__, self) or isinstance(instance, Negotiation)
+
+
+class _FoolscapOrHttps(Protocol, metaclass=_PretendToBeNegotiation):
+    """
+    Based on initial query, decide whether we're talking Foolscap or HTTP.
+
+    Additionally, pretends to be a ``foolscap.negotiate.Negotiation`` instance,
+    since these are created by Foolscap's ``Tub``, by setting this to be the
+    tub's ``negotiationClass``.
+
+    Do not instantiate directly, use ``create_tub_with_https_support(...)``
+    instead.  The way this class works is that a new subclass is created for a
+    specific ``Tub`` instance.
+    """
+
+    # These are class attributes; they will be set by
+    # create_tub_with_https_support() and add_storage_server().
+
+    # The Twisted HTTPS protocol factory wrapping the storage server HTTP API:
+    https_factory: TLSMemoryBIOFactory
+    # The tub that created us:
+    tub: Tub
+
+    @classmethod
+    def add_storage_server(
+        cls, storage_server: StorageServer, swissnum: bytes
+    ) -> set[DecodedURL]:
+        """
+        Update a ``_FoolscapOrHttps`` subclass for a specific ``Tub`` instance
+        with the class attributes it requires for a specific storage server.
+
+        Returns the resulting NURLs.
+        """
+        # We need to be a subclass:
+        assert cls != _FoolscapOrHttps
+        # The tub instance must already be set:
+        assert hasattr(cls, "tub")
+        assert isinstance(cls.tub, Tub)
+
+        # Tub.myCertificate is a twisted.internet.ssl.PrivateCertificate
+        # instance.
+        certificate_options = CertificateOptions(
+            privateKey=cls.tub.myCertificate.privateKey.original,
+            certificate=cls.tub.myCertificate.original,
+        )
+
+        http_storage_server = HTTPServer(storage_server, swissnum)
+        cls.https_factory = TLSMemoryBIOFactory(
+            certificate_options,
+            False,
+            Site(http_storage_server.get_resource()),
+        )
+
+        storage_nurls = set()
+        # Individual hints can be in the form
+        # "tcp:host:port,tcp:host:port,tcp:host:port".
+        for location_hint in chain.from_iterable(
+            hints.split(",") for hints in cls.tub.locationHints
+        ):
+            if location_hint.startswith("tcp:"):
+                _, hostname, port = location_hint.split(":")
+                port = int(port)
+                storage_nurls.add(
+                    build_nurl(
+                        hostname,
+                        port,
+                        str(swissnum, "ascii"),
+                        cls.tub.myCertificate.original.to_cryptography(),
+                    )
+                )
+            # TODO this is probably where we'll have to support Tor and I2P?
+            # See https://tahoe-lafs.org/trac/tahoe-lafs/ticket/3888#comment:9
+            # for discussion (there will be separate tickets added for those at
+            # some point.)
+        return storage_nurls
+
+    def __init__(self, *args, **kwargs):
+        self._foolscap: Negotiation = Negotiation(*args, **kwargs)
+
+    def __setattr__(self, name, value):
+        if name in {"_foolscap", "_buffer", "transport", "__class__", "_timeout"}:
+            object.__setattr__(self, name, value)
+        else:
+            setattr(self._foolscap, name, value)
+
+    def __getattr__(self, name):
+        return getattr(self._foolscap, name)
+
+    def _convert_to_negotiation(self):
+        """
+        Convert self to a ``Negotiation`` instance.
+        """
+        self.__class__ = Negotiation  # type: ignore
+        self.__dict__ = self._foolscap.__dict__
+
+    def initClient(self, *args, **kwargs):
+        # After creation, a Negotiation instance either has initClient() or
+        # initServer() called. Since this is a client, we're never going to do
+        # HTTP, so we can immediately become a Negotiation instance.
+        assert not hasattr(self, "_buffer")
+        self._convert_to_negotiation()
+        return self.initClient(*args, **kwargs)
+
+    def connectionMade(self):
+        self._buffer: bytes = b""
+        self._timeout: IDelayedCall = reactor.callLater(
+            30, self.transport.abortConnection
+        )
+
+    def connectionLost(self, reason):
+        if self._timeout.active():
+            self._timeout.cancel()
+
+    def dataReceived(self, data: bytes) -> None:
+        """Handle incoming data.
+
+        Once we've decided which protocol we are, update self.__class__, at
+        which point all methods will be called on the new class.
+        """
+        self._buffer += data
+        if len(self._buffer) < 8:
+            return
+
+        # Check if it looks like a Foolscap request. If so, it can handle this
+        # and later data, otherwise assume HTTPS.
+        self._timeout.cancel()
+        if self._buffer.startswith(b"GET /id/"):
+            # We're a Foolscap Negotiation server protocol instance:
+            transport = self.transport
+            buf = self._buffer
+            self._convert_to_negotiation()
+            self.makeConnection(transport)
+            self.dataReceived(buf)
+            return
+        else:
+            # We're a HTTPS protocol instance, serving the storage protocol:
+            assert self.transport is not None
+            protocol = self.https_factory.buildProtocol(self.transport.getPeer())
+            protocol.makeConnection(self.transport)
+            protocol.dataReceived(self._buffer)
+
+            # Update the factory so it knows we're transforming to a new
+            # protocol object (we'll do that next)
+            value = self.https_factory.protocols.pop(protocol)
+            self.https_factory.protocols[self] = value
+
+            # Transform self into the TLS protocol 🪄
+            self.__class__ = protocol.__class__
+            self.__dict__ = protocol.__dict__
+
+
+def create_tub_with_https_support(**kwargs) -> Tub:
+    """
+    Create a new Tub that also supports HTTPS.
+
+    This involves creating a new protocol switch class for the specific ``Tub``
+    instance.
+    """
+    the_tub = Tub(**kwargs)
+
+    class FoolscapOrHttpForTub(_FoolscapOrHttps):
+        tub = the_tub
+
+    the_tub.negotiationClass = FoolscapOrHttpForTub  # type: ignore
+    return the_tub

src/allmydata/storage/http_client.py

@@ -392,7 +392,7 @@ class StorageClientGeneral(object):
         """
         Return the version metadata for the server.
         """
-        url = self._client.relative_url("/v1/version")
+        url = self._client.relative_url("/storage/v1/version")
         response = yield self._client.request("GET", url)
         decoded_response = yield _decode_cbor(response, _SCHEMAS["get_version"])
         returnValue(decoded_response)
@@ -408,7 +408,7 @@ class StorageClientGeneral(object):
         Otherwise a new lease is added.
         """
         url = self._client.relative_url(
-            "/v1/lease/{}".format(_encode_si(storage_index))
+            "/storage/v1/lease/{}".format(_encode_si(storage_index))
         )
         response = yield self._client.request(
             "PUT",
@@ -457,7 +457,9 @@ def read_share_chunk(
     always provided by the current callers.
     """
     url = client.relative_url(
-        "/v1/{}/{}/{}".format(share_type, _encode_si(storage_index), share_number)
+        "/storage/v1/{}/{}/{}".format(
+            share_type, _encode_si(storage_index), share_number
+        )
     )
     response = yield client.request(
         "GET",
@@ -518,7 +520,7 @@ async def advise_corrupt_share(
 ):
     assert isinstance(reason, str)
     url = client.relative_url(
-        "/v1/{}/{}/{}/corrupt".format(
+        "/storage/v1/{}/{}/{}/corrupt".format(
             share_type, _encode_si(storage_index), share_number
         )
     )
@@ -563,7 +565,9 @@ class StorageClientImmutables(object):
         Result fires when creating the storage index succeeded, if creating the
         storage index failed the result will fire with an exception.
         """
-        url = self._client.relative_url("/v1/immutable/" + _encode_si(storage_index))
+        url = self._client.relative_url(
+            "/storage/v1/immutable/" + _encode_si(storage_index)
+        )
         message = {"share-numbers": share_numbers, "allocated-size": allocated_size}
 
         response = yield self._client.request(
@@ -588,7 +592,9 @@ class StorageClientImmutables(object):
     ) -> Deferred[None]:
         """Abort the upload."""
         url = self._client.relative_url(
-            "/v1/immutable/{}/{}/abort".format(_encode_si(storage_index), share_number)
+            "/storage/v1/immutable/{}/{}/abort".format(
+                _encode_si(storage_index), share_number
+            )
         )
         response = yield self._client.request(
             "PUT",
@@ -620,7 +626,9 @@ class StorageClientImmutables(object):
         been uploaded.
         """
         url = self._client.relative_url(
-            "/v1/immutable/{}/{}".format(_encode_si(storage_index), share_number)
+            "/storage/v1/immutable/{}/{}".format(
+                _encode_si(storage_index), share_number
+            )
         )
         response = yield self._client.request(
             "PATCH",
@@ -668,7 +676,7 @@ class StorageClientImmutables(object):
         Return the set of shares for a given storage index.
         """
         url = self._client.relative_url(
-            "/v1/immutable/{}/shares".format(_encode_si(storage_index))
+            "/storage/v1/immutable/{}/shares".format(_encode_si(storage_index))
         )
         response = yield self._client.request(
             "GET",
@@ -774,7 +782,7 @@ class StorageClientMutables:
         are done and if they are valid the writes are done.
         """
         url = self._client.relative_url(
-            "/v1/mutable/{}/read-test-write".format(_encode_si(storage_index))
+            "/storage/v1/mutable/{}/read-test-write".format(_encode_si(storage_index))
         )
         message = {
             "test-write-vectors": {
@@ -817,7 +825,7 @@ class StorageClientMutables:
         List the share numbers for a given storage index.
         """
         url = self._client.relative_url(
-            "/v1/mutable/{}/shares".format(_encode_si(storage_index))
+            "/storage/v1/mutable/{}/shares".format(_encode_si(storage_index))
         )
         response = await self._client.request("GET", url)
         if response.code == http.OK:

src/allmydata/storage/http_server.py

@@ -4,12 +4,13 @@ HTTP server for storage.
 
 from __future__ import annotations
 
-from typing import Dict, List, Set, Tuple, Any, Callable, Union
+from typing import Dict, List, Set, Tuple, Any, Callable, Union, cast
 from functools import wraps
 from base64 import b64decode
 import binascii
 from tempfile import TemporaryFile
 
+from cryptography.x509 import Certificate as CryptoCertificate
 from zope.interface import implementer
 from klein import Klein
 from twisted.web import http
@@ -18,6 +19,7 @@ from twisted.internet.interfaces import (
     IStreamServerEndpoint,
     IPullProducer,
 )
+from twisted.internet.address import IPv4Address, IPv6Address
 from twisted.internet.defer import Deferred
 from twisted.internet.ssl import CertificateOptions, Certificate, PrivateCertificate
 from twisted.web.server import Site, Request
@@ -193,7 +195,12 @@ class UploadsInProgress(object):
 
     def remove_write_bucket(self, bucket: BucketWriter):
         """Stop tracking the given ``BucketWriter``."""
-        storage_index, share_number = self._bucketwriters.pop(bucket)
+        try:
+            storage_index, share_number = self._bucketwriters.pop(bucket)
+        except KeyError:
+            # This is probably a BucketWriter created by Foolscap, so just
+            # ignore it.
+            return
         uploads_index = self._uploads[storage_index]
         uploads_index.shares.pop(share_number)
         uploads_index.upload_secrets.pop(share_number)
@@ -545,7 +552,7 @@ class HTTPServer(object):
 
     ##### Generic APIs #####
 
-    @_authorized_route(_app, set(), "/v1/version", methods=["GET"])
+    @_authorized_route(_app, set(), "/storage/v1/version", methods=["GET"])
     def version(self, request, authorization):
         """Return version information."""
         return self._send_encoded(request, self._storage_server.get_version())
@@ -555,7 +562,7 @@ class HTTPServer(object):
     @_authorized_route(
         _app,
         {Secrets.LEASE_RENEW, Secrets.LEASE_CANCEL, Secrets.UPLOAD},
-        "/v1/immutable/<storage_index:storage_index>",
+        "/storage/v1/immutable/<storage_index:storage_index>",
         methods=["POST"],
     )
     def allocate_buckets(self, request, authorization, storage_index):
@@ -591,7 +598,7 @@ class HTTPServer(object):
     @_authorized_route(
         _app,
         {Secrets.UPLOAD},
-        "/v1/immutable/<storage_index:storage_index>/<int(signed=False):share_number>/abort",
+        "/storage/v1/immutable/<storage_index:storage_index>/<int(signed=False):share_number>/abort",
         methods=["PUT"],
     )
     def abort_share_upload(self, request, authorization, storage_index, share_number):
@@ -622,7 +629,7 @@ class HTTPServer(object):
     @_authorized_route(
         _app,
         {Secrets.UPLOAD},
-        "/v1/immutable/<storage_index:storage_index>/<int(signed=False):share_number>",
+        "/storage/v1/immutable/<storage_index:storage_index>/<int(signed=False):share_number>",
         methods=["PATCH"],
     )
     def write_share_data(self, request, authorization, storage_index, share_number):
@@ -665,7 +672,7 @@ class HTTPServer(object):
     @_authorized_route(
         _app,
         set(),
-        "/v1/immutable/<storage_index:storage_index>/shares",
+        "/storage/v1/immutable/<storage_index:storage_index>/shares",
         methods=["GET"],
     )
     def list_shares(self, request, authorization, storage_index):
@@ -678,7 +685,7 @@ class HTTPServer(object):
     @_authorized_route(
         _app,
         set(),
-        "/v1/immutable/<storage_index:storage_index>/<int(signed=False):share_number>",
+        "/storage/v1/immutable/<storage_index:storage_index>/<int(signed=False):share_number>",
         methods=["GET"],
     )
     def read_share_chunk(self, request, authorization, storage_index, share_number):
@@ -694,7 +701,7 @@ class HTTPServer(object):
     @_authorized_route(
         _app,
         {Secrets.LEASE_RENEW, Secrets.LEASE_CANCEL},
-        "/v1/lease/<storage_index:storage_index>",
+        "/storage/v1/lease/<storage_index:storage_index>",
         methods=["PUT"],
     )
     def add_or_renew_lease(self, request, authorization, storage_index):
@@ -715,7 +722,7 @@ class HTTPServer(object):
     @_authorized_route(
         _app,
         set(),
-        "/v1/immutable/<storage_index:storage_index>/<int(signed=False):share_number>/corrupt",
+        "/storage/v1/immutable/<storage_index:storage_index>/<int(signed=False):share_number>/corrupt",
         methods=["POST"],
     )
     def advise_corrupt_share_immutable(
@@ -736,7 +743,7 @@ class HTTPServer(object):
     @_authorized_route(
         _app,
         {Secrets.LEASE_RENEW, Secrets.LEASE_CANCEL, Secrets.WRITE_ENABLER},
-        "/v1/mutable/<storage_index:storage_index>/read-test-write",
+        "/storage/v1/mutable/<storage_index:storage_index>/read-test-write",
         methods=["POST"],
     )
     def mutable_read_test_write(self, request, authorization, storage_index):
@@ -771,7 +778,7 @@ class HTTPServer(object):
     @_authorized_route(
         _app,
         set(),
-        "/v1/mutable/<storage_index:storage_index>/<int(signed=False):share_number>",
+        "/storage/v1/mutable/<storage_index:storage_index>/<int(signed=False):share_number>",
         methods=["GET"],
     )
     def read_mutable_chunk(self, request, authorization, storage_index, share_number):
@@ -795,7 +802,10 @@ class HTTPServer(object):
         return read_range(request, read_data, share_length)
 
     @_authorized_route(
-        _app, set(), "/v1/mutable/<storage_index:storage_index>/shares", methods=["GET"]
+        _app,
+        set(),
+        "/storage/v1/mutable/<storage_index:storage_index>/shares",
+        methods=["GET"],
     )
     def enumerate_mutable_shares(self, request, authorization, storage_index):
         """List mutable shares for a storage index."""
@@ -805,7 +815,7 @@ class HTTPServer(object):
     @_authorized_route(
         _app,
         set(),
-        "/v1/mutable/<storage_index:storage_index>/<int(signed=False):share_number>/corrupt",
+        "/storage/v1/mutable/<storage_index:storage_index>/<int(signed=False):share_number>/corrupt",
         methods=["POST"],
     )
     def advise_corrupt_share_mutable(
@@ -859,6 +869,29 @@ class _TLSEndpointWrapper(object):
         )
 
 
+def build_nurl(
+    hostname: str, port: int, swissnum: str, certificate: CryptoCertificate
+) -> DecodedURL:
+    """
+    Construct a HTTPS NURL, given the hostname, port, server swissnum, and x509
+    certificate for the server.  Clients can then connect to the server using
+    this NURL.
+    """
+    return DecodedURL().replace(
+        fragment="v=1",  # how we know this NURL is HTTP-based (i.e. not Foolscap)
+        host=hostname,
+        port=port,
+        path=(swissnum,),
+        userinfo=(
+            str(
+                get_spki_hash(certificate),
+                "ascii",
+            ),
+        ),
+        scheme="pb",
+    )
+
+
 def listen_tls(
     server: HTTPServer,
     hostname: str,
@@ -878,22 +911,15 @@ def listen_tls(
     """
     endpoint = _TLSEndpointWrapper.from_paths(endpoint, private_key_path, cert_path)
 
-    def build_nurl(listening_port: IListeningPort) -> DecodedURL:
+    def get_nurl(listening_port: IListeningPort) -> DecodedURL:
-        nurl = DecodedURL().replace(
+        address = cast(Union[IPv4Address, IPv6Address], listening_port.getHost())
-            fragment="v=1",  # how we know this NURL is HTTP-based (i.e. not Foolscap)
+        return build_nurl(
-            host=hostname,
+            hostname,
-            port=listening_port.getHost().port,
+            address.port,
-            path=(str(server._swissnum, "ascii"),),
+            str(server._swissnum, "ascii"),
-            userinfo=(
+            load_pem_x509_certificate(cert_path.getContent()),
-                str(
-                    get_spki_hash(load_pem_x509_certificate(cert_path.getContent())),
-                    "ascii",
-                ),
-            ),
-            scheme="pb",
         )
-        return nurl
 
     return endpoint.listen(Site(server.get_resource())).addCallback(
-        lambda listening_port: (build_nurl(listening_port), listening_port)
+        lambda listening_port: (get_nurl(listening_port), listening_port)
     )

src/allmydata/test/test_istorageserver.py

@@ -18,21 +18,14 @@ from unittest import SkipTest
 from twisted.internet.defer import inlineCallbacks, returnValue, succeed
 from twisted.internet.task import Clock
 from twisted.internet import reactor
-from twisted.internet.endpoints import serverFromString
-from twisted.python.filepath import FilePath
 from foolscap.api import Referenceable, RemoteException
 
-from allmydata.interfaces import IStorageServer  # really, IStorageClient
+# A better name for this would be IStorageClient...
+from allmydata.interfaces import IStorageServer
+
 from .common_system import SystemTestMixin
-from .common import AsyncTestCase, SameProcessStreamEndpointAssigner
+from .common import AsyncTestCase
-from .certs import (
-    generate_certificate,
-    generate_private_key,
-    private_key_to_file,
-    cert_to_file,
-)
 from allmydata.storage.server import StorageServer  # not a IStorageServer!!
-from allmydata.storage.http_server import HTTPServer, listen_tls
 from allmydata.storage.http_client import StorageClient
 from allmydata.storage_client import _HTTPStorageServer
 
@@ -1084,40 +1077,17 @@ class _FoolscapMixin(_SharedMixin):
 class _HTTPMixin(_SharedMixin):
     """Run tests on the HTTP version of ``IStorageServer``."""
 
-    def setUp(self):
-        self._port_assigner = SameProcessStreamEndpointAssigner()
-        self._port_assigner.setUp()
-        self.addCleanup(self._port_assigner.tearDown)
-        return _SharedMixin.setUp(self)
-
-    @inlineCallbacks
     def _get_istorage_server(self):
-        swissnum = b"1234"
+        nurl = list(self.clients[0].storage_nurls)[0]
-        http_storage_server = HTTPServer(self.server, swissnum)
-
-        # Listen on randomly assigned port, using self-signed cert:
-        private_key = generate_private_key()
-        certificate = generate_certificate(private_key)
-        _, endpoint_string = self._port_assigner.assign(reactor)
-        nurl, listening_port = yield listen_tls(
-            http_storage_server,
-            "127.0.0.1",
-            serverFromString(reactor, endpoint_string),
-            private_key_to_file(FilePath(self.mktemp()), private_key),
-            cert_to_file(FilePath(self.mktemp()), certificate),
-        )
-        self.addCleanup(listening_port.stopListening)
 
         # Create HTTP client with non-persistent connections, so we don't leak
         # state across tests:
-        returnValue(
+        client: IStorageServer = _HTTPStorageServer.from_http_client(
-            _HTTPStorageServer.from_http_client(
+            StorageClient.from_nurl(nurl, reactor, persistent=False)
-                StorageClient.from_nurl(nurl, reactor, persistent=False)
-            )
         )
+        self.assertTrue(IStorageServer.providedBy(client))
 
-        # Eventually should also:
+        return succeed(client)
-        #  self.assertTrue(IStorageServer.providedBy(client))
 
 
 class FoolscapSharedAPIsTests(

src/allmydata/test/test_protocol_switch.py

@@ -0,0 +1,43 @@
+"""
+Unit tests for ``allmydata.protocol_switch``.
+
+By its nature, most of the testing needs to be end-to-end; essentially any test
+that uses real Foolscap (``test_system.py``, integration tests) ensures
+Foolscap still works.  ``test_istorageserver.py`` tests the HTTP support.
+"""
+
+from foolscap.negotiate import Negotiation
+
+from .common import TestCase
+from ..protocol_switch import _PretendToBeNegotiation
+
+
+class UtilityTests(TestCase):
+    """Tests for utilities in the protocol switch code."""
+
+    def test_metaclass(self):
+        """
+        A class that has the ``_PretendToBeNegotiation`` metaclass will support
+        ``isinstance()``'s normal semantics on its own instances, but will also
+        indicate that ``Negotiation`` instances are its instances.
+        """
+
+        class Parent(metaclass=_PretendToBeNegotiation):
+            pass
+
+        class Child(Parent):
+            pass
+
+        class Other:
+            pass
+
+        p = Parent()
+        self.assertIsInstance(p, Parent)
+        self.assertIsInstance(Negotiation(), Parent)
+        self.assertNotIsInstance(Other(), Parent)
+
+        c = Child()
+        self.assertIsInstance(c, Child)
+        self.assertIsInstance(c, Parent)
+        self.assertIsInstance(Negotiation(), Child)
+        self.assertNotIsInstance(Other(), Child)

src/allmydata/test/test_storage_http.py

@@ -255,7 +255,7 @@ class TestApp(object):
         else:
             return "BAD: {}".format(authorization)
 
-    @_authorized_route(_app, set(), "/v1/version", methods=["GET"])
+    @_authorized_route(_app, set(), "/storage/v1/version", methods=["GET"])
     def bad_version(self, request, authorization):
         """Return version result that violates the expected schema."""
         request.setHeader("content-type", CBOR_MIME_TYPE)
@@ -534,7 +534,7 @@ class GenericHTTPAPITests(SyncTestCase):
         lease_secret = urandom(32)
         storage_index = urandom(16)
         url = self.http.client.relative_url(
-            "/v1/immutable/" + _encode_si(storage_index)
+            "/storage/v1/immutable/" + _encode_si(storage_index)
         )
         message = {"bad-message": "missing expected keys"}
 
@@ -1418,7 +1418,7 @@ class SharedImmutableMutableTestsMixin:
             self.http.client.request(
                 "GET",
                 self.http.client.relative_url(
-                    "/v1/{}/{}/1".format(self.KIND, _encode_si(storage_index))
+                    "/storage/v1/{}/{}/1".format(self.KIND, _encode_si(storage_index))
                 ),
             )
         )
@@ -1441,7 +1441,7 @@ class SharedImmutableMutableTestsMixin:
                 self.http.client.request(
                     "GET",
                     self.http.client.relative_url(
-                        "/v1/{}/{}/1".format(self.KIND, _encode_si(storage_index))
+                        "/storage/v1/{}/{}/1".format(self.KIND, _encode_si(storage_index))
                     ),
                     headers=headers,
                 )

src/allmydata/test/test_storage_https.py

@@ -198,6 +198,10 @@ class PinningHTTPSValidation(AsyncTestCase):
             response = await self.request(url, certificate)
             self.assertEqual(await response.content(), b"YOYODYNE")
 
+        # We keep getting TLSMemoryBIOProtocol being left around, so try harder
+        # to wait for it to finish.
+        await deferLater(reactor, 0.001)
+
     # A potential attack to test is a private key that doesn't match the
     # certificate... but OpenSSL (quite rightly) won't let you listen with that
     # so I don't know how to test that! See