Merge remote-tracking branch 'origin/master' into 3871-http-list-storage-index-shares

.github/CONTRIBUTING.rst

@@ -18,3 +18,9 @@ Examples of contributions include:
 
 Before authoring or reviewing a patch,
 please familiarize yourself with the `Coding Standards <https://tahoe-lafs.org/trac/tahoe-lafs/wiki/CodingStandards>`_ and the `Contributor Code of Conduct <../docs/CODE_OF_CONDUCT.md>`_.
+
+
+🥳 First Contribution?
+======================
+
+If you are committing to Tahoe for the very first time, consider adding your name to our contributor list in `CREDITS <../CREDITS>`__

.github/workflows/ci.yml

@@ -217,7 +217,7 @@ jobs:
           fetch-depth: 0
 
       - name: Set up Python ${{ matrix.python-version }}
-        uses: actions/setup-python@v1
+        uses: actions/setup-python@v2
         with:
           python-version: ${{ matrix.python-version }}
 
@@ -277,7 +277,7 @@ jobs:
           fetch-depth: 0
 
       - name: Set up Python ${{ matrix.python-version }}
-        uses: actions/setup-python@v1
+        uses: actions/setup-python@v2
         with:
           python-version: ${{ matrix.python-version }}
 

README.rst

@@ -95,7 +95,14 @@ As a community-driven open source project, Tahoe-LAFS welcomes contributions of
 
 -  `Patch reviews <https://tahoe-lafs.org/trac/tahoe-lafs/wiki/PatchReviewProcess>`__
 
-Before authoring or reviewing a patch, please familiarize yourself with the `Coding Standard <https://tahoe-lafs.org/trac/tahoe-lafs/wiki/CodingStandards>`__ and the `Contributor Code of Conduct <docs/CODE_OF_CONDUCT.md>`__.
+Before authoring or reviewing a patch, please familiarize yourself with the `Coding Standard <https://tahoe-lafs.org/trac/tahoe-lafs/wiki/CodingStandards>`__ and the `Contributor Code of Conduct <docs/CODE_OF_CONDUCT.md>`__. 
+
+
+🥳 First Contribution?
+----------------------
+
+If you are committing to Tahoe for the very first time, it's required that you add your name to our contributor list in `CREDITS <CREDITS>`__. Please ensure that this addition has it's own commit within your first contribution.
+
 
 🤝 Supporters
 --------------

docs/gpg-setup.rst

@@ -0,0 +1,18 @@
+Preparing to Authenticate Release (Setting up GPG)
+--------------------------------------------------
+
+In other to keep releases authentic it's required that releases are signed before being
+published. This ensure's that users of Tahoe are able to verify that the version of Tahoe
+they are using is coming from a trusted or at the very least known source.
+
+The authentication is done using the ``GPG`` implementation of ``OpenGPG`` to be able to complete 
+the release steps you would have to download the ``GPG`` software and setup a key(identity).
+
+- `Download <https://www.gnupg.org/download/>`__ and install GPG for your operating system.
+- Generate a key pair using ``gpg --gen-key``. *Some questions would be asked to personalize your key configuration.*
+
+You might take additional steps including:
+
+- Setting up a revocation certificate (Incase you lose your secret key)
+- Backing up your key pair
+- Upload your fingerprint to a keyserver such as `openpgp.org <https://keys.openpgp.org/>`__

docs/index.rst

@@ -29,6 +29,7 @@ Contents:
    contributing
    CODE_OF_CONDUCT
    release-checklist
+   gpg-setup
 
    servers
    helper

docs/release-checklist.rst

@@ -3,9 +3,8 @@
 Release Checklist
 =================
 
-These instructions were produced while making the 1.15.0 release. They
-are based on the original instructions (in old revisions in the file
-`docs/how_to_make_a_tahoe-lafs_release.org`).
+This release checklist specifies a series of checks that anyone engaged in 
+releasing a version of Tahoe should follow.
 
 Any contributor can do the first part of the release preparation. Only
 certain contributors can perform other parts. These are the two main
@@ -13,9 +12,12 @@ sections of this checklist (and could be done by different people).
 
 A final section describes how to announce the release.
 
+This checklist is based on the original instructions (in old revisions in the file
+`docs/how_to_make_a_tahoe-lafs_release.org`).
+
 
 Any Contributor
----------------
+===============
 
 Anyone who can create normal PRs should be able to complete this
 portion of the release process.
@@ -32,13 +34,35 @@ Tuesday if you want to get anything in").
 
 - Create a ticket for the release in Trac
 - Ticket number needed in next section
+- Making first release? See `GPG Setup Instructions <gpg-setup.rst>`__ to make sure you can sign releases. [One time setup]
+
+Get a clean checkout
+````````````````````
+
+The release proccess involves compressing source files and putting them in formats 
+suitable for distribution such as ``.tar.gz`` and ``zip``. That said, it's neccesary to 
+the release process begins with a clean checkout to avoid making a release with
+previously generated files.
+
+- Inside the tahoe root dir run ``git clone . ../tahoe-release-x.x.x`` where (x.x.x is the release number such as 1.16.0). 
+
+.. note:: 
+     The above command would create a new directory at the same level as your original clone named ``tahoe-release-x.x.x``. You can name this folder however you want but it would be a good 
+     practice to give it the release name. You MAY also discard this directory once the release
+     process is complete.
+
+Get into the release directory and install dependencies by running 
+
+- cd ../tahoe-release-x.x.x (assuming you are still in your original clone)
+- python -m venv venv
+- ./venv/bin/pip install --editable .[test]
 
 
 Create Branch and Apply Updates
 ```````````````````````````````
 
-- Create a branch for release-candidates (e.g. `XXXX.release-1.15.0.rc0`)
-- run `tox -e news` to produce a new NEWS.txt file (this does a commit)
+- Create a branch for the release/candidate (e.g. ``XXXX.release-1.16.0``)
+- run tox -e news to produce a new NEWS.txt file (this does a commit)
 - create the news for the release
 
   - newsfragments/<ticket number>.minor
@@ -46,7 +70,7 @@ Create Branch and Apply Updates
 
 - manually fix NEWS.txt
 
-  - proper title for latest release ("Release 1.15.0" instead of "Release ...post1432")
+  - proper title for latest release ("Release 1.16.0" instead of "Release ...post1432")
   - double-check date (maybe release will be in the future)
   - spot-check the release notes (these come from the newsfragments
     files though so don't do heavy editing)
@@ -54,7 +78,7 @@ Create Branch and Apply Updates
 
 - update "relnotes.txt"
 
-  - update all mentions of 1.14.0 -> 1.15.0
+  - update all mentions of ``1.16.0`` to new and higher release version for example ``1.16.1``
   - update "previous release" statement and date
   - summarize major changes
   - commit it
@@ -63,12 +87,6 @@ Create Branch and Apply Updates
 
   - change the value given for `version` from `OLD.post1` to `NEW.post1`
 
-- update "CREDITS"
-
-  - are there any new contributors in this release?
-  - one way: git log release-1.14.0.. | grep Author | sort | uniq
-  - commit it
-
 - update "docs/known_issues.rst" if appropriate
 - Push the branch to github
 - Create a (draft) PR; this should trigger CI (note that github
@@ -94,10 +112,14 @@ they will need to evaluate which contributors' signatures they trust.
 - (all steps above are completed)
 - sign the release
 
-  - git tag -s -u 0xE34E62D06D0E69CFCA4179FFBDE0D31D68666A7A -m "release Tahoe-LAFS-1.15.0rc0" tahoe-lafs-1.15.0rc0
-  - (replace the key-id above with your own)
+  - git tag -s -u 0xE34E62D06D0E69CFCA4179FFBDE0D31D68666A7A -m "release Tahoe-LAFS-1.16.0rc0" tahoe-lafs-1.16.0rc0
+
+.. note:: 
+    - Replace the key-id above with your own, which can simply be your email if it's attached to your fingerprint.
+    - Don't forget to put the correct tag message and name. In this example, the tag message is "release Tahoe-LAFS-1.16.0rc0" and the tag name is ``tahoe-lafs-1.16.0rc0`` 
 
 - build all code locally
+
   - these should all pass:
 
     - tox -e py27,codechecks,docs,integration
@@ -114,8 +136,9 @@ they will need to evaluate which contributors' signatures they trust.
 - build tarballs
 
   - tox -e tarballs
-  - confirm it at least exists:
-  - ls dist/ | grep 1.15.0rc0
+  - Confirm that release tarballs exist by runnig: 
+
+    - ls dist/ | grep 1.16.0rc0
 
 - inspect and test the tarballs
 
@@ -124,12 +147,12 @@ they will need to evaluate which contributors' signatures they trust.
 
 - when satisfied, sign the tarballs:
 
-  - gpg --pinentry=loopback --armor --detach-sign dist/tahoe_lafs-1.15.0rc0-py2.py3-none-any.whl
-  - gpg --pinentry=loopback --armor --detach-sign dist/tahoe_lafs-1.15.0rc0.tar.gz
+  - gpg --pinentry=loopback --armor --detach-sign dist/tahoe_lafs-1.16.0rc0-py2.py3-none-any.whl
+  - gpg --pinentry=loopback --armor --detach-sign dist/tahoe_lafs-1.16.0rc0.tar.gz
 
 
 Privileged Contributor
------------------------
+======================
 
 Steps in this portion require special access to keys or
 infrastructure. For example, **access to tahoe-lafs.org** to upload

src/allmydata/test/test_istorageserver.py

@@ -25,23 +25,23 @@ else:
 from random import Random
 from unittest import SkipTest
 
-from twisted.internet.defer import inlineCallbacks, returnValue
+from twisted.internet.defer import inlineCallbacks, returnValue, succeed
 from twisted.internet.task import Clock
 from twisted.internet import reactor
+from twisted.internet.endpoints import serverFromString
 from twisted.web.server import Site
-from twisted.web.client import HTTPConnectionPool
+from twisted.web.client import Agent, HTTPConnectionPool
 from hyperlink import DecodedURL
-from treq.api import set_global_pool as set_treq_pool
+from treq.client import HTTPClient
 
 from foolscap.api import Referenceable, RemoteException
 
 from allmydata.interfaces import IStorageServer  # really, IStorageClient
 from .common_system import SystemTestMixin
-from .common import AsyncTestCase
+from .common import AsyncTestCase, SameProcessStreamEndpointAssigner
 from allmydata.storage.server import StorageServer  # not a IStorageServer!!
 from allmydata.storage.http_server import HTTPServer
 from allmydata.storage.http_client import StorageClient
-from allmydata.util.iputil import allocate_tcp_port
 from allmydata.storage_client import _HTTPStorageServer
 
 
@@ -1029,6 +1029,11 @@ class _SharedMixin(SystemTestMixin):
             )
 
         AsyncTestCase.setUp(self)
+
+        self._port_assigner = SameProcessStreamEndpointAssigner()
+        self._port_assigner.setUp()
+        self.addCleanup(self._port_assigner.tearDown)
+
         self.basedir = "test_istorageserver/" + self.id()
         yield SystemTestMixin.setUp(self)
         yield self.set_up_nodes(1)
@@ -1041,7 +1046,7 @@ class _SharedMixin(SystemTestMixin):
         self._clock = Clock()
         self._clock.advance(123456)
         self.server._clock = self._clock
-        self.storage_client = self._get_istorage_server()
+        self.storage_client = yield self._get_istorage_server()
 
     def fake_time(self):
         """Return the current fake, test-controlled, time."""
@@ -1073,7 +1078,7 @@ class _FoolscapMixin(_SharedMixin):
     def _get_istorage_server(self):
         client = self._get_native_server().get_storage_server()
         self.assertTrue(IStorageServer.providedBy(client))
-        return client
+        return succeed(client)
 
     @inlineCallbacks
     def disconnect(self):
@@ -1094,30 +1099,37 @@ class _HTTPMixin(_SharedMixin):
             self.skipTest("Not going to bother supporting Python 2")
         return _SharedMixin.setUp(self)
 
+    @inlineCallbacks
     def _get_istorage_server(self):
-        set_treq_pool(HTTPConnectionPool(reactor, persistent=False))
         swissnum = b"1234"
-        self._http_storage_server = HTTPServer(self.server, swissnum)
-        self._port_number = allocate_tcp_port()
-        self._listening_port = reactor.listenTCP(
-            self._port_number,
-            Site(self._http_storage_server.get_resource()),
-            interface="127.0.0.1",
+        http_storage_server = HTTPServer(self.server, swissnum)
+
+        # Listen on randomly assigned port:
+        tcp_address, endpoint_string = self._port_assigner.assign(reactor)
+        _, host, port = tcp_address.split(":")
+        port = int(port)
+        endpoint = serverFromString(reactor, endpoint_string)
+        listening_port = yield endpoint.listen(Site(http_storage_server.get_resource()))
+        self.addCleanup(listening_port.stopListening)
+
+        # Create HTTP client with non-persistent connections, so we don't leak
+        # state across tests:
+        treq_client = HTTPClient(
+            Agent(reactor, HTTPConnectionPool(reactor, persistent=False))
         )
-        return _HTTPStorageServer.from_http_client(
-            StorageClient(
-                DecodedURL.from_text("http://127.0.0.1:{}".format(self._port_number)),
-                swissnum,
+
+        returnValue(
+            _HTTPStorageServer.from_http_client(
+                StorageClient(
+                    DecodedURL().replace(scheme="http", host=host, port=port),
+                    swissnum,
+                    treq=treq_client,
+                )
             )
         )
         # Eventually should also:
         #  self.assertTrue(IStorageServer.providedBy(client))
 
-    @inlineCallbacks
-    def tearDown(self):
-        yield _SharedMixin.tearDown(self)
-        yield self._listening_port.stopListening()
-
 
 class FoolscapSharedAPIsTests(
     _FoolscapMixin, IStorageServerSharedAPIsTestsMixin, AsyncTestCase