From 914e7ac013ac97adda9b8798200065f3328a3b76 Mon Sep 17 00:00:00 2001 From: Jean-Paul Calderone Date: Tue, 8 Dec 2020 15:38:46 -0500 Subject: [PATCH 01/54] news fragment --- newsfragments/3523.minor | 0 1 file changed, 0 insertions(+), 0 deletions(-) create mode 100644 newsfragments/3523.minor diff --git a/newsfragments/3523.minor b/newsfragments/3523.minor new file mode 100644 index 000000000..e69de29bb From 20dd4d55ad0052649ffb91fa7991ecd7c53590e1 Mon Sep 17 00:00:00 2001 From: Jean-Paul Calderone Date: Tue, 8 Dec 2020 15:44:12 -0500 Subject: [PATCH 02/54] Use "tahoe run" --- docs/anonymity-configuration.rst | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/docs/anonymity-configuration.rst b/docs/anonymity-configuration.rst index 5ad9ae740..d25f8ad41 100644 --- a/docs/anonymity-configuration.rst +++ b/docs/anonymity-configuration.rst @@ -273,7 +273,7 @@ Then, do the following: [connections] tcp = tor -* Launch the Tahoe server with ``tahoe start $NODEDIR`` +* Launch the Tahoe server with ``tahoe run $NODEDIR`` The ``tub.port`` section will cause the Tahoe server to listen on PORT, but bind the listening socket to the loopback interface, which is not reachable @@ -435,4 +435,3 @@ It is therefore important that your I2P router is sharing bandwidth with other routers, so that you can give back as you use I2P. This will never impair the performance of your Tahoe-LAFS node, because your I2P router will always prioritize your own traffic. - From 9fe3dddac4c76c9628bdb702db48717b8556e5f3 Mon Sep 17 00:00:00 2001 From: Jean-Paul Calderone Date: Tue, 8 Dec 2020 15:44:19 -0500 Subject: [PATCH 03/54] use "tahoe run" --- docs/frontends/CLI.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/frontends/CLI.rst b/docs/frontends/CLI.rst index e46936bad..007032e5d 100644 --- a/docs/frontends/CLI.rst +++ b/docs/frontends/CLI.rst @@ -85,7 +85,7 @@ Node Management "``tahoe create-node [NODEDIR]``" is the basic make-a-new-node command. It creates a new directory and populates it with files that -will allow the "``tahoe start``" and related commands to use it later +will allow the "``tahoe run``" and related commands to use it later on. ``tahoe create-node`` creates nodes that have client functionality (upload/download files), web API services (controlled by the '[node]web.port' configuration), and storage services (unless From 35a21b6b3667a2f23f49a4da8c1d6ec8f33f9243 Mon Sep 17 00:00:00 2001 From: Jean-Paul Calderone Date: Tue, 8 Dec 2020 15:44:28 -0500 Subject: [PATCH 04/54] don't use "tahoe start" and "tahoe stop" --- docs/running.rst | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/docs/running.rst b/docs/running.rst index ef6ba42ed..49fccdf9c 100644 --- a/docs/running.rst +++ b/docs/running.rst @@ -81,9 +81,7 @@ does not offer its disk space to other nodes. To configure other behavior, use “``tahoe create-node``” or see :doc:`configuration`. The “``tahoe run``” command above will run the node in the foreground. -On Unix, you can run it in the background instead by using the -“``tahoe start``” command. To stop a node started in this way, use -“``tahoe stop``”. ``tahoe --help`` gives a summary of all commands. +``tahoe --help`` gives a summary of all commands. Running a Server or Introducer From 44cb2d048fe6cc7e810ee53c31bb1dd935659211 Mon Sep 17 00:00:00 2001 From: Jean-Paul Calderone Date: Tue, 8 Dec 2020 15:44:36 -0500 Subject: [PATCH 05/54] use "tahoe run" --- docs/running.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/running.rst b/docs/running.rst index 49fccdf9c..0ed8983f7 100644 --- a/docs/running.rst +++ b/docs/running.rst @@ -97,7 +97,7 @@ and ``--location`` arguments. To construct an introducer, create a new base directory for it (the name of the directory is up to you), ``cd`` into it, and run “``tahoe create-introducer --hostname=example.net .``” (but using the hostname of -your VPS). Now run the introducer using “``tahoe start .``”. After it +your VPS). Now run the introducer using “``tahoe run .``”. After it starts, it will write a file named ``introducer.furl`` into the ``private/`` subdirectory of that base directory. This file contains the URL the other nodes must use in order to connect to this introducer. From 4e6077fae1c80d2910997c2d98319627a35998d0 Mon Sep 17 00:00:00 2001 From: Jean-Paul Calderone Date: Tue, 8 Dec 2020 15:45:03 -0500 Subject: [PATCH 06/54] Nope, actually it doesn't. It uses ~/.tahoe. --- docs/frontends/CLI.rst | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/docs/frontends/CLI.rst b/docs/frontends/CLI.rst index 007032e5d..9a001c479 100644 --- a/docs/frontends/CLI.rst +++ b/docs/frontends/CLI.rst @@ -94,8 +94,7 @@ on. ``tahoe create-node`` creates nodes that have client functionality NODEDIR defaults to ``~/.tahoe/`` , and newly-created nodes default to publishing a web server on port 3456 (limited to the loopback interface, at 127.0.0.1, to restrict access to other programs on the same host). All of the -other "``tahoe``" subcommands use corresponding defaults (with the exception -that "``tahoe run``" defaults to running a node in the current directory). +other "``tahoe``" subcommands use corresponding defaults. "``tahoe create-client [NODEDIR]``" creates a node with no storage service. That is, it behaves like "``tahoe create-node --no-storage [NODEDIR]``". From d9ea26b0e55a733594300a5905948cdf6076992a Mon Sep 17 00:00:00 2001 From: Jean-Paul Calderone Date: Tue, 8 Dec 2020 15:45:32 -0500 Subject: [PATCH 07/54] It works now, that issue is resolved. --- docs/running.rst | 2 -- 1 file changed, 2 deletions(-) diff --git a/docs/running.rst b/docs/running.rst index 0ed8983f7..6d82a97f2 100644 --- a/docs/running.rst +++ b/docs/running.rst @@ -101,8 +101,6 @@ your VPS). Now run the introducer using “``tahoe run .``”. After it starts, it will write a file named ``introducer.furl`` into the ``private/`` subdirectory of that base directory. This file contains the URL the other nodes must use in order to connect to this introducer. -(Note that “``tahoe run .``” doesn't work for introducers, this is a -known issue: `#937`_.) You can distribute your Introducer fURL securely to new clients by using the ``tahoe invite`` command. This will prepare some JSON to send to the From aca5397d459f1ad06c9e4da466ab2e4ee4d74c2d Mon Sep 17 00:00:00 2001 From: Jean-Paul Calderone Date: Tue, 8 Dec 2020 15:58:30 -0500 Subject: [PATCH 08/54] Don't use "tahoe start", "tahoe stop", and "tahoe restart" --- docs/frontends/CLI.rst | 19 ------------------- 1 file changed, 19 deletions(-) diff --git a/docs/frontends/CLI.rst b/docs/frontends/CLI.rst index 9a001c479..0badede98 100644 --- a/docs/frontends/CLI.rst +++ b/docs/frontends/CLI.rst @@ -116,25 +116,6 @@ the same way on all platforms and logs to stdout. If you want to run the process as a daemon, it is recommended that you use your favourite daemonization tool. -The now-deprecated "``tahoe start [NODEDIR]``" command will launch a -previously-created node. It will launch the node into the background -using ``tahoe daemonize`` (and internal-only command, not for user -use). On some platforms (including Windows) this command is unable to -run a daemon in the background; in that case it behaves in the same -way as "``tahoe run``". ``tahoe start`` also monitors the logs for up -to 5 seconds looking for either a succesful startup message or for -early failure messages and produces an appropriate exit code. You are -encouraged to use ``tahoe run`` along with your favourite -daemonization tool instead of this. ``tahoe start`` is maintained for -backwards compatibility of users already using it; new scripts should -depend on ``tahoe run``. - -"``tahoe stop [NODEDIR]``" will shut down a running node. "``tahoe -restart [NODEDIR]``" will stop and then restart a running -node. Similar to above, you should use ``tahoe run`` instead alongside -your favourite daemonization tool. - - File Store Manipulation ======================= From db9eb6d80792d0c06ba9d9aec843253091138881 Mon Sep 17 00:00:00 2001 From: Jean-Paul Calderone Date: Tue, 8 Dec 2020 15:58:42 -0500 Subject: [PATCH 09/54] maybe "tahoe run" works on incident gatherers? I don't know. I can't actually create one. This feature seems broken in flogtool. So it probably doesn't matter. --- docs/logging.rst | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/docs/logging.rst b/docs/logging.rst index 88cdebc00..893981096 100644 --- a/docs/logging.rst +++ b/docs/logging.rst @@ -128,10 +128,9 @@ provided in ``misc/incident-gatherer/support_classifiers.py`` . There is roughly one category for each ``log.WEIRD``-or-higher level event in the Tahoe source code. -The incident gatherer is created with the "``flogtool -create-incident-gatherer WORKDIR``" command, and started with "``tahoe -start``". The generated "``gatherer.tac``" file should be modified to add -classifier functions. +The incident gatherer is created with the "``flogtool create-incident-gatherer +WORKDIR``" command, and started with "``tahoe run``". The generated +"``gatherer.tac``" file should be modified to add classifier functions. The incident gatherer writes incident names (which are simply the relative pathname of the ``incident-\*.flog.bz2`` file) into ``classified/CATEGORY``. From 9a10673e8a6f6b72eab1c947e567989a06109e3f Mon Sep 17 00:00:00 2001 From: Jean-Paul Calderone Date: Tue, 8 Dec 2020 15:59:14 -0500 Subject: [PATCH 10/54] "tahoe start" doesn't even work on these but "twistd -ny" does --- docs/logging.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/logging.rst b/docs/logging.rst index 893981096..11835b283 100644 --- a/docs/logging.rst +++ b/docs/logging.rst @@ -174,7 +174,7 @@ things that happened on multiple machines (such as comparing a client node making a request with the storage servers that respond to that request). Create the Log Gatherer with the "``flogtool create-gatherer WORKDIR``" -command, and start it with "``tahoe start``". Then copy the contents of the +command, and start it with "``twistd -ny gatherer.tac``". Then copy the contents of the ``log_gatherer.furl`` file it creates into the ``BASEDIR/tahoe.cfg`` file (under the key ``log_gatherer.furl`` of the section ``[node]``) of all nodes that should be sending it log events. (See :doc:`configuration`) From ae35ba84de3cdfe42f6c3ee891f5eccbd1b85215 Mon Sep 17 00:00:00 2001 From: Jean-Paul Calderone Date: Tue, 8 Dec 2020 16:02:15 -0500 Subject: [PATCH 11/54] Use "tahoe run" not "tahoe start" --- docs/configuration.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/configuration.rst b/docs/configuration.rst index 5bd89eec9..540107cbb 100644 --- a/docs/configuration.rst +++ b/docs/configuration.rst @@ -365,7 +365,7 @@ set the ``tub.location`` option described below. also generally reduced when operating in private mode. When False, any of the following configuration problems will cause - ``tahoe start`` to throw a PrivacyError instead of starting the node: + ``tahoe run`` to throw a PrivacyError instead of starting the node: * ``[node] tub.location`` contains any ``tcp:`` hints From 7dda680cb28ae0f12e394b637290f93740843613 Mon Sep 17 00:00:00 2001 From: Jean-Paul Calderone Date: Tue, 8 Dec 2020 16:02:26 -0500 Subject: [PATCH 12/54] Already using "tahoe run", de-emphasize "tahoe start" --- integration/conftest.py | 10 ++++------ integration/util.py | 6 ++---- 2 files changed, 6 insertions(+), 10 deletions(-) diff --git a/integration/conftest.py b/integration/conftest.py index ca18230cd..f37ec9353 100644 --- a/integration/conftest.py +++ b/integration/conftest.py @@ -201,9 +201,8 @@ log_gatherer.furl = {log_furl} with open(join(intro_dir, 'tahoe.cfg'), 'w') as f: f.write(config) - # on windows, "tahoe start" means: run forever in the foreground, - # but on linux it means daemonize. "tahoe run" is consistent - # between platforms. + # "tahoe run" is consistent across Linux/macOS/Windows, unlike the old + # "start" command. protocol = _MagicTextProtocol('introducer running') transport = _tahoe_runner_optional_coverage( protocol, @@ -278,9 +277,8 @@ log_gatherer.furl = {log_furl} with open(join(intro_dir, 'tahoe.cfg'), 'w') as f: f.write(config) - # on windows, "tahoe start" means: run forever in the foreground, - # but on linux it means daemonize. "tahoe run" is consistent - # between platforms. + # "tahoe run" is consistent across Linux/macOS/Windows, unlike the old + # "start" command. protocol = _MagicTextProtocol('introducer running') transport = _tahoe_runner_optional_coverage( protocol, diff --git a/integration/util.py b/integration/util.py index f916240ff..eed073225 100644 --- a/integration/util.py +++ b/integration/util.py @@ -189,10 +189,8 @@ def _run_node(reactor, node_dir, request, magic_text): magic_text = "client running" protocol = _MagicTextProtocol(magic_text) - # on windows, "tahoe start" means: run forever in the foreground, - # but on linux it means daemonize. "tahoe run" is consistent - # between platforms. - + # "tahoe run" is consistent across Linux/macOS/Windows, unlike the old + # "start" command. transport = _tahoe_runner_optional_coverage( protocol, reactor, From a34fca8e7af67fd137afd71efd6dd5ef889a7e0b Mon Sep 17 00:00:00 2001 From: Jean-Paul Calderone Date: Tue, 8 Dec 2020 18:21:26 -0500 Subject: [PATCH 13/54] Don't think about "tahoe start" --- src/allmydata/scripts/common.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/allmydata/scripts/common.py b/src/allmydata/scripts/common.py index 29bb1d5f1..106dad3f2 100644 --- a/src/allmydata/scripts/common.py +++ b/src/allmydata/scripts/common.py @@ -37,7 +37,7 @@ class BaseOptions(usage.Options): super(BaseOptions, self).__init__() self.command_name = os.path.basename(sys.argv[0]) - # Only allow "tahoe --version", not e.g. "tahoe start --version" + # Only allow "tahoe --version", not e.g. "tahoe --version" def opt_version(self): raise usage.UsageError("--version not allowed on subcommands") From 4d28b0ec27ac8631f5d4f2ff33a8dad0d29596ff Mon Sep 17 00:00:00 2001 From: Jean-Paul Calderone Date: Tue, 8 Dec 2020 18:22:01 -0500 Subject: [PATCH 14/54] Get rid of "tahoe start", "tahoe daemonize", "tahoe stop", "tahoe restart" --- src/allmydata/scripts/runner.py | 11 +- src/allmydata/scripts/tahoe_daemonize.py | 16 -- src/allmydata/scripts/tahoe_restart.py | 21 -- src/allmydata/scripts/tahoe_start.py | 152 ------------- src/allmydata/scripts/tahoe_stop.py | 85 ------- src/allmydata/test/cli/test_daemonize.py | 202 ----------------- src/allmydata/test/cli/test_start.py | 273 ----------------------- 7 files changed, 1 insertion(+), 759 deletions(-) delete mode 100644 src/allmydata/scripts/tahoe_daemonize.py delete mode 100644 src/allmydata/scripts/tahoe_restart.py delete mode 100644 src/allmydata/scripts/tahoe_start.py delete mode 100644 src/allmydata/scripts/tahoe_stop.py delete mode 100644 src/allmydata/test/cli/test_daemonize.py delete mode 100644 src/allmydata/test/cli/test_start.py diff --git a/src/allmydata/scripts/runner.py b/src/allmydata/scripts/runner.py index 273a05af1..1f993fda1 100644 --- a/src/allmydata/scripts/runner.py +++ b/src/allmydata/scripts/runner.py @@ -9,8 +9,7 @@ from twisted.internet import defer, task, threads from allmydata.scripts.common import get_default_nodedir from allmydata.scripts import debug, create_node, cli, \ - admin, tahoe_daemonize, tahoe_start, \ - tahoe_stop, tahoe_restart, tahoe_run, tahoe_invite + admin, tahoe_run, tahoe_invite from allmydata.util.encodingutil import quote_output, quote_local_unicode_path, get_io_encoding from allmydata.util.eliotutil import ( opt_eliot_destination, @@ -37,19 +36,11 @@ if _default_nodedir: # XXX all this 'dispatch' stuff needs to be unified + fixed up _control_node_dispatch = { - "daemonize": tahoe_daemonize.daemonize, - "start": tahoe_start.start, "run": tahoe_run.run, - "stop": tahoe_stop.stop, - "restart": tahoe_restart.restart, } process_control_commands = [ ["run", None, tahoe_run.RunOptions, "run a node without daemonizing"], - ["daemonize", None, tahoe_daemonize.DaemonizeOptions, "(deprecated) run a node in the background"], - ["start", None, tahoe_start.StartOptions, "(deprecated) start a node in the background and confirm it started"], - ["stop", None, tahoe_stop.StopOptions, "(deprecated) stop a node"], - ["restart", None, tahoe_restart.RestartOptions, "(deprecated) restart a node"], ] diff --git a/src/allmydata/scripts/tahoe_daemonize.py b/src/allmydata/scripts/tahoe_daemonize.py deleted file mode 100644 index ad2f92355..000000000 --- a/src/allmydata/scripts/tahoe_daemonize.py +++ /dev/null @@ -1,16 +0,0 @@ -from .run_common import ( - RunOptions as _RunOptions, - run, -) - -__all__ = [ - "DaemonizeOptions", - "daemonize", -] - -class DaemonizeOptions(_RunOptions): - subcommand_name = "daemonize" - -def daemonize(config): - print("'tahoe daemonize' is deprecated; see 'tahoe run'") - return run(config) diff --git a/src/allmydata/scripts/tahoe_restart.py b/src/allmydata/scripts/tahoe_restart.py deleted file mode 100644 index 339db862f..000000000 --- a/src/allmydata/scripts/tahoe_restart.py +++ /dev/null @@ -1,21 +0,0 @@ -from __future__ import print_function - -from .tahoe_start import StartOptions, start -from .tahoe_stop import stop, COULD_NOT_STOP - - -class RestartOptions(StartOptions): - subcommand_name = "restart" - - -def restart(config): - print("'tahoe restart' is deprecated; see 'tahoe run'") - stderr = config.stderr - rc = stop(config) - if rc == COULD_NOT_STOP: - print("ignoring couldn't-stop", file=stderr) - rc = 0 - if rc: - print("not restarting", file=stderr) - return rc - return start(config) diff --git a/src/allmydata/scripts/tahoe_start.py b/src/allmydata/scripts/tahoe_start.py deleted file mode 100644 index bc076d1b7..000000000 --- a/src/allmydata/scripts/tahoe_start.py +++ /dev/null @@ -1,152 +0,0 @@ -from __future__ import print_function - -import os -import io -import sys -import time -import subprocess -from os.path import join, exists - -from allmydata.scripts.common import BasedirOptions -from allmydata.scripts.default_nodedir import _default_nodedir -from allmydata.util.encodingutil import quote_local_unicode_path - -from .run_common import MyTwistdConfig, identify_node_type - - -class StartOptions(BasedirOptions): - subcommand_name = "start" - optParameters = [ - ("basedir", "C", None, - "Specify which Tahoe base directory should be used." - " This has the same effect as the global --node-directory option." - " [default: %s]" % quote_local_unicode_path(_default_nodedir)), - ] - - def parseArgs(self, basedir=None, *twistd_args): - # This can't handle e.g. 'tahoe start --nodaemon', since '--nodaemon' - # looks like an option to the tahoe subcommand, not to twistd. So you - # can either use 'tahoe start' or 'tahoe start NODEDIR - # --TWISTD-OPTIONS'. Note that 'tahoe --node-directory=NODEDIR start - # --TWISTD-OPTIONS' also isn't allowed, unfortunately. - - BasedirOptions.parseArgs(self, basedir) - self.twistd_args = twistd_args - - def getSynopsis(self): - return ("Usage: %s [global-options] %s [options]" - " [NODEDIR [twistd-options]]" - % (self.command_name, self.subcommand_name)) - - def getUsage(self, width=None): - t = BasedirOptions.getUsage(self, width) + "\n" - twistd_options = str(MyTwistdConfig()).partition("\n")[2].partition("\n\n")[0] - t += twistd_options.replace("Options:", "twistd-options:", 1) - t += """ - -Note that if any twistd-options are used, NODEDIR must be specified explicitly -(not by default or using -C/--basedir or -d/--node-directory), and followed by -the twistd-options. -""" - return t - - -def start(config): - """ - Start a tahoe node (daemonize it and confirm startup) - - We run 'tahoe daemonize' with all the options given to 'tahoe - start' and then watch the log files for the correct text to appear - (e.g. "introducer started"). If that doesn't happen within a few - seconds, an error is printed along with all collected logs. - """ - print("'tahoe start' is deprecated; see 'tahoe run'") - out = config.stdout - err = config.stderr - basedir = config['basedir'] - quoted_basedir = quote_local_unicode_path(basedir) - print("STARTING", quoted_basedir, file=out) - if not os.path.isdir(basedir): - print("%s does not look like a directory at all" % quoted_basedir, file=err) - return 1 - nodetype = identify_node_type(basedir) - if not nodetype: - print("%s is not a recognizable node directory" % quoted_basedir, file=err) - return 1 - - # "tahoe start" attempts to monitor the logs for successful - # startup -- but we can't always do that. - - can_monitor_logs = False - if (nodetype in (u"client", u"introducer") - and "--nodaemon" not in config.twistd_args - and "--syslog" not in config.twistd_args - and "--logfile" not in config.twistd_args): - can_monitor_logs = True - - if "--help" in config.twistd_args: - return 0 - - if not can_monitor_logs: - print("Custom logging options; can't monitor logs for proper startup messages", file=out) - return 1 - - # before we spawn tahoe, we check if "the log file" exists or not, - # and if so remember how big it is -- essentially, we're doing - # "tail -f" to see what "this" incarnation of "tahoe daemonize" - # spews forth. - starting_offset = 0 - log_fname = join(basedir, 'logs', 'twistd.log') - if exists(log_fname): - with open(log_fname, 'r') as f: - f.seek(0, 2) - starting_offset = f.tell() - - # spawn tahoe. Note that since this daemonizes, it should return - # "pretty fast" and with a zero return-code, or else something - # Very Bad has happened. - try: - args = [sys.executable] if not getattr(sys, 'frozen', False) else [] - for i, arg in enumerate(sys.argv): - if arg in ['start', 'restart']: - args.append('daemonize') - else: - args.append(arg) - subprocess.check_call(args) - except subprocess.CalledProcessError as e: - return e.returncode - - # now, we have to determine if tahoe has actually started up - # successfully or not. so, we start sucking up log files and - # looking for "the magic string", which depends on the node type. - - magic_string = u'{} running'.format(nodetype) - with io.open(log_fname, 'r') as f: - f.seek(starting_offset) - - collected = u'' - overall_start = time.time() - while time.time() - overall_start < 60: - this_start = time.time() - while time.time() - this_start < 5: - collected += f.read() - if magic_string in collected: - if not config.parent['quiet']: - print("Node has started successfully", file=out) - return 0 - if 'Traceback ' in collected: - print("Error starting node; see '{}' for more:\n\n{}".format( - log_fname, - collected, - ), file=err) - return 1 - time.sleep(0.1) - print("Still waiting up to {}s for node startup".format( - 60 - int(time.time() - overall_start) - ), file=out) - - print("Something has gone wrong starting the node.", file=out) - print("Logs are available in '{}'".format(log_fname), file=out) - print("Collected for this run:", file=out) - print(collected, file=out) - return 1 diff --git a/src/allmydata/scripts/tahoe_stop.py b/src/allmydata/scripts/tahoe_stop.py deleted file mode 100644 index 28c0f8131..000000000 --- a/src/allmydata/scripts/tahoe_stop.py +++ /dev/null @@ -1,85 +0,0 @@ -from __future__ import print_function - -import os -import time -import signal - -from allmydata.scripts.common import BasedirOptions -from allmydata.util.encodingutil import quote_local_unicode_path -from .run_common import get_pidfile, get_pid_from_pidfile - -COULD_NOT_STOP = 2 - - -class StopOptions(BasedirOptions): - def parseArgs(self, basedir=None): - BasedirOptions.parseArgs(self, basedir) - - def getSynopsis(self): - return ("Usage: %s [global-options] stop [options] [NODEDIR]" - % (self.command_name,)) - - -def stop(config): - print("'tahoe stop' is deprecated; see 'tahoe run'") - out = config.stdout - err = config.stderr - basedir = config['basedir'] - quoted_basedir = quote_local_unicode_path(basedir) - print("STOPPING", quoted_basedir, file=out) - pidfile = get_pidfile(basedir) - pid = get_pid_from_pidfile(pidfile) - if pid is None: - print("%s does not look like a running node directory (no twistd.pid)" % quoted_basedir, file=err) - # we define rc=2 to mean "nothing is running, but it wasn't me who - # stopped it" - return COULD_NOT_STOP - elif pid == -1: - print("%s contains an invalid PID file" % basedir, file=err) - # we define rc=2 to mean "nothing is running, but it wasn't me who - # stopped it" - return COULD_NOT_STOP - - # kill it hard (SIGKILL), delete the twistd.pid file, then wait for the - # process itself to go away. If it hasn't gone away after 20 seconds, warn - # the user but keep waiting until they give up. - try: - os.kill(pid, signal.SIGKILL) - except OSError as oserr: - if oserr.errno == 3: - print(oserr.strerror) - # the process didn't exist, so wipe the pid file - os.remove(pidfile) - return COULD_NOT_STOP - else: - raise - try: - os.remove(pidfile) - except EnvironmentError: - pass - start = time.time() - time.sleep(0.1) - wait = 40 - first_time = True - while True: - # poll once per second until we see the process is no longer running - try: - os.kill(pid, 0) - except OSError: - print("process %d is dead" % pid, file=out) - return - wait -= 1 - if wait < 0: - if first_time: - print("It looks like pid %d is still running " - "after %d seconds" % (pid, - (time.time() - start)), file=err) - print("I will keep watching it until you interrupt me.", file=err) - wait = 10 - first_time = False - else: - print("pid %d still running after %d seconds" % \ - (pid, (time.time() - start)), file=err) - wait = 10 - time.sleep(1) - # control never reaches here: no timeout diff --git a/src/allmydata/test/cli/test_daemonize.py b/src/allmydata/test/cli/test_daemonize.py deleted file mode 100644 index b1365329a..000000000 --- a/src/allmydata/test/cli/test_daemonize.py +++ /dev/null @@ -1,202 +0,0 @@ -import os -from io import ( - BytesIO, -) -from os.path import dirname, join -from mock import patch, Mock -from six.moves import StringIO -from sys import getfilesystemencoding -from twisted.trial import unittest -from allmydata.scripts import runner -from allmydata.scripts.run_common import ( - identify_node_type, - DaemonizeTahoeNodePlugin, - MyTwistdConfig, -) -from allmydata.scripts.tahoe_daemonize import ( - DaemonizeOptions, -) - - -class Util(unittest.TestCase): - def setUp(self): - self.twistd_options = MyTwistdConfig() - self.twistd_options.parseOptions(["DaemonizeTahoeNode"]) - self.options = self.twistd_options.subOptions - - def test_node_type_nothing(self): - tmpdir = self.mktemp() - base = dirname(tmpdir).decode(getfilesystemencoding()) - - t = identify_node_type(base) - - self.assertIs(None, t) - - def test_node_type_introducer(self): - tmpdir = self.mktemp() - base = dirname(tmpdir).decode(getfilesystemencoding()) - with open(join(dirname(tmpdir), 'introducer.tac'), 'w') as f: - f.write("test placeholder") - - t = identify_node_type(base) - - self.assertEqual(u"introducer", t) - - def test_daemonize(self): - tmpdir = self.mktemp() - plug = DaemonizeTahoeNodePlugin('client', tmpdir) - - with patch('twisted.internet.reactor') as r: - def call(fn, *args, **kw): - fn() - r.stop = lambda: None - r.callWhenRunning = call - service = plug.makeService(self.options) - service.parent = Mock() - service.startService() - - self.assertTrue(service is not None) - - def test_daemonize_no_keygen(self): - tmpdir = self.mktemp() - stderr = BytesIO() - plug = DaemonizeTahoeNodePlugin('key-generator', tmpdir) - - with patch('twisted.internet.reactor') as r: - def call(fn, *args, **kw): - d = fn() - d.addErrback(lambda _: None) # ignore the error we'll trigger - r.callWhenRunning = call - service = plug.makeService(self.options) - service.stderr = stderr - service.parent = Mock() - # we'll raise ValueError because there's no key-generator - # .. BUT we do this in an async function called via - # "callWhenRunning" .. hence using a hook - d = service.set_hook('running') - service.startService() - def done(f): - self.assertIn( - "key-generator support removed", - stderr.getvalue(), - ) - return None - d.addBoth(done) - return d - - def test_daemonize_unknown_nodetype(self): - tmpdir = self.mktemp() - plug = DaemonizeTahoeNodePlugin('an-unknown-service', tmpdir) - - with patch('twisted.internet.reactor') as r: - def call(fn, *args, **kw): - fn() - r.stop = lambda: None - r.callWhenRunning = call - service = plug.makeService(self.options) - service.parent = Mock() - with self.assertRaises(ValueError) as ctx: - service.startService() - self.assertIn( - "unknown nodetype", - str(ctx.exception) - ) - - def test_daemonize_options(self): - parent = runner.Options() - opts = DaemonizeOptions() - opts.parent = parent - opts.parseArgs() - - # just gratuitous coverage, ensureing we don't blow up on - # these methods. - opts.getSynopsis() - opts.getUsage() - - -class RunDaemonizeTests(unittest.TestCase): - - def setUp(self): - # no test should change our working directory - self._working = os.path.abspath('.') - d = super(RunDaemonizeTests, self).setUp() - self._reactor = patch('twisted.internet.reactor') - self._reactor.stop = lambda: None - self._twistd = patch('allmydata.scripts.run_common.twistd') - self.node_dir = self.mktemp() - os.mkdir(self.node_dir) - for cm in [self._reactor, self._twistd]: - cm.__enter__() - return d - - def tearDown(self): - d = super(RunDaemonizeTests, self).tearDown() - for cm in [self._reactor, self._twistd]: - cm.__exit__(None, None, None) - # Note: if you raise an exception (e.g. via self.assertEqual - # or raise RuntimeError) it is apparently just ignored and the - # test passes anyway... - if self._working != os.path.abspath('.'): - print("WARNING: a test just changed the working dir; putting it back") - os.chdir(self._working) - return d - - def _placeholder_nodetype(self, nodetype): - fname = join(self.node_dir, '{}.tac'.format(nodetype)) - with open(fname, 'w') as f: - f.write("test placeholder") - - def test_daemonize_defaults(self): - self._placeholder_nodetype('introducer') - - config = runner.parse_or_exit_with_explanation([ - # have to do this so the tests don't much around in - # ~/.tahoe (the default) - '--node-directory', self.node_dir, - 'daemonize', - ]) - i, o, e = StringIO(), StringIO(), StringIO() - with patch('allmydata.scripts.runner.sys') as s: - exit_code = [None] - def _exit(code): - exit_code[0] = code - s.exit = _exit - runner.dispatch(config, i, o, e) - - self.assertEqual(0, exit_code[0]) - - def test_daemonize_wrong_nodetype(self): - self._placeholder_nodetype('invalid') - - config = runner.parse_or_exit_with_explanation([ - # have to do this so the tests don't much around in - # ~/.tahoe (the default) - '--node-directory', self.node_dir, - 'daemonize', - ]) - i, o, e = StringIO(), StringIO(), StringIO() - with patch('allmydata.scripts.runner.sys') as s: - exit_code = [None] - def _exit(code): - exit_code[0] = code - s.exit = _exit - runner.dispatch(config, i, o, e) - - self.assertEqual(0, exit_code[0]) - - def test_daemonize_run(self): - self._placeholder_nodetype('client') - - config = runner.parse_or_exit_with_explanation([ - # have to do this so the tests don't much around in - # ~/.tahoe (the default) - '--node-directory', self.node_dir, - 'daemonize', - ]) - with patch('allmydata.scripts.runner.sys') as s: - exit_code = [None] - def _exit(code): - exit_code[0] = code - s.exit = _exit - from allmydata.scripts.tahoe_daemonize import daemonize - daemonize(config) diff --git a/src/allmydata/test/cli/test_start.py b/src/allmydata/test/cli/test_start.py deleted file mode 100644 index 42c70f024..000000000 --- a/src/allmydata/test/cli/test_start.py +++ /dev/null @@ -1,273 +0,0 @@ -import os -import shutil -import subprocess -from os.path import join -from mock import patch -from six.moves import StringIO -from functools import partial - -from twisted.trial import unittest -from allmydata.scripts import runner - - -#@patch('twisted.internet.reactor') -@patch('allmydata.scripts.tahoe_start.subprocess') -class RunStartTests(unittest.TestCase): - - def setUp(self): - d = super(RunStartTests, self).setUp() - self.node_dir = self.mktemp() - os.mkdir(self.node_dir) - return d - - def _placeholder_nodetype(self, nodetype): - fname = join(self.node_dir, '{}.tac'.format(nodetype)) - with open(fname, 'w') as f: - f.write("test placeholder") - - def _pid_file(self, pid): - fname = join(self.node_dir, 'twistd.pid') - with open(fname, 'w') as f: - f.write(u"{}\n".format(pid)) - - def _logs(self, logs): - os.mkdir(join(self.node_dir, 'logs')) - fname = join(self.node_dir, 'logs', 'twistd.log') - with open(fname, 'w') as f: - f.write(logs) - - def test_start_defaults(self, _subprocess): - self._placeholder_nodetype('client') - self._pid_file(1234) - self._logs('one log\ntwo log\nred log\nblue log\n') - - config = runner.parse_or_exit_with_explanation([ - # have to do this so the tests don't muck around in - # ~/.tahoe (the default) - '--node-directory', self.node_dir, - 'start', - ]) - i, o, e = StringIO(), StringIO(), StringIO() - try: - with patch('allmydata.scripts.tahoe_start.os'): - with patch('allmydata.scripts.runner.sys') as s: - exit_code = [None] - def _exit(code): - exit_code[0] = code - s.exit = _exit - - def launch(*args, **kw): - with open(join(self.node_dir, 'logs', 'twistd.log'), 'a') as f: - f.write('client running\n') # "the magic" - _subprocess.check_call = launch - runner.dispatch(config, i, o, e) - except Exception: - pass - - self.assertEqual([0], exit_code) - self.assertTrue('Node has started' in o.getvalue()) - - def test_start_fails(self, _subprocess): - self._placeholder_nodetype('client') - self._logs('existing log line\n') - - config = runner.parse_or_exit_with_explanation([ - # have to do this so the tests don't muck around in - # ~/.tahoe (the default) - '--node-directory', self.node_dir, - 'start', - ]) - - i, o, e = StringIO(), StringIO(), StringIO() - with patch('allmydata.scripts.tahoe_start.time') as t: - with patch('allmydata.scripts.runner.sys') as s: - exit_code = [None] - def _exit(code): - exit_code[0] = code - s.exit = _exit - - thetime = [0] - def _time(): - thetime[0] += 0.1 - return thetime[0] - t.time = _time - - def launch(*args, **kw): - with open(join(self.node_dir, 'logs', 'twistd.log'), 'a') as f: - f.write('a new log line\n') - _subprocess.check_call = launch - - runner.dispatch(config, i, o, e) - - # should print out the collected logs and an error-code - self.assertTrue("a new log line" in o.getvalue()) - self.assertEqual([1], exit_code) - - def test_start_subprocess_fails(self, _subprocess): - self._placeholder_nodetype('client') - self._logs('existing log line\n') - - config = runner.parse_or_exit_with_explanation([ - # have to do this so the tests don't muck around in - # ~/.tahoe (the default) - '--node-directory', self.node_dir, - 'start', - ]) - - i, o, e = StringIO(), StringIO(), StringIO() - with patch('allmydata.scripts.tahoe_start.time'): - with patch('allmydata.scripts.runner.sys') as s: - # undo patch for the exception-class - _subprocess.CalledProcessError = subprocess.CalledProcessError - exit_code = [None] - def _exit(code): - exit_code[0] = code - s.exit = _exit - - def launch(*args, **kw): - raise subprocess.CalledProcessError(42, "tahoe") - _subprocess.check_call = launch - - runner.dispatch(config, i, o, e) - - # should get our "odd" error-code - self.assertEqual([42], exit_code) - - def test_start_help(self, _subprocess): - self._placeholder_nodetype('client') - - std = StringIO() - with patch('sys.stdout') as stdo: - stdo.write = std.write - try: - runner.parse_or_exit_with_explanation([ - # have to do this so the tests don't muck around in - # ~/.tahoe (the default) - '--node-directory', self.node_dir, - 'start', - '--help', - ], stdout=std) - self.fail("Should get exit") - except SystemExit as e: - print(e) - - self.assertIn( - "Usage:", - std.getvalue() - ) - - def test_start_unknown_node_type(self, _subprocess): - self._placeholder_nodetype('bogus') - - config = runner.parse_or_exit_with_explanation([ - # have to do this so the tests don't muck around in - # ~/.tahoe (the default) - '--node-directory', self.node_dir, - 'start', - ]) - - i, o, e = StringIO(), StringIO(), StringIO() - with patch('allmydata.scripts.runner.sys') as s: - exit_code = [None] - def _exit(code): - exit_code[0] = code - s.exit = _exit - - runner.dispatch(config, i, o, e) - - # should print out the collected logs and an error-code - self.assertIn( - "is not a recognizable node directory", - e.getvalue() - ) - self.assertEqual([1], exit_code) - - def test_start_nodedir_not_dir(self, _subprocess): - shutil.rmtree(self.node_dir) - assert not os.path.isdir(self.node_dir) - - config = runner.parse_or_exit_with_explanation([ - # have to do this so the tests don't muck around in - # ~/.tahoe (the default) - '--node-directory', self.node_dir, - 'start', - ]) - - i, o, e = StringIO(), StringIO(), StringIO() - with patch('allmydata.scripts.runner.sys') as s: - exit_code = [None] - def _exit(code): - exit_code[0] = code - s.exit = _exit - - runner.dispatch(config, i, o, e) - - # should print out the collected logs and an error-code - self.assertIn( - "does not look like a directory at all", - e.getvalue() - ) - self.assertEqual([1], exit_code) - - -class RunTests(unittest.TestCase): - """ - Tests confirming end-user behavior of CLI commands - """ - - def setUp(self): - d = super(RunTests, self).setUp() - self.addCleanup(partial(os.chdir, os.getcwd())) - self.node_dir = self.mktemp() - os.mkdir(self.node_dir) - return d - - @patch('twisted.internet.reactor') - def test_run_invalid_config(self, reactor): - """ - Configuration that's invalid should be obvious to the user - """ - - def cwr(fn, *args, **kw): - fn() - - def stop(*args, **kw): - stopped.append(None) - stopped = [] - reactor.callWhenRunning = cwr - reactor.stop = stop - - with open(os.path.join(self.node_dir, "client.tac"), "w") as f: - f.write('test') - - with open(os.path.join(self.node_dir, "tahoe.cfg"), "w") as f: - f.write( - "[invalid section]\n" - "foo = bar\n" - ) - - config = runner.parse_or_exit_with_explanation([ - # have to do this so the tests don't muck around in - # ~/.tahoe (the default) - '--node-directory', self.node_dir, - 'run', - ]) - - i, o, e = StringIO(), StringIO(), StringIO() - d = runner.dispatch(config, i, o, e) - - self.assertFailure(d, SystemExit) - - output = e.getvalue() - # should print out the collected logs and an error-code - self.assertIn( - "invalid section", - output, - ) - self.assertIn( - "Configuration error:", - output, - ) - # ensure reactor.stop was actually called - self.assertEqual([None], stopped) - return d From ca92fa4eb5b977dd879a7fd900e7f13259c481bf Mon Sep 17 00:00:00 2001 From: Jean-Paul Calderone Date: Tue, 8 Dec 2020 18:22:14 -0500 Subject: [PATCH 15/54] Don't think about "tahoe start" --- src/allmydata/scripts/run_common.py | 8 ++++---- src/allmydata/test/test_runner.py | 13 ++++--------- 2 files changed, 8 insertions(+), 13 deletions(-) diff --git a/src/allmydata/scripts/run_common.py b/src/allmydata/scripts/run_common.py index 71934414d..f62534434 100644 --- a/src/allmydata/scripts/run_common.py +++ b/src/allmydata/scripts/run_common.py @@ -73,10 +73,10 @@ class RunOptions(BasedirOptions): ] def parseArgs(self, basedir=None, *twistd_args): - # This can't handle e.g. 'tahoe start --nodaemon', since '--nodaemon' - # looks like an option to the tahoe subcommand, not to twistd. So you - # can either use 'tahoe start' or 'tahoe start NODEDIR - # --TWISTD-OPTIONS'. Note that 'tahoe --node-directory=NODEDIR start + # This can't handle e.g. 'tahoe run --reactor=foo', since + # '--reactor=foo' looks like an option to the tahoe subcommand, not to + # twistd. So you can either use 'tahoe run' or 'tahoe run NODEDIR + # --TWISTD-OPTIONS'. Note that 'tahoe --node-directory=NODEDIR run # --TWISTD-OPTIONS' also isn't allowed, unfortunately. BasedirOptions.parseArgs(self, basedir) diff --git a/src/allmydata/test/test_runner.py b/src/allmydata/test/test_runner.py index 2ec871231..a258542b9 100644 --- a/src/allmydata/test/test_runner.py +++ b/src/allmydata/test/test_runner.py @@ -252,15 +252,10 @@ class RunNode(common_util.SignalMixin, unittest.TestCase, pollmixin.PollMixin, RunBinTahoeMixin): """ exercise "tahoe run" for both introducer, client node, and key-generator, - by spawning "tahoe run" (or "tahoe start") as a subprocess. This doesn't - get us line-level coverage, but it does a better job of confirming that - the user can actually run "./bin/tahoe run" and expect it to work. This - verifies that bin/tahoe sets up PYTHONPATH and the like correctly. - - This doesn't work on cygwin (it hangs forever), so we skip this test - when we're on cygwin. It is likely that "tahoe start" itself doesn't - work on cygwin: twisted seems unable to provide a version of - spawnProcess which really works there. + by spawning "tahoe run" as a subprocess. This doesn't get us line-level + coverage, but it does a better job of confirming that the user can + actually run "./bin/tahoe run" and expect it to work. This verifies that + bin/tahoe sets up PYTHONPATH and the like correctly. """ def workdir(self, name): From d346c90c6e035d726b81db5a2761f88e6ed53e7b Mon Sep 17 00:00:00 2001 From: Jean-Paul Calderone Date: Tue, 8 Dec 2020 18:22:58 -0500 Subject: [PATCH 16/54] This is gonna take some work --- src/allmydata/test/check_grid.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/allmydata/test/check_grid.py b/src/allmydata/test/check_grid.py index d3993ee5e..266b7688b 100644 --- a/src/allmydata/test/check_grid.py +++ b/src/allmydata/test/check_grid.py @@ -24,7 +24,7 @@ To set up the client node, do the following: tahoe create-client DIR populate DIR/introducer.furl - tahoe start DIR + tahoe start DIR XXXX tahoe add-alias -d DIR testgrid `tahoe mkdir -d DIR` pick a 10kB-ish test file, compute its md5sum tahoe put -d DIR FILE testgrid:old.MD5SUM @@ -117,7 +117,7 @@ class GridTester(object): def start_node(self): print("tahoe start", self.nodedir) - self.command(self.tahoe, "start", self.nodedir) + self.command(self.tahoe, "start", self.nodedir) # XXXX time.sleep(5) def stop_node(self): From 5b0190b9a1eef8a9735a8f46a72596178ba2b3c0 Mon Sep 17 00:00:00 2001 From: Jean-Paul Calderone Date: Wed, 9 Dec 2020 07:24:51 -0500 Subject: [PATCH 17/54] Remove some more test code related to start/restart/stop --- src/allmydata/test/cli/test_cli.py | 81 +++++++++--------------------- src/allmydata/test/cli_node_api.py | 40 +++++++-------- src/allmydata/test/test_runner.py | 20 ++------ 3 files changed, 46 insertions(+), 95 deletions(-) diff --git a/src/allmydata/test/cli/test_cli.py b/src/allmydata/test/cli/test_cli.py index 7f4f4140e..27af29520 100644 --- a/src/allmydata/test/cli/test_cli.py +++ b/src/allmydata/test/cli/test_cli.py @@ -20,14 +20,14 @@ from allmydata.scripts.common_http import socket_error import allmydata.scripts.common_http # Test that the scripts can be imported. -from allmydata.scripts import create_node, debug, tahoe_start, tahoe_restart, \ +from allmydata.scripts import create_node, debug, \ tahoe_add_alias, tahoe_backup, tahoe_check, tahoe_cp, tahoe_get, tahoe_ls, \ tahoe_manifest, tahoe_mkdir, tahoe_mv, tahoe_put, tahoe_unlink, tahoe_webopen, \ - tahoe_stop, tahoe_daemonize, tahoe_run -_hush_pyflakes = [create_node, debug, tahoe_start, tahoe_restart, tahoe_stop, + tahoe_run +_hush_pyflakes = [create_node, debug, tahoe_add_alias, tahoe_backup, tahoe_check, tahoe_cp, tahoe_get, tahoe_ls, tahoe_manifest, tahoe_mkdir, tahoe_mv, tahoe_put, tahoe_unlink, tahoe_webopen, - tahoe_daemonize, tahoe_run] + tahoe_run] from allmydata.scripts import common from allmydata.scripts.common import DEFAULT_ALIAS, get_aliases, get_alias, \ @@ -626,18 +626,6 @@ class Help(unittest.TestCase): help = str(cli.ListAliasesOptions()) self.failUnlessIn("[options]", help) - def test_start(self): - help = str(tahoe_start.StartOptions()) - self.failUnlessIn("[options] [NODEDIR [twistd-options]]", help) - - def test_stop(self): - help = str(tahoe_stop.StopOptions()) - self.failUnlessIn("[options] [NODEDIR]", help) - - def test_restart(self): - help = str(tahoe_restart.RestartOptions()) - self.failUnlessIn("[options] [NODEDIR [twistd-options]]", help) - def test_run(self): help = str(tahoe_run.RunOptions()) self.failUnlessIn("[options] [NODEDIR [twistd-options]]", help) @@ -1269,79 +1257,58 @@ class Options(ReallyEqualMixin, unittest.TestCase): self.failUnlessIn(allmydata.__full_version__, stdout.getvalue()) # but "tahoe SUBCOMMAND --version" should be rejected self.failUnlessRaises(usage.UsageError, self.parse, - ["start", "--version"]) + ["run", "--version"]) self.failUnlessRaises(usage.UsageError, self.parse, - ["start", "--version-and-path"]) + ["run", "--version-and-path"]) def test_quiet(self): # accepted as an overall option, but not on subcommands - o = self.parse(["--quiet", "start"]) + o = self.parse(["--quiet", "run"]) self.failUnless(o.parent["quiet"]) self.failUnlessRaises(usage.UsageError, self.parse, - ["start", "--quiet"]) + ["run", "--quiet"]) def test_basedir(self): # accept a --node-directory option before the verb, or a --basedir # option after, or a basedir argument after, but none in the wrong # place, and not more than one of the three. - o = self.parse(["start"]) + o = self.parse(["run"]) self.failUnlessReallyEqual(o["basedir"], os.path.join(fileutil.abspath_expanduser_unicode(u"~"), u".tahoe")) - o = self.parse(["start", "here"]) + o = self.parse(["run", "here"]) self.failUnlessReallyEqual(o["basedir"], fileutil.abspath_expanduser_unicode(u"here")) - o = self.parse(["start", "--basedir", "there"]) + o = self.parse(["run", "--basedir", "there"]) self.failUnlessReallyEqual(o["basedir"], fileutil.abspath_expanduser_unicode(u"there")) - o = self.parse(["--node-directory", "there", "start"]) + o = self.parse(["--node-directory", "there", "run"]) self.failUnlessReallyEqual(o["basedir"], fileutil.abspath_expanduser_unicode(u"there")) - o = self.parse(["start", "here", "--nodaemon"]) + o = self.parse(["run", "here", "--nodaemon"]) self.failUnlessReallyEqual(o["basedir"], fileutil.abspath_expanduser_unicode(u"here")) self.failUnlessRaises(usage.UsageError, self.parse, - ["--basedir", "there", "start"]) + ["--basedir", "there", "run"]) self.failUnlessRaises(usage.UsageError, self.parse, - ["start", "--node-directory", "there"]) + ["run", "--node-directory", "there"]) self.failUnlessRaises(usage.UsageError, self.parse, ["--node-directory=there", - "start", "--basedir=here"]) + "run", "--basedir=here"]) self.failUnlessRaises(usage.UsageError, self.parse, - ["start", "--basedir=here", "anywhere"]) + ["run", "--basedir=here", "anywhere"]) self.failUnlessRaises(usage.UsageError, self.parse, ["--node-directory=there", - "start", "anywhere"]) + "run", "anywhere"]) self.failUnlessRaises(usage.UsageError, self.parse, ["--node-directory=there", - "start", "--basedir=here", "anywhere"]) + "run", "--basedir=here", "anywhere"]) self.failUnlessRaises(usage.UsageError, self.parse, - ["--node-directory=there", "start", "--nodaemon"]) + ["--node-directory=there", "run", "--nodaemon"]) self.failUnlessRaises(usage.UsageError, self.parse, - ["start", "--basedir=here", "--nodaemon"]) + ["run", "--basedir=here", "--nodaemon"]) -class Stop(unittest.TestCase): - def test_non_numeric_pid(self): - """ - If the pidfile exists but does not contain a numeric value, a complaint to - this effect is written to stderr and the non-success result is - returned. - """ - basedir = FilePath(self.mktemp().decode("ascii")) - basedir.makedirs() - basedir.child(u"twistd.pid").setContent(b"foo") - - config = tahoe_stop.StopOptions() - config.stdout = StringIO() - config.stderr = StringIO() - config['basedir'] = basedir.path - - result_code = tahoe_stop.stop(config) - self.assertEqual(2, result_code) - self.assertIn("invalid PID file", config.stderr.getvalue()) - - -class Start(unittest.TestCase): +class Run(unittest.TestCase): @patch('allmydata.scripts.run_common.os.chdir') @patch('allmydata.scripts.run_common.twistd') @@ -1355,13 +1322,13 @@ class Start(unittest.TestCase): basedir.child(u"twistd.pid").setContent(b"foo") basedir.child(u"tahoe-client.tac").setContent(b"") - config = tahoe_daemonize.DaemonizeOptions() + config = tahoe_run.RunOptions() config.stdout = StringIO() config.stderr = StringIO() config['basedir'] = basedir.path config.twistd_args = [] - result_code = tahoe_daemonize.daemonize(config) + result_code = tahoe_run.run(config) self.assertIn("invalid PID file", config.stderr.getvalue()) self.assertTrue(len(mock_twistd.mock_calls), 1) self.assertEqual(mock_twistd.mock_calls[0][0], 'runApp') diff --git a/src/allmydata/test/cli_node_api.py b/src/allmydata/test/cli_node_api.py index 8453fbca2..09e96bce9 100644 --- a/src/allmydata/test/cli_node_api.py +++ b/src/allmydata/test/cli_node_api.py @@ -14,6 +14,10 @@ from errno import ENOENT import attr +from eliot import ( + log_call, +) + from twisted.internet.error import ( ProcessDone, ProcessTerminated, @@ -42,11 +46,9 @@ from twisted.internet.task import ( from ..client import ( _Client, ) -from ..scripts.tahoe_stop import ( - COULD_NOT_STOP, -) from ..util.eliotutil import ( inline_callbacks, + log_call_deferred, ) class Expect(Protocol, object): @@ -156,6 +158,7 @@ class CLINodeAPI(object): env=os.environ, ) + @log_call(action_type="test:cli-api:run", include_args=["extra_tahoe_args"]) def run(self, protocol, extra_tahoe_args=()): """ Start the node running. @@ -176,28 +179,21 @@ class CLINodeAPI(object): if ENOENT != e.errno: raise + @log_call_deferred(action_type="test:cli-api:stop") def stop(self, protocol): - self._execute( - protocol, - [u"stop", self.basedir.asTextMode().path], - ) + return self.stop_and_wait() + @log_call_deferred(action_type="test:cli-api:stop-and-wait") @inline_callbacks def stop_and_wait(self): - if platform.isWindows(): - # On Windows there is no PID file and no "tahoe stop". - if self.process is not None: - while True: - try: - self.process.signalProcess("TERM") - except ProcessExitedAlready: - break - else: - yield deferLater(self.reactor, 0.1, lambda: None) - else: - protocol, ended = wait_for_exit() - self.stop(protocol) - yield ended + if self.process is not None: + while True: + try: + self.process.signalProcess("TERM") + except ProcessExitedAlready: + break + else: + yield deferLater(self.reactor, 0.1, lambda: None) def active(self): # By writing this file, we get two minutes before the client will @@ -208,8 +204,6 @@ class CLINodeAPI(object): def _check_cleanup_reason(self, reason): # Let it fail because the process has already exited. reason.trap(ProcessTerminated) - if reason.value.exitCode != COULD_NOT_STOP: - return reason return None def cleanup(self): diff --git a/src/allmydata/test/test_runner.py b/src/allmydata/test/test_runner.py index a258542b9..636880621 100644 --- a/src/allmydata/test/test_runner.py +++ b/src/allmydata/test/test_runner.py @@ -34,6 +34,7 @@ from ._twisted_9607 import ( ) from ..util.eliotutil import ( inline_callbacks, + log_call_deferred, ) def get_root_from_file(src): @@ -54,6 +55,7 @@ rootdir = get_root_from_file(srcfile) class RunBinTahoeMixin(object): + @log_call_deferred(action_type="run-bin-tahoe") def run_bintahoe(self, args, stdin=None, python_options=[], env=None): command = sys.executable argv = python_options + ["-m", "allmydata.scripts.runner"] + args @@ -335,7 +337,7 @@ class RunNode(common_util.SignalMixin, unittest.TestCase, pollmixin.PollMixin, @inline_callbacks def test_client(self): """ - Test many things. + Test too many things. 0) Verify that "tahoe create-node" takes a --webport option and writes the value to the configuration file. @@ -343,9 +345,9 @@ class RunNode(common_util.SignalMixin, unittest.TestCase, pollmixin.PollMixin, 1) Verify that "tahoe run" writes a pid file and a node url file (on POSIX). 2) Verify that the storage furl file has a stable value across a - "tahoe run" / "tahoe stop" / "tahoe run" sequence. + "tahoe run" / stop / "tahoe run" sequence. - 3) Verify that the pid file is removed after "tahoe stop" succeeds (on POSIX). + 3) Verify that the pid file is removed after SIGTERM (on POSIX). """ basedir = self.workdir("test_client") c1 = os.path.join(basedir, "c1") @@ -449,18 +451,6 @@ class RunNode(common_util.SignalMixin, unittest.TestCase, pollmixin.PollMixin, "does not look like a directory at all" ) - def test_stop_bad_directory(self): - """ - If ``tahoe run`` is pointed at a directory where no node is running, it - reports an error and exits. - """ - return self._bad_directory_test( - u"test_stop_bad_directory", - "tahoe stop", - lambda tahoe, p: tahoe.stop(p), - "does not look like a running node directory", - ) - @inline_callbacks def _bad_directory_test(self, workdir, description, operation, expected_message): """ From bc9019943e5728132534e51300c65aede175b753 Mon Sep 17 00:00:00 2001 From: Jean-Paul Calderone Date: Wed, 9 Dec 2020 07:28:16 -0500 Subject: [PATCH 18/54] no more "tahoe restart" --- docs/frontends/webapi.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/frontends/webapi.rst b/docs/frontends/webapi.rst index 2ee348080..99fa44979 100644 --- a/docs/frontends/webapi.rst +++ b/docs/frontends/webapi.rst @@ -2145,7 +2145,7 @@ you could do the following:: tahoe debug dump-cap URI:CHK:n7r3m6wmomelk4sep3kw5cvduq:os7ijw5c3maek7pg65e5254k2fzjflavtpejjyhshpsxuqzhcwwq:3:20:14861 -> storage index: whpepioyrnff7orecjolvbudeu echo "whpepioyrnff7orecjolvbudeu my puppy told me to" >>$NODEDIR/access.blacklist - tahoe restart $NODEDIR + # ... restart the node to re-read configuration ... tahoe get URI:CHK:n7r3m6wmomelk4sep3kw5cvduq:os7ijw5c3maek7pg65e5254k2fzjflavtpejjyhshpsxuqzhcwwq:3:20:14861 -> error, 403 Access Prohibited: my puppy told me to From 179f0bb9ec942e27a49369461f73efad8573ce73 Mon Sep 17 00:00:00 2001 From: Jean-Paul Calderone Date: Wed, 9 Dec 2020 10:55:13 -0500 Subject: [PATCH 19/54] news fragment --- newsfragments/3523.minor | 0 newsfragments/3550.removed | 1 + 2 files changed, 1 insertion(+) delete mode 100644 newsfragments/3523.minor create mode 100644 newsfragments/3550.removed diff --git a/newsfragments/3523.minor b/newsfragments/3523.minor deleted file mode 100644 index e69de29bb..000000000 diff --git a/newsfragments/3550.removed b/newsfragments/3550.removed new file mode 100644 index 000000000..2074bf676 --- /dev/null +++ b/newsfragments/3550.removed @@ -0,0 +1 @@ +The deprecated ``tahoe`` start, restart, stop, and daemonize sub-commands have been removed. \ No newline at end of file From 74c39904561c1d0f1b3383c5057f8255760e6b6b Mon Sep 17 00:00:00 2001 From: Jean-Paul Calderone Date: Wed, 9 Dec 2020 10:57:02 -0500 Subject: [PATCH 20/54] This extra stop complexity is no longer needed --- src/allmydata/test/cli_node_api.py | 24 +----------------------- 1 file changed, 1 insertion(+), 23 deletions(-) diff --git a/src/allmydata/test/cli_node_api.py b/src/allmydata/test/cli_node_api.py index 09e96bce9..34d73a199 100644 --- a/src/allmydata/test/cli_node_api.py +++ b/src/allmydata/test/cli_node_api.py @@ -5,7 +5,6 @@ __all__ = [ "on_stdout", "on_stdout_and_stderr", "on_different", - "wait_for_exit", ] import os @@ -19,7 +18,6 @@ from eliot import ( ) from twisted.internet.error import ( - ProcessDone, ProcessTerminated, ProcessExitedAlready, ) @@ -29,9 +27,6 @@ from twisted.internet.interfaces import ( from twisted.python.filepath import ( FilePath, ) -from twisted.python.runtime import ( - platform, -) from twisted.internet.protocol import ( Protocol, ProcessProtocol, @@ -180,7 +175,7 @@ class CLINodeAPI(object): raise @log_call_deferred(action_type="test:cli-api:stop") - def stop(self, protocol): + def stop(self): return self.stop_and_wait() @log_call_deferred(action_type="test:cli-api:stop-and-wait") @@ -210,20 +205,3 @@ class CLINodeAPI(object): stopping = self.stop_and_wait() stopping.addErrback(self._check_cleanup_reason) return stopping - - -class _WaitForEnd(ProcessProtocol, object): - def __init__(self, ended): - self._ended = ended - - def processEnded(self, reason): - if reason.check(ProcessDone): - self._ended.callback(None) - else: - self._ended.errback(reason) - - -def wait_for_exit(): - ended = Deferred() - protocol = _WaitForEnd(ended) - return protocol, ended From 8b890d255ee583463390244b78286fc6cfebb46c Mon Sep 17 00:00:00 2001 From: Jean-Paul Calderone Date: Wed, 9 Dec 2020 11:02:29 -0500 Subject: [PATCH 21/54] extra news fragments --- newsfragments/3523.minor | 0 newsfragments/3524.minor | 0 2 files changed, 0 insertions(+), 0 deletions(-) create mode 100644 newsfragments/3523.minor create mode 100644 newsfragments/3524.minor diff --git a/newsfragments/3523.minor b/newsfragments/3523.minor new file mode 100644 index 000000000..e69de29bb diff --git a/newsfragments/3524.minor b/newsfragments/3524.minor new file mode 100644 index 000000000..e69de29bb From 48bef7db9937ebbaf3440e7b3b6a2aa98e1871aa Mon Sep 17 00:00:00 2001 From: Itamar Turner-Trauring Date: Wed, 9 Dec 2020 13:42:35 -0500 Subject: [PATCH 22/54] Some straightforward changes to support Python 3. --- src/allmydata/dirnode.py | 34 +++++++++++++++++----------------- 1 file changed, 17 insertions(+), 17 deletions(-) diff --git a/src/allmydata/dirnode.py b/src/allmydata/dirnode.py index 59ebd73ba..1af655574 100644 --- a/src/allmydata/dirnode.py +++ b/src/allmydata/dirnode.py @@ -179,7 +179,7 @@ class Adder(object): def modify(self, old_contents, servermap, first_time): children = self.node._unpack_contents(old_contents) now = time.time() - for (namex, (child, new_metadata)) in self.entries.iteritems(): + for (namex, (child, new_metadata)) in list(self.entries.items()): name = normalize(namex) precondition(IFilesystemNode.providedBy(child), child) @@ -205,8 +205,8 @@ class Adder(object): return new_contents def _encrypt_rw_uri(writekey, rw_uri): - precondition(isinstance(rw_uri, str), rw_uri) - precondition(isinstance(writekey, str), writekey) + precondition(isinstance(rw_uri, bytes), rw_uri) + precondition(isinstance(writekey, bytes), writekey) salt = hashutil.mutable_rwcap_salt_hash(rw_uri) key = hashutil.mutable_rwcap_key_hash(salt, writekey) @@ -221,7 +221,7 @@ def _encrypt_rw_uri(writekey, rw_uri): def pack_children(childrenx, writekey, deep_immutable=False): # initial_children must have metadata (i.e. {} instead of None) children = {} - for (namex, (node, metadata)) in childrenx.iteritems(): + for (namex, (node, metadata)) in list(childrenx.items()): precondition(isinstance(metadata, dict), "directory creation requires metadata to be a dict, not None", metadata) children[normalize(namex)] = (node, metadata) @@ -245,7 +245,7 @@ def _pack_normalized_children(children, writekey, deep_immutable=False): If deep_immutable is True, I will require that all my children are deeply immutable, and will raise a MustBeDeepImmutableError if not. """ - precondition((writekey is None) or isinstance(writekey, str), writekey) + precondition((writekey is None) or isinstance(writekey, bytes), writekey) has_aux = isinstance(children, AuxValueDict) entries = [] @@ -264,26 +264,26 @@ def _pack_normalized_children(children, writekey, deep_immutable=False): assert isinstance(metadata, dict) rw_uri = child.get_write_uri() if rw_uri is None: - rw_uri = "" - assert isinstance(rw_uri, str), rw_uri + rw_uri = b"" + assert isinstance(rw_uri, bytes), rw_uri # should be prevented by MustBeDeepImmutableError check above assert not (rw_uri and deep_immutable) ro_uri = child.get_readonly_uri() if ro_uri is None: - ro_uri = "" - assert isinstance(ro_uri, str), ro_uri + ro_uri = b"" + assert isinstance(ro_uri, bytes), ro_uri if writekey is not None: writecap = netstring(_encrypt_rw_uri(writekey, rw_uri)) else: writecap = ZERO_LEN_NETSTR - entry = "".join([netstring(name.encode("utf-8")), + entry = b"".join([netstring(name.encode("utf-8")), netstring(strip_prefix_for_ro(ro_uri, deep_immutable)), writecap, - netstring(json.dumps(metadata))]) + netstring(json.dumps(metadata).encode("utf-8"))]) entries.append(netstring(entry)) - return "".join(entries) + return b"".join(entries) @implementer(IDirectoryNode, ICheckable, IDeepCheckable) class DirectoryNode(object): @@ -352,7 +352,7 @@ class DirectoryNode(object): # cleartext. The 'name' is UTF-8 encoded, and should be normalized to NFC. # The rwcapdata is formatted as: # pack("16ss32s", iv, AES(H(writekey+iv), plaintext_rw_uri), mac) - assert isinstance(data, str), (repr(data), type(data)) + assert isinstance(data, bytes), (repr(data), type(data)) # an empty directory is serialized as an empty string if data == "": return AuxValueDict() @@ -555,8 +555,8 @@ class DirectoryNode(object): return d def set_uri(self, namex, writecap, readcap, metadata=None, overwrite=True): - precondition(isinstance(writecap, (str,type(None))), writecap) - precondition(isinstance(readcap, (str,type(None))), readcap) + precondition(isinstance(writecap, (bytes, type(None))), writecap) + precondition(isinstance(readcap, (bytes, type(None))), readcap) # We now allow packing unknown nodes, provided they are valid # for this type of directory. @@ -577,8 +577,8 @@ class DirectoryNode(object): else: assert len(e) == 3 writecap, readcap, metadata = e - precondition(isinstance(writecap, (str,type(None))), writecap) - precondition(isinstance(readcap, (str,type(None))), readcap) + precondition(isinstance(writecap, (bytes,type(None))), writecap) + precondition(isinstance(readcap, (bytes,type(None))), readcap) # We now allow packing unknown nodes, provided they are valid # for this type of directory. From 6b8fd2f29d321a16c552bd03d450eeaa4d977f31 Mon Sep 17 00:00:00 2001 From: Itamar Turner-Trauring Date: Wed, 9 Dec 2020 13:45:31 -0500 Subject: [PATCH 23/54] Some progress towards passing tests on Python 3. --- src/allmydata/dirnode.py | 4 +-- src/allmydata/nodemaker.py | 2 +- src/allmydata/test/test_dirnode.py | 42 +++++++++++++++--------------- 3 files changed, 24 insertions(+), 24 deletions(-) diff --git a/src/allmydata/dirnode.py b/src/allmydata/dirnode.py index 1af655574..74d11a40f 100644 --- a/src/allmydata/dirnode.py +++ b/src/allmydata/dirnode.py @@ -384,8 +384,8 @@ class DirectoryNode(object): # ro_uri is treated in the same way for consistency. # rw_uri and ro_uri will be either None or a non-empty string. - rw_uri = rw_uri.rstrip(' ') or None - ro_uri = ro_uri.rstrip(' ') or None + rw_uri = rw_uri.rstrip(b' ') or None + ro_uri = ro_uri.rstrip(b' ') or None try: child = self._create_and_validate_node(rw_uri, ro_uri, name) diff --git a/src/allmydata/nodemaker.py b/src/allmydata/nodemaker.py index c3ba1ba7b..ab455ce48 100644 --- a/src/allmydata/nodemaker.py +++ b/src/allmydata/nodemaker.py @@ -126,7 +126,7 @@ class NodeMaker(object): def create_new_mutable_directory(self, initial_children={}, version=None): # initial_children must have metadata (i.e. {} instead of None) - for (name, (node, metadata)) in initial_children.iteritems(): + for (name, (node, metadata)) in initial_children.items(): precondition(isinstance(metadata, dict), "create_new_mutable_directory requires metadata to be a dict, not None", metadata) node.raise_error() diff --git a/src/allmydata/test/test_dirnode.py b/src/allmydata/test/test_dirnode.py index 48ffff45a..1845c3ac3 100644 --- a/src/allmydata/test/test_dirnode.py +++ b/src/allmydata/test/test_dirnode.py @@ -48,16 +48,16 @@ class MemAccum(object): self.data = data self.producer.resumeProducing() -setup_py_uri = "URI:CHK:n7r3m6wmomelk4sep3kw5cvduq:os7ijw5c3maek7pg65e5254k2fzjflavtpejjyhshpsxuqzhcwwq:3:20:14861" -one_uri = "URI:LIT:n5xgk" # LIT for "one" -mut_write_uri = "URI:SSK:vfvcbdfbszyrsaxchgevhmmlii:euw4iw7bbnkrrwpzuburbhppuxhc3gwxv26f6imekhz7zyw2ojnq" -mdmf_write_uri = "URI:MDMF:x533rhbm6kiehzl5kj3s44n5ie:4gif5rhneyd763ouo5qjrgnsoa3bg43xycy4robj2rf3tvmhdl3a" -empty_litdir_uri = "URI:DIR2-LIT:" -tiny_litdir_uri = "URI:DIR2-LIT:gqytunj2onug64tufqzdcosvkjetutcjkq5gw4tvm5vwszdgnz5hgyzufqydulbshj5x2lbm" # contains one child which is itself also LIT -mut_read_uri = "URI:SSK-RO:jf6wkflosyvntwxqcdo7a54jvm:euw4iw7bbnkrrwpzuburbhppuxhc3gwxv26f6imekhz7zyw2ojnq" -mdmf_read_uri = "URI:MDMF-RO:d4cydxselputycfzkw6qgz4zv4:4gif5rhneyd763ouo5qjrgnsoa3bg43xycy4robj2rf3tvmhdl3a" -future_write_uri = "x-tahoe-crazy://I_am_from_the_future." -future_read_uri = "x-tahoe-crazy-readonly://I_am_from_the_future." +setup_py_uri = b"URI:CHK:n7r3m6wmomelk4sep3kw5cvduq:os7ijw5c3maek7pg65e5254k2fzjflavtpejjyhshpsxuqzhcwwq:3:20:14861" +one_uri = b"URI:LIT:n5xgk" # LIT for "one" +mut_write_uri = b"URI:SSK:vfvcbdfbszyrsaxchgevhmmlii:euw4iw7bbnkrrwpzuburbhppuxhc3gwxv26f6imekhz7zyw2ojnq" +mdmf_write_uri = b"URI:MDMF:x533rhbm6kiehzl5kj3s44n5ie:4gif5rhneyd763ouo5qjrgnsoa3bg43xycy4robj2rf3tvmhdl3a" +empty_litdir_uri = b"URI:DIR2-LIT:" +tiny_litdir_uri = b"URI:DIR2-LIT:gqytunj2onug64tufqzdcosvkjetutcjkq5gw4tvm5vwszdgnz5hgyzufqydulbshj5x2lbm" # contains one child which is itself also LIT +mut_read_uri = b"URI:SSK-RO:jf6wkflosyvntwxqcdo7a54jvm:euw4iw7bbnkrrwpzuburbhppuxhc3gwxv26f6imekhz7zyw2ojnq" +mdmf_read_uri = b"URI:MDMF-RO:d4cydxselputycfzkw6qgz4zv4:4gif5rhneyd763ouo5qjrgnsoa3bg43xycy4robj2rf3tvmhdl3a" +future_write_uri = b"x-tahoe-crazy://I_am_from_the_future." +future_read_uri = b"x-tahoe-crazy-readonly://I_am_from_the_future." future_nonascii_write_uri = u"x-tahoe-even-more-crazy://I_am_from_the_future_rw_\u263A".encode('utf-8') future_nonascii_read_uri = u"x-tahoe-even-more-crazy-readonly://I_am_from_the_future_ro_\u263A".encode('utf-8') @@ -423,7 +423,7 @@ class Dirnode(GridTestMixin, unittest.TestCase, # moved on to stdlib "json" which doesn't have it either. d.addCallback(self.stall, 0.1) d.addCallback(lambda res: n.add_file(u"timestamps", - upload.Data("stamp me", convergence="some convergence string"))) + upload.Data(b"stamp me", convergence=b"some convergence string"))) d.addCallback(self.stall, 0.1) def _stop(res): self._stop_timestamp = time.time() @@ -472,11 +472,11 @@ class Dirnode(GridTestMixin, unittest.TestCase, self.failUnlessReallyEqual(set(children.keys()), set([u"child"]))) - uploadable1 = upload.Data("some data", convergence="converge") + uploadable1 = upload.Data(b"some data", convergence=b"converge") d.addCallback(lambda res: n.add_file(u"newfile", uploadable1)) d.addCallback(lambda newnode: self.failUnless(IImmutableFileNode.providedBy(newnode))) - uploadable2 = upload.Data("some data", convergence="stuff") + uploadable2 = upload.Data(b"some data", convergence=b"stuff") d.addCallback(lambda res: self.shouldFail(ExistingChildError, "add_file-no", "child 'newfile' already exists", @@ -491,7 +491,7 @@ class Dirnode(GridTestMixin, unittest.TestCase, d.addCallback(lambda metadata: self.failUnlessEqual(set(metadata.keys()), set(["tahoe"]))) - uploadable3 = upload.Data("some data", convergence="converge") + uploadable3 = upload.Data(b"some data", convergence=b"converge") d.addCallback(lambda res: n.add_file(u"newfile-metadata", uploadable3, {"key": "value"})) @@ -507,8 +507,8 @@ class Dirnode(GridTestMixin, unittest.TestCase, def _created2(subdir2): self.subdir2 = subdir2 # put something in the way, to make sure it gets overwritten - return subdir2.add_file(u"child", upload.Data("overwrite me", - "converge")) + return subdir2.add_file(u"child", upload.Data(b"overwrite me", + b"converge")) d.addCallback(_created2) d.addCallback(lambda res: @@ -1074,7 +1074,7 @@ class Dirnode(GridTestMixin, unittest.TestCase, d.addCallback(_created_root) def _created_subdir(subdir): self._subdir = subdir - d = subdir.add_file(u"file1", upload.Data("data"*100, None)) + d = subdir.add_file(u"file1", upload.Data(b"data"*100, None)) d.addCallback(lambda res: subdir.set_node(u"link", self._rootnode)) d.addCallback(lambda res: c.create_dirnode()) d.addCallback(lambda dn: @@ -1250,7 +1250,7 @@ class Dirnode(GridTestMixin, unittest.TestCase, nm = c.nodemaker filecap = make_chk_file_uri(1234) filenode = nm.create_from_cap(filecap) - uploadable = upload.Data("some data", convergence="some convergence string") + uploadable = upload.Data(b"some data", convergence=b"some convergence string") d = c.create_dirnode(version=version) def _created(rw_dn): @@ -1867,7 +1867,7 @@ class Deleter(GridTestMixin, testutil.ReallyEqualMixin, unittest.TestCase): self.set_up_grid(oneshare=True) c0 = self.g.clients[0] d = c0.create_dirnode() - small = upload.Data("Small enough for a LIT", None) + small = upload.Data(b"Small enough for a LIT", None) def _created_dir(dn): self.root = dn self.root_uri = dn.get_uri() @@ -1909,10 +1909,10 @@ class Adder(GridTestMixin, unittest.TestCase, testutil.ShouldFailMixin): # root/file1 # root/file2 # root/dir1 - d = root_node.add_file(u'file1', upload.Data("Important Things", + d = root_node.add_file(u'file1', upload.Data(b"Important Things", None)) d.addCallback(lambda res: - root_node.add_file(u'file2', upload.Data("Sekrit Codes", None))) + root_node.add_file(u'file2', upload.Data(b"Sekrit Codes", None))) d.addCallback(lambda res: root_node.create_subdirectory(u"dir1")) d.addCallback(lambda res: root_node) From 016240d6e6056601079cb80216e445b4a8f25c78 Mon Sep 17 00:00:00 2001 From: Itamar Turner-Trauring Date: Wed, 9 Dec 2020 13:50:15 -0500 Subject: [PATCH 24/54] More passing tests on Python 3. --- src/allmydata/dirnode.py | 6 ++--- src/allmydata/test/test_dirnode.py | 37 ++++++++++++++++++------------ 2 files changed, 25 insertions(+), 18 deletions(-) diff --git a/src/allmydata/dirnode.py b/src/allmydata/dirnode.py index 74d11a40f..137ff9699 100644 --- a/src/allmydata/dirnode.py +++ b/src/allmydata/dirnode.py @@ -468,7 +468,7 @@ class DirectoryNode(object): exists a child of the given name, False if not.""" name = normalize(namex) d = self._read() - d.addCallback(lambda children: children.has_key(name)) + d.addCallback(lambda children: name in children) return d def _get(self, children, name): @@ -569,7 +569,7 @@ class DirectoryNode(object): # this takes URIs a = Adder(self, overwrite=overwrite, create_readonly_node=self._create_readonly_node) - for (namex, e) in entries.iteritems(): + for (namex, e) in entries.items(): assert isinstance(namex, unicode), namex if len(e) == 2: writecap, readcap = e @@ -779,7 +779,7 @@ class DirectoryNode(object): # in the nodecache) seem to consume about 2000 bytes. dirkids = [] filekids = [] - for name, (child, metadata) in sorted(children.iteritems()): + for name, (child, metadata) in sorted(children.items()): childpath = path + [name] if isinstance(child, UnknownNode): walker.add_node(child, childpath) diff --git a/src/allmydata/test/test_dirnode.py b/src/allmydata/test/test_dirnode.py index 1845c3ac3..68e774501 100644 --- a/src/allmydata/test/test_dirnode.py +++ b/src/allmydata/test/test_dirnode.py @@ -1,4 +1,11 @@ """Tests for the dirnode module.""" +# from __future__ import absolute_import +# from __future__ import division +# from __future__ import print_function +# from __future__ import unicode_literals + +from past.builtins import unicode, long + import six import time import unicodedata @@ -95,13 +102,13 @@ class Dirnode(GridTestMixin, unittest.TestCase, self.failUnless(u) cap_formats = [] if mdmf: - cap_formats = ["URI:DIR2-MDMF:", - "URI:DIR2-MDMF-RO:", - "URI:DIR2-MDMF-Verifier:"] + cap_formats = [b"URI:DIR2-MDMF:", + b"URI:DIR2-MDMF-RO:", + b"URI:DIR2-MDMF-Verifier:"] else: - cap_formats = ["URI:DIR2:", - "URI:DIR2-RO", - "URI:DIR2-Verifier:"] + cap_formats = [b"URI:DIR2:", + b"URI:DIR2-RO", + b"URI:DIR2-Verifier:"] rw, ro, v = cap_formats self.failUnless(u.startswith(rw), u) u_ro = n.get_readonly_uri() @@ -149,7 +156,7 @@ class Dirnode(GridTestMixin, unittest.TestCase, self.failUnless(isinstance(subdir, dirnode.DirectoryNode)) self.subdir = subdir new_v = subdir.get_verify_cap().to_string() - assert isinstance(new_v, str) + assert isinstance(new_v, bytes) self.expected_manifest.append( ((u"subdir",), subdir.get_uri()) ) self.expected_verifycaps.add(new_v) si = subdir.get_storage_index() @@ -182,7 +189,7 @@ class Dirnode(GridTestMixin, unittest.TestCase, "largest-directory-children": 2, "largest-immutable-file": 0, } - for k,v in expected.iteritems(): + for k,v in expected.items(): self.failUnlessReallyEqual(stats[k], v, "stats[%s] was %s, not %s" % (k, stats[k], v)) @@ -272,8 +279,8 @@ class Dirnode(GridTestMixin, unittest.TestCase, { 'tahoe': {'linkcrtime': "bogus"}})) d.addCallback(lambda res: n.get_metadata_for(u"c2")) def _has_good_linkcrtime(metadata): - self.failUnless(metadata.has_key('tahoe')) - self.failUnless(metadata['tahoe'].has_key('linkcrtime')) + self.failUnless('tahoe' in metadata) + self.failUnless('linkcrtime' in metadata['tahoe']) self.failIfEqual(metadata['tahoe']['linkcrtime'], 'bogus') d.addCallback(_has_good_linkcrtime) @@ -918,7 +925,7 @@ class Dirnode(GridTestMixin, unittest.TestCase, self.cap = cap return dn.list() d.addCallback(_created_small) - d.addCallback(lambda kids: self.failUnlessReallyEqual(kids.keys(), [u"o"])) + d.addCallback(lambda kids: self.failUnlessReallyEqual(list(kids.keys()), [u"o"])) # now test n.create_subdirectory(mutable=False) d.addCallback(lambda ign: c.create_dirnode()) @@ -928,7 +935,7 @@ class Dirnode(GridTestMixin, unittest.TestCase, d.addCallback(_check_kids) d.addCallback(lambda ign: n.list()) d.addCallback(lambda children: - self.failUnlessReallyEqual(children.keys(), [u"subdir"])) + self.failUnlessReallyEqual(list(children.keys()), [u"subdir"])) d.addCallback(lambda ign: n.get(u"subdir")) d.addCallback(lambda sd: sd.list()) d.addCallback(_check_kids) @@ -1417,9 +1424,9 @@ class Packing(testutil.ReallyEqualMixin, unittest.TestCase): def _check_children(self, children): # Are all the expected child nodes there? - self.failUnless(children.has_key(u'file1')) - self.failUnless(children.has_key(u'file2')) - self.failUnless(children.has_key(u'file3')) + self.failUnless(u'file1' in children) + self.failUnless(u'file2' in children) + self.failUnless(u'file3' in children) # Are the metadata for child 3 right? file3_rocap = "URI:CHK:cmtcxq7hwxvfxan34yiev6ivhy:qvcekmjtoetdcw4kmi7b3rtblvgx7544crnwaqtiewemdliqsokq:3:10:5" From 59968d099cfa139cf10f4242af7818ecc5ad391c Mon Sep 17 00:00:00 2001 From: Itamar Turner-Trauring Date: Wed, 9 Dec 2020 14:05:03 -0500 Subject: [PATCH 25/54] More passing tests. --- src/allmydata/dirnode.py | 4 ++-- src/allmydata/test/test_dirnode.py | 36 +++++++++++++++--------------- src/allmydata/unknown.py | 4 ++-- 3 files changed, 22 insertions(+), 22 deletions(-) diff --git a/src/allmydata/dirnode.py b/src/allmydata/dirnode.py index 137ff9699..7b0c97540 100644 --- a/src/allmydata/dirnode.py +++ b/src/allmydata/dirnode.py @@ -354,7 +354,7 @@ class DirectoryNode(object): # pack("16ss32s", iv, AES(H(writekey+iv), plaintext_rw_uri), mac) assert isinstance(data, bytes), (repr(data), type(data)) # an empty directory is serialized as an empty string - if data == "": + if data == b"": return AuxValueDict() writeable = not self.is_readonly() mutable = self.is_mutable() @@ -373,7 +373,7 @@ class DirectoryNode(object): # Therefore we normalize names going both in and out of directories. name = normalize(namex_utf8.decode("utf-8")) - rw_uri = "" + rw_uri = b"" if writeable: rw_uri = self._decrypt_rwcapdata(rwcapdata) diff --git a/src/allmydata/test/test_dirnode.py b/src/allmydata/test/test_dirnode.py index 68e774501..23dcb3bbd 100644 --- a/src/allmydata/test/test_dirnode.py +++ b/src/allmydata/test/test_dirnode.py @@ -704,7 +704,7 @@ class Dirnode(GridTestMixin, unittest.TestCase, set([u"short"]))) d2.addCallback(lambda ignored: tinylit_node.list()) d2.addCallback(lambda children: children[u"short"][0].read(MemAccum())) - d2.addCallback(lambda accum: self.failUnlessReallyEqual(accum.data, "The end.")) + d2.addCallback(lambda accum: self.failUnlessReallyEqual(accum.data, b"The end.")) return d2 d.addCallback(_check_kids) @@ -789,7 +789,7 @@ class Dirnode(GridTestMixin, unittest.TestCase, rep = str(dn) self.failUnless("RO-IMM" in rep) cap = dn.get_cap() - self.failUnlessIn("CHK", cap.to_string()) + self.failUnlessIn(b"CHK", cap.to_string()) self.cap = cap return dn.list() d.addCallback(_created) @@ -901,8 +901,8 @@ class Dirnode(GridTestMixin, unittest.TestCase, rep = str(dn) self.failUnless("RO-IMM" in rep) cap = dn.get_cap() - self.failUnlessIn("LIT", cap.to_string()) - self.failUnlessReallyEqual(cap.to_string(), "URI:DIR2-LIT:") + self.failUnlessIn(b"LIT", cap.to_string()) + self.failUnlessReallyEqual(cap.to_string(), b"URI:DIR2-LIT:") self.cap = cap return dn.list() d.addCallback(_created_empty) @@ -919,9 +919,9 @@ class Dirnode(GridTestMixin, unittest.TestCase, rep = str(dn) self.failUnless("RO-IMM" in rep) cap = dn.get_cap() - self.failUnlessIn("LIT", cap.to_string()) + self.failUnlessIn(b"LIT", cap.to_string()) self.failUnlessReallyEqual(cap.to_string(), - "URI:DIR2-LIT:gi4tumj2n4wdcmz2kvjesosmjfkdu3rvpbtwwlbqhiwdeot3puwcy") + b"URI:DIR2-LIT:gi4tumj2n4wdcmz2kvjesosmjfkdu3rvpbtwwlbqhiwdeot3puwcy") self.cap = cap return dn.list() d.addCallback(_created_small) @@ -969,14 +969,14 @@ class Dirnode(GridTestMixin, unittest.TestCase, # It also tests that we store child names as UTF-8 NFC, and normalize # them again when retrieving them. - stripped_write_uri = "lafs://from_the_future\t" - stripped_read_uri = "lafs://readonly_from_the_future\t" - spacedout_write_uri = stripped_write_uri + " " - spacedout_read_uri = stripped_read_uri + " " + stripped_write_uri = b"lafs://from_the_future\t" + stripped_read_uri = b"lafs://readonly_from_the_future\t" + spacedout_write_uri = stripped_write_uri + b" " + spacedout_read_uri = stripped_read_uri + b" " child = nm.create_from_cap(spacedout_write_uri, spacedout_read_uri) self.failUnlessReallyEqual(child.get_write_uri(), spacedout_write_uri) - self.failUnlessReallyEqual(child.get_readonly_uri(), "ro." + spacedout_read_uri) + self.failUnlessReallyEqual(child.get_readonly_uri(), b"ro." + spacedout_read_uri) child_dottedi = u"ch\u0131\u0307ld" @@ -1393,7 +1393,7 @@ class Dirnode(GridTestMixin, unittest.TestCase, class MinimalFakeMutableFile(object): def get_writekey(self): - return "writekey" + return b"writekey" class Packing(testutil.ReallyEqualMixin, unittest.TestCase): # This is a base32-encoded representation of the directory tree @@ -1535,10 +1535,10 @@ class Packing(testutil.ReallyEqualMixin, unittest.TestCase): @implementer(IMutableFileNode) class FakeMutableFile(object): counter = 0 - def __init__(self, initial_contents=""): + def __init__(self, initial_contents=b""): data = self._get_initial_contents(initial_contents) self.data = data.read(data.get_size()) - self.data = "".join(self.data) + self.data = b"".join(self.data) counter = FakeMutableFile.counter FakeMutableFile.counter += 1 @@ -1547,10 +1547,10 @@ class FakeMutableFile(object): self.uri = uri.WriteableSSKFileURI(writekey, fingerprint) def _get_initial_contents(self, contents): - if isinstance(contents, str): + if isinstance(contents, bytes): return contents if contents is None: - return "" + return b"" assert callable(contents), "%s should be callable, not %s" % \ (contents, type(contents)) return contents(self) @@ -1568,7 +1568,7 @@ class FakeMutableFile(object): return defer.succeed(self.data) def get_writekey(self): - return "writekey" + return b"writekey" def is_readonly(self): return False @@ -1591,7 +1591,7 @@ class FakeMutableFile(object): return defer.succeed(None) class FakeNodeMaker(NodeMaker): - def create_mutable_file(self, contents="", keysize=None, version=None): + def create_mutable_file(self, contents=b"", keysize=None, version=None): return defer.succeed(FakeMutableFile(contents)) class FakeClient2(_Client): diff --git a/src/allmydata/unknown.py b/src/allmydata/unknown.py index 6c970e484..34d084136 100644 --- a/src/allmydata/unknown.py +++ b/src/allmydata/unknown.py @@ -31,8 +31,8 @@ class UnknownNode(object): def __init__(self, given_rw_uri, given_ro_uri, deep_immutable=False, name=u""): - assert given_rw_uri is None or isinstance(given_rw_uri, str) - assert given_ro_uri is None or isinstance(given_ro_uri, str) + assert given_rw_uri is None or isinstance(given_rw_uri, bytes) + assert given_ro_uri is None or isinstance(given_ro_uri, bytes) given_rw_uri = given_rw_uri or None given_ro_uri = given_ro_uri or None From ff6443228246276f2da454fb9299fe783e0ac243 Mon Sep 17 00:00:00 2001 From: Itamar Turner-Trauring Date: Wed, 9 Dec 2020 14:14:07 -0500 Subject: [PATCH 26/54] More passing on Python 3.. --- src/allmydata/test/test_dirnode.py | 160 ++++++++++++++--------------- 1 file changed, 80 insertions(+), 80 deletions(-) diff --git a/src/allmydata/test/test_dirnode.py b/src/allmydata/test/test_dirnode.py index 23dcb3bbd..b3deb9df8 100644 --- a/src/allmydata/test/test_dirnode.py +++ b/src/allmydata/test/test_dirnode.py @@ -673,22 +673,22 @@ class Dirnode(GridTestMixin, unittest.TestCase, self.failUnless(fut_node.is_unknown()) self.failUnlessReallyEqual(fut_node.get_uri(), future_write_uri) - self.failUnlessReallyEqual(fut_node.get_readonly_uri(), "ro." + future_read_uri) + self.failUnlessReallyEqual(fut_node.get_readonly_uri(), b"ro." + future_read_uri) self.failUnless(isinstance(fut_metadata, dict), fut_metadata) self.failUnless(futna_node.is_unknown()) self.failUnlessReallyEqual(futna_node.get_uri(), future_nonascii_write_uri) - self.failUnlessReallyEqual(futna_node.get_readonly_uri(), "ro." + future_nonascii_read_uri) + self.failUnlessReallyEqual(futna_node.get_readonly_uri(), b"ro." + future_nonascii_read_uri) self.failUnless(isinstance(futna_metadata, dict), futna_metadata) self.failUnless(fro_node.is_unknown()) - self.failUnlessReallyEqual(fro_node.get_uri(), "ro." + future_read_uri) - self.failUnlessReallyEqual(fut_node.get_readonly_uri(), "ro." + future_read_uri) + self.failUnlessReallyEqual(fro_node.get_uri(), b"ro." + future_read_uri) + self.failUnlessReallyEqual(fut_node.get_readonly_uri(), b"ro." + future_read_uri) self.failUnless(isinstance(fro_metadata, dict), fro_metadata) self.failUnless(frona_node.is_unknown()) - self.failUnlessReallyEqual(frona_node.get_uri(), "ro." + future_nonascii_read_uri) - self.failUnlessReallyEqual(futna_node.get_readonly_uri(), "ro." + future_nonascii_read_uri) + self.failUnlessReallyEqual(frona_node.get_uri(), b"ro." + future_nonascii_read_uri) + self.failUnlessReallyEqual(futna_node.get_readonly_uri(), b"ro." + future_nonascii_read_uri) self.failUnless(isinstance(frona_metadata, dict), frona_metadata) self.failIf(emptylit_node.is_unknown()) @@ -815,13 +815,13 @@ class Dirnode(GridTestMixin, unittest.TestCase, self.failUnlessEqual(two_metadata["metakey"], "metavalue") self.failUnless(fut_node.is_unknown()) - self.failUnlessReallyEqual(fut_node.get_uri(), "imm." + future_read_uri) - self.failUnlessReallyEqual(fut_node.get_readonly_uri(), "imm." + future_read_uri) + self.failUnlessReallyEqual(fut_node.get_uri(), b"imm." + future_read_uri) + self.failUnlessReallyEqual(fut_node.get_readonly_uri(), b"imm." + future_read_uri) self.failUnless(isinstance(fut_metadata, dict), fut_metadata) self.failUnless(futna_node.is_unknown()) - self.failUnlessReallyEqual(futna_node.get_uri(), "imm." + future_nonascii_read_uri) - self.failUnlessReallyEqual(futna_node.get_readonly_uri(), "imm." + future_nonascii_read_uri) + self.failUnlessReallyEqual(futna_node.get_uri(), b"imm." + future_nonascii_read_uri) + self.failUnlessReallyEqual(futna_node.get_readonly_uri(), b"imm." + future_nonascii_read_uri) self.failUnless(isinstance(futna_metadata, dict), futna_metadata) self.failIf(emptylit_node.is_unknown()) @@ -837,7 +837,7 @@ class Dirnode(GridTestMixin, unittest.TestCase, set([u"short"]))) d2.addCallback(lambda ignored: tinylit_node.list()) d2.addCallback(lambda children: children[u"short"][0].read(MemAccum())) - d2.addCallback(lambda accum: self.failUnlessReallyEqual(accum.data, "The end.")) + d2.addCallback(lambda accum: self.failUnlessReallyEqual(accum.data, b"The end.")) return d2 d.addCallback(_check_kids) @@ -1010,7 +1010,7 @@ class Dirnode(GridTestMixin, unittest.TestCase, self.failUnlessIn(name, kids_out) (expected_child, ign) = kids_out[name] self.failUnlessReallyEqual(rw_uri, expected_child.get_write_uri()) - self.failUnlessReallyEqual("ro." + ro_uri, expected_child.get_readonly_uri()) + self.failUnlessReallyEqual(b"ro." + ro_uri, expected_child.get_readonly_uri()) numkids += 1 self.failUnlessReallyEqual(numkids, len(kids_out)) @@ -1046,7 +1046,7 @@ class Dirnode(GridTestMixin, unittest.TestCase, child_node, child_metadata = children[u"child"] self.failUnlessReallyEqual(child_node.get_write_uri(), stripped_write_uri) - self.failUnlessReallyEqual(child_node.get_readonly_uri(), "ro." + stripped_read_uri) + self.failUnlessReallyEqual(child_node.get_readonly_uri(), b"ro." + stripped_read_uri) d.addCallback(_check_kids) d.addCallback(lambda ign: nm.create_from_cap(self.cap.to_string())) @@ -1459,12 +1459,12 @@ class Packing(testutil.ReallyEqualMixin, unittest.TestCase): children[u'file1'][0].get_uri()) def _make_kids(self, nm, which): - caps = {"imm": "URI:CHK:n7r3m6wmomelk4sep3kw5cvduq:os7ijw5c3maek7pg65e5254k2fzjflavtpejjyhshpsxuqzhcwwq:3:20:14861", - "lit": "URI:LIT:n5xgk", # LIT for "one" - "write": "URI:SSK:vfvcbdfbszyrsaxchgevhmmlii:euw4iw7bbnkrrwpzuburbhppuxhc3gwxv26f6imekhz7zyw2ojnq", - "read": "URI:SSK-RO:e3mdrzfwhoq42hy5ubcz6rp3o4:ybyibhnp3vvwuq2vaw2ckjmesgkklfs6ghxleztqidihjyofgw7q", - "dirwrite": "URI:DIR2:n6x24zd3seu725yluj75q5boaa:mm6yoqjhl6ueh7iereldqxue4nene4wl7rqfjfybqrehdqmqskvq", - "dirread": "URI:DIR2-RO:b7sr5qsifnicca7cbk3rhrhbvq:mm6yoqjhl6ueh7iereldqxue4nene4wl7rqfjfybqrehdqmqskvq", + caps = {"imm": b"URI:CHK:n7r3m6wmomelk4sep3kw5cvduq:os7ijw5c3maek7pg65e5254k2fzjflavtpejjyhshpsxuqzhcwwq:3:20:14861", + "lit": b"URI:LIT:n5xgk", # LIT for "one" + "write": b"URI:SSK:vfvcbdfbszyrsaxchgevhmmlii:euw4iw7bbnkrrwpzuburbhppuxhc3gwxv26f6imekhz7zyw2ojnq", + "read": b"URI:SSK-RO:e3mdrzfwhoq42hy5ubcz6rp3o4:ybyibhnp3vvwuq2vaw2ckjmesgkklfs6ghxleztqidihjyofgw7q", + "dirwrite": b"URI:DIR2:n6x24zd3seu725yluj75q5boaa:mm6yoqjhl6ueh7iereldqxue4nene4wl7rqfjfybqrehdqmqskvq", + "dirread": b"URI:DIR2-RO:b7sr5qsifnicca7cbk3rhrhbvq:mm6yoqjhl6ueh7iereldqxue4nene4wl7rqfjfybqrehdqmqskvq", } kids = {} for name in which: @@ -1542,8 +1542,8 @@ class FakeMutableFile(object): counter = FakeMutableFile.counter FakeMutableFile.counter += 1 - writekey = hashutil.ssk_writekey_hash(str(counter)) - fingerprint = hashutil.ssk_pubkey_fingerprint_hash(str(counter)) + writekey = hashutil.ssk_writekey_hash(b"%d" % counter) + fingerprint = hashutil.ssk_pubkey_fingerprint_hash(b"%d" % counter) self.uri = uri.WriteableSSKFileURI(writekey, fingerprint) def _get_initial_contents(self, contents): @@ -1638,9 +1638,9 @@ class Dirnode2(testutil.ReallyEqualMixin, testutil.ShouldFailMixin, unittest.Tes # and to add an URI prefixed with "ro." or "imm." when it is given in a # write slot (or URL parameter). d.addCallback(lambda ign: self._node.set_uri(u"add-ro", - "ro." + future_read_uri, None)) + b"ro." + future_read_uri, None)) d.addCallback(lambda ign: self._node.set_uri(u"add-imm", - "imm." + future_imm_uri, None)) + b"imm." + future_imm_uri, None)) d.addCallback(lambda ign: self._node.list()) def _check(children): @@ -1649,25 +1649,25 @@ class Dirnode2(testutil.ReallyEqualMixin, testutil.ShouldFailMixin, unittest.Tes self.failUnless(isinstance(fn, UnknownNode), fn) self.failUnlessReallyEqual(fn.get_uri(), future_write_uri) self.failUnlessReallyEqual(fn.get_write_uri(), future_write_uri) - self.failUnlessReallyEqual(fn.get_readonly_uri(), "ro." + future_read_uri) + self.failUnlessReallyEqual(fn.get_readonly_uri(), b"ro." + future_read_uri) (fn2, metadata2) = children[u"add-pair"] self.failUnless(isinstance(fn2, UnknownNode), fn2) self.failUnlessReallyEqual(fn2.get_uri(), future_write_uri) self.failUnlessReallyEqual(fn2.get_write_uri(), future_write_uri) - self.failUnlessReallyEqual(fn2.get_readonly_uri(), "ro." + future_read_uri) + self.failUnlessReallyEqual(fn2.get_readonly_uri(), b"ro." + future_read_uri) (fn3, metadata3) = children[u"add-ro"] self.failUnless(isinstance(fn3, UnknownNode), fn3) - self.failUnlessReallyEqual(fn3.get_uri(), "ro." + future_read_uri) + self.failUnlessReallyEqual(fn3.get_uri(), b"ro." + future_read_uri) self.failUnlessReallyEqual(fn3.get_write_uri(), None) - self.failUnlessReallyEqual(fn3.get_readonly_uri(), "ro." + future_read_uri) + self.failUnlessReallyEqual(fn3.get_readonly_uri(), b"ro." + future_read_uri) (fn4, metadata4) = children[u"add-imm"] self.failUnless(isinstance(fn4, UnknownNode), fn4) - self.failUnlessReallyEqual(fn4.get_uri(), "imm." + future_imm_uri) + self.failUnlessReallyEqual(fn4.get_uri(), b"imm." + future_imm_uri) self.failUnlessReallyEqual(fn4.get_write_uri(), None) - self.failUnlessReallyEqual(fn4.get_readonly_uri(), "imm." + future_imm_uri) + self.failUnlessReallyEqual(fn4.get_readonly_uri(), b"imm." + future_imm_uri) # We should also be allowed to copy the "future" UnknownNode, because # it contains all the information that was in the original directory @@ -1682,17 +1682,17 @@ class Dirnode2(testutil.ReallyEqualMixin, testutil.ShouldFailMixin, unittest.Tes self.failUnless(isinstance(fn, UnknownNode), fn) self.failUnlessReallyEqual(fn.get_uri(), future_write_uri) self.failUnlessReallyEqual(fn.get_write_uri(), future_write_uri) - self.failUnlessReallyEqual(fn.get_readonly_uri(), "ro." + future_read_uri) + self.failUnlessReallyEqual(fn.get_readonly_uri(), b"ro." + future_read_uri) d.addCallback(_check2) return d def test_unknown_strip_prefix_for_ro(self): - self.failUnlessReallyEqual(strip_prefix_for_ro("foo", False), "foo") - self.failUnlessReallyEqual(strip_prefix_for_ro("ro.foo", False), "foo") - self.failUnlessReallyEqual(strip_prefix_for_ro("imm.foo", False), "imm.foo") - self.failUnlessReallyEqual(strip_prefix_for_ro("foo", True), "foo") - self.failUnlessReallyEqual(strip_prefix_for_ro("ro.foo", True), "foo") - self.failUnlessReallyEqual(strip_prefix_for_ro("imm.foo", True), "foo") + self.failUnlessReallyEqual(strip_prefix_for_ro(b"foo", False), b"foo") + self.failUnlessReallyEqual(strip_prefix_for_ro(b"ro.foo", False), b"foo") + self.failUnlessReallyEqual(strip_prefix_for_ro(b"imm.foo", False), b"imm.foo") + self.failUnlessReallyEqual(strip_prefix_for_ro(b"foo", True), b"foo") + self.failUnlessReallyEqual(strip_prefix_for_ro(b"ro.foo", True), b"foo") + self.failUnlessReallyEqual(strip_prefix_for_ro(b"imm.foo", True), b"foo") def test_unknownnode(self): lit_uri = one_uri @@ -1704,58 +1704,58 @@ class Dirnode2(testutil.ReallyEqualMixin, testutil.ShouldFailMixin, unittest.Tes ] unknown_rw = [# These are errors because we're only given a rw_uri, and we can't # diminish it. - ( 2, UnknownNode("foo", None)), - ( 3, UnknownNode("foo", None, deep_immutable=True)), - ( 4, UnknownNode("ro.foo", None, deep_immutable=True)), - ( 5, UnknownNode("ro." + mut_read_uri, None, deep_immutable=True)), - ( 5.1, UnknownNode("ro." + mdmf_read_uri, None, deep_immutable=True)), - ( 6, UnknownNode("URI:SSK-RO:foo", None, deep_immutable=True)), - ( 7, UnknownNode("URI:SSK:foo", None)), + ( 2, UnknownNode(b"foo", None)), + ( 3, UnknownNode(b"foo", None, deep_immutable=True)), + ( 4, UnknownNode(b"ro.foo", None, deep_immutable=True)), + ( 5, UnknownNode(b"ro." + mut_read_uri, None, deep_immutable=True)), + ( 5.1, UnknownNode(b"ro." + mdmf_read_uri, None, deep_immutable=True)), + ( 6, UnknownNode(b"URI:SSK-RO:foo", None, deep_immutable=True)), + ( 7, UnknownNode(b"URI:SSK:foo", None)), ] must_be_ro = [# These are errors because a readonly constraint is not met. - ( 8, UnknownNode("ro." + mut_write_uri, None)), - ( 8.1, UnknownNode("ro." + mdmf_write_uri, None)), - ( 9, UnknownNode(None, "ro." + mut_write_uri)), - ( 9.1, UnknownNode(None, "ro." + mdmf_write_uri)), + ( 8, UnknownNode(b"ro." + mut_write_uri, None)), + ( 8.1, UnknownNode(b"ro." + mdmf_write_uri, None)), + ( 9, UnknownNode(None, b"ro." + mut_write_uri)), + ( 9.1, UnknownNode(None, b"ro." + mdmf_write_uri)), ] must_be_imm = [# These are errors because an immutable constraint is not met. - (10, UnknownNode(None, "ro.URI:SSK-RO:foo", deep_immutable=True)), - (11, UnknownNode(None, "imm.URI:SSK:foo")), - (12, UnknownNode(None, "imm.URI:SSK-RO:foo")), - (13, UnknownNode("bar", "ro.foo", deep_immutable=True)), - (14, UnknownNode("bar", "imm.foo", deep_immutable=True)), - (15, UnknownNode("bar", "imm." + lit_uri, deep_immutable=True)), - (16, UnknownNode("imm." + mut_write_uri, None)), - (16.1, UnknownNode("imm." + mdmf_write_uri, None)), - (17, UnknownNode("imm." + mut_read_uri, None)), - (17.1, UnknownNode("imm." + mdmf_read_uri, None)), - (18, UnknownNode("bar", "imm.foo")), + (10, UnknownNode(None, b"ro.URI:SSK-RO:foo", deep_immutable=True)), + (11, UnknownNode(None, b"imm.URI:SSK:foo")), + (12, UnknownNode(None, b"imm.URI:SSK-RO:foo")), + (13, UnknownNode(b"bar", b"ro.foo", deep_immutable=True)), + (14, UnknownNode(b"bar", b"imm.foo", deep_immutable=True)), + (15, UnknownNode(b"bar", b"imm." + lit_uri, deep_immutable=True)), + (16, UnknownNode(b"imm." + mut_write_uri, None)), + (16.1, UnknownNode(b"imm." + mdmf_write_uri, None)), + (17, UnknownNode(b"imm." + mut_read_uri, None)), + (17.1, UnknownNode(b"imm." + mdmf_read_uri, None)), + (18, UnknownNode(b"bar", b"imm.foo")), ] bad_uri = [# These are errors because the URI is bad once we've stripped the prefix. - (19, UnknownNode("ro.URI:SSK-RO:foo", None)), - (20, UnknownNode("imm.URI:CHK:foo", None, deep_immutable=True)), - (21, UnknownNode(None, "URI:CHK:foo")), - (22, UnknownNode(None, "URI:CHK:foo", deep_immutable=True)), + (19, UnknownNode(b"ro.URI:SSK-RO:foo", None)), + (20, UnknownNode(b"imm.URI:CHK:foo", None, deep_immutable=True)), + (21, UnknownNode(None, b"URI:CHK:foo")), + (22, UnknownNode(None, b"URI:CHK:foo", deep_immutable=True)), ] ro_prefixed = [# These are valid, and the readcap should end up with a ro. prefix. - (23, UnknownNode(None, "foo")), - (24, UnknownNode(None, "ro.foo")), - (25, UnknownNode(None, "ro." + lit_uri)), - (26, UnknownNode("bar", "foo")), - (27, UnknownNode("bar", "ro.foo")), - (28, UnknownNode("bar", "ro." + lit_uri)), - (29, UnknownNode("ro.foo", None)), - (30, UnknownNode("ro." + lit_uri, None)), + (23, UnknownNode(None, b"foo")), + (24, UnknownNode(None, b"ro.foo")), + (25, UnknownNode(None, b"ro." + lit_uri)), + (26, UnknownNode(b"bar", b"foo")), + (27, UnknownNode(b"bar", b"ro.foo")), + (28, UnknownNode(b"bar", b"ro." + lit_uri)), + (29, UnknownNode(b"ro.foo", None)), + (30, UnknownNode(b"ro." + lit_uri, None)), ] imm_prefixed = [# These are valid, and the readcap should end up with an imm. prefix. - (31, UnknownNode(None, "foo", deep_immutable=True)), - (32, UnknownNode(None, "ro.foo", deep_immutable=True)), - (33, UnknownNode(None, "imm.foo")), - (34, UnknownNode(None, "imm.foo", deep_immutable=True)), - (35, UnknownNode("imm." + lit_uri, None)), - (36, UnknownNode("imm." + lit_uri, None, deep_immutable=True)), - (37, UnknownNode(None, "imm." + lit_uri)), - (38, UnknownNode(None, "imm." + lit_uri, deep_immutable=True)), + (31, UnknownNode(None, b"foo", deep_immutable=True)), + (32, UnknownNode(None, b"ro.foo", deep_immutable=True)), + (33, UnknownNode(None, b"imm.foo")), + (34, UnknownNode(None, b"imm.foo", deep_immutable=True)), + (35, UnknownNode(b"imm." + lit_uri, None)), + (36, UnknownNode(b"imm." + lit_uri, None, deep_immutable=True)), + (37, UnknownNode(None, b"imm." + lit_uri)), + (38, UnknownNode(None, b"imm." + lit_uri, deep_immutable=True)), ] error = unknown_rw + must_be_ro + must_be_imm + bad_uri ok = ro_prefixed + imm_prefixed @@ -1787,10 +1787,10 @@ class Dirnode2(testutil.ReallyEqualMixin, testutil.ShouldFailMixin, unittest.Tes self.failIf(n.get_readonly_uri() is None, i) for (i, n) in ro_prefixed: - self.failUnless(n.get_readonly_uri().startswith("ro."), i) + self.failUnless(n.get_readonly_uri().startswith(b"ro."), i) for (i, n) in imm_prefixed: - self.failUnless(n.get_readonly_uri().startswith("imm."), i) + self.failUnless(n.get_readonly_uri().startswith(b"imm."), i) From b1800c457dc4f598585ecb3753413ecff968734f Mon Sep 17 00:00:00 2001 From: Itamar Turner-Trauring Date: Wed, 9 Dec 2020 14:33:56 -0500 Subject: [PATCH 27/54] All tests pass on Python 3. --- src/allmydata/test/test_dirnode.py | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/src/allmydata/test/test_dirnode.py b/src/allmydata/test/test_dirnode.py index b3deb9df8..f37926fa4 100644 --- a/src/allmydata/test/test_dirnode.py +++ b/src/allmydata/test/test_dirnode.py @@ -1412,7 +1412,7 @@ class Packing(testutil.ReallyEqualMixin, unittest.TestCase): nodemaker = NodeMaker(None, None, None, None, None, {"k": 3, "n": 10}, None, None) - write_uri = "URI:SSK-RO:e3mdrzfwhoq42hy5ubcz6rp3o4:ybyibhnp3vvwuq2vaw2ckjmesgkklfs6ghxleztqidihjyofgw7q" + write_uri = b"URI:SSK-RO:e3mdrzfwhoq42hy5ubcz6rp3o4:ybyibhnp3vvwuq2vaw2ckjmesgkklfs6ghxleztqidihjyofgw7q" filenode = nodemaker.create_from_cap(write_uri) node = dirnode.DirectoryNode(filenode, nodemaker, None) children = node._unpack_contents(known_tree) @@ -1429,8 +1429,8 @@ class Packing(testutil.ReallyEqualMixin, unittest.TestCase): self.failUnless(u'file3' in children) # Are the metadata for child 3 right? - file3_rocap = "URI:CHK:cmtcxq7hwxvfxan34yiev6ivhy:qvcekmjtoetdcw4kmi7b3rtblvgx7544crnwaqtiewemdliqsokq:3:10:5" - file3_rwcap = "URI:CHK:cmtcxq7hwxvfxan34yiev6ivhy:qvcekmjtoetdcw4kmi7b3rtblvgx7544crnwaqtiewemdliqsokq:3:10:5" + file3_rocap = b"URI:CHK:cmtcxq7hwxvfxan34yiev6ivhy:qvcekmjtoetdcw4kmi7b3rtblvgx7544crnwaqtiewemdliqsokq:3:10:5" + file3_rwcap = b"URI:CHK:cmtcxq7hwxvfxan34yiev6ivhy:qvcekmjtoetdcw4kmi7b3rtblvgx7544crnwaqtiewemdliqsokq:3:10:5" file3_metadata = {'ctime': 1246663897.4336269, 'tahoe': {'linkmotime': 1246663897.4336269, 'linkcrtime': 1246663897.4336269}, 'mtime': 1246663897.4336269} self.failUnlessEqual(file3_metadata, children[u'file3'][1]) self.failUnlessReallyEqual(file3_rocap, @@ -1439,8 +1439,8 @@ class Packing(testutil.ReallyEqualMixin, unittest.TestCase): children[u'file3'][0].get_uri()) # Are the metadata for child 2 right? - file2_rocap = "URI:CHK:apegrpehshwugkbh3jlt5ei6hq:5oougnemcl5xgx4ijgiumtdojlipibctjkbwvyygdymdphib2fvq:3:10:4" - file2_rwcap = "URI:CHK:apegrpehshwugkbh3jlt5ei6hq:5oougnemcl5xgx4ijgiumtdojlipibctjkbwvyygdymdphib2fvq:3:10:4" + file2_rocap = b"URI:CHK:apegrpehshwugkbh3jlt5ei6hq:5oougnemcl5xgx4ijgiumtdojlipibctjkbwvyygdymdphib2fvq:3:10:4" + file2_rwcap = b"URI:CHK:apegrpehshwugkbh3jlt5ei6hq:5oougnemcl5xgx4ijgiumtdojlipibctjkbwvyygdymdphib2fvq:3:10:4" file2_metadata = {'ctime': 1246663897.430218, 'tahoe': {'linkmotime': 1246663897.430218, 'linkcrtime': 1246663897.430218}, 'mtime': 1246663897.430218} self.failUnlessEqual(file2_metadata, children[u'file2'][1]) self.failUnlessReallyEqual(file2_rocap, @@ -1449,8 +1449,8 @@ class Packing(testutil.ReallyEqualMixin, unittest.TestCase): children[u'file2'][0].get_uri()) # Are the metadata for child 1 right? - file1_rocap = "URI:CHK:olxtimympo7f27jvhtgqlnbtn4:emzdnhk2um4seixozlkw3qx2nfijvdkx3ky7i7izl47yedl6e64a:3:10:10" - file1_rwcap = "URI:CHK:olxtimympo7f27jvhtgqlnbtn4:emzdnhk2um4seixozlkw3qx2nfijvdkx3ky7i7izl47yedl6e64a:3:10:10" + file1_rocap = b"URI:CHK:olxtimympo7f27jvhtgqlnbtn4:emzdnhk2um4seixozlkw3qx2nfijvdkx3ky7i7izl47yedl6e64a:3:10:10" + file1_rwcap = b"URI:CHK:olxtimympo7f27jvhtgqlnbtn4:emzdnhk2um4seixozlkw3qx2nfijvdkx3ky7i7izl47yedl6e64a:3:10:10" file1_metadata = {'ctime': 1246663897.4275661, 'tahoe': {'linkmotime': 1246663897.4275661, 'linkcrtime': 1246663897.4275661}, 'mtime': 1246663897.4275661} self.failUnlessEqual(file1_metadata, children[u'file1'][1]) self.failUnlessReallyEqual(file1_rocap, @@ -1492,7 +1492,7 @@ class Packing(testutil.ReallyEqualMixin, unittest.TestCase): name: (LiteralFileNode(uri.from_string(one_uri)), {}), } packed = dirnode.pack_children(kids, fn.get_writekey(), deep_immutable=False) - write_uri = "URI:SSK-RO:e3mdrzfwhoq42hy5ubcz6rp3o4:ybyibhnp3vvwuq2vaw2ckjmesgkklfs6ghxleztqidihjyofgw7q" + write_uri = b"URI:SSK-RO:e3mdrzfwhoq42hy5ubcz6rp3o4:ybyibhnp3vvwuq2vaw2ckjmesgkklfs6ghxleztqidihjyofgw7q" filenode = nm.create_from_cap(write_uri) dn = dirnode.DirectoryNode(filenode, nm, None) unkids = dn._unpack_contents(packed) @@ -1505,11 +1505,11 @@ class Packing(testutil.ReallyEqualMixin, unittest.TestCase): kids = self._make_kids(nm, ["imm", "lit", "write", "read", "dirwrite", "dirread"]) packed = dirnode.pack_children(kids, fn.get_writekey(), deep_immutable=False) - self.failUnlessIn("lit", packed) + self.failUnlessIn(b"lit", packed) kids = self._make_kids(nm, ["imm", "lit"]) packed = dirnode.pack_children(kids, fn.get_writekey(), deep_immutable=True) - self.failUnlessIn("lit", packed) + self.failUnlessIn(b"lit", packed) kids = self._make_kids(nm, ["imm", "lit", "write"]) self.failUnlessRaises(dirnode.MustBeDeepImmutableError, From 5cba8a438008387d52eecd9942a715ba0fee2b87 Mon Sep 17 00:00:00 2001 From: Itamar Turner-Trauring Date: Wed, 9 Dec 2020 14:45:07 -0500 Subject: [PATCH 28/54] Port to Python 3. --- src/allmydata/test/test_dirnode.py | 22 +++++++++++++++------- src/allmydata/util/_python3.py | 1 + 2 files changed, 16 insertions(+), 7 deletions(-) diff --git a/src/allmydata/test/test_dirnode.py b/src/allmydata/test/test_dirnode.py index f37926fa4..586d89731 100644 --- a/src/allmydata/test/test_dirnode.py +++ b/src/allmydata/test/test_dirnode.py @@ -1,10 +1,18 @@ -"""Tests for the dirnode module.""" -# from __future__ import absolute_import -# from __future__ import division -# from __future__ import print_function -# from __future__ import unicode_literals +"""Tests for the dirnode module. -from past.builtins import unicode, long +Ported to Python 3. +""" +from __future__ import absolute_import +from __future__ import division +from __future__ import print_function +from __future__ import unicode_literals + +from past.builtins import long + +from future.utils import PY2 +if PY2: + # Skip list() since it results in spurious test failures + from future.builtins import filter, map, zip, ascii, chr, hex, input, next, oct, open, pow, round, super, bytes, dict, object, range, str, max, min # noqa: F401 import six import time @@ -1468,7 +1476,7 @@ class Packing(testutil.ReallyEqualMixin, unittest.TestCase): } kids = {} for name in which: - kids[unicode(name)] = (nm.create_from_cap(caps[name]), {}) + kids[str(name)] = (nm.create_from_cap(caps[name]), {}) return kids @given(text(min_size=1, max_size=20)) diff --git a/src/allmydata/util/_python3.py b/src/allmydata/util/_python3.py index 9763c35d7..7b454cbb7 100644 --- a/src/allmydata/util/_python3.py +++ b/src/allmydata/util/_python3.py @@ -134,6 +134,7 @@ PORTED_TEST_MODULES = [ "allmydata.test.test_crypto", "allmydata.test.test_deferredutil", "allmydata.test.test_dictutil", + "allmydata.test.test_dirnode", "allmydata.test.test_download", "allmydata.test.test_encode", "allmydata.test.test_encodingutil", From bb06067c33d6a226fbc5d7233e309db63ccf27e5 Mon Sep 17 00:00:00 2001 From: Itamar Turner-Trauring Date: Wed, 9 Dec 2020 14:48:33 -0500 Subject: [PATCH 29/54] Port to Python 3. --- src/allmydata/dirnode.py | 21 +++++++++++++++++---- src/allmydata/util/_python3.py | 1 + 2 files changed, 18 insertions(+), 4 deletions(-) diff --git a/src/allmydata/dirnode.py b/src/allmydata/dirnode.py index 7b0c97540..6c4af7461 100644 --- a/src/allmydata/dirnode.py +++ b/src/allmydata/dirnode.py @@ -1,4 +1,15 @@ -"""Directory Node implementation.""" +"""Directory Node implementation. + +Ported to Python 3. +""" +from __future__ import absolute_import +from __future__ import division +from __future__ import print_function +from __future__ import unicode_literals + +from future.utils import PY2 +if PY2: + from future.builtins import filter, map, zip, ascii, chr, hex, input, next, oct, open, pow, round, super, bytes, dict, list, object, range, str, max, min # noqa: F401 from past.builtins import unicode import time @@ -37,6 +48,8 @@ from eliot.twisted import ( NAME = Field.for_types( u"name", + # Make sure this works on Python 2; with str, it gets Future str which + # breaks Eliot. [unicode], u"The name linking the parent to this node.", ) @@ -250,7 +263,7 @@ def _pack_normalized_children(children, writekey, deep_immutable=False): has_aux = isinstance(children, AuxValueDict) entries = [] for name in sorted(children.keys()): - assert isinstance(name, unicode) + assert isinstance(name, str) entry = None (child, metadata) = children[name] child.raise_error() @@ -543,7 +556,7 @@ class DirectoryNode(object): else: pathx = pathx.split("/") for p in pathx: - assert isinstance(p, unicode), p + assert isinstance(p, str), p childnamex = pathx[0] remaining_pathx = pathx[1:] if remaining_pathx: @@ -570,7 +583,7 @@ class DirectoryNode(object): a = Adder(self, overwrite=overwrite, create_readonly_node=self._create_readonly_node) for (namex, e) in entries.items(): - assert isinstance(namex, unicode), namex + assert isinstance(namex, str), namex if len(e) == 2: writecap, readcap = e metadata = None diff --git a/src/allmydata/util/_python3.py b/src/allmydata/util/_python3.py index 7b454cbb7..79f09af02 100644 --- a/src/allmydata/util/_python3.py +++ b/src/allmydata/util/_python3.py @@ -34,6 +34,7 @@ PORTED_MODULES = [ "allmydata.crypto.error", "allmydata.crypto.rsa", "allmydata.crypto.util", + "allmydata.dirnode", "allmydata.hashtree", "allmydata.immutable.downloader", "allmydata.immutable.downloader.common", From ba424837419482de7498731c06c5eb56f767dadc Mon Sep 17 00:00:00 2001 From: Itamar Turner-Trauring Date: Wed, 9 Dec 2020 14:50:28 -0500 Subject: [PATCH 30/54] News file. --- newsfragments/3553.minor | 0 1 file changed, 0 insertions(+), 0 deletions(-) create mode 100644 newsfragments/3553.minor diff --git a/newsfragments/3553.minor b/newsfragments/3553.minor new file mode 100644 index 000000000..e69de29bb From 96fd1861d2ffb35c9bc81230ab6f57c3aff493b3 Mon Sep 17 00:00:00 2001 From: Itamar Turner-Trauring Date: Wed, 9 Dec 2020 14:58:54 -0500 Subject: [PATCH 31/54] Port to Python 3. --- src/allmydata/dirnode.py | 8 +++++--- src/allmydata/nodemaker.py | 12 ++++++++++++ src/allmydata/test/test_dirnode.py | 4 ---- src/allmydata/util/_python3.py | 1 + 4 files changed, 18 insertions(+), 7 deletions(-) diff --git a/src/allmydata/dirnode.py b/src/allmydata/dirnode.py index 6c4af7461..fd6a9cc8c 100644 --- a/src/allmydata/dirnode.py +++ b/src/allmydata/dirnode.py @@ -9,7 +9,8 @@ from __future__ import unicode_literals from future.utils import PY2 if PY2: - from future.builtins import filter, map, zip, ascii, chr, hex, input, next, oct, open, pow, round, super, bytes, dict, list, object, range, str, max, min # noqa: F401 + # Skip dict so it doesn't break things. + from future.builtins import filter, map, zip, ascii, chr, hex, input, next, oct, open, pow, round, super, bytes, list, object, range, str, max, min # noqa: F401 from past.builtins import unicode import time @@ -268,8 +269,9 @@ def _pack_normalized_children(children, writekey, deep_immutable=False): (child, metadata) = children[name] child.raise_error() if deep_immutable and not child.is_allowed_in_immutable_directory(): - raise MustBeDeepImmutableError("child %s is not allowed in an immutable directory" % - quote_output(name, encoding='utf-8'), name) + raise MustBeDeepImmutableError( + "child %r is not allowed in an immutable directory" % (name,), + name) if has_aux: entry = children.get_aux(name) if not entry: diff --git a/src/allmydata/nodemaker.py b/src/allmydata/nodemaker.py index ab455ce48..6b0b77c5c 100644 --- a/src/allmydata/nodemaker.py +++ b/src/allmydata/nodemaker.py @@ -1,3 +1,15 @@ +""" +Ported to Python 3. +""" +from __future__ import absolute_import +from __future__ import division +from __future__ import print_function +from __future__ import unicode_literals + +from future.utils import PY2 +if PY2: + from future.builtins import filter, map, zip, ascii, chr, hex, input, next, oct, open, pow, round, super, bytes, dict, list, object, range, str, max, min # noqa: F401 + import weakref from zope.interface import implementer from allmydata.util.assertutil import precondition diff --git a/src/allmydata/test/test_dirnode.py b/src/allmydata/test/test_dirnode.py index 586d89731..3a073dafe 100644 --- a/src/allmydata/test/test_dirnode.py +++ b/src/allmydata/test/test_dirnode.py @@ -14,7 +14,6 @@ if PY2: # Skip list() since it results in spurious test failures from future.builtins import filter, map, zip, ascii, chr, hex, input, next, oct, open, pow, round, super, bytes, dict, object, range, str, max, min # noqa: F401 -import six import time import unicodedata from zope.interface import implementer @@ -46,9 +45,6 @@ import allmydata.test.common_util as testutil from hypothesis import given from hypothesis.strategies import text -if six.PY3: - long = int - @implementer(IConsumer) class MemAccum(object): diff --git a/src/allmydata/util/_python3.py b/src/allmydata/util/_python3.py index 79f09af02..604e2ccab 100644 --- a/src/allmydata/util/_python3.py +++ b/src/allmydata/util/_python3.py @@ -66,6 +66,7 @@ PORTED_MODULES = [ "allmydata.mutable.retrieve", "allmydata.mutable.servermap", "allmydata.node", + "allmydata.nodemaker", "allmydata.storage_client", "allmydata.storage.common", "allmydata.storage.crawler", From 713e45626574a3835a2675e87566ac8811cd23cb Mon Sep 17 00:00:00 2001 From: Sajith Sasidharan Date: Fri, 11 Dec 2020 10:19:07 -0500 Subject: [PATCH 32/54] Mention stats gatherer removal in notes about old configuration --- docs/historical/configuration.rst | 7 ++++++- newsfragments/3555.minor | 0 2 files changed, 6 insertions(+), 1 deletion(-) create mode 100644 newsfragments/3555.minor diff --git a/docs/historical/configuration.rst b/docs/historical/configuration.rst index 7d9b9fbe4..1ad512883 100644 --- a/docs/historical/configuration.rst +++ b/docs/historical/configuration.rst @@ -23,7 +23,7 @@ Config setting File Comment ``BASEDIR/introducer.furl`` ``BASEDIR/private/introducers.yaml`` ``[client]helper.furl`` ``BASEDIR/helper.furl`` ``[client]key_generator.furl`` ``BASEDIR/key_generator.furl`` -``[client]stats_gatherer.furl`` ``BASEDIR/stats_gatherer.furl`` +``[client]stats_gatherer.furl`` ``BASEDIR/stats_gatherer.furl`` Stats gatherer has been removed. ``[storage]enabled`` ``BASEDIR/no_storage`` (``False`` if ``no_storage`` exists) ``[storage]readonly`` ``BASEDIR/readonly_storage`` (``True`` if ``readonly_storage`` exists) ``[storage]sizelimit`` ``BASEDIR/sizelimit`` @@ -47,3 +47,8 @@ the now (since Tahoe-LAFS v1.3.0) unsupported addresses specified in ``advertised_ip_addresses`` were used in addition to any that were automatically discovered), whereas the new ``tahoe.cfg`` directive is not (``tub.location`` is used verbatim). + +The stats gatherer has been broken at least since Tahoe-LAFS 1.13.0, +and has been removed. The functionality of +``[client].stats_gatherer.furl`` (which was previously in +``BASEDIR/stats_gatherer.furl``), is no longer available. diff --git a/newsfragments/3555.minor b/newsfragments/3555.minor new file mode 100644 index 000000000..e69de29bb From 94906bed1746eeca09dc6472b4d614bccab45dac Mon Sep 17 00:00:00 2001 From: Sajith Sasidharan Date: Fri, 11 Dec 2020 10:32:41 -0500 Subject: [PATCH 33/54] Rewrite notes about stats gatherer removal --- docs/historical/configuration.rst | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/historical/configuration.rst b/docs/historical/configuration.rst index 1ad512883..5f7a8725b 100644 --- a/docs/historical/configuration.rst +++ b/docs/historical/configuration.rst @@ -23,7 +23,7 @@ Config setting File Comment ``BASEDIR/introducer.furl`` ``BASEDIR/private/introducers.yaml`` ``[client]helper.furl`` ``BASEDIR/helper.furl`` ``[client]key_generator.furl`` ``BASEDIR/key_generator.furl`` -``[client]stats_gatherer.furl`` ``BASEDIR/stats_gatherer.furl`` Stats gatherer has been removed. + ``BASEDIR/stats_gatherer.furl`` Stats gatherer has been removed. ``[storage]enabled`` ``BASEDIR/no_storage`` (``False`` if ``no_storage`` exists) ``[storage]readonly`` ``BASEDIR/readonly_storage`` (``True`` if ``readonly_storage`` exists) ``[storage]sizelimit`` ``BASEDIR/sizelimit`` @@ -48,7 +48,7 @@ addresses specified in ``advertised_ip_addresses`` were used in addition to any that were automatically discovered), whereas the new ``tahoe.cfg`` directive is not (``tub.location`` is used verbatim). -The stats gatherer has been broken at least since Tahoe-LAFS 1.13.0, -and has been removed. The functionality of -``[client].stats_gatherer.furl`` (which was previously in -``BASEDIR/stats_gatherer.furl``), is no longer available. +The stats gatherer has been broken at least since Tahoe-LAFS 1.13.0. +The functionality of ``[client].stats_gatherer.furl`` (which was +previously in ``BASEDIR/stats_gatherer.furl``), will not be available +after Tahoe-LAFS v1.15.0 rc0. From 36e53caaeb31547fa7117d8d35e640adc124c019 Mon Sep 17 00:00:00 2001 From: Itamar Turner-Trauring Date: Fri, 11 Dec 2020 11:14:50 -0500 Subject: [PATCH 34/54] Add test coverage for packing UnknownNode with missing read-only URI. --- src/allmydata/test/test_dirnode.py | 24 ++++++++++++++++++++++++ src/allmydata/unknown.py | 8 ++++++++ 2 files changed, 32 insertions(+) diff --git a/src/allmydata/test/test_dirnode.py b/src/allmydata/test/test_dirnode.py index 3a073dafe..1c265492b 100644 --- a/src/allmydata/test/test_dirnode.py +++ b/src/allmydata/test/test_dirnode.py @@ -1475,6 +1475,30 @@ class Packing(testutil.ReallyEqualMixin, unittest.TestCase): kids[str(name)] = (nm.create_from_cap(caps[name]), {}) return kids + def test_pack_unpack_unknown(self): + """ + Minimal testing for roundtripping unknown URIs. + """ + nm = NodeMaker(None, None, None, None, None, {"k": 3, "n": 10}, None, None) + fn = MinimalFakeMutableFile() + # UnknownNode has massively complex rules about when it's an error. + # Just force it not to be an error. + unknown_rw = UnknownNode(b"whatevs://write", None) + unknown_rw.error = None + unknown_ro = UnknownNode(None, b"whatevs://readonly") + unknown_ro.error = None + kids = { + "unknown_rw": (unknown_rw, {}), + "unknown_ro": (unknown_ro, {}) + } + packed = dirnode.pack_children(kids, fn.get_writekey(), deep_immutable=False) + + write_uri = b"URI:SSK-RO:e3mdrzfwhoq42hy5ubcz6rp3o4:ybyibhnp3vvwuq2vaw2ckjmesgkklfs6ghxleztqidihjyofgw7q" + filenode = nm.create_from_cap(write_uri) + dn = dirnode.DirectoryNode(filenode, nm, None) + unkids = dn._unpack_contents(packed) + self.assertEqual(kids, unkids) + @given(text(min_size=1, max_size=20)) def test_pack_unpack_unicode_hypothesis(self, name): """ diff --git a/src/allmydata/unknown.py b/src/allmydata/unknown.py index 34d084136..f79c88415 100644 --- a/src/allmydata/unknown.py +++ b/src/allmydata/unknown.py @@ -182,3 +182,11 @@ class UnknownNode(object): def check_and_repair(self, monitor, verify, add_lease): return defer.succeed(None) + + def __eq__(self, other): + if not isinstance(other, UnknownNode): + return False + return other.ro_uri == self.ro_uri and other.rw_uri == self.rw_uri + + def __ne__(self, other): + return not (self == other) From 0e02f0cd85f543d5fa9aa1e938961755f093cf85 Mon Sep 17 00:00:00 2001 From: Sajith Sasidharan Date: Fri, 11 Dec 2020 11:19:27 -0500 Subject: [PATCH 35/54] Remove extra "." --- docs/historical/configuration.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/historical/configuration.rst b/docs/historical/configuration.rst index 5f7a8725b..de71ffd38 100644 --- a/docs/historical/configuration.rst +++ b/docs/historical/configuration.rst @@ -49,6 +49,6 @@ addition to any that were automatically discovered), whereas the new ``tahoe.cfg`` directive is not (``tub.location`` is used verbatim). The stats gatherer has been broken at least since Tahoe-LAFS 1.13.0. -The functionality of ``[client].stats_gatherer.furl`` (which was +The functionality of ``[client]stats_gatherer.furl`` (which was previously in ``BASEDIR/stats_gatherer.furl``), will not be available after Tahoe-LAFS v1.15.0 rc0. From da3d5e9f6c7d1b002c71f590378c9fc2f21331c2 Mon Sep 17 00:00:00 2001 From: Sajith Sasidharan Date: Fri, 11 Dec 2020 11:26:31 -0500 Subject: [PATCH 36/54] Update note about stats gatherer removal --- docs/historical/configuration.rst | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/docs/historical/configuration.rst b/docs/historical/configuration.rst index de71ffd38..a31e4897c 100644 --- a/docs/historical/configuration.rst +++ b/docs/historical/configuration.rst @@ -49,6 +49,8 @@ addition to any that were automatically discovered), whereas the new ``tahoe.cfg`` directive is not (``tub.location`` is used verbatim). The stats gatherer has been broken at least since Tahoe-LAFS 1.13.0. -The functionality of ``[client]stats_gatherer.furl`` (which was -previously in ``BASEDIR/stats_gatherer.furl``), will not be available -after Tahoe-LAFS v1.15.0 rc0. +The (broken) functionality of ``[client]stats_gatherer.furl`` (which +was previously in ``BASEDIR/stats_gatherer.furl``), is scheduled to be +completely removed after Tahoe-LAFS v1.15.0. After that point, if +your configuration contains a ``[client]stats_gatherer.furl``, your +node will refuse to start. From a0479a52e39b7b17919741177c574122a8e44130 Mon Sep 17 00:00:00 2001 From: Sajith Sasidharan Date: Fri, 11 Dec 2020 11:30:03 -0500 Subject: [PATCH 37/54] Be consistent when mentioning versions --- docs/historical/configuration.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/historical/configuration.rst b/docs/historical/configuration.rst index a31e4897c..867ed0de0 100644 --- a/docs/historical/configuration.rst +++ b/docs/historical/configuration.rst @@ -48,7 +48,7 @@ addresses specified in ``advertised_ip_addresses`` were used in addition to any that were automatically discovered), whereas the new ``tahoe.cfg`` directive is not (``tub.location`` is used verbatim). -The stats gatherer has been broken at least since Tahoe-LAFS 1.13.0. +The stats gatherer has been broken at least since Tahoe-LAFS v1.13.0. The (broken) functionality of ``[client]stats_gatherer.furl`` (which was previously in ``BASEDIR/stats_gatherer.furl``), is scheduled to be completely removed after Tahoe-LAFS v1.15.0. After that point, if From 51e50671e5a6c986b41d235b5d541337dcf2e40d Mon Sep 17 00:00:00 2001 From: Jean-Paul Calderone Date: Fri, 11 Dec 2020 15:32:24 -0500 Subject: [PATCH 38/54] Get rid of the "tahoe start" and "tahoe stop" and fix the obvious problems This just requires the client node to already be running now. --- src/allmydata/test/check_grid.py | 43 ++++++++------------------------ 1 file changed, 11 insertions(+), 32 deletions(-) diff --git a/src/allmydata/test/check_grid.py b/src/allmydata/test/check_grid.py index 266b7688b..82323a5c5 100644 --- a/src/allmydata/test/check_grid.py +++ b/src/allmydata/test/check_grid.py @@ -23,15 +23,15 @@ down the node after the test finishes. To set up the client node, do the following: tahoe create-client DIR - populate DIR/introducer.furl - tahoe start DIR XXXX - tahoe add-alias -d DIR testgrid `tahoe mkdir -d DIR` - pick a 10kB-ish test file, compute its md5sum - tahoe put -d DIR FILE testgrid:old.MD5SUM - tahoe put -d DIR FILE testgrid:recent.MD5SUM - tahoe put -d DIR FILE testgrid:recentdir/recent.MD5SUM - echo "" | tahoe put -d DIR --mutable testgrid:log - echo "" | tahoe put -d DIR --mutable testgrid:recentlog + populate DIR/private/introducers.yaml + $DAEMONIZE tahoe run DIR + tahoe -d DIR create-alias testgrid + # pick a 10kB-ish test file, compute its md5sum + tahoe -d DIR put FILE testgrid:old.MD5SUM + tahoe -d DIR put FILE testgrid:recent.MD5SUM + tahoe -d DIR put FILE testgrid:recentdir/recent.MD5SUM + echo "" | tahoe -d DIR put --mutable - testgrid:log + echo "" | tahoe -d DIR put --mutable - testgrid:recentlog This script will perform the following steps (the kind of compatibility that is being tested is in [brackets]): @@ -104,26 +104,13 @@ class GridTester(object): def cli(self, cmd, *args, **kwargs): print("tahoe", cmd, " ".join(args)) - stdout, stderr = self.command(self.tahoe, cmd, "-d", self.nodedir, + stdout, stderr = self.command(self.tahoe, "-d", self.nodedir, cmd, *args, **kwargs) if not kwargs.get("ignore_stderr", False) and stderr != "": raise CommandFailed("command '%s' had stderr: %s" % (" ".join(args), stderr)) return stdout - def stop_old_node(self): - print("tahoe stop", self.nodedir, "(force)") - self.command(self.tahoe, "stop", self.nodedir, expected_rc=None) - - def start_node(self): - print("tahoe start", self.nodedir) - self.command(self.tahoe, "start", self.nodedir) # XXXX - time.sleep(5) - - def stop_node(self): - print("tahoe stop", self.nodedir) - self.command(self.tahoe, "stop", self.nodedir) - def read_and_check(self, f): expected_md5_s = f[f.find(".")+1:] out = self.cli("get", "testgrid:" + f) @@ -204,19 +191,11 @@ class GridTester(object): fn = prefix + "." + md5sum return fn, data - def run(self): - self.stop_old_node() - self.start_node() - try: - self.do_test() - finally: - self.stop_node() - def main(): config = GridTesterOptions() config.parseOptions() gt = GridTester(config) - gt.run() + gt.do_test() if __name__ == "__main__": main() From f17a5dfafc3420582dc9c05ad29c45c5ca183003 Mon Sep 17 00:00:00 2001 From: Jean-Paul Calderone Date: Sat, 12 Dec 2020 17:40:13 -0500 Subject: [PATCH 39/54] key-generator was apparently removed long ago --- src/allmydata/scripts/run_common.py | 5 ++--- src/allmydata/test/test_runner.py | 14 +++++++------- 2 files changed, 9 insertions(+), 10 deletions(-) diff --git a/src/allmydata/scripts/run_common.py b/src/allmydata/scripts/run_common.py index f62534434..da02c6517 100644 --- a/src/allmydata/scripts/run_common.py +++ b/src/allmydata/scripts/run_common.py @@ -46,8 +46,7 @@ def get_pid_from_pidfile(pidfile): def identify_node_type(basedir): """ - :return unicode: None or one of: 'client', 'introducer', or - 'key-generator' + :return unicode: None or one of: 'client' or 'introducer'. """ tac = u'' try: @@ -58,7 +57,7 @@ def identify_node_type(basedir): except OSError: return None - for t in (u"client", u"introducer", u"key-generator"): + for t in (u"client", u"introducer"): if t in tac: return t return None diff --git a/src/allmydata/test/test_runner.py b/src/allmydata/test/test_runner.py index 636880621..ef2b99a19 100644 --- a/src/allmydata/test/test_runner.py +++ b/src/allmydata/test/test_runner.py @@ -144,8 +144,8 @@ class BinTahoe(common_util.SignalMixin, unittest.TestCase, RunBinTahoeMixin): class CreateNode(unittest.TestCase): - # exercise "tahoe create-node", create-introducer, and - # create-key-generator by calling the corresponding code as a subroutine. + # exercise "tahoe create-node" and "tahoe create-introducer" by calling + # the corresponding code as a subroutine. def workdir(self, name): basedir = os.path.join("test_runner", "CreateNode", name) @@ -253,11 +253,11 @@ class CreateNode(unittest.TestCase): class RunNode(common_util.SignalMixin, unittest.TestCase, pollmixin.PollMixin, RunBinTahoeMixin): """ - exercise "tahoe run" for both introducer, client node, and key-generator, - by spawning "tahoe run" as a subprocess. This doesn't get us line-level - coverage, but it does a better job of confirming that the user can - actually run "./bin/tahoe run" and expect it to work. This verifies that - bin/tahoe sets up PYTHONPATH and the like correctly. + exercise "tahoe run" for both introducer and client node, by spawning + "tahoe run" as a subprocess. This doesn't get us line-level coverage, but + it does a better job of confirming that the user can actually run + "./bin/tahoe run" and expect it to work. This verifies that bin/tahoe sets + up PYTHONPATH and the like correctly. """ def workdir(self, name): From d8da61205522e28921185c1ae103c98a2905f33d Mon Sep 17 00:00:00 2001 From: Jean-Paul Calderone Date: Sat, 12 Dec 2020 18:20:09 -0500 Subject: [PATCH 40/54] Move the --nodaemon logic out of tahoe_run We're always going to --nodaemon from now on --- src/allmydata/scripts/run_common.py | 8 +------- src/allmydata/scripts/tahoe_run.py | 3 --- 2 files changed, 1 insertion(+), 10 deletions(-) diff --git a/src/allmydata/scripts/run_common.py b/src/allmydata/scripts/run_common.py index da02c6517..da2e93363 100644 --- a/src/allmydata/scripts/run_common.py +++ b/src/allmydata/scripts/run_common.py @@ -195,13 +195,7 @@ def run(config): # Now prepare to turn into a twistd process. This os.chdir is the point # of no return. os.chdir(basedir) - twistd_args = [] - if (nodetype in (u"client", u"introducer") - and "--nodaemon" not in config.twistd_args - and "--syslog" not in config.twistd_args - and "--logfile" not in config.twistd_args): - fileutil.make_dirs(os.path.join(basedir, u"logs")) - twistd_args.extend(["--logfile", os.path.join("logs", "twistd.log")]) + twistd_args = ["--nodaemon"] twistd_args.extend(config.twistd_args) twistd_args.append("DaemonizeTahoeNode") # point at our DaemonizeTahoeNodePlugin diff --git a/src/allmydata/scripts/tahoe_run.py b/src/allmydata/scripts/tahoe_run.py index 0a921cc71..cea9bda27 100644 --- a/src/allmydata/scripts/tahoe_run.py +++ b/src/allmydata/scripts/tahoe_run.py @@ -10,6 +10,3 @@ __all__ = [ class RunOptions(_RunOptions): subcommand_name = "run" - - def postOptions(self): - self.twistd_args += ("--nodaemon",) From ed2152e2c822b83163e1972cbcf566691f39f33a Mon Sep 17 00:00:00 2001 From: Jean-Paul Calderone Date: Sat, 12 Dec 2020 18:20:26 -0500 Subject: [PATCH 41/54] We don't need to check this condition. We're always running. --- src/allmydata/scripts/run_common.py | 34 ++--------------------------- 1 file changed, 2 insertions(+), 32 deletions(-) diff --git a/src/allmydata/scripts/run_common.py b/src/allmydata/scripts/run_common.py index da2e93363..86d0330a5 100644 --- a/src/allmydata/scripts/run_common.py +++ b/src/allmydata/scripts/run_common.py @@ -217,38 +217,8 @@ def run(config): print("found invalid PID file in %s - deleting it" % basedir, file=err) os.remove(pidfile) - # On Unix-like platforms: - # Unless --nodaemon was provided, the twistd.runApp() below spawns off a - # child process, and the parent calls os._exit(0), so there's no way for - # us to get control afterwards, even with 'except SystemExit'. If - # application setup fails (e.g. ImportError), runApp() will raise an - # exception. - # - # So if we wanted to do anything with the running child, we'd have two - # options: - # - # * fork first, and have our child wait for the runApp() child to get - # running. (note: just fork(). This is easier than fork+exec, since we - # don't have to get PATH and PYTHONPATH set up, since we're not - # starting a *different* process, just cloning a new instance of the - # current process) - # * or have the user run a separate command some time after this one - # exits. - # - # For Tahoe, we don't need to do anything with the child, so we can just - # let it exit. - # - # On Windows: - # twistd does not fork; it just runs in the current process whether or not - # --nodaemon is specified. (As on Unix, --nodaemon does have the side effect - # of causing us to log to stdout/stderr.) - - if "--nodaemon" in twistd_args or sys.platform == "win32": - verb = "running" - else: - verb = "starting" - - print("%s node in %s" % (verb, quoted_basedir), file=out) + # We always pass --nodaemon so twistd.runApp does not daemonize. + print("running node in %s" % (quoted_basedir,), file=out) twistd.runApp(twistd_config) # we should only reach here if --nodaemon or equivalent was used return 0 From 692285ada3758d6c966cfec6d6ceced141d64489 Mon Sep 17 00:00:00 2001 From: Jean-Paul Calderone Date: Sat, 12 Dec 2020 18:21:16 -0500 Subject: [PATCH 42/54] key-generator was removed --- src/allmydata/scripts/run_common.py | 4 ---- 1 file changed, 4 deletions(-) diff --git a/src/allmydata/scripts/run_common.py b/src/allmydata/scripts/run_common.py index 86d0330a5..7abb9dd32 100644 --- a/src/allmydata/scripts/run_common.py +++ b/src/allmydata/scripts/run_common.py @@ -126,14 +126,10 @@ class DaemonizeTheRealService(Service, HookMixin): def startService(self): - def key_generator_removed(): - return fail(ValueError("key-generator support removed, see #2783")) - def start(): node_to_instance = { u"client": lambda: maybeDeferred(namedAny("allmydata.client.create_client"), self.basedir), u"introducer": lambda: maybeDeferred(namedAny("allmydata.introducer.server.create_introducer"), self.basedir), - u"key-generator": key_generator_removed, } try: From 25c98d742150cd532e1cc7723a005c105b69465a Mon Sep 17 00:00:00 2001 From: Jean-Paul Calderone Date: Sat, 12 Dec 2020 18:21:35 -0500 Subject: [PATCH 43/54] don't shadow the global --- src/allmydata/scripts/run_common.py | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/src/allmydata/scripts/run_common.py b/src/allmydata/scripts/run_common.py index 7abb9dd32..8978d892a 100644 --- a/src/allmydata/scripts/run_common.py +++ b/src/allmydata/scripts/run_common.py @@ -137,12 +137,12 @@ class DaemonizeTheRealService(Service, HookMixin): except KeyError: raise ValueError("unknown nodetype %s" % self.nodetype) - def handle_config_error(fail): - if fail.check(UnknownConfigError): - self.stderr.write("\nConfiguration error:\n{}\n\n".format(fail.value)) + def handle_config_error(reason): + if reason.check(UnknownConfigError): + self.stderr.write("\nConfiguration error:\n{}\n\n".format(reason.value)) else: self.stderr.write("\nUnknown error\n") - fail.printTraceback(self.stderr) + reason.printTraceback(self.stderr) reactor.stop() d = service_factory() From 39631a90bf0ed261bcd0fb09fb9484c09fcecddd Mon Sep 17 00:00:00 2001 From: Jean-Paul Calderone Date: Sat, 12 Dec 2020 18:23:01 -0500 Subject: [PATCH 44/54] we always use --nodaemon --- src/allmydata/scripts/run_common.py | 1 - 1 file changed, 1 deletion(-) diff --git a/src/allmydata/scripts/run_common.py b/src/allmydata/scripts/run_common.py index 8978d892a..698680e78 100644 --- a/src/allmydata/scripts/run_common.py +++ b/src/allmydata/scripts/run_common.py @@ -216,5 +216,4 @@ def run(config): # We always pass --nodaemon so twistd.runApp does not daemonize. print("running node in %s" % (quoted_basedir,), file=out) twistd.runApp(twistd_config) - # we should only reach here if --nodaemon or equivalent was used return 0 From b58b07a9d7d3df20b774298186321e3674700c6f Mon Sep 17 00:00:00 2001 From: Jean-Paul Calderone Date: Sat, 12 Dec 2020 18:26:22 -0500 Subject: [PATCH 45/54] Fold run_common into tahoe_run since there are no other run-like commands anymore --- src/allmydata/scripts/run_common.py | 219 --------------------------- src/allmydata/scripts/tahoe_run.py | 223 +++++++++++++++++++++++++++- src/allmydata/test/cli/test_cli.py | 4 +- 3 files changed, 220 insertions(+), 226 deletions(-) delete mode 100644 src/allmydata/scripts/run_common.py diff --git a/src/allmydata/scripts/run_common.py b/src/allmydata/scripts/run_common.py deleted file mode 100644 index 698680e78..000000000 --- a/src/allmydata/scripts/run_common.py +++ /dev/null @@ -1,219 +0,0 @@ -from __future__ import print_function - -import os, sys -from allmydata.scripts.common import BasedirOptions -from twisted.scripts import twistd -from twisted.python import usage -from twisted.python.reflect import namedAny -from twisted.internet.defer import maybeDeferred, fail -from twisted.application.service import Service - -from allmydata.scripts.default_nodedir import _default_nodedir -from allmydata.util import fileutil -from allmydata.util.encodingutil import listdir_unicode, quote_local_unicode_path -from allmydata.util.configutil import UnknownConfigError -from allmydata.util.deferredutil import HookMixin - - -def get_pidfile(basedir): - """ - Returns the path to the PID file. - :param basedir: the node's base directory - :returns: the path to the PID file - """ - return os.path.join(basedir, u"twistd.pid") - -def get_pid_from_pidfile(pidfile): - """ - Tries to read and return the PID stored in the node's PID file - (twistd.pid). - :param pidfile: try to read this PID file - :returns: A numeric PID on success, ``None`` if PID file absent or - inaccessible, ``-1`` if PID file invalid. - """ - try: - with open(pidfile, "r") as f: - pid = f.read() - except EnvironmentError: - return None - - try: - pid = int(pid) - except ValueError: - return -1 - - return pid - -def identify_node_type(basedir): - """ - :return unicode: None or one of: 'client' or 'introducer'. - """ - tac = u'' - try: - for fn in listdir_unicode(basedir): - if fn.endswith(u".tac"): - tac = fn - break - except OSError: - return None - - for t in (u"client", u"introducer"): - if t in tac: - return t - return None - - -class RunOptions(BasedirOptions): - optParameters = [ - ("basedir", "C", None, - "Specify which Tahoe base directory should be used." - " This has the same effect as the global --node-directory option." - " [default: %s]" % quote_local_unicode_path(_default_nodedir)), - ] - - def parseArgs(self, basedir=None, *twistd_args): - # This can't handle e.g. 'tahoe run --reactor=foo', since - # '--reactor=foo' looks like an option to the tahoe subcommand, not to - # twistd. So you can either use 'tahoe run' or 'tahoe run NODEDIR - # --TWISTD-OPTIONS'. Note that 'tahoe --node-directory=NODEDIR run - # --TWISTD-OPTIONS' also isn't allowed, unfortunately. - - BasedirOptions.parseArgs(self, basedir) - self.twistd_args = twistd_args - - def getSynopsis(self): - return ("Usage: %s [global-options] %s [options]" - " [NODEDIR [twistd-options]]" - % (self.command_name, self.subcommand_name)) - - def getUsage(self, width=None): - t = BasedirOptions.getUsage(self, width) + "\n" - twistd_options = str(MyTwistdConfig()).partition("\n")[2].partition("\n\n")[0] - t += twistd_options.replace("Options:", "twistd-options:", 1) - t += """ - -Note that if any twistd-options are used, NODEDIR must be specified explicitly -(not by default or using -C/--basedir or -d/--node-directory), and followed by -the twistd-options. -""" - return t - - -class MyTwistdConfig(twistd.ServerOptions): - subCommands = [("DaemonizeTahoeNode", None, usage.Options, "node")] - - stderr = sys.stderr - - -class DaemonizeTheRealService(Service, HookMixin): - """ - this HookMixin should really be a helper; our hooks: - - - 'running': triggered when startup has completed; it triggers - with None of successful or a Failure otherwise. - """ - stderr = sys.stderr - - def __init__(self, nodetype, basedir, options): - super(DaemonizeTheRealService, self).__init__() - self.nodetype = nodetype - self.basedir = basedir - # setup for HookMixin - self._hooks = { - "running": None, - } - self.stderr = options.parent.stderr - - def startService(self): - - def start(): - node_to_instance = { - u"client": lambda: maybeDeferred(namedAny("allmydata.client.create_client"), self.basedir), - u"introducer": lambda: maybeDeferred(namedAny("allmydata.introducer.server.create_introducer"), self.basedir), - } - - try: - service_factory = node_to_instance[self.nodetype] - except KeyError: - raise ValueError("unknown nodetype %s" % self.nodetype) - - def handle_config_error(reason): - if reason.check(UnknownConfigError): - self.stderr.write("\nConfiguration error:\n{}\n\n".format(reason.value)) - else: - self.stderr.write("\nUnknown error\n") - reason.printTraceback(self.stderr) - reactor.stop() - - d = service_factory() - - def created(srv): - srv.setServiceParent(self.parent) - d.addCallback(created) - d.addErrback(handle_config_error) - d.addBoth(self._call_hook, 'running') - return d - - from twisted.internet import reactor - reactor.callWhenRunning(start) - - -class DaemonizeTahoeNodePlugin(object): - tapname = "tahoenode" - def __init__(self, nodetype, basedir): - self.nodetype = nodetype - self.basedir = basedir - - def makeService(self, so): - return DaemonizeTheRealService(self.nodetype, self.basedir, so) - - -def run(config): - """ - Runs a Tahoe-LAFS node in the foreground. - - Sets up the IService instance corresponding to the type of node - that's starting and uses Twisted's twistd runner to disconnect our - process from the terminal. - """ - out = config.stdout - err = config.stderr - basedir = config['basedir'] - quoted_basedir = quote_local_unicode_path(basedir) - print("'tahoe {}' in {}".format(config.subcommand_name, quoted_basedir), file=out) - if not os.path.isdir(basedir): - print("%s does not look like a directory at all" % quoted_basedir, file=err) - return 1 - nodetype = identify_node_type(basedir) - if not nodetype: - print("%s is not a recognizable node directory" % quoted_basedir, file=err) - return 1 - # Now prepare to turn into a twistd process. This os.chdir is the point - # of no return. - os.chdir(basedir) - twistd_args = ["--nodaemon"] - twistd_args.extend(config.twistd_args) - twistd_args.append("DaemonizeTahoeNode") # point at our DaemonizeTahoeNodePlugin - - twistd_config = MyTwistdConfig() - twistd_config.stdout = out - twistd_config.stderr = err - try: - twistd_config.parseOptions(twistd_args) - except usage.error as ue: - # these arguments were unsuitable for 'twistd' - print(config, file=err) - print("tahoe %s: usage error from twistd: %s\n" % (config.subcommand_name, ue), file=err) - return 1 - twistd_config.loadedPlugins = {"DaemonizeTahoeNode": DaemonizeTahoeNodePlugin(nodetype, basedir)} - - # handle invalid PID file (twistd might not start otherwise) - pidfile = get_pidfile(basedir) - if get_pid_from_pidfile(pidfile) == -1: - print("found invalid PID file in %s - deleting it" % basedir, file=err) - os.remove(pidfile) - - # We always pass --nodaemon so twistd.runApp does not daemonize. - print("running node in %s" % (quoted_basedir,), file=out) - twistd.runApp(twistd_config) - return 0 diff --git a/src/allmydata/scripts/tahoe_run.py b/src/allmydata/scripts/tahoe_run.py index cea9bda27..3ee1bf3d9 100644 --- a/src/allmydata/scripts/tahoe_run.py +++ b/src/allmydata/scripts/tahoe_run.py @@ -1,12 +1,225 @@ -from .run_common import ( - RunOptions as _RunOptions, - run, -) +from __future__ import print_function __all__ = [ "RunOptions", "run", ] -class RunOptions(_RunOptions): +import os, sys +from allmydata.scripts.common import BasedirOptions +from twisted.scripts import twistd +from twisted.python import usage +from twisted.python.reflect import namedAny +from twisted.internet.defer import maybeDeferred +from twisted.application.service import Service + +from allmydata.scripts.default_nodedir import _default_nodedir +from allmydata.util.encodingutil import listdir_unicode, quote_local_unicode_path +from allmydata.util.configutil import UnknownConfigError +from allmydata.util.deferredutil import HookMixin + + +def get_pidfile(basedir): + """ + Returns the path to the PID file. + :param basedir: the node's base directory + :returns: the path to the PID file + """ + return os.path.join(basedir, u"twistd.pid") + +def get_pid_from_pidfile(pidfile): + """ + Tries to read and return the PID stored in the node's PID file + (twistd.pid). + :param pidfile: try to read this PID file + :returns: A numeric PID on success, ``None`` if PID file absent or + inaccessible, ``-1`` if PID file invalid. + """ + try: + with open(pidfile, "r") as f: + pid = f.read() + except EnvironmentError: + return None + + try: + pid = int(pid) + except ValueError: + return -1 + + return pid + +def identify_node_type(basedir): + """ + :return unicode: None or one of: 'client' or 'introducer'. + """ + tac = u'' + try: + for fn in listdir_unicode(basedir): + if fn.endswith(u".tac"): + tac = fn + break + except OSError: + return None + + for t in (u"client", u"introducer"): + if t in tac: + return t + return None + + +class RunOptions(BasedirOptions): subcommand_name = "run" + + optParameters = [ + ("basedir", "C", None, + "Specify which Tahoe base directory should be used." + " This has the same effect as the global --node-directory option." + " [default: %s]" % quote_local_unicode_path(_default_nodedir)), + ] + + def parseArgs(self, basedir=None, *twistd_args): + # This can't handle e.g. 'tahoe run --reactor=foo', since + # '--reactor=foo' looks like an option to the tahoe subcommand, not to + # twistd. So you can either use 'tahoe run' or 'tahoe run NODEDIR + # --TWISTD-OPTIONS'. Note that 'tahoe --node-directory=NODEDIR run + # --TWISTD-OPTIONS' also isn't allowed, unfortunately. + + BasedirOptions.parseArgs(self, basedir) + self.twistd_args = twistd_args + + def getSynopsis(self): + return ("Usage: %s [global-options] %s [options]" + " [NODEDIR [twistd-options]]" + % (self.command_name, self.subcommand_name)) + + def getUsage(self, width=None): + t = BasedirOptions.getUsage(self, width) + "\n" + twistd_options = str(MyTwistdConfig()).partition("\n")[2].partition("\n\n")[0] + t += twistd_options.replace("Options:", "twistd-options:", 1) + t += """ + +Note that if any twistd-options are used, NODEDIR must be specified explicitly +(not by default or using -C/--basedir or -d/--node-directory), and followed by +the twistd-options. +""" + return t + + +class MyTwistdConfig(twistd.ServerOptions): + subCommands = [("DaemonizeTahoeNode", None, usage.Options, "node")] + + stderr = sys.stderr + + +class DaemonizeTheRealService(Service, HookMixin): + """ + this HookMixin should really be a helper; our hooks: + + - 'running': triggered when startup has completed; it triggers + with None of successful or a Failure otherwise. + """ + stderr = sys.stderr + + def __init__(self, nodetype, basedir, options): + super(DaemonizeTheRealService, self).__init__() + self.nodetype = nodetype + self.basedir = basedir + # setup for HookMixin + self._hooks = { + "running": None, + } + self.stderr = options.parent.stderr + + def startService(self): + + def start(): + node_to_instance = { + u"client": lambda: maybeDeferred(namedAny("allmydata.client.create_client"), self.basedir), + u"introducer": lambda: maybeDeferred(namedAny("allmydata.introducer.server.create_introducer"), self.basedir), + } + + try: + service_factory = node_to_instance[self.nodetype] + except KeyError: + raise ValueError("unknown nodetype %s" % self.nodetype) + + def handle_config_error(reason): + if reason.check(UnknownConfigError): + self.stderr.write("\nConfiguration error:\n{}\n\n".format(reason.value)) + else: + self.stderr.write("\nUnknown error\n") + reason.printTraceback(self.stderr) + reactor.stop() + + d = service_factory() + + def created(srv): + srv.setServiceParent(self.parent) + d.addCallback(created) + d.addErrback(handle_config_error) + d.addBoth(self._call_hook, 'running') + return d + + from twisted.internet import reactor + reactor.callWhenRunning(start) + + +class DaemonizeTahoeNodePlugin(object): + tapname = "tahoenode" + def __init__(self, nodetype, basedir): + self.nodetype = nodetype + self.basedir = basedir + + def makeService(self, so): + return DaemonizeTheRealService(self.nodetype, self.basedir, so) + + +def run(config): + """ + Runs a Tahoe-LAFS node in the foreground. + + Sets up the IService instance corresponding to the type of node + that's starting and uses Twisted's twistd runner to disconnect our + process from the terminal. + """ + out = config.stdout + err = config.stderr + basedir = config['basedir'] + quoted_basedir = quote_local_unicode_path(basedir) + print("'tahoe {}' in {}".format(config.subcommand_name, quoted_basedir), file=out) + if not os.path.isdir(basedir): + print("%s does not look like a directory at all" % quoted_basedir, file=err) + return 1 + nodetype = identify_node_type(basedir) + if not nodetype: + print("%s is not a recognizable node directory" % quoted_basedir, file=err) + return 1 + # Now prepare to turn into a twistd process. This os.chdir is the point + # of no return. + os.chdir(basedir) + twistd_args = ["--nodaemon"] + twistd_args.extend(config.twistd_args) + twistd_args.append("DaemonizeTahoeNode") # point at our DaemonizeTahoeNodePlugin + + twistd_config = MyTwistdConfig() + twistd_config.stdout = out + twistd_config.stderr = err + try: + twistd_config.parseOptions(twistd_args) + except usage.error as ue: + # these arguments were unsuitable for 'twistd' + print(config, file=err) + print("tahoe %s: usage error from twistd: %s\n" % (config.subcommand_name, ue), file=err) + return 1 + twistd_config.loadedPlugins = {"DaemonizeTahoeNode": DaemonizeTahoeNodePlugin(nodetype, basedir)} + + # handle invalid PID file (twistd might not start otherwise) + pidfile = get_pidfile(basedir) + if get_pid_from_pidfile(pidfile) == -1: + print("found invalid PID file in %s - deleting it" % basedir, file=err) + os.remove(pidfile) + + # We always pass --nodaemon so twistd.runApp does not daemonize. + print("running node in %s" % (quoted_basedir,), file=out) + twistd.runApp(twistd_config) + return 0 diff --git a/src/allmydata/test/cli/test_cli.py b/src/allmydata/test/cli/test_cli.py index 27af29520..2d1e9f57a 100644 --- a/src/allmydata/test/cli/test_cli.py +++ b/src/allmydata/test/cli/test_cli.py @@ -1310,8 +1310,8 @@ class Options(ReallyEqualMixin, unittest.TestCase): class Run(unittest.TestCase): - @patch('allmydata.scripts.run_common.os.chdir') - @patch('allmydata.scripts.run_common.twistd') + @patch('allmydata.scripts.tahoe_run.os.chdir') + @patch('allmydata.scripts.tahoe_run.twistd') def test_non_numeric_pid(self, mock_twistd, chdir): """ If the pidfile exists but does not contain a numeric value, a complaint to From 34cd1efaa477ba28e0bd7ce15605db3c72631b7a Mon Sep 17 00:00:00 2001 From: Jean-Paul Calderone Date: Sat, 12 Dec 2020 18:34:49 -0500 Subject: [PATCH 46/54] For the sake of clarity, stop talking about daemons here --- src/allmydata/test/cli/test_cli.py | 14 +++++++++++--- 1 file changed, 11 insertions(+), 3 deletions(-) diff --git a/src/allmydata/test/cli/test_cli.py b/src/allmydata/test/cli/test_cli.py index 2d1e9f57a..2b1bc1c86 100644 --- a/src/allmydata/test/cli/test_cli.py +++ b/src/allmydata/test/cli/test_cli.py @@ -1272,6 +1272,14 @@ class Options(ReallyEqualMixin, unittest.TestCase): # accept a --node-directory option before the verb, or a --basedir # option after, or a basedir argument after, but none in the wrong # place, and not more than one of the three. + + # Here is some option twistd recognizes but we don't. Depending on + # where it appears, it should be passed through to twistd. It doesn't + # really matter which option it is (it doesn't even have to be a valid + # option). This test does not actually run any of the twistd argument + # parsing. + some_twistd_option = "--spew" + o = self.parse(["run"]) self.failUnlessReallyEqual(o["basedir"], os.path.join(fileutil.abspath_expanduser_unicode(u"~"), u".tahoe")) @@ -1282,7 +1290,7 @@ class Options(ReallyEqualMixin, unittest.TestCase): o = self.parse(["--node-directory", "there", "run"]) self.failUnlessReallyEqual(o["basedir"], fileutil.abspath_expanduser_unicode(u"there")) - o = self.parse(["run", "here", "--nodaemon"]) + o = self.parse(["run", "here", some_twistd_option]) self.failUnlessReallyEqual(o["basedir"], fileutil.abspath_expanduser_unicode(u"here")) self.failUnlessRaises(usage.UsageError, self.parse, @@ -1303,9 +1311,9 @@ class Options(ReallyEqualMixin, unittest.TestCase): "run", "--basedir=here", "anywhere"]) self.failUnlessRaises(usage.UsageError, self.parse, - ["--node-directory=there", "run", "--nodaemon"]) + ["--node-directory=there", "run", some_twistd_option]) self.failUnlessRaises(usage.UsageError, self.parse, - ["run", "--basedir=here", "--nodaemon"]) + ["run", "--basedir=here", some_twistd_option]) class Run(unittest.TestCase): From 0a1c2386b975ae0de1b618055d3d4e72a973898d Mon Sep 17 00:00:00 2001 From: Jean-Paul Calderone Date: Mon, 14 Dec 2020 09:37:04 -0500 Subject: [PATCH 47/54] client must be running already --- src/allmydata/test/check_grid.py | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/src/allmydata/test/check_grid.py b/src/allmydata/test/check_grid.py index 82323a5c5..5c46c41a8 100644 --- a/src/allmydata/test/check_grid.py +++ b/src/allmydata/test/check_grid.py @@ -16,9 +16,7 @@ that this script does not import anything from tahoe directly, so it doesn't matter what its PYTHONPATH is, as long as the bin/tahoe that it uses is functional. -This script expects that the client node will be not running when the script -starts, but it will forcibly shut down the node just to be sure. It will shut -down the node after the test finishes. +This script expects the client node to be running already. To set up the client node, do the following: @@ -52,7 +50,6 @@ is being tested is in [brackets]): This script will also keep track of speeds and latencies and will write them in a machine-readable logfile. - """ import time, subprocess, md5, os.path, random From a0931f4999ec0e5f589080e1583046d497d9e115 Mon Sep 17 00:00:00 2001 From: Jean-Paul Calderone Date: Mon, 14 Dec 2020 09:38:06 -0500 Subject: [PATCH 48/54] You can pass the introducer on the command line --- src/allmydata/test/check_grid.py | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/src/allmydata/test/check_grid.py b/src/allmydata/test/check_grid.py index 5c46c41a8..f4bfcae31 100644 --- a/src/allmydata/test/check_grid.py +++ b/src/allmydata/test/check_grid.py @@ -20,9 +20,8 @@ This script expects the client node to be running already. To set up the client node, do the following: - tahoe create-client DIR - populate DIR/private/introducers.yaml $DAEMONIZE tahoe run DIR + tahoe create-client --introducer=INTRODUCER_FURL DIR tahoe -d DIR create-alias testgrid # pick a 10kB-ish test file, compute its md5sum tahoe -d DIR put FILE testgrid:old.MD5SUM From bdb7c50fac2d7cab17e6974390d247cfafe11472 Mon Sep 17 00:00:00 2001 From: Jean-Paul Calderone Date: Mon, 14 Dec 2020 09:38:16 -0500 Subject: [PATCH 49/54] You can just use multiple terminals If you know how to daemonize stuff you can figure it out yourself I guess. --- src/allmydata/test/check_grid.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/allmydata/test/check_grid.py b/src/allmydata/test/check_grid.py index f4bfcae31..0a68ed899 100644 --- a/src/allmydata/test/check_grid.py +++ b/src/allmydata/test/check_grid.py @@ -20,8 +20,8 @@ This script expects the client node to be running already. To set up the client node, do the following: - $DAEMONIZE tahoe run DIR tahoe create-client --introducer=INTRODUCER_FURL DIR + tahoe run DIR tahoe -d DIR create-alias testgrid # pick a 10kB-ish test file, compute its md5sum tahoe -d DIR put FILE testgrid:old.MD5SUM From 0357eeb924d0691a9153125b99070a56a8cbd5fb Mon Sep 17 00:00:00 2001 From: Jean-Paul Calderone Date: Mon, 14 Dec 2020 09:24:57 -0500 Subject: [PATCH 50/54] news fragment --- newsfragments/3558.minor | 0 1 file changed, 0 insertions(+), 0 deletions(-) create mode 100644 newsfragments/3558.minor diff --git a/newsfragments/3558.minor b/newsfragments/3558.minor new file mode 100644 index 000000000..e69de29bb From 41fc7d2c3c55e499dd1b29fa7618d31a39b82b10 Mon Sep 17 00:00:00 2001 From: Jean-Paul Calderone Date: Mon, 14 Dec 2020 10:24:26 -0500 Subject: [PATCH 51/54] Can we force it to report for the right project? --- .circleci/config.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.circleci/config.yml b/.circleci/config.yml index afa3fafa1..5039b0d7f 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -270,7 +270,7 @@ jobs: name: "Submit coverage results" command: | if [ -n "${UPLOAD_COVERAGE}" ]; then - /tmp/venv/bin/codecov + /tmp/venv/bin/codecov --slug tahoe-lafs/tahoe-lafs fi From 85c1b2472972d7ed8ec90257cfcab041d21888d9 Mon Sep 17 00:00:00 2001 From: Jean-Paul Calderone Date: Mon, 14 Dec 2020 10:36:24 -0500 Subject: [PATCH 52/54] That didn't work. Using `--slug tahoe-lafs/tahoe-lafs` does not fix the issue. It causes codecov to fail with `Commit sha does not match Circle build.` --- .circleci/config.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.circleci/config.yml b/.circleci/config.yml index 5039b0d7f..afa3fafa1 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -270,7 +270,7 @@ jobs: name: "Submit coverage results" command: | if [ -n "${UPLOAD_COVERAGE}" ]; then - /tmp/venv/bin/codecov --slug tahoe-lafs/tahoe-lafs + /tmp/venv/bin/codecov fi From 7be13f5c7f462af15d09c65ac3eef93c2a66ae1e Mon Sep 17 00:00:00 2001 From: Jean-Paul Calderone Date: Mon, 14 Dec 2020 10:36:45 -0500 Subject: [PATCH 53/54] Try providing an upload token instead --- .codecov.yml | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/.codecov.yml b/.codecov.yml index 57abf7c0a..b596def25 100644 --- a/.codecov.yml +++ b/.codecov.yml @@ -32,3 +32,11 @@ coverage: patch: default: threshold: 1% + + +codecov: + # This is a public repository so supposedly we don't "need" to use an upload + # token. However, using one makes sure that CI jobs running against forked + # repositories have coverage uploaded to the right place in codecov so + # they're reports aren't incomplete. + token: "abf679b6-e2e6-4b33-b7b5-6cfbd41ee691" From f1be6a50a2a4c3d5d543c24450a8d27eec3e4900 Mon Sep 17 00:00:00 2001 From: Jean-Paul Calderone Date: Mon, 14 Dec 2020 11:04:31 -0500 Subject: [PATCH 54/54] So embarassing --- .codecov.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.codecov.yml b/.codecov.yml index b596def25..df1eb5e01 100644 --- a/.codecov.yml +++ b/.codecov.yml @@ -38,5 +38,5 @@ codecov: # This is a public repository so supposedly we don't "need" to use an upload # token. However, using one makes sure that CI jobs running against forked # repositories have coverage uploaded to the right place in codecov so - # they're reports aren't incomplete. + # their reports aren't incomplete. token: "abf679b6-e2e6-4b33-b7b5-6cfbd41ee691"