ExternalVendorCode/tahoe-lafs
1
0
Fork 0
mirror of https://github.com/tahoe-lafs/tahoe-lafs.git synced 2025-01-06 21:18:43 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity
14,675 Commits 185 Branches 90 Tags 89 MiB
6ee5c75880
Commit Graph

436 Commits

Author SHA1 Message Date
meejah
da0fe23082 _load_grid_manager_certificates -> method 2020-11-24 14:13:48 -07:00
Jean-Paul Calderone
1a5efa5ec9 Just let the test use the real SFTPServer service 2020-11-24 14:05:00 -05:00
Jean-Paul Calderone
c694e8c7e2 Delete allmydata.version_checks and related functionality 
It is not Tahoe-LAFS' job to manage package installation in this way.
Instead, we can declare our dependencies in setup.py and rely on installation
management tools and packagers to create a suitable execution environment.

Making this statement in the past required going much further out on a limb
than it does today.  This code has served its purpose and can now be retired.
 2020-11-23 15:10:18 -05:00
Itamar Turner-Trauring
8029a1befc First passing test on Python 3. 2020-11-19 11:45:32 -05:00
Jean-Paul Calderone
46955202e2 Hook into Twisted Web to control where request bodies are written 2020-11-18 15:47:06 -05:00
Jean-Paul Calderone
e0f69dcfcf Get the path manipulation into _Config too 2020-11-14 09:26:07 -05:00
Jean-Paul Calderone
bef5ccd0ca Move the introducer config reading code into _Config 2020-11-14 09:12:14 -05:00
meejah
5b4f5e89e1 explicit unicode 2020-11-13 22:22:27 -07:00
meejah
c7f4f36c8a merge config 2020-11-13 21:30:23 -07:00
meejah
b6fbfeee48 more utests 2020-11-13 18:23:21 -07:00
meejah
25ff7c6f55 Merge branch 'master' into 2916.grid-manager-proposal.5 2020-11-13 16:48:14 -07:00
Itamar Turner-Trauring
6264cf15b1 Fix some failing tests. 2020-10-30 14:08:09 -04:00
Itamar Turner-Trauring
bcb6822171 Merge branch '3485.backported-configparser-for-py-2' into 3479.test-node-python-3 2020-10-27 14:06:35 -04:00
Itamar Turner-Trauring
4a54377208 Some more fixes. 2020-10-26 16:48:18 -04:00
Itamar Turner-Trauring
4dc1adc817 Some progress towards passing Python 2 tests. 2020-10-26 16:37:00 -04:00
Itamar Turner-Trauring
375ed5096c Config parsing now always returns Unicode. 2020-10-26 12:12:49 -04:00
Itamar Turner-Trauring
f50fd8e474 Switch to new configparser backport. 2020-10-26 12:12:46 -04:00
Itamar Turner-Trauring
51d472e221 More progress towards passing tests on Python 3. 2020-10-16 11:13:11 -04:00
Itamar Turner-Trauring
f689d59a40 More passing tests on Python 3. 2020-10-16 10:55:33 -04:00
Itamar Turner-Trauring
bcd7cdf86f Some passing tests on Python 3. 2020-10-16 10:47:49 -04:00
Ross Patterson
da046108e2 feat(py3): Fix section name unicode type clash 
Before this change, there was an error in the tests in python 3.  I couldn't find any
clues from the history as to why explicit byte-string literals were used here.  This
change addresses the error under Python 3 and doesn't cause any regressions under Python
2 in the test suite.  This changes two tests from failures to passing under Python 3 and
increases coverage a bit:

```diff
--- ../../.tox/make-test-py3-all-old.log	2020-10-04 21:42:22.931028265 -0700
+++ ../../.tox/make-test-py3-all-new.log	2020-10-04 21:49:19.164127097 -0700
@@ -313,7 +313,7 @@
 ####-##-##T##:##:##-###0 [twisted.scripts._twistd_unix.UnixAppLogger#info] twistd 20.3.0 (/home/rpatterson/src/work/sfu/tahoe-lafs/.tox/py36-coverage/bin/python 3.6.12) starting up.
 ####-##-##T##:##:##-###0 [twisted.scripts._twistd_unix.UnixAppLogger#info] reactor class: mock.mock.MagicMock.
 ####-##-##T##:##:##-###0 [twisted.scripts._twistd_unix.UnixAppLogger#info] Server Shut Down.
-                                         [FAIL]
+                                           [OK]
 allmydata.test
   cli
     test_status ...                                                     [ERROR]
@@ -1191,7 +1191,7 @@
     test_disabled_but_helper ...                                         [FAIL]
     test_disabled_but_storage ...                                        [FAIL]
   Configuration
-    test_create_client_invalid_config ...                                [FAIL]
+    test_create_client_invalid_config ...                                  [OK]
     test_read_invalid_config ...                                           [OK]
   IntroducerNotListening
     test_port_none_introducer ...                                        [FAIL]
@@ -2024,18 +2024,6 @@
 ===============================================================================
 [FAIL]
 Traceback (most recent call last):
-  File "/home/rpatterson/src/work/sfu/tahoe-lafs/.tox/py36-coverage/lib/python3.6/site-packages/mock/mock.py", line 1369, in patched
-    return func(*newargs, **newkeywargs)
-  File "/home/rpatterson/src/work/sfu/tahoe-lafs/src/allmydata/test/cli/test_start.py", line 265, in test_run_invalid_config
-    output,
-  File "/home/rpatterson/src/work/sfu/tahoe-lafs/.tox/py36-coverage/lib/python3.6/site-packages/twisted/trial/_synctest.py", line 494, in assertIn
-    % (containee, container))
-twisted.trial.unittest.FailTest: 'invalid section' not in '\nUnknown error\nTraceback (most recent call last):\n  File "/home/rpatterson/src/work/sfu/tahoe-lafs/src/allmydata/test/cli/test_start.py", line 232, in cwr\n    fn()\n  File "/home/rpatterson/src/work/sfu/tahoe-lafs/src/allmydata/scripts/run_common.py", line 155, in start\n    d = service_factory()\n  File "/home/rpatterson/src/work/sfu/tahoe-lafs/src/allmydata/scripts/run_common.py", line 136, in <lambda>\n    u"client": lambda: maybeDeferred(namedAny("allmydata.client.create_client"), self.basedir),\n  File "/home/rpatterson/src/work/sfu/tahoe-lafs/.tox/py36-coverage/lib/python3.6/site-packages/twisted/internet/defer.py", line 151, in maybeDeferred\n    result = f(*args, **kw)\n--- <exception caught here> ---\n  File "/home/rpatterson/src/work/sfu/tahoe-lafs/src/allmydata/client.py", line 243, in create_client\n    config = read_config(basedir, u"client.port")\n  File "/home/rpatterson/src/work/sfu/tahoe-lafs/src/allmydata/client.py", line 219, in read_config\n    _valid_config=_valid_config(),\n  File "/home/rpatterson/src/work/sfu/tahoe-lafs/src/allmydata/node.py", line 192, in read_config\n    configutil.validate_config(config_fname, parser, _valid_config)\n  File "/home/rpatterson/src/work/sfu/tahoe-lafs/src/allmydata/util/configutil.py", line 72, in validate_config\n    if not valid_config.is_valid_section(section):\n  File "/home/rpatterson/src/work/sfu/tahoe-lafs/src/allmydata/util/configutil.py", line 113, in is_valid_section\n    self._is_valid_section(section_name)\n  File "/home/rpatterson/src/work/sfu/tahoe-lafs/src/allmydata/util/configutil.py", line 141, in <lambda>\n    return lambda *a, **kw: f(*a, **kw) or g(*a, **kw)\n  File "/home/rpatterson/src/work/sfu/tahoe-lafs/src/allmydata/client.py", line 72, in _is_valid_section\n    section_name.startswith(b"storageserver.plugins.") or\nbuiltins.TypeError: startswith first arg must be str or a tuple of str, not bytes\n'
-
-allmydata.test.cli.test_start.RunTests.test_run_invalid_config
-===============================================================================
-[FAIL]
-Traceback (most recent call last):
   File "/home/rpatterson/src/work/sfu/tahoe-lafs/.tox/py36-coverage/lib/python3.6/site-packages/twisted/trial/_asynctest.py", line 75, in _eb
     raise self.failureException(output)
 twisted.trial.unittest.FailTest:
@@ -2222,34 +2210,6 @@
     result = result.throwExceptionIntoGenerator(g)
   File "/home/rpatterson/src/work/sfu/tahoe-lafs/.tox/py36-coverage/lib/python3.6/site-packages/twisted/python/failure.py", line 512, in throwExceptionIntoGenerator
     return g.throw(self.type, self.value, self.tb)
-  File "/home/rpatterson/src/work/sfu/tahoe-lafs/src/allmydata/test/test_node.py", line 741, in test_create_client_invalid_config
-    yield client.create_client(self.basedir)
-  File "/home/rpatterson/src/work/sfu/tahoe-lafs/.tox/py36-coverage/lib/python3.6/site-packages/twisted/trial/_synctest.py", line 355, in __exit__
-    self._expectedName, reason.getTraceback()),
-twisted.trial.unittest.FailTest: builtins.TypeError raised instead of UnknownConfigError:
- Traceback (most recent call last):
-  File "/home/rpatterson/src/work/sfu/tahoe-lafs/.tox/py36-coverage/lib/python3.6/site-packages/twisted/internet/defer.py", line 1529, in _cancellableInlineCallbacks
-    _inlineCallbacks(None, g, status)
-  File "/home/rpatterson/src/work/sfu/tahoe-lafs/.tox/py36-coverage/lib/python3.6/site-packages/twisted/internet/defer.py", line 1416, in _inlineCallbacks
-    result = result.throwExceptionIntoGenerator(g)
-  File "/home/rpatterson/src/work/sfu/tahoe-lafs/.tox/py36-coverage/lib/python3.6/site-packages/twisted/python/failure.py", line 512, in throwExceptionIntoGenerator
-    return g.throw(self.type, self.value, self.tb)
-  File "/home/rpatterson/src/work/sfu/tahoe-lafs/src/allmydata/test/test_node.py", line 741, in test_create_client_invalid_config
-    yield client.create_client(self.basedir)
---- <exception caught here> ---
-  File "/home/rpatterson/src/work/sfu/tahoe-lafs/src/allmydata/test/test_node.py", line 741, in test_create_client_invalid_config
-    yield client.create_client(self.basedir)
-builtins.TypeError: startswith first arg must be str or a tuple of str, not bytes
-
-
-allmydata.test.test_node.Configuration.test_create_client_invalid_config
-===============================================================================
-[FAIL]
-Traceback (most recent call last):
-  File "/home/rpatterson/src/work/sfu/tahoe-lafs/.tox/py36-coverage/lib/python3.6/site-packages/twisted/internet/defer.py", line 1416, in _inlineCallbacks
-    result = result.throwExceptionIntoGenerator(g)
-  File "/home/rpatterson/src/work/sfu/tahoe-lafs/.tox/py36-coverage/lib/python3.6/site-packages/twisted/python/failure.py", line 512, in throwExceptionIntoGenerator
-    return g.throw(self.type, self.value, self.tb)
   File "/home/rpatterson/src/work/sfu/tahoe-lafs/src/allmydata/test/test_node.py", line 704, in test_port_none_introducer
     yield create_introducer(basedir)
   File "/home/rpatterson/src/work/sfu/tahoe-lafs/.tox/py36-coverage/lib/python3.6/site-packages/twisted/trial/_synctest.py", line 355, in __exit__
@@ -8717,7 +8677,7 @@
 -------------------------------------------------------------------------------
 Ran 1300 tests in ###.###s

-FAILED (skips=42, expectedFailures=1, failures=35, errors=531, successes=707)
+FAILED (skips=42, expectedFailures=1, failures=33, errors=531, successes=709)

 Unknown error
 Traceback (most recent call last):
@@ -8806,7 +8766,7 @@
 src/allmydata/scripts/create_node.py                   302     98    114     10    66%   224-229, 235, 257-260, 262-265, 268-269, 289-292, 295-298, 329, 339, 347-380, 391-445, 461-477, 223->224, 234->235, 256->257, 261->262, 266->277, 267->268, 288->289, 294->295, 328->329, 338->339
 src/allmydata/scripts/debug.py                         719    638    202      0     9%   14, 31-32, 35-49, 52-60, 63-142, 146-154, 157-164, 168-217, 220-304, 307-401, 407, 417, 437-465, 468-485, 488-602, 606, 609-611, 637-648, 653-656, 659, 683-689, 692-810, 813-842, 845-848, 851-865, 869, 888, 891-940, 946, 949-950, 957, 960-961, 967-972, 984-985, 999-1000, 1003-1004, 1020-1021, 1025-1031, 1046-1050
 src/allmydata/scripts/default_nodedir.py                15      5      6      2    57%   10-14, 9->10, 16->exit
-src/allmydata/scripts/run_common.py                    135     18     24      6    85%   37, 41-46, 59-60, 149, 158, 192-193, 216-220, 226-227, 55->62, 135->exit, 135->exit, 148->149, 191->192, 225->226
+src/allmydata/scripts/run_common.py                    135     17     24      5    86%   37, 41-46, 59-60, 158, 192-193, 216-220, 226-227, 55->62, 135->exit, 135->exit, 191->192, 225->226
 src/allmydata/scripts/runner.py                        138     49     42      5    61%   84-85, 91, 97-99, 150, 153-160, 174-181, 188-192, 202-232, 237-252, 255, 31->36, 149->150, 151->153, 185->188, 254->255
 src/allmydata/scripts/slow_operation.py                 69     56     22      0    14%   15-44, 47-52, 55-61, 64-83
 src/allmydata/scripts/stats_gatherer.py                 44     16     12      3    59%   8, 30, 75-79, 84-93, 7->8, 29->30, 74->75
@@ -8893,7 +8853,7 @@
 src/allmydata/windows/fixups.py                        133    133     54      0     0%   1-237
 src/allmydata/windows/registry.py                       42     42     12      0     0%   1-77
 ------------------------------------------------------------------------------------------------
-TOTAL                                                27477  11782   8244    605    54%
+TOTAL                                                27477  11781   8244    604    54%

 18 files skipped due to complete coverage.
 + '[' '!' -z 1 ']'
```
 2020-10-04 21:50:09 -07:00
meejah
9c89100f6d Merge master 2020-10-01 19:04:01 -06:00
Itamar Turner-Trauring
7d8320b843 Python 3 fixes. 2020-09-16 11:13:23 -04:00
Itamar Turner-Trauring
a758f32edf Try to make test_storage import on Python 3. 2020-08-26 10:53:02 -04:00
Maciej Fijalkowski
93c474c39b make test_util pass under both pythons 2020-07-29 09:00:50 +02:00
meejah
0bdfae845e actually put grid-manager-certificates in announcement 2020-05-07 22:30:05 -06:00
meejah
e6cb700bcc incorrect merge conflict-resolution 2020-05-07 20:43:37 -06:00
meejah
38069e51fb keyutil -> ed25519 2020-05-07 20:43:17 -06:00
meejah
91af588c9b post-merge fixups (keyutil, preferred_peers) 2020-05-07 20:08:02 -06:00
meejah
60fea95e23 Merge branch 'master' into 2916.grid-manager-proposal.5 2020-05-07 18:58:55 -06:00
meejah
85d8e24421 ascii -> utf8 2020-05-07 15:27:47 -06:00
Jean-Paul Calderone
ef73e18f3d
Remove the greater part of the implementation 2020-02-13 10:43:50 -05:00
Jean-Paul Calderone
a55719cdc4 Raise UnknownConfigError when a server is configured with an unknown storage plugin 2019-10-16 19:56:29 -04:00
Jean-Paul Calderone
8c1f536ba4 Merge remote-tracking branch 'origin/master' into integration/storage-economics 2019-10-04 12:02:00 -04:00
meejah
9050fdcea6 explicitly save refs to webish and operations (instead of using Service methods) 2019-08-24 12:31:08 -06:00
Jean-Paul Calderone
9940beaae1 Thread a _Config all the way down 2019-08-19 16:09:26 -04:00
Jean-Paul Calderone
a47463e032 Pass _Config instead of a smaller dict to get_client_resource 2019-08-19 11:21:03 -04:00
Jean-Paul Calderone
21d735ece9 whitespace more conforming to PEP8 2019-08-03 06:39:31 -04:00
Jean-Paul Calderone
d69de15664 implement the feature 
improve the test slightly, too, to verify the configuration supplied to the
plugin is as expected.
 2019-08-03 06:39:31 -04:00
Jean-Paul Calderone
9842447a07 Don't start or announce anonymous access if config says not to 2019-08-03 06:39:31 -04:00
Jean-Paul Calderone
853cf62530 Allow [storage]anonymous through the validator 
And provide a helpful accessor for reading it
 2019-08-03 06:39:31 -04:00
Jean-Paul Calderone
311afa8a75 Test & fix supplying plugin configuration 2019-08-03 06:39:31 -04:00
Jean-Paul Calderone
b5a2c70a4a create a StorageClientConfig object 
Make it easier to pass more storage configuration down into StorageFarmBroker
and beyond
 2019-08-03 06:39:31 -04:00
Jean-Paul Calderone
6b7e0dd700 add a test for the negative case 2019-08-03 06:39:31 -04:00
Jean-Paul Calderone
895cf37a84 docstrings 2019-08-03 06:19:01 -04:00
Jean-Paul Calderone
53861e2a0f Change the shape of the storage announcement(s) 
Instead of generating a sequence of announcements like:

    - anonymous storage server announcement
    - plugin 1 storage server announcement
    - ...
    - plugin N storage server announcement

The client now generates a single announcement like:

    - anonymous storage server details
    - storage-options
      - plugin 1 storage server details
      - ...
      - plugin N storage server details
 2019-08-03 06:19:01 -04:00
Jean-Paul Calderone
9c240b61ac Persist the furl ourselves rather than relying on Foolscap 
Going via our config abstraction here will let us change how config is
persisted more easily, later.
 2019-08-03 06:19:01 -04:00
Jean-Paul Calderone
6068b6c1b2 don't reach through the tahoe-lafs config object 2019-08-03 06:19:01 -04:00
Jean-Paul Calderone
d2e16df5cc link to a ticket about implementing better missing-plugin behavior 2019-08-03 06:19:01 -04:00
Jean-Paul Calderone
251eda0b80 rename introducer_factory parameter to be private 2019-08-03 06:19:01 -04:00
Jean-Paul Calderone
58db131787 remove unused imports 2019-08-03 06:19:01 -04:00
Jean-Paul Calderone
fd9ae24149 fix indentation 2019-08-03 06:19:01 -04:00
Jean-Paul Calderone
756c21c251 actually provide validating client-config-from-string function 2019-08-03 06:19:01 -04:00
Jean-Paul Calderone
6cf48f7d4f Separate async initialization from _Client.__init__ 2019-08-03 06:19:01 -04:00
Jean-Paul Calderone
a6959d111c Log init_storage and its result (particularly failures) 2019-08-03 06:19:01 -04:00
Jean-Paul Calderone
7919cf205e Test the actual interface 
get_storage_server is supposed to return a Deferred
 2019-08-03 06:19:01 -04:00
Jean-Paul Calderone
a45e2bebfe Allow the new plugins item in the [storage] section 2019-08-03 06:19:01 -04:00
Jean-Paul Calderone
f606beb065 Test and support plugins without any configuration 2019-08-03 06:19:01 -04:00
Jean-Paul Calderone
e2982c0129 Support multiple plugins 2019-08-03 06:19:01 -04:00
Jean-Paul Calderone
646cd452b9 Add tests for announcements for plugins 
And a basic implementation
 2019-08-03 06:19:01 -04:00
Jean-Paul Calderone
212f96dfe7 Baseline tests for anonymous storage server announcements 2019-08-03 05:34:21 -04:00
Jean-Paul Calderone
fb4c5cf91f Allow for dynamic configuration validation rules 2019-08-03 05:34:21 -04:00
meejah
5644f421fe get rid of ed25519.bytes_from_* methods 2019-07-08 12:46:22 -06:00
meejah
47633fb01e unused import 2019-06-26 14:55:35 -06:00
meejah
dd55accec8 Merge branch 'master' into 3031-replace-pycryptopp 2019-06-26 14:42:43 -06:00
meejah
47ccdb0177 refactor ed25519 helpers to functional style 
eliminates the wrapper classes and uses some more-explicit
names throughout (e.g "sk" -> "signing_key")
 2019-06-13 22:19:28 -06:00
meejah
df4671f90e refactor away from pycryptopp "helper" classes 2019-06-13 22:19:28 -06:00
heartsucker
8063d93c6d replaced pytcryptopp rsa with our own wrapper 2019-06-13 22:19:28 -06:00
heartsucker
9e31bfe2f4 update code/test to use new ed25512 module 2019-06-13 22:19:28 -06:00
heartsucker
3a5a0fb572 replaced uses of pycryptopp's ed25519 with our own 2019-06-13 22:18:55 -06:00
Jean-Paul Calderone
b604d08463 Add a test for the success case 2019-06-12 16:47:25 -04:00
meejah
f0e3b69f90 switch around how we do config (avoid space-separated filenames) 2019-05-30 16:01:46 -06:00
meejah
2cf36cf884 flake8 2019-05-30 16:01:46 -06:00
meejah
c88409afc5 add a note 2019-05-30 16:01:46 -06:00
meejah
4f481bbb4c remove FIXME 2019-05-30 16:01:45 -06:00
meejah
e453db879a make options work, config API usage 2019-05-30 16:01:45 -06:00
meejah
82a74e8986 updates and a bunch of debugging for cert/grid-manager handling 2019-05-30 16:01:45 -06:00
meejah
36c471fdef don't need our pubkey for anything 2019-05-30 16:01:45 -06:00
meejah
e5f608f80e fixes to Options use 2019-05-30 16:01:45 -06:00
meejah
2031b723b8 various fixes (flake8, errors) 2019-05-30 16:01:45 -06:00
meejah
ee3e1cbcf2 wip; updates to grid-manager impl 2019-05-30 16:01:45 -06:00
meejah
64eb9d7c30 hack in grid-manager announcements to storage-servers 2019-05-30 16:01:45 -06:00
tpltnt
c8f11dc2d3 ported old-style classes to new-style 2019-05-26 08:28:18 +02:00
advanced4
ef127f113d
update client.py to use DEFAULT_MAX_SEGMENT_SIZE 2019-05-08 16:04:57 -04:00
Jean-Paul Calderone
53b40ef0e0 don't double start; don't try to handle double start. 2019-03-28 16:10:36 -04:00
meejah
1ad2174df9 windows yaml.safe_load returns None on unreadable files.. 2018-09-11 11:14:41 -06:00
meejah
3d35723a59 windows yamlutil.safe_load returns None when files are unreadable 2018-09-10 21:58:28 -06:00
meejah
692128ad1d better defaults handling 2018-09-10 21:58:28 -06:00
meejah
f6b62ca192 import 2018-09-10 21:58:28 -06:00
meejah
a344bb24dc immutable default 2018-09-10 21:58:27 -06:00
meejah
dee8b72206 dead code 2018-09-10 21:58:27 -06:00
meejah
9aae4bd459 only handle ENOENT 2018-09-10 21:58:27 -06:00
meejah
a5287add80 clarify comments 2018-09-10 21:58:27 -06:00
meejah
efce7b1f6a use Failure for all errors from async methods 2018-09-10 21:58:27 -06:00
meejah
714b0887dc more docstrings 2018-09-10 21:58:27 -06:00
meejah
c22d7c632e docstring improvements 2018-09-10 21:58:27 -06:00
meejah
3974af6660 redundant imports 2018-09-10 21:58:27 -06:00
meejah
edc50f655b get rid of is_tub_listening 2018-09-10 21:58:27 -06:00
meejah
a82aa4ba2c code cleanup, docs 2018-09-10 21:58:27 -06:00
meejah
e74d2a7d01 get rid of redundant 'introducer_clients' var 2018-09-10 21:58:27 -06:00