ExternalVendorCode/tahoe-lafs
1
0
Fork 0
mirror of https://github.com/tahoe-lafs/tahoe-lafs.git synced 2025-01-22 12:28:05 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity
15,242 Commits 185 Branches 90 Tags 91 MiB
7bcd90cd91
Commit Graph

8959 Commits

Author SHA1 Message Date
Itamar Turner-Trauring
74e77685a3 Get rid of DeferredList. 2023-02-23 10:07:57 -05:00
meejah
6aff94dd8f flake8, more frozen 2023-02-22 00:15:32 -07:00
meejah
c587c9ed19 Merge branch '2916.grid-manager-proposal.6' of https://github.com/tahoe-lafs/tahoe-lafs into 2916.grid-manager-proposal.6 2023-02-21 23:21:09 -07:00
meejah
0ae7da7352 prop up the fragile scaffolding 2023-02-21 23:20:28 -07:00
meejah
69a480dc08 rewrite test 2023-02-21 18:44:09 -07:00
meejah
fb10e13c68 might want to use these at parse time 2023-02-21 18:16:45 -07:00
meejah
70459cfbf1 use possibly-overriden stderr 2023-02-21 16:47:58 -07:00
meejah
6ee5c75880 comment -> ticket 2023-02-21 16:13:11 -07:00
meejah
00ef4661a6 ISO dates, not seconds 2023-02-21 16:06:24 -07:00
meejah
d91bfcb1d2 clarify 2023-02-21 14:38:43 -07:00
meejah
82045b4298 store signature as raw bytes, not base32 2023-02-21 14:19:44 -07:00
meejah
1b6d5e1bda Revert "use attrs directly" 
This reverts commit 5b14561ec0.
 2023-02-21 11:56:46 -07:00
Itamar Turner-Trauring
32768e310a Unit test for _pick_a_http_server. 2023-02-21 11:30:47 -05:00
Itamar Turner-Trauring
2ac6580c26 Welcome to the world of tomorrow. 2023-02-21 11:00:14 -05:00
Itamar Turner-Trauring
95bb7afba7 Sketch of happy eyeballs. 2023-02-21 10:42:06 -05:00
Itamar Turner-Trauring
2811c80dc3 Fix timeout in parallel benchmark. 2023-02-21 09:57:21 -05:00
Itamar Turner-Trauring
154f1ce143 No need for sleep. 2023-02-21 09:38:54 -05:00
Itamar Turner-Trauring
c9d140b05d Merge remote-tracking branch 'origin/master' into 3957-mutable-over-http-speed 2023-02-21 09:38:36 -05:00
meejah
38669cc3ce define -> frozen 2023-02-21 02:15:20 -07:00
meejah
032b852bab define -> frozen 2023-02-21 01:51:13 -07:00
meejah
43d29986a6
Update src/allmydata/cli/grid_manager.py 
Co-authored-by: Jean-Paul Calderone <exarkun@twistedmatrix.com>
 2023-02-21 00:04:03 -07:00
meejah
affe0cb37b fine? 2023-02-21 00:02:53 -07:00
meejah
bdf4c49a34 fine, move the if statement 2023-02-20 22:57:15 -07:00
meejah
a6cf06cc6d http needs access to GridManager 2023-02-20 12:26:44 -07:00
meejah
5b14561ec0 use attrs directly 2023-02-20 12:02:34 -07:00
meejah
464498f6bf Merge branch 'master' into 2916.grid-manager-proposal.6 2023-02-20 11:24:20 -07:00
Itamar Turner-Trauring
acc9cd2f9f Merge remote-tracking branch 'origin/master' into 3946-less-chatty-downloads 2023-02-20 12:45:27 -05:00
Itamar Turner-Trauring
b14b2d0409 Use a nicer shutdown mechanism. 2023-02-20 12:01:14 -05:00
dlee
86dbcb21ce Refactored verify function to update deprecated getargspec function with getfullargspec and maintained strictness 2023-02-17 16:24:32 -06:00
dlee
badba97ff2 Type annotations added for wormholetesting.py 2023-02-17 16:20:29 -06:00
Itamar Turner-Trauring
3686fe7fa3 Merge remote-tracking branch 'origin/master' into 3946-less-chatty-downloads 2023-02-14 14:46:36 -05:00
dlee
a7ddcbf868 Changes requested 2023-02-14 12:50:37 -06:00
dlee
b7cadfc53a Fixes bad practice of naming variable a built-in type 2023-02-14 11:38:35 -06:00
dlee
19e58f19ca Fixes truthy conditional 2023-02-14 11:21:57 -06:00
Jean-Paul Calderone
97686f6d42 Merge remote-tracking branch 'origin/master' into 3957-mutable-over-http-speed 2023-02-13 09:19:49 -05:00
Itamar Turner-Trauring
3bc3cf39d0 Test using an integration test. 2023-02-07 09:45:50 -05:00
Itamar Turner-Trauring
51d44ba676 Document. 2023-02-07 09:06:57 -05:00
Itamar Turner-Trauring
f4255cdaa3 More accurate names. 2023-02-07 09:03:45 -05:00
dlee
eb26c97ef7 implicit_optional flag added and errors related to flag fixed 2023-02-06 15:29:53 -06:00
Itamar Turner-Trauring
c4114e032e Fix type signature. 2023-02-06 15:33:08 -05:00
Itamar Turner-Trauring
4576d10915 Add an explanation. 2023-02-06 13:57:33 -05:00
Itamar Turner-Trauring
5909f451e3 Use the CPU thread pool for CBOR validation. 2023-02-06 13:54:47 -05:00
Itamar Turner-Trauring
b221954946 A working thread pool. 2023-02-06 13:09:22 -05:00
dlee
31c5b78e6a Add back rsa.py accidentally removed file on website 2023-02-03 21:35:55 -06:00
Daniel
80db4a9de4
Delete rsa.py 2023-02-03 21:25:24 -06:00
dlee
e2e33933a8 Forgot to push newsfragment 2023-02-03 16:48:06 -06:00
Itamar Turner-Trauring
b477c59e15 Actually have a working run-in-thread code path 2023-01-31 15:53:16 -05:00
Itamar Turner-Trauring
921a2083dc Make sure (immutable) test vectors are run with the segment size that was used 
to generate them.
 2023-01-25 15:47:35 -05:00
Itamar Turner-Trauring
28ca5abe3c Merge remote-tracking branch 'origin/master' into 3957-mutable-over-http-speed 2023-01-25 10:53:15 -05:00
Itamar Turner-Trauring
af53f5b8b9 Merge remote-tracking branch 'origin/master' into 3946-less-chatty-downloads 2023-01-25 10:12:20 -05:00
Itamar Turner-Trauring
cf4d767535 Fix whitespace. 2023-01-24 13:28:24 -05:00
Itamar Turner-Trauring
6ccccde9e0 Increase the max default segment size. 2023-01-24 13:15:55 -05:00
Itamar Turner-Trauring
d5f5d394dd Test changes in max default segment size (both directions). 2023-01-24 13:15:40 -05:00
Itamar Turner-Trauring
ba793e2c16 Make it actually work. 2023-01-24 11:10:50 -05:00
Itamar Turner-Trauring
3e6778268f Merge remote-tracking branch 'origin/master' into 3957-mutable-over-http-speed 2023-01-23 11:39:54 -05:00
Itamar Turner-Trauring
1f3993b689 Don't block on CDDL validation. 2023-01-23 11:37:13 -05:00
Jean-Paul Calderone
e2eac5855c Remove handling for older versions of pycddl 
>=0.4 is now a hard-requirement
 2023-01-23 08:52:00 -05:00
Jean-Paul Calderone
1a807a0232 mollify the type checker 2023-01-12 16:32:32 -05:00
Jean-Paul Calderone
d5628f9c84 Merge remote-tracking branch 'origin/master' into 3962.pre-determined-rsa-keys 2023-01-12 16:17:09 -05:00
Jean-Paul Calderone
c856f1aa29 Censor private key values in the HTTP log, too. 2023-01-12 16:16:55 -05:00
Jean-Paul Calderone
47ec418f7a Test that we can also download the mutable data via the web interface 2023-01-12 16:09:29 -05:00
Jean-Paul Calderone
e6ef45d337 test that we can also download the mutable 2023-01-12 15:37:12 -05:00
Jean-Paul Calderone
2d23e2e640 some doc improvements 2023-01-12 15:37:07 -05:00
Jean-Paul Calderone
2490f0f58a some minor rationalization of the return type 2023-01-12 15:33:37 -05:00
Itamar Turner-Trauring
828fc588c5 Add minimal docstrings. 2023-01-10 11:10:31 -05:00
Itamar Turner-Trauring
ccb5956645 0 is also valid FD. 2023-01-10 11:04:22 -05:00
Itamar Turner-Trauring
7c506057b5 Merge remote-tracking branch 'origin/master' into 3956-mutable-uploads 2023-01-09 17:05:31 -05:00
Itamar Turner-Trauring
f6d9c5a1b2 Fix PyInstaller. 2023-01-09 10:46:09 -05:00
Itamar Turner-Trauring
22227c7094 Support old pycddl too so nix can keep working. 2023-01-09 10:31:48 -05:00
Jean-Paul Calderone
3ce5ee6f03 get Literal from somewhere it is more likely to be 2023-01-07 07:17:40 -05:00
Jean-Paul Calderone
e893d06cb3 RSAPrivateKey certainly does have this method 
I don't know why mypy fails to see it.
 2023-01-06 21:00:21 -05:00
Jean-Paul Calderone
8c56ccad72 fall back to *with* validation, not without 2023-01-06 21:00:10 -05:00
Jean-Paul Calderone
85234b07a0 load the right kind of key! 2023-01-06 21:00:04 -05:00
Jean-Paul Calderone
c9e23dea13 we should always be able to get these 
and we always need overload now
 2023-01-06 20:59:48 -05:00
Jean-Paul Calderone
a806b2faba Fix some more mypy errors 2023-01-06 18:11:47 -05:00
Jean-Paul Calderone
01b14fe05c Merge remote-tracking branch 'origin/master' into 3962.pre-determined-rsa-keys 2023-01-06 17:30:01 -05:00
Jean-Paul Calderone
2dc6466ef5 fix some errors reported by mypy 2023-01-06 17:12:59 -05:00
Jean-Paul Calderone
e829b891b3 important data file ... 2023-01-06 15:51:59 -05:00
Jean-Paul Calderone
3ff9c45e95 expose the private-key feature in the tahoe put cli 2023-01-06 15:40:48 -05:00
Jean-Paul Calderone
e236cc95a5 Move get_keypair to a shared location 2023-01-06 15:36:08 -05:00
Itamar Turner-Trauring
3de5ebde48 Merge branch '3956-mutable-uploads' into 3957-mutable-over-http-speed 2023-01-06 11:05:11 -05:00
Jean-Paul Calderone
3423bfb351 Expose the pre-constructed keypair functionality to the HTTP API 2023-01-03 11:31:29 -05:00
Jean-Paul Calderone
c7bb190290 Factor some SSK "signature" key handling code into a more reusable shape 
This gives the test suite access to the derivation function so it can
re-derive certain values to use as expected results to compare against actual
results.
 2023-01-03 10:38:18 -05:00
Jean-Paul Calderone
5bad92cfc5 Another Python 2 remnant cleanup 2023-01-03 10:34:39 -05:00
Jean-Paul Calderone
a58d8a567a Clean up some more Python 2 remnants 2023-01-03 10:33:37 -05:00
Jean-Paul Calderone
6b58b66677 Clean up some Python 2 remnants 2023-01-03 10:32:03 -05:00
Jean-Paul Calderone
f6d9c33526 Give slightly better error messages from rsa key validation failure 2023-01-03 10:28:59 -05:00
Jean-Paul Calderone
23f2d8b019 add some type annotations to allmydata.crypto.rsa 2023-01-03 10:28:32 -05:00
Jean-Paul Calderone
15e22dcc52 Add keypair to NodeMaker.create_mutable_file 
Previously `NodeMaker` always took responsibility for generating a keypair to
use.  Now the caller may supply one.
 2023-01-02 19:29:13 -05:00
Jean-Paul Calderone
3f8bec0ecc Merge remote-tracking branch 'origin/master' into 3914.faster-rsa-tests 2023-01-02 11:46:10 -05:00
Jean-Paul Calderone
1c643ebbaf
Merge pull request #1237 from tahoe-lafs/3954.faster-http 
Faster HTTP for immutable downloads

Fixes: ticket:3954
 2023-01-02 10:09:55 -05:00
Itamar Turner-Trauring
d1b464d0d8 Writing large files can involve many writes. 2022-12-21 09:35:10 -05:00
Itamar Turner-Trauring
54da6eb60a Remove unneeded imports. 2022-12-21 09:34:25 -05:00
Itamar Turner-Trauring
1a4dcc70e2 Support large mutable uploads in a memory-efficient manner. 2022-12-21 09:24:31 -05:00
Jean-Paul Calderone
6b57b03157 Merge remote-tracking branch 'origin/master' into 3942.missing-authorization-handling 2022-12-21 05:51:39 -05:00
Jean-Paul Calderone
80caf04966
Merge pull request #1234 from tahoe-lafs/3947.mutable-test_version-async-def 
Convert some tests for mutables to use `async def`

Fixes: ticket:3947
 2022-12-21 05:50:07 -05:00
Itamar Turner-Trauring
1d3464a430 Add end-to-end MDMF test. 2022-12-19 13:37:20 -05:00
Itamar Turner-Trauring
96347e22e2 Make a test demonstrating the problem. 2022-12-15 13:14:49 -05:00
Jean-Paul Calderone
c014ad55b1 remove Python 2 boilerplate 2022-12-14 08:48:02 -05:00
Jean-Paul Calderone
05c7450376 Try to use an upcoming python-cryptography feature to avoid some costs 
If the key is the wrong number of bits then we don't care about any other
validation results because we're just going to reject it.  So, check that
before applying other validation, if possible.

This is untested since the version of python-cryptography that supports it is
not released yet and I don't feel like setting up a Rust build tool chain at
the moment.
 2022-12-14 08:47:05 -05:00
Itamar Turner-Trauring
6ae40a932d A much more reasonable number of HTTP connections. 2022-12-13 15:54:19 -05:00
Itamar Turner-Trauring
1eba202c08
Merge pull request #1231 from tahoe-lafs/3939-faster-http-protocol 
Faster http protocol, part 1 (and maybe faster Foolscap too, while we're at it)

Fixes ticket:3939
 2022-12-05 14:05:55 -05:00
Jean-Paul Calderone
daecfc63b9
Merge pull request #1233 from tahoe-lafs/3874.fix-testing-helper-encoding 
Fix a testing helper
 2022-12-02 10:27:58 -05:00
Jean-Paul Calderone
c542b84637 remove the annotations 
everything is broken on older pythons
 2022-12-02 08:47:07 -05:00
Jean-Paul Calderone
c6cc3708f4 Fixup the annotations a bit 2022-12-02 08:38:46 -05:00
Jean-Paul Calderone
b40d882fce remove unused import 2022-12-02 08:28:22 -05:00
Jean-Paul Calderone
a84b278ecd support older pythons 2022-12-02 08:26:15 -05:00
Jean-Paul Calderone
9619e286f4 Switch the web testing double to BytesKeyDict 
This will catch more str/bytes errors by default than `dict`
 2022-12-02 08:16:02 -05:00
meejah
20b3594d12 exarkun wants a helper 2022-12-01 17:05:58 -07:00
meejah
36ed554627 proto -> transport 2022-12-01 17:03:48 -07:00
meejah
3d43cbccc9 log less-specific failures 2022-12-01 17:01:38 -07:00
meejah
c57d1aa8a9 Merge branch 'master' into 3921.exit-on-stdin-close 2022-12-01 16:58:37 -07:00
meejah
7ffcfcdb67
Update src/allmydata/test/test_runner.py 
Co-authored-by: Jean-Paul Calderone <exarkun@twistedmatrix.com>
 2022-12-01 16:47:40 -07:00
Itamar Turner-Trauring
c93ff23da7 Don't send empty string writes. 2022-12-01 14:54:28 -05:00
Itamar Turner-Trauring
8ed333b171 Correct explanation. 2022-12-01 14:45:45 -05:00
Itamar Turner-Trauring
0ba58070cd Tweaks. 2022-12-01 14:45:39 -05:00
Itamar Turner-Trauring
d4c202307c Better method name. 2022-12-01 14:43:49 -05:00
Itamar Turner-Trauring
17dfda6b5a More direct API. 2022-12-01 14:42:52 -05:00
Fon E. Noel NFEBE
b193ad3ed4 Correct addCleanup reference 
Some test_storage.py classes contain calls to cleanup methods
instead of references. This commit fixes that.

Signed-off-by: Fon E. Noel NFEBE <fenn25.fn@gmail.com>
 2022-11-30 16:53:22 +01:00
Jean-Paul Calderone
05dfa875a7 Quite a mypy warning about formatting bytes into a string 2022-11-30 09:46:13 -05:00
Jean-Paul Calderone
156954c621 no longer any need to "daisy chain" this value 2022-11-30 09:43:01 -05:00
Jean-Paul Calderone
a11eeaf240 Convert all of the asynchronous functions to use async and await 2022-11-30 09:30:37 -05:00
Jean-Paul Calderone
1acf8604ef Remove the Py2/Py3 compatibility header 2022-11-30 09:30:08 -05:00
Jean-Paul Calderone
5cebe91406 update the module docstring 2022-11-30 09:29:57 -05:00
Jean-Paul Calderone
02aeb68f17 Take care with str vs bytes in the implementation 
Also replace the intentional BAD_REQUEST with GONE for this case.
 2022-11-29 10:51:07 -05:00
Jean-Paul Calderone
d7fe25f7c7 Correct the assertion about how "not found" should be handled 
Behavior verified visually against a live client node:

```
❯ curl -v 'http://localhost:3456/uri/URI:CHK:cmtcxq7hwxvfxan34yiev6ivhy:qvcekmjtoetdcw4kmi7b3rtblvgx7544crnwaqtiewemdliqsokq:1:1:1'
*   Trying 127.0.0.1:3456...
* Connected to localhost (127.0.0.1) port 3456 (#0)
> GET /uri/URI:CHK:cmtcxq7hwxvfxan34yiev6ivhy:qvcekmjtoetdcw4kmi7b3rtblvgx7544crnwaqtiewemdliqsokq:1:1:1 HTTP/1.1
> Host: localhost:3456
> User-Agent: curl/7.83.1
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 410 Gone
< X-Frame-Options: DENY
< Referrer-Policy: no-referrer
< Server: TwistedWeb/22.10.0
< Date: Tue, 29 Nov 2022 15:39:47 GMT
< Content-Type: text/plain;charset=utf-8
< Accept-Ranges: bytes
< Content-Length: 294
< ETag: ui2tnwl5lltj5clzpyff42jdce-
<
NoSharesError: no shares could be found. Zero shares usually indicates a corrupt URI, or that no servers were connected, but it might also indicate severe corruption. You should perform a filecheck on this object to learn more.

The full error message is:
* Connection #0 to host localhost left intact
no shares (need 1). Last failure: None
```
 2022-11-29 10:49:20 -05:00
Jean-Paul Calderone
920467dcea Treat missing Authorization as the same as empty Authorization 2022-11-29 10:19:01 -05:00
Jean-Paul Calderone
f5b24d51e9 Add a test for missing Authorization 2022-11-29 10:18:57 -05:00
Itamar Turner-Trauring
06b57cd835 Merge remote-tracking branch 'origin/master' into 3939-faster-http-protocol 2022-11-28 11:07:01 -05:00
Itamar Turner-Trauring
aa80c9ef47 Be more robust. 2022-11-28 10:21:59 -05:00
Itamar Turner-Trauring
3ba166c2cb A bit more robust code. 2022-11-28 10:20:12 -05:00
Itamar Turner-Trauring
0f4dc91295 Refactor so internal attributes needn't leak. 2022-11-28 10:12:08 -05:00
Itamar Turner-Trauring
38d7430c57 Simplify. 2022-11-28 10:03:42 -05:00
Itamar Turner-Trauring
2ab8e3e8d2 Cancel timeout on failures too. 2022-11-28 10:02:56 -05:00
Fon E. Noel NFEBE
6b0fa64236 Clean up test_storage.py after refactor 
This PR cleans up errorneous changes resulting from
1d85a2c5cf and adds a
few improvements such as calling `super` implementations.

Making sure classes with functions returning deferreds
use `AsyncTestCase`

Signed-off-by: Fon E. Noel NFEBE <fenn25.fn@gmail.com>
 2022-11-25 13:36:22 +01:00
Itamar Turner-Trauring
41533f162e Not used anymore. 2022-11-23 10:20:32 -05:00
Itamar Turner-Trauring
3790280264 Documentation. 2022-11-23 10:16:49 -05:00
Itamar Turner-Trauring
fd9e50adf1 Simplify _WriteBuffer slightly. 2022-11-23 10:13:18 -05:00
Itamar Turner-Trauring
d1deda5fdd Unit tests for _WriteBuffer. 2022-11-23 10:09:53 -05:00
Itamar Turner-Trauring
7f1d7d4f46 Better explanation. 2022-11-23 09:53:07 -05:00
Itamar Turner-Trauring
d86d578034 Refactor to make core data structure easier to test in isolation. 2022-11-22 15:17:56 -05:00
Itamar Turner-Trauring
f638aec0af Refactor to use BytesIO. 2022-11-22 14:22:54 -05:00
Itamar Turner-Trauring
a4787ca45e Batch writes much more aggressively. 2022-11-22 14:12:14 -05:00
Itamar Turner-Trauring
c80469b50b Handle the Windows waker too. 2022-11-22 11:55:56 -05:00
Itamar Turner-Trauring
3a613aee70 Try a different approach to timeouts: dynamic, instead of hardcoded. 2022-11-21 12:24:50 -05:00
Fon E. Noel NFEBE
1d85a2c5cf Refactor more test_storage.py classes 
As a follow up to commit fbc8baa238
this refactor continues to remove deprecated methods and ensures
test classes either extend `SyncTestCase` or `AsyncTestCase`

Classes refactored:

- `MutableServer`
- `MDMFProxies`
- `Stats`
- `MutableShareFileTests`

Signed-off-by: Fon E. Noel NFEBE <fenn25.fn@gmail.com>
 2022-11-20 15:38:08 +01:00
Fon E. Noel NFEBE
4c18890e84 Merge branch 'master' into 3917-refactor-test-storage-py 2022-11-19 23:35:59 +01:00
Itamar Turner-Trauring
8cfdae2ab4 sigh 2022-11-18 15:26:02 -05:00
Itamar Turner-Trauring
aeaced848d Merge remote-tracking branch 'origin/master' into 3940-http-timeouts 2022-11-18 15:25:44 -05:00
Itamar Turner-Trauring
4c0c75a034 Fix DelayedCall leak in tests. 2022-11-18 13:56:54 -05:00
Itamar Turner-Trauring
9b21f1da90 Increase how many statuses are stored. 2022-11-17 11:35:10 -05:00
Itamar Turner-Trauring
d182a2f186 Add the delay to appropriate test. 2022-11-17 11:01:12 -05:00
Itamar Turner-Trauring
097d918a24 Sigh 2022-11-16 13:37:50 -05:00
Itamar Turner-Trauring
3531737347 Make timeouts less aggressive, CI machines are slow? 2022-11-16 10:36:11 -05:00
Itamar Turner-Trauring
2ab172ffca Try to set more aggressive timeouts when testing. 2022-11-16 10:26:29 -05:00
Itamar Turner-Trauring
9f5f287473 Nope, not helpful. 2022-11-16 09:57:39 -05:00
Itamar Turner-Trauring
a20943e10c As an experiment, see if this fixes failing CI. 2022-11-16 09:33:01 -05:00
Itamar Turner-Trauring
2a5e8e5971 Better cleanup. 2022-11-15 15:05:27 -05:00
Itamar Turner-Trauring
30a9877236 Merge branch '3783-storage-client-http' into 3940-http-timeouts 2022-11-15 14:38:48 -05:00
Itamar Turner-Trauring
f3fc426830 Switch to [storage] force_foolscap. 2022-11-15 14:36:14 -05:00
Itamar Turner-Trauring
fb52b4d302 Delete some garbage. 2022-11-15 14:22:30 -05:00
Itamar Turner-Trauring
8e4ac69032 Stop test mode when done. 2022-11-15 14:21:31 -05:00
Itamar Turner-Trauring
4aeb62b66c Use a constant. 2022-11-15 14:16:41 -05:00
Itamar Turner-Trauring
d700163aec Remove no-longer-relevant comment. 2022-11-15 14:14:29 -05:00
Itamar Turner-Trauring
6c80ad5290 Not necessary. 2022-11-15 14:13:50 -05:00
Itamar Turner-Trauring
d1287df629 The short timeout should be specific to the storage client's needs. 2022-11-15 14:02:19 -05:00
meejah
039c1d8037
Update src/allmydata/grid_manager.py 
Co-authored-by: Jean-Paul Calderone <exarkun@twistedmatrix.com>
 2022-11-09 10:52:42 -07:00
meejah
b2431f3a89
Update src/allmydata/cli/grid_manager.py 
Co-authored-by: Jean-Paul Calderone <exarkun@twistedmatrix.com>
 2022-11-09 10:46:31 -07:00
Itamar Turner-Trauring
8d678fe3de Increase timeout. 2022-11-07 11:41:50 -05:00
Itamar Turner-Trauring
65a7945fd9 Don't need a connection timeout since we have request-level timeouts. 2022-11-07 11:39:45 -05:00
Itamar Turner-Trauring
2c911eeac1 Make sure everything is using the same clock. 2022-11-07 11:28:36 -05:00
Itamar Turner-Trauring
f8b9607fc2 Finish up limited_content() timeout code. 2022-11-07 11:26:11 -05:00
Itamar Turner-Trauring
ce59ddc1ea Merge branch '3783-storage-client-http' into 3940-http-timeouts 2022-11-07 11:20:54 -05:00
Itamar Turner-Trauring
c4772482ef WIP 2022-11-07 11:19:00 -05:00
Itamar Turner-Trauring
414b463556 Use built-in treq timeout feature. 2022-11-07 09:23:04 -05:00
Itamar Turner-Trauring
1e50e96e24 Update to new test API. 2022-11-03 15:04:41 -04:00
Itamar Turner-Trauring
8bebb09edd Less test-specific way to make test_rref pass. 2022-11-03 14:38:59 -04:00
Itamar Turner-Trauring
262d9d85b9 Switch to using persistent connections in tests too. 2022-11-03 14:32:43 -04:00
Itamar Turner-Trauring
e05136c238 Less aggressive timeout, to try to make tests pass on CI. 2022-11-02 13:13:21 -04:00
Itamar Turner-Trauring
48ae729c0d Don't reuse basedir across tests. 2022-11-02 10:18:23 -04:00
Itamar Turner-Trauring
73271afa65 Merge remote-tracking branch 'origin/master' into 3783-storage-client-http 2022-11-02 09:23:29 -04:00
Itamar Turner-Trauring
1a3e3a86c3 Require latest pycddl, and work around a regression. 2022-10-14 11:27:04 -04:00
Itamar Turner-Trauring
0febc87456 Don't include reactor in comparison. 2022-10-14 10:03:06 -04:00
Itamar Turner-Trauring
e409262e86 Fix some flakes. 2022-10-14 09:50:07 -04:00
Itamar Turner-Trauring
42d3843343 Run test_system with both Foolscap and HTTP storage protocols, plus some 
resulting cleanups.
 2022-10-14 09:16:59 -04:00
Itamar Turner-Trauring
0f31e3cd4b Leave HTTP off by default for now. 2022-10-13 14:41:59 -04:00
Itamar Turner-Trauring
b80a215ae1 test_rref passes now. 2022-10-13 14:15:10 -04:00
Itamar Turner-Trauring
0d23237b11 Some progress towards passing test_rref. 2022-10-13 13:44:49 -04:00
Itamar Turner-Trauring
435df4923f
Merge branch 'master' into 2916.grid-manager-proposal.6 2022-10-05 10:18:09 -04:00
Itamar Turner-Trauring
8b0ddf406e Make HTTP and Foolscap match in another edge case. 2022-10-04 11:17:19 -04:00
Itamar Turner-Trauring
8190eea489 Fix bug introduced in previous commit. 2022-10-04 11:02:36 -04:00
Itamar Turner-Trauring
ea1d248611 These objects get stored in a context where they need to be hashed, sometimes. 2022-10-04 10:51:43 -04:00
Itamar Turner-Trauring
1294baa82e LoopingCall may already have been stopped. 2022-10-04 10:30:27 -04:00
Itamar Turner-Trauring
fd07c092ed close() is called while writes are still happening. 2022-10-04 10:30:07 -04:00
Itamar Turner-Trauring
8b2884cf3a Make changes work again. 2022-10-04 09:44:30 -04:00
Itamar Turner-Trauring
1ea716c8bf Merge branch '3902-listen-storage-http' into 3783-storage-client-http 2022-10-04 09:22:49 -04:00
Itamar Turner-Trauring
c13be0c89b Try harder to cleanup. 2022-10-04 09:19:48 -04:00
Itamar Turner-Trauring
58247799c1 Fix remaining references to refactored-out-of-existence API. 2022-10-03 11:27:19 -04:00
Itamar Turner-Trauring
3034f35c7b Document type expectations. 2022-10-03 11:24:53 -04:00
Itamar Turner-Trauring
5d53cd4a17 Nicer API. 2022-10-03 11:16:30 -04:00
Itamar Turner-Trauring
d918135a0d Use parser instead of ad-hoc parser. 2022-10-03 11:10:36 -04:00
Itamar Turner-Trauring
d753bb58da Better type for storage_nurls. 2022-10-03 11:05:56 -04:00
Itamar Turner-Trauring
b0fb72e379 Link to design issue. 2022-10-03 11:02:48 -04:00
Itamar Turner-Trauring
8179ea7738 Merge remote-tracking branch 'origin/master' into 3902-listen-storage-http 2022-10-03 10:56:17 -04:00
Itamar Turner-Trauring
bd74c3d971
Merge pull request #1216 from tahoe-lafs/3904-http-storage-holistic 
Address various http storage holistic review comments
 2022-10-03 09:13:30 -04:00
meejah
d2dd211420
Merge pull request #1219 from meejah/3926.pid-time-reactor 
3926: upgrade PID files to contain start-time, do file-locking and exist on Windows
 2022-09-28 11:02:46 -06:00
meejah
bef71978b6 don't need start_new_session 2022-09-28 10:15:03 -06:00
meejah
ca522a5293 sys.argv not inline 2022-09-28 10:14:59 -06:00
meejah
8991509f8c blackslashes.... 2022-09-25 00:16:40 -06:00
meejah
f2cfd96b5e typo, longer timeout 2022-09-23 01:04:58 -06:00
meejah
7fdeb8797e hardcoding bad 2022-09-23 00:26:39 -06:00
meejah
62b92585c6 simplify 2022-09-22 23:57:19 -06:00
meejah
a182a25079 backslashes 2022-09-22 21:43:20 -06:00
meejah
8ebe331c35 maybe a newline helps 2022-09-22 00:11:20 -06:00
meejah
4f5a1ac372 naming? 2022-09-21 23:36:23 -06:00
meejah
8d8b0e6f01 cleanup 2022-09-21 20:40:25 -06:00
meejah
fedea96964 less state 2022-09-21 20:26:14 -06:00
meejah
8474ecf83d typo 2022-09-21 20:15:07 -06:00
meejah
930f4029f3 properly write pid, create-time 2022-09-21 20:07:46 -06:00
meejah
0a09d23525 more docstring 2022-09-21 19:29:40 -06:00
meejah
e111694b3e get rid of find_process= 2022-09-21 19:28:09 -06:00
meejah
56775dde19 refactor: parsing in a function 2022-09-21 19:05:30 -06:00
meejah
ea39e4ca69 docstring 2022-09-21 19:01:28 -06:00
meejah
5973196931 refactor: use filelock and test it 2022-09-21 19:00:27 -06:00
meejah
1f29cc9c29 windows special 2022-09-20 14:50:46 -06:00
meejah
77bc83d341 incorrectly removed 2022-09-20 14:45:19 -06:00
meejah
0eeb11c9cd after shutdown 2022-09-20 14:44:51 -06:00
meejah
6db1476dac comment typo 2022-09-20 14:44:21 -06:00
meejah
81c8e1c57b windows is special 2022-09-20 14:24:02 -06:00
Itamar Turner-Trauring
4b2725df00 Try to prevent leaking timeouts. 2022-09-20 10:09:43 -04:00
Itamar Turner-Trauring
3b3ed9db2b Merge remote-tracking branch 'origin/master' into 3902-listen-storage-http 2022-09-19 10:07:00 -04:00
meejah
fe80126e3f fixups 2022-09-18 22:39:25 -06:00
meejah
254a994eb5 flake8 2022-09-17 16:41:17 -06:00
meejah
8b2cb79070 cleanup via reactor 2022-09-17 16:38:25 -06:00
meejah
aef2e96139 refactor: dispatch with our reactor, pass to tahoe_run 2022-09-17 16:38:24 -06:00
meejah
114d5e1ed8 pidfile on windows now 2022-09-17 16:37:33 -06:00
meejah
228bbbc2fe new pid-file 2022-09-17 16:37:33 -06:00
meejah
82c72ddede cleanup 2022-09-17 16:37:30 -06:00
meejah
642b604753 use stdin-closing for pidfile cleanup too 2022-09-17 16:36:55 -06:00
meejah
6048d1d9a9 in case hypothesis finds the magic 2022-09-17 16:36:26 -06:00
meejah
0e0ebf6687 more testing 2022-09-17 16:36:26 -06:00
meejah
cad162bb8f should have pid-file on windows too, now 2022-09-17 16:36:26 -06:00
meejah
3bfb60c6f4 back to context-manager, simplify 2022-09-17 16:36:25 -06:00
meejah
fb532a71ef own pid-file checks 2022-09-17 16:31:49 -06:00
Itamar Turner-Trauring
444bc724c5 A better approach to MAX_UEB_SIZE: just delete the code since it's not used in practice. 2022-09-16 10:38:29 -04:00
Itamar Turner-Trauring
4a573ede34 Download the actual data we need, instead of relying on bad reading-beyond-the-end semantics. 2022-09-15 11:29:32 -04:00
Itamar Turner-Trauring
b1aa93e022 Switch prefix. 2022-09-15 10:34:59 -04:00
Itamar Turner-Trauring
8d5f08771a Minimal check on parameters' contents. 2022-09-15 09:45:46 -04:00
Itamar Turner-Trauring
373a532829 Detect corrupted UEB length more consistently. 2022-09-15 09:36:56 -04:00
Fon E. Noel NFEBE
fbc8baa238 Refactor Server class in test_storage.py 
As a follow up to commit: 48283ea6f8
this refactor adds better methods and cleans up the test to be
consistent with methods that used in classes that extend the `AsyncTestCase`.

Signed-off-by: Fon E. Noel NFEBE <fenn25.fn@gmail.com>
 2022-09-14 22:55:34 +01:00
meejah
1058e50c50 close properly 2022-09-08 16:30:30 -06:00
Itamar Turner-Trauring
c82bb5f21c Use a more meaningful constant. 2022-09-08 13:44:22 -04:00
Itamar Turner-Trauring
9975fddd88 Get rid of garbage. 2022-09-08 13:42:19 -04:00
Itamar Turner-Trauring
b72b51d22d Merge remote-tracking branch 'origin/master' into 3915-immutable-size 2022-09-08 13:41:46 -04:00
Fon E. Noel NFEBE
48283ea6f8 Refactor test_storage.py 
There are base test classes namely `SyncTestCase` and
`AsyncTestCase` which we would like all test classes in
this code base to extend.

This commit extends the listed classes in test_storage.py
to extend the above mentioned base classes:

  * UtilTests
  * BucketProxy
  * Server

Signed-off-by: Fon E. Noel NFEBE <fenn25.fn@gmail.com>
 2022-09-07 22:36:01 +01:00
meejah
711f6d39e7 missing proto 2022-09-01 22:29:19 -06:00
meejah
decb36a8f6 refactor for Windows testing 2022-09-01 22:20:07 -06:00
meejah
00c785ec76 debug windows 2022-09-01 21:47:28 -06:00
meejah
768829e993 more robust 2022-09-01 21:22:45 -06:00
meejah
488a04cb9b exit when stdin closes 2022-09-01 17:42:06 -06:00
Itamar Turner-Trauring
bdb4aac0de Pass in the missing argument. 2022-08-17 13:15:27 -04:00
Itamar Turner-Trauring
92662d802c Don't drop a Deferred on the ground. 2022-08-17 13:15:13 -04:00
Itamar Turner-Trauring
cd81e5a01c Hint for future debugging. 2022-08-17 13:13:22 -04:00
Itamar Turner-Trauring
3464637bbb Fix unit tests. 2022-08-17 12:54:26 -04:00
Itamar Turner-Trauring
9d03c476d1 Make sure we write all the bytes we say we're sending. 2022-08-17 12:49:45 -04:00
Itamar Turner-Trauring
c9084a2a45 Disable assertion we can't, sadly, enable. 2022-08-17 12:49:26 -04:00
Fon E. Noel NFEBE
7aa97336a0 Refactor FakeWebTest & MemoryConsumerTest classes 
There are base test classes namely `SyncTestCase` and
`AsyncTestCase` which we would like all test classes in
this code base to extend.

This commit refactors two test classes to use the `SyncTestCase`
with the newer assert methods.

Signed-off-by: Fon E. Noel NFEBE <fenn25.fn@gmail.com>
 2022-08-17 16:03:06 +01:00
Itamar Turner-Trauring
d50c98a1e9 Calculate URI extension size upfront, instead of hand-waving with a larger value. 2022-08-16 14:34:40 -04:00
Itamar Turner-Trauring
2e5662aa91 Temporarily enforce requirement that allocated size matches actual size of an immutable. 2022-08-16 13:11:06 -04:00
Itamar Turner-Trauring
c1bcfab7f8 Repeatedly poll status of server. 2022-08-15 11:38:02 -04:00
Itamar Turner-Trauring
6d48c1ca34 Merge branch '3902-listen-storage-http' into 3783-storage-client-http 2022-08-15 10:09:31 -04:00
Itamar Turner-Trauring
555fa6e942 Merge remote-tracking branch 'origin/master' into 3902-listen-storage-http 2022-08-15 10:09:21 -04:00
Itamar Turner-Trauring
71b7e9b643 Support comma-separated multi-location hints. 2022-08-15 10:09:04 -04:00
Itamar Turner-Trauring
78b421462d Merge remote-tracking branch 'origin/master' into 3783-storage-client-http 2022-08-15 09:51:45 -04:00
Itamar Turner-Trauring
b9ac10293a Merge remote-tracking branch 'origin/master' into 2916.grid-manager-proposal.6 2022-08-15 09:51:08 -04:00
Itamar Turner-Trauring
42e818f0a7 Refer to appropriate attributes, hopefully. 2022-08-12 11:47:08 -04:00
Itamar Turner-Trauring
3fbc4d7eea Let's make this a little clearer 2022-08-12 11:45:37 -04:00
Itamar Turner-Trauring
636b8a9e2d Fix a bytes-vs-str bug. 2022-08-12 11:28:08 -04:00
Itamar Turner-Trauring
e8609ac2df test_istorageserver passes with both Foolscap and HTTP again. 2022-08-12 11:24:41 -04:00
Itamar Turner-Trauring
09d778c2cf Allow nodes to disable the HTTPS storage protocol. 2022-08-12 11:13:09 -04:00
Itamar Turner-Trauring
f671fb04a1 A lot closer to working end-to-end. 2022-08-11 16:24:33 -04:00
Itamar Turner-Trauring
9ad4e844e8 Do status change notification. 2022-08-11 16:16:17 -04:00
Itamar Turner-Trauring
94be227aaa Hopefully don't actually need that. 2022-08-11 16:15:21 -04:00
Itamar Turner-Trauring
c3b159a3fd Continue simplified sketch of HTTPNativeStorageServer. 2022-08-11 16:12:57 -04:00
Itamar Turner-Trauring
c3e4158813 Remove duplication. 2022-08-11 15:55:14 -04:00
Itamar Turner-Trauring
709f139c85 Start refactoring to enable HTTP storage client. 2022-08-11 15:51:30 -04:00
Itamar Turner-Trauring
8b3280bf31 Simplify more. 2022-07-29 10:51:17 -04:00
Itamar Turner-Trauring
d4c73f19fe A unittest for the metaclass. 2022-07-29 10:42:56 -04:00
Itamar Turner-Trauring
533d2a7ac9 Note Tor and I2P support. 2022-07-29 10:15:23 -04:00
Itamar Turner-Trauring
1cd2185be7 More cleanups. 2022-07-29 10:12:24 -04:00
Itamar Turner-Trauring
34518f9d0d Fix lints. 2022-07-29 10:01:09 -04:00
Itamar Turner-Trauring
822b652d99 Improve factoring. 2022-07-29 09:57:18 -04:00
Itamar Turner-Trauring
21bb9e50f6 Merge remote-tracking branch 'origin/master' into 3902-listen-storage-http 2022-07-29 09:45:17 -04:00
Itamar Turner-Trauring
4cc5c4d55d Merge remote-tracking branch 'origin/master' into 2916.grid-manager-proposal.6 2022-07-25 11:55:47 -04:00
Itamar Turner-Trauring
c9e36411ac Merge remote-tracking branch 'origin/master' into 3880-http-storage-logging 2022-07-25 10:38:21 -04:00
Itamar Turner-Trauring
94e0568653 Actually we do need it. 2022-07-22 11:57:32 -04:00
Itamar Turner-Trauring
83f9c0788b Use more direct API. 2022-07-22 11:56:18 -04:00
Itamar Turner-Trauring
be963e2324 Docstrings. 2022-07-22 11:55:33 -04:00
Itamar Turner-Trauring
2b3a8ddeec Docstring. 2022-07-22 11:55:00 -04:00
Itamar Turner-Trauring
36b96a8776 Fix typo. 2022-07-22 11:53:28 -04:00