ExternalVendorCode/tahoe-lafs
1
0
Fork 0
mirror of https://github.com/tahoe-lafs/tahoe-lafs.git synced 2024-12-24 07:06:41 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity
13,449 Commits 185 Branches 90 Tags 89 MiB
ccb6e65c04
Commit Graph

7945 Commits

Author SHA1 Message Date
meejah
d4fc14f9ad docstring 2021-11-16 15:08:31 -07:00
meejah
89c2aacadc working test of 'in the wild' data, working converters 2021-11-16 15:08:31 -07:00
meejah
bf5e682d71 test upgrade of main state works 2021-11-16 15:08:31 -07:00
meejah
f81e4e2d25 refactor to use serializers / pickle->json upgraders 2021-11-16 15:08:31 -07:00
meejah
fa6950f08d an old pickle-format lease-checker state file 2021-11-16 15:08:31 -07:00
meejah
bb70e00065 Make internal state JSON-able for lease-crawler 2021-11-16 15:08:31 -07:00
meejah
f7b385f954 play nice with subclasses 2021-11-16 15:08:31 -07:00
meejah
3b69df36b0 crawler: pickle -> json 2021-11-16 15:08:31 -07:00
Itamar Turner-Trauring
f549488bb5 Don't use a deprecated API. 2021-11-16 11:47:09 -05:00
Itamar Turner-Trauring
a64778ddb0 Flakes. 2021-11-16 11:28:13 -05:00
Itamar Turner-Trauring
c195f895db Python 2 support. 2021-11-16 11:16:26 -05:00
Itamar Turner-Trauring
171d1053ec CBOR content-type on responses. 2021-11-16 11:09:17 -05:00
Itamar Turner-Trauring
671b670154 Some type annotations. 2021-11-16 11:09:08 -05:00
Itamar Turner-Trauring
41ec63f758 Passing first tests. 2021-11-16 10:56:21 -05:00
Jean-Paul Calderone
fcd634fc43 some direct tests for the new utility function 2021-11-15 13:34:46 -05:00
Jean-Paul Calderone
84c19f5468 clarify renew_leases docs on slot_testv_and_readv_and_writev 2021-11-15 08:12:07 -05:00
Jean-Paul Calderone
ad6017e63d clarify renew_leases docs on allocate_buckets 2021-11-15 08:11:53 -05:00
Jean-Paul Calderone
6c2e85e991 put the comment back 2021-11-12 16:25:36 -05:00
Jean-Paul Calderone
dece67ee3a it is not the remote interface that varies anymore 2021-11-12 16:24:29 -05:00
Jean-Paul Calderone
85977e48a7 put this comment back and merge info from the two versions 2021-11-12 16:23:15 -05:00
Jean-Paul Calderone
c3cb0ebaea Switch to per-call parameter for controlling lease renewal behavior 
This is closer to an implementation where you could have two frontends, say a
Foolscap frontend and an HTTP frontend or even just two different HTTP
frontends, which had different opinions about what the behaviour should be.
 2021-11-12 16:20:27 -05:00
Itamar Turner-Trauring
c101dd4dc9 Closer to first passing test. 2021-11-12 13:13:19 -05:00
Itamar Turner-Trauring
12cbf8a901 First sketch of HTTP testing infrastructure. 2021-11-12 13:03:53 -05:00
Itamar Turner-Trauring
ddd2780bd2 First sketch of HTTP client. 2021-11-12 12:51:52 -05:00
Itamar Turner-Trauring
a32c6be978 A sketch of what the HTTP server will look like. 2021-11-12 12:02:58 -05:00
Jean-Paul Calderone
2742de6f7c drop some ancient cruft 
allocated_size not used anywhere, so why have it
 2021-11-10 16:08:53 -05:00
Jean-Paul Calderone
9af81d21c5 add a way to turn off implicit bucket lease renewal too 2021-11-10 16:08:40 -05:00
Jean-Paul Calderone
617a1eac9d refactor lease hashing logic to avoid mutable/immutable duplication 2021-11-05 15:30:49 -04:00
Jean-Paul Calderone
456df65a07 Add v2 of the mutable container schema 
It uses hashed lease secrets, like v2 of the immutable container schema.
 2021-11-05 14:16:43 -04:00
Jean-Paul Calderone
3de9c73b0b preserve the type when renewing HashedLeaseInfo 
does this mean immutable lease renewal is untested?  maybe
 2021-11-05 14:11:05 -04:00
Jean-Paul Calderone
5d703d9893 some type annotations 2021-11-05 14:10:27 -04:00
Jean-Paul Calderone
0cd96ed713 fix the debug tool for the hashed lease secret case 2021-11-05 14:09:46 -04:00
Jean-Paul Calderone
8adff050a7 compare without breaking out all of the fields 
HashedLeaseInfo doesn't have all of these attributes
 2021-11-05 13:51:46 -04:00
Jean-Paul Calderone
07ffde3bec
Merge pull request #1158 from tahoe-lafs/3838.more-main-tub-cleanup 
More "main tub" initialization cleanup in test code

Fixes: ticket:3838
 2021-11-05 11:13:23 -04:00
Jean-Paul Calderone
728638fe23 apply the MutableShareFile tests to all known schemas 2021-11-04 15:37:29 -04:00
Jean-Paul Calderone
931ddf85a5 introduce an explicit representation of the v1 mutable container schema 
This is only a partial representation, sufficient to express the changes that
are coming in v2.
 2021-11-04 15:26:58 -04:00
Jean-Paul Calderone
2186bfcc37 silence some mypy errors :/ 
I don't know the "right" way to make mypy happy with these things
 2021-11-04 14:40:43 -04:00
Jean-Paul Calderone
6889ab2a76 fix syntax of type hint 2021-11-04 14:16:55 -04:00
Jean-Paul Calderone
7a59aa83bb add missing import 2021-11-04 14:12:54 -04:00
Jean-Paul Calderone
b69e8d013b introduce immutable container schema version 2 
This version used on-disk hashed secrets to reduce the chance of secrets
leaking to unintended parties.
 2021-11-04 14:07:49 -04:00
Jean-Paul Calderone
234b8dcde2 Formalize LeaseInfo interface in preparation for another implementation 2021-11-04 12:07:24 -04:00
Jean-Paul Calderone
3b41419523 apply the ShareFile tests to all schema versions using hypothesis 2021-11-04 10:32:59 -04:00
Jean-Paul Calderone
10724a91f9 introduce an explicit representation of the v1 immutable container schema 
This is only a partial representation, sufficient to express the changes that
are coming in v2.
 2021-11-04 10:17:36 -04:00
Jean-Paul Calderone
274dc6e837 Introduce UnknownContainerVersionError base w/ structured args 2021-11-04 10:06:26 -04:00
Jean-Paul Calderone
a71b1d31bf Merge remote-tracking branch 'upstream/master' 2021-11-04 10:02:54 -04:00
Jean-Paul Calderone
780be2691b assign a tub.port to all system test nodes 2021-11-03 16:11:28 -04:00
Jean-Paul Calderone
5caa80fe38 use UseNode more in test_client.py 
Also make write_introducer more lenient about filesystem state
 2021-11-03 16:11:08 -04:00
Jean-Paul Calderone
5a71774bf8 use port assigner and UseNode more in test_node.py 2021-11-03 16:10:32 -04:00
Jean-Paul Calderone
31649890ef Teach UseNode to use a port assigner for tub.port 
Then use it to assign ports for tub.port unless the caller supplied their own
value.
 2021-11-03 16:08:08 -04:00
Jean-Paul Calderone
797e099459 make create_introducer_webish assign a main tub port 2021-11-03 16:05:28 -04:00
Jean-Paul Calderone
7335b2a597 remove unused import 2021-11-03 13:16:15 -04:00
Jean-Paul Calderone
8e150cce6a add explicit direct tests for the new methods 2021-11-03 13:14:55 -04:00
Jean-Paul Calderone
70a7500c56 Merge remote-tracking branch 'origin/master' into 3834.renew_secret_comparison_helper 2021-11-03 13:05:41 -04:00
Jean-Paul Calderone
e8ed711d26 Merge remote-tracking branch 'origin/master' into 3833.container-format-abstraction 2021-11-03 12:25:38 -04:00
Jean-Paul Calderone
5bfe6d5b24
Merge pull request #1156 from tahoe-lafs/3836.systemtests-introducer-main-tub-port 
Use the "port assigner" helper on the introducer main tub in SystemTestsMixin

Fixes: 3836
 2021-11-03 12:25:09 -04:00
Jean-Paul Calderone
39c4a2c4eb tidy up some corners 2021-11-02 15:10:54 -04:00
fenn-cs
fcfc89e3ae moved new tests/update for eliotutils 
Signed-off-by: fenn-cs <fenn25.fn@gmail.com>
 2021-11-02 14:32:20 +01:00
fenn-cs
11f45b5c3c Merge branch 'master' of github.com:tahoe-lafs/tahoe-lafs into 3758.refactor.web-tests-grid-logs-root 2021-11-02 10:36:44 +01:00
fenn-cs
5ba636c7b1 removed deferred logger from basic function in test_logs 
Signed-off-by: fenn-cs <fenn25.fn@gmail.com>
 2021-11-02 10:36:32 +01:00
Jean-Paul Calderone
0856218a89 Merge remote-tracking branch 'origin/master' into 3833.container-format-abstraction 2021-10-30 07:35:50 -04:00
Jean-Paul Calderone
892b468365 use the port assigner to assign a port for the main tub 2021-10-29 09:03:37 -04:00
Jean-Paul Calderone
ff511bbcf2 Merge remote-tracking branch 'origin/master' into 3832.backdated-leases 2021-10-29 07:17:23 -04:00
Jean-Paul Calderone
c05464f420
Merge pull request #1154 from tahoe-lafs/3835.document-some-testing-code 
Document some testing code

Fixes: ticket:3835
 2021-10-29 07:16:02 -04:00
Jean-Paul Calderone
b5f882ffa6 introduce and use LeaseInfo.is_cancel_secret 2021-10-28 15:21:51 -04:00
Jean-Paul Calderone
8b976b441e add LeaseInfo.is_renew_secret and use it 2021-10-28 15:21:51 -04:00
Jean-Paul Calderone
78dbe76994 remove unused import 2021-10-28 15:20:44 -04:00
Jean-Paul Calderone
f8655f149b fix the type annotations and such 2021-10-28 15:04:19 -04:00
Jean-Paul Calderone
8d5727977b it doesn't typecheck, nevermind 2021-10-28 14:47:42 -04:00
Jean-Paul Calderone
54d80222c9 switch to monkey-patching from other sources 
This is not much of an improvement to the tests themselves, unfortunately.
However, it does get us one step closer to dropping `mock` as a dependency.
 2021-10-28 14:36:30 -04:00
Jean-Paul Calderone
d0ee17d99e some docstrings 2021-10-28 12:39:01 -04:00
Jean-Paul Calderone
1b46ac7a24 add MutableShareFile.is_valid_header and use it 2021-10-28 11:38:18 -04:00
Jean-Paul Calderone
66845c9a17 Add ShareFile.is_valid_header and use it instead of manual header inspection 2021-10-28 11:12:20 -04:00
Jean-Paul Calderone
65d3ab6142 move backdating logic into mutable/immutable share files 2021-10-28 10:53:52 -04:00
Jean-Paul Calderone
3514995068 some versions of mypy don't like this so nevermind 2021-10-28 10:26:30 -04:00
Jean-Paul Calderone
76caf46347 make the alternate LeaseInfo constructors into class methods 2021-10-28 10:23:58 -04:00
Jean-Paul Calderone
70fb5d563a Get rid of the public expiration_time attribute 
LeaseInfo now has a getter and a setter for this attribute.
LeaseInfo is now also immutable by way of `attrs`.
LeaseInfo is now also comparable by way of `attrs`.
 2021-10-28 09:48:26 -04:00
Jean-Paul Calderone
f02f14a6e2 Merge remote-tracking branch 'upstream/master' 2021-10-26 09:23:49 -04:00
fenn-cs
69d335c1e1 update test overlap function docstring 
Signed-off-by: fenn-cs <fenn25.fn@gmail.com>
 2021-10-26 13:14:26 +01:00
fenn-cs
9764ac740a test kwargs overlap with params in start_action 
Signed-off-by: fenn-cs <fenn25.fn@gmail.com>
 2021-10-26 11:22:32 +01:00
Jean-Paul Calderone
84dfb360ce
Merge pull request #1141 from LeastAuthority/3814.remove-control-port 
Remove the "control port" and its associated Tub

Fixes: ticket:3814
 2021-10-25 20:57:23 -04:00
Jean-Paul Calderone
3de481ab6b remove unused imports 2021-10-25 20:52:35 -04:00
Jean-Paul Calderone
5878a64890 Remove password-based authentication from the SFTP frontend 2021-10-25 20:50:19 -04:00
Jean-Paul Calderone
aa6360f08e Merge remote-tracking branch 'remediate/master' 2021-10-24 07:38:42 -04:00
Jean-Paul Calderone
b51f0ac8ff storage_index is a byte string 
and Python 3 cares
 2021-10-23 08:04:19 -04:00
Jean-Paul Calderone
0ada9d93f7 remove unused local 2021-10-23 07:43:22 -04:00
Jean-Paul Calderone
470657b337 Drop corruption advisories if we don't have a matching share 2021-10-23 07:35:05 -04:00
Jean-Paul Calderone
21f848f0e3 Merge remote-tracking branch 'upstream/master' 2021-10-23 07:32:26 -04:00
Jean-Paul Calderone
cb675df48d remove unused encoding of storage index 2021-10-22 16:10:24 -04:00
Jean-Paul Calderone
194499aafe remove unused import 2021-10-22 16:09:54 -04:00
Jean-Paul Calderone
6449ad03de Do not record corruption advisories if there is no available space 2021-10-22 15:56:45 -04:00
Jean-Paul Calderone
f789339a79 Have MutableShare file only write a new lease if there is room for it 
This is analagous to the earlier ShareFile change.
 2021-10-22 15:56:45 -04:00
Jean-Paul Calderone
dd1ab2afe8 Add a helper to compute the size of a lease in a mutable share 2021-10-22 15:56:45 -04:00
Jean-Paul Calderone
e0ed04c103 use SyncTestCase to get expectThat 2021-10-22 15:56:45 -04:00
Jean-Paul Calderone
4defc641a2 Have ShareFile only write a new lease if there is room for it 
StorageServer passes available space down so it can make the decision.
ShareFile has to do it because `add_or_renew_lease` only *sometimes* adds a
lease and only ShareFile knows when that is.
 2021-10-22 15:56:45 -04:00
Jean-Paul Calderone
1264c3be1e Use _add_or_renew_leases helper consistently in StorageServer 
This will make it easier to add a new argument to the underlying
`add_or_renew_lease` call.
 2021-10-22 15:56:45 -04:00
Jean-Paul Calderone
b3aa1e224f Add a helper to LeaseInfo for computing size 
This lets some code LBYL and avoid writing if the lease won't fit in the
immutable share in the space available.
 2021-10-22 15:56:45 -04:00
Jean-Paul Calderone
c774256937 Add a test for `remote_add_lease` with respect to reserved space 2021-10-22 15:56:45 -04:00
Jean-Paul Calderone
82e4704415 Merge remote-tracking branch 'remediate/master' into LFS-01-008 2021-10-22 15:27:15 -04:00
Jean-Paul Calderone
ce30f9dd06 clean up copyediting errors 2021-10-22 15:04:45 -04:00
Jean-Paul Calderone
bb31215134 Merge remote-tracking branch 'remediate/master' into LFS-01-007 2021-10-22 12:35:57 -04:00
Jean-Paul Calderone
d8c466e9a7 try to explain lease_count_format more clearly 2021-10-22 12:35:11 -04:00
Jean-Paul Calderone
a22787e879
Merge pull request #30 from tahoe-lafs/LFS-01-001 
Fix item LFS-01-001 from the Cure53 audit

Fixes: ticket:3819
 2021-10-22 12:20:38 -04:00
Jean-Paul Calderone
6c0ca0b885 try making windows let us use longer paths 2021-10-22 11:42:13 -04:00
fenn-cs
20ad6cd9e7 iterate over args directly without indexing 
Signed-off-by: fenn-cs <fenn25.fn@gmail.com>
 2021-10-19 23:57:52 +01:00
Itamar Turner-Trauring
2a5dbcb05e Tests for mutable leases. 2021-10-19 09:30:38 -04:00
Itamar Turner-Trauring
4b8b6052f3 Finish testing leases on immutables. 2021-10-19 09:05:48 -04:00
Jean-Paul Calderone
3cd9a02c81 Reject negative lengths in MutableShareFile._read_share_data and readv 2021-10-18 20:22:52 -04:00
Jean-Paul Calderone
5e58b62979 Add a test for negative offset or length to MutableShareFile.readv 2021-10-18 20:22:52 -04:00
Jean-Paul Calderone
df64bbb1e4 fail to encode the lease count *before* changing anything 
This preserves the failure behavior - `struct.error` is raised - but leaves
the actual share file contents untouched if the new lease count cannot be
encoded.

There are still two separate write operations so it is conceivable that some
other problem could cause `write_lease_record` to happen but
`write_encoded_num_leases` not to happen.  As far as I can tell we have
severely limited options for addressing that problem in general as long as
share files are backed by a POSIX filesystem.

However, by removing the failure mode that depends on user input, it may be
that this is all that is needed to close the *security* hole.
 2021-10-18 20:17:41 -04:00
Jean-Paul Calderone
f60bbbd3e2 make it possible to test this behavior of add_lease 2021-10-18 20:17:41 -04:00
Itamar Turner-Trauring
b7be91e3d0 First test for leases. 2021-10-18 13:17:07 -04:00
Itamar Turner-Trauring
2b40610a27 "Server" is extremely ambiguous, so let's just call this a client, which it is. 2021-10-18 12:55:30 -04:00
Itamar Turner-Trauring
58112ba75b Plan of implementation for lease tests. 2021-10-18 12:50:29 -04:00
Jean-Paul Calderone
1c347c5931 replace sensitive introducer fURL with path where it can be found 2021-10-18 08:05:40 -04:00
Jean-Paul Calderone
ad216e0f23 remove unused import 2021-10-15 13:13:37 -04:00
Jean-Paul Calderone
0611af6b0b Stop passing even a dummy value for control tub into Nodes 2021-10-15 13:10:18 -04:00
Jean-Paul Calderone
fe2e2cc1d6 Get rid of the control service 2021-10-15 12:12:19 -04:00
Jean-Paul Calderone
1de480dc37 Stop offering an API to create a control tub or handling the control tub 2021-10-15 12:12:03 -04:00
Jean-Paul Calderone
ddf5f461bf Stop half-pretending to have a control port 2021-10-15 12:11:53 -04:00
Jean-Paul Calderone
e0312eae57 stop creating a control tub for client nodes 2021-10-15 12:02:24 -04:00
Jean-Paul Calderone
95b765e309 stop creating a control tub for the introducer 2021-10-15 11:54:18 -04:00
fenn-cs
1a12a8acdf don't throw away unserializable parameter 
Signed-off-by: fenn-cs <fenn25.fn@gmail.com>
 2021-10-15 00:50:11 +01:00
Itamar Turner-Trauring
e099bc6736 Tests for IStorageServer.advise_corrupt_share. 2021-10-14 12:52:56 -04:00
Itamar Turner-Trauring
22aab98fcf When callRemoteOnly as removed, these probably should've been changed to return 
a Deferred.
 2021-10-14 12:52:40 -04:00
Jean-Paul Calderone
efc9dc831b Revert "a stab at using setup.cfg and setuptools_scm" 
This reverts commit 68e8e0a7d5.
 2021-10-14 11:01:37 -04:00
Jean-Paul Calderone
68e8e0a7d5 a stab at using setup.cfg and setuptools_scm 2021-10-14 10:58:41 -04:00
fenn-cs
57a0f76e1f maintain list of positional arguments as tuple 
Signed-off-by: fenn-cs <fenn25.fn@gmail.com>
 2021-10-13 23:41:42 +01:00
fenn-cs
5803d9999d remove unseriable args in log_call_deferred passed to start_action 
Signed-off-by: fenn-cs <fenn25.fn@gmail.com>
 2021-10-11 13:49:29 +01:00
fenn-cs
9da1456e5b Merge branch 'master' of github.com:tahoe-lafs/tahoe-lafs into 3758.refactor.web-tests-grid-logs-root 2021-10-08 23:37:34 +01:00
fenn-cs
32e00af023 Merge branch 'master' of github.com:tahoe-lafs/tahoe-lafs into 3758.refactor.web-tests-grid-logs-root 2021-10-07 17:46:20 +01:00
Itamar Turner-Trauring
807363adc9 Reduce duplication. 2021-10-06 16:41:50 -04:00
Itamar Turner-Trauring
3cf09ef888 Merge remote-tracking branch 'origin/master' into 3798-bucket-write-abort-tests 2021-10-06 16:07:39 -04:00
Itamar Turner-Trauring
0a605530d8
Merge pull request #1134 from tahoe-lafs/3793-persistent-bucketwriter 
Don't tie BucketWriter lifetime to Foolscap connection lifetime

Fixes ticket:3793
 2021-10-06 16:07:17 -04:00
Itamar Turner-Trauring
f3ebc71410
Merge pull request #1133 from tahoe-lafs/3805-istorageserver-tests-slot-readv 
IStorageServer tests: slot_readv

Fixes ticket:3805
 2021-10-06 15:41:29 -04:00
Itamar Turner-Trauring
bf176144c5 Handle double-disconnect, should it happen by mistake. 2021-10-06 15:18:00 -04:00
Itamar Turner-Trauring
82cbce6b7e Better explanation. 2021-10-06 15:12:22 -04:00
Itamar Turner-Trauring
17a670dfb5 Test for aborting bucket upload. 2021-10-04 10:37:37 -04:00
Itamar Turner-Trauring
b09b7fd01b Merge branch '3793-persistent-bucketwriter' into 3798-bucket-write-abort-tests 2021-10-04 10:30:41 -04:00
Itamar Turner-Trauring
23fd11be43 Expand explanation. 2021-09-29 14:13:18 -04:00
Itamar Turner-Trauring
016d6b4530 Fix spurious type checking error. 2021-09-29 14:10:14 -04:00
Itamar Turner-Trauring
f8604e2394 Fix flakes. 2021-09-29 14:00:11 -04:00
Itamar Turner-Trauring
58d7e2f627 Simplify implementation. 2021-09-29 13:58:53 -04:00
Itamar Turner-Trauring
8fb6afee1b Refactor BucketWriters such that disconnection can be limited Foolscap. 2021-09-29 13:42:17 -04:00
Itamar Turner-Trauring
a4153b7125 Implementation plan. 2021-09-29 11:56:04 -04:00
Itamar Turner-Trauring
51e8b5e197 Disconnection test works now. 2021-09-29 11:17:33 -04:00
Itamar Turner-Trauring
45c564f3c8
Merge pull request #1131 from tahoe-lafs/3797-istorageserver-tests-part-3 
IStorageServer tests, part 3: slot_testv_and_writev_and_readv

Fixes ticket:3797
 2021-09-29 10:28:42 -04:00
meejah
0a072a98c8
Merge pull request #1125 from hacklschorsch/3786.openmetrics 
3786 Add OpenMetrics (Prometheus) statistics endpoint
 2021-09-28 13:21:46 -06:00
Itamar Turner-Trauring
e64c397fc5 WIP disconnection test. 2021-09-28 13:51:31 -04:00
Itamar Turner-Trauring
a02d5f4c9c Just stick to current behavior. 2021-09-28 13:02:01 -04:00
Itamar Turner-Trauring
9970559019 Tests for slot_readv. 2021-09-28 10:46:21 -04:00