ExternalVendorCode/tahoe-lafs
1
0
Fork 0
mirror of https://github.com/tahoe-lafs/tahoe-lafs.git synced 2024-12-31 18:27:00 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity
6,829 Commits 185 Branches 90 Tags 89 MiB
57bed47495
Commit Graph

4022 Commits

Author SHA1 Message Date
Brian Warner
af1e2a1e2f node.py: advise twistd.log readers about flogtool 2016-08-09 15:27:33 -04:00
meejah
69b86ebe9a Improve startup error-reporting 
This keeps re-trying the initial magic-folder scan and alerts
the user (via logs only :/) until it succeeds at least once.

After this happens and the node has started up, it will continue
to re-try if enough storage servers go away later such that the
remote collection can't be retrieved.
 2016-08-09 14:16:11 -04:00
Brian Warner
8ca653c30d no_network: turn off control_tub and log_tub 
These aren't needed for most tests, and saves about 7% of the total test
runtime (on my laptop, it drops from 844s to 786s).
 2016-08-05 19:33:49 -04:00
Brian Warner
a35073cb3d test.common: allow test_repairer to be run independently 
Previously, this file importing "allmydata.immutable" but assuming that
"allmydata.immutable.upload" was available, which only worked if some
other file had imported upload.py . This didn't affect running the
entire test suite (something imported upload.py before anything else
needed it), but caused errors when running specific tests like
test_repairer.py .
 2016-08-05 18:26:18 -04:00
Brian Warner
cd083b9a04 move SystemTestMixin from common.py to test_system.py 
It's only being used in test_system.py anyways, so having it live in a
"common" file didn't provide much value.
 2016-08-04 01:09:12 -04:00
Brian Warner
4f0e71db4a split test_web.py into smaller files 
there's more to be done, but this is a start
 2016-08-03 20:46:12 -04:00
Brian Warner
421520ade4 test_magic_folder: use oneshare=True too 
I can't currently test this (my OS-X laptop can't run those tests), but
based on how much time test_magic_folder takes on the buildbots, I
expect oneshare=True to help considerably.
 2016-08-03 19:11:01 -04:00
Brian Warner
6dd58ceb2a tests: use oneshare=True on more test cases 
This saves more time (as measured on my laptop):

* test_sftp: 17.7s -> 13s
* test_dirnode: 26.5s -> 20s
* test_ftp, test_configutil, test_web show negligible speedups

As before, some tests care about the number of shares, generally ones
which delete or corrupt shares and then expect to see the errors get
noticed or fixed. Those tests continue to use k=3/N=10.
 2016-08-03 19:11:01 -04:00
Brian Warner
22d032ef2e test.cli: use oneshare=True, reduce runtime by 50% 
Most of the CLI tests don't care about the actual shares. Configuring
the test client to use k=N=1 reduces the runtime from 180s to 90s on my
laptop.

A few tests *do* care, like test_check (which delete some shares, then
assert that 'tahoe check' shows the damage). These still use k=3/N=10.
 2016-08-03 19:11:01 -04:00
Brian Warner
50df45e3a0 add GridTestMixin.set_up_grid(oneshare=True) 
This configures client[0] to use k=N=1, which is a lot faster for tests
that don't care about the actual shares (e.g. most of the CLI tests).
 2016-08-03 19:11:01 -04:00
Brian Warner
e431faf58c test_cli_*: move files into test/cli/* 2016-08-03 19:10:53 -04:00
Brian Warner
f11769560c test.mutable.Update: stop exercising max_shares 
Many of the test cases would exercise two copies of each file: one with
k=3/N=10, and a second with k=127/N=255 (255 being the maximum supported
by zfec).

Large number of shares increases the overhead of the testing apparatus,
which is pushing those shares to lots of local servers.

I don't think the "max_shares" case is necessary, and it takes forever.
Because of it, "mutable.Update" was consuming 15% of the total test
runtime, and a third of that was just a single
function (test_replace_locations_max_shares, now deleted). On a
Raspberry Pi 3 (our "slow computer" benchmark), including branch
coverage, this one class took 42 minutes to complete, and requires
disabling a bunch of timeouts to finish at all.

The total number of shares in a file ("N") affects one thing: the
width (and thus height) of the share hash tree. This should be exercised
in test_hashtree.

The number of required shares ("k") affects one thing: the segment size
must be a multiple of k. I don't think we need to exercise this, but if
so, it could be exercised by a few small values for k, rather than 127.

Removing the max_shares cases saves 82% of the mutable.update
runtime (on top of the previous three-segment fix), reducing it from 64s
to 11.3s on my laptop.
 2016-08-03 15:32:01 -04:00
Brian Warner
1bdae637e1 test.mutable.update: reduce data size in half 
We only need three segments to exercise everything. This saves about 15%
of the test.mutable runtime (from 76s to 64s on my laptop).
 2016-08-03 15:08:10 -04:00
Brian Warner
dbd78f19d2 test.mutable.update: minor refactoring 2016-08-03 15:07:51 -04:00
Brian Warner
4f56c4b053 split test_mutable.py into smaller files 2016-08-03 14:54:32 -04:00
Brian Warner
6f8c96e262 oops, fix NoNetworkStorageBroker to match 2016-07-21 18:13:28 -07:00
Brian Warner
1b64ab5e85 simplify when_connected_enough() 
This seems happier as a method on StorageBroker, rather than a
completely separate helper class.
 2016-07-21 17:23:22 -07:00
meejah
88cd4cb6c2 skip all tests w/o inotify support 2016-07-21 12:35:59 -07:00
meejah
87acfe968e pyflakes warnings 2016-07-21 12:35:59 -07:00
meejah
4e0a30ab91 get rid of more win32 special-casing 2016-07-21 12:35:59 -07:00
meejah
1ac6c9bdf6 remove special-case win32 2016-07-21 12:35:59 -07:00
meejah
161c876f2f make tests reflect reality 2016-07-21 12:35:59 -07:00
meejah
46f1f952e9 tighten delay on windows tests 2016-07-21 12:35:59 -07:00
meejah
e182562dbe remove tub.location from smoke_test config for storage nodes 2016-07-21 12:35:59 -07:00
meejah
041aabd8c4 Error-handling and --debug option to print stack-trace for all magic-folder commands 2016-07-21 12:35:59 -07:00
meejah
4509c7dafd Fix magic-folder 'status' command 2016-07-21 12:35:59 -07:00
meejah
db40417886 make check_magicfolder_smoke easier to run, cross-platform 2016-07-21 12:35:59 -07:00
meejah
1b72683741 get rid of more spew 2016-07-21 12:35:59 -07:00
meejah
e9343c8715 fix typo 2016-07-21 12:35:59 -07:00
meejah
efef4ae632 only use long delay on windows 2016-07-21 12:35:59 -07:00
meejah
fc5f820202 whitespace 2016-07-21 12:35:59 -07:00
meejah
516af88252 put longer timeout for test_alice_bob back in 2016-07-21 12:35:59 -07:00
meejah
a533aef960 remove delayedcall debugging 2016-07-21 12:35:59 -07:00
meejah
304da362f7 remove prints 2016-07-21 12:35:59 -07:00
meejah
ec11551404 slightly better way to find tahoe binary 2016-07-21 12:35:59 -07:00
meejah
18f799bdc8 remove incorrect override 2016-07-21 12:35:59 -07:00
meejah
34aed4bbad fixup remove debug 2016-07-21 12:35:59 -07:00
meejah
e060c584d7 fixup for pending-call calls 2016-07-21 12:35:59 -07:00
meejah
3de2f9c756 get rid of a bunch of debug 2016-07-21 12:35:59 -07:00
meejah
db6717951f properly cancel pending-call things in windows inotify 2016-07-21 12:35:59 -07:00
meejah
fce5b789dc fixup rebase conflict mis-resolutions 2016-07-21 12:35:59 -07:00
Daira Hopwood
ba9abe6a2b Repair fileutil tests. 
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2016-07-21 12:35:59 -07:00
Daira Hopwood
12773ea034 Increase timeout for test_alice_bob to 5 minutes. 
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2016-07-21 12:35:59 -07:00
Daira Hopwood
504e0d02f3 refactor to use context managers 2016-07-21 12:35:59 -07:00
meejah
ec37d52942 more 2016-07-21 12:35:59 -07:00
meejah
d41477bd5f fix/change some assertions 2016-07-21 12:35:59 -07:00
meejah
575c00113a fix typo for .conflict files 2016-07-21 12:35:59 -07:00
Daira Hopwood
c978db4c90 more fixes 2016-07-21 12:35:59 -07:00
meejah
c087a87b4a fixes to test_alice_bob 2016-07-21 12:35:59 -07:00
Daira Hopwood
5c01d4781f Temporarily increase delay to make sure we receive notifications. 
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2016-07-21 12:35:59 -07:00
Daira Hopwood
874b4cf343 Debug prints in Windows inotify emulation. 
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2016-07-21 12:35:59 -07:00
Daira Hopwood
e845f7ff65 Filter out FILE_ACTION_MODIFIED for directories. 
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2016-07-21 12:35:59 -07:00
Daira Hopwood
23f4b9dbfd Repair tests after changing the semantics of objects_succeeded. 
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2016-07-21 12:35:59 -07:00
Daira Hopwood
8a83cff14d Make _process return a boolean to indicate whether processing succeeded, 
and only increment objects_succeeded if that is the case.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2016-07-21 12:35:59 -07:00
Daira Hopwood
0abe5baaf2 WIP: temporarily work around the AccessDenied errors after a test failure. 
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2016-07-21 12:35:59 -07:00
Daira Hopwood
afc7ab43a1 Fix a subtle typo. 
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2016-07-21 12:35:59 -07:00
Daira Hopwood
b949082f8e Change type of mtime and ctime stored in magicfolderdb to integer nanoseconds. 
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2016-07-21 12:35:59 -07:00
meejah
7c85de813f (some) fixes for smoke_magicfolder 2016-07-21 12:35:59 -07:00
Daira Hopwood
237f4719ee Remove the flush_volume call in RealTest as well as in RealTestAliceBob. 
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2016-07-21 12:35:59 -07:00
Daira Hopwood
b05b701532 Try to avoid spurious failures when shutting down the INotify loop on Windows. 
This only partially solves the problem.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2016-07-21 12:35:59 -07:00
Daira Hopwood
23a1f311af Fix pyflakes errors, add a comment about a suspicious test. 
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2016-07-21 12:35:59 -07:00
Daira Hopwood
a78e9c750a WIP: add close method for MagicFolderDB and make MagicFolderDbTests use it. 
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2016-07-21 12:35:59 -07:00
meejah
a50d0e5c8b WIP and debugging things 
Conflicts:
	src/allmydata/test/test_magic_folder.py
 2016-07-21 12:35:59 -07:00
meejah
58f9641179 Correct magic-folder use of StorageFarmBroker 
When landing the "connected enough" changes on master,
we changed the API; this fixes up usage of it in magic-folder.
 2016-07-21 12:35:59 -07:00
Daira Hopwood
e67d23b8fa Fix negative counts for object_queued. fixes ticket:2731 
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2016-07-21 12:35:59 -07:00
meejah
625c5a42cd daira points out bug 2016-07-21 12:35:59 -07:00
meejah
5f073c6987 more startup waiting 2016-07-21 12:35:59 -07:00
meejah
faed2574d6 possible less race-y startup 2016-07-21 12:35:59 -07:00
meejah
378ce27cbd Make retry_download test work, and some bonus comments 2016-07-21 12:35:59 -07:00
Daira Hopwood
9e5b01b3ae Swallow the error from Downloader._when_queue_is_empty. 
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2016-07-21 12:35:59 -07:00
Daira Hopwood
811d597cd8 Add WIP test_download_retry. 
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2016-07-21 12:35:59 -07:00
meejah
e6104cd1a2 Replace _lazy_tail with iterative loop, cleanup tests (#2412) 
* uses @inlineCallbacks to turn the _lazy_tail recursion into
   a "real" looking loop;
 * remove the need for "immediate" vs delayed iteration of said loop;
 * make it easier for the unit-tests to control the behavior of the
   uploader/downloader;
 * consolidates (some) setup/teardown code into the setUp and tearDown
   hooks provided by unittest so unit-tests aren't doing that themselves
 * re-factors some of the unit-tests to use an @inlineCallbacks style
   so they're easier to follow and debug

This doesn't tackle the "how to know when our inotify events have arrived"
problem the unit-tests still have, nor does it eliminate the myriad bits
of state that get added to tests via all the MixIns.
 2016-07-21 12:35:59 -07:00
Daira Hopwood
e03e243c67 Expand test for Magic Folder statistics. 2016-07-21 12:35:59 -07:00
Daira Hopwood
37d22efaea Improve test for Magic Folder statistics and move it from test_system.py to test_magic_folder.py. 2016-07-21 12:35:59 -07:00
Daira Hopwood
dd0597f869 Work in progress to fix incorrect statistics output for Magic Folder. refs ticket:2709 2016-07-21 12:35:59 -07:00
Daira Hopwood
e3dfc8f3f4 Fix test_magic_folder.py to use absolute unicode paths when calling write_downloaded_file. 2016-07-21 12:35:59 -07:00
David Stainton
d2ae4255c8 Add daira's implementation of make_dirs_with_absolute_mode 2016-07-21 12:35:59 -07:00
David Stainton
2963e9e50c Use os.path.dirname instead of split and rename var to initial_path_u 2016-07-21 12:35:59 -07:00
David Stainton
cd390b90e4 remove superfluous trailing comma from make_dirs_with_absolute_mode def 2016-07-21 12:35:59 -07:00
David Stainton
ba35d7262b Add unit test and make corrections to make_dirs_with_absolute_mode 2016-07-21 12:35:59 -07:00
David Stainton
ee44732d03 Break out our chmod while loop into fielutils.py 2016-07-21 12:35:59 -07:00
David Stainton
51ccec7a33 Use chmod instead of changing umask 
Conflicts:
	src/allmydata/frontends/magic_folder.py
 2016-07-21 12:35:59 -07:00
David Stainton
3d909682a4 Add comment about FUDGE_SECONDS and refer to our design doc 2016-07-21 12:35:59 -07:00
Daira Hopwood
807cfbf0dc Fix pyflakes warnings. 
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2016-07-21 12:35:59 -07:00
meejah
9cfcb1b3aa remove humanize, use internal method, teach internal method to understand timedelta 2016-07-21 12:35:59 -07:00
meejah
573a5481d1 add Bob leaving to smoke-tests 2016-07-21 12:35:59 -07:00
meejah
c217d1f8f3 fix return-valid of leave 2016-07-21 12:35:59 -07:00
meejah
bf3bf70814 add docstrings 2016-07-21 12:35:59 -07:00
meejah
8ba2912ff4 make history longer 2016-07-21 12:35:59 -07:00
meejah
7e84b0cf57 flesh out IQueuedItem 2016-07-21 12:35:59 -07:00
meejah
05ec62c4b1 humanize doesn't report a proper version 2016-07-21 12:35:59 -07:00
meejah
5ed24913d2 add humanize dependency 2016-07-21 12:35:59 -07:00
meejah
658470ba6e Add some magicfolder database tests 2016-07-21 12:35:59 -07:00
meejah
2573cf18ef Add simple auth-token to get JSON data 2016-07-21 12:35:59 -07:00
meejah
86abe56d91 Flesh out "tahoe magic-folder status" command 
Adds:

 - a JSON endpoint
 - CLI to display information
 - QueuedItem + IQueuedItem for uploader/downloader
 - IProgress interface + PercentProgress implementation
 - progress= args to many upload/download APIs
 2016-07-21 12:35:59 -07:00
David Stainton
3df0a82a38 Add cli stub for magic-folder status command 2016-07-21 12:35:59 -07:00
David Stainton
af10ab43ee Daira's fix during session with David and Meejah. 2016-07-21 12:35:59 -07:00
David Stainton
71e3164e92 Further work in progress refinements to unit tests 
wip from pairing with Daira and Meejah
 2016-07-21 12:35:59 -07:00
David Stainton
9483d24f67 Fix cleanup after test_periodic_full_scan. 2016-07-21 12:35:59 -07:00
David Stainton
38dd49ee5a Add logging for Downloader.stop. 2016-07-21 12:35:59 -07:00