Commit Graph

469 Commits

Author SHA1 Message Date
Jean-Paul Calderone
4f238d0f64 Clarify possible usage here 2019-02-25 13:11:52 -05:00
Jean-Paul Calderone
abae1be9c6 Add helpers for configuring and using Eliot logging 2019-02-25 11:52:50 -05:00
Jean-Paul Calderone
f90a137552 Basic housekeeping previously missed 2019-02-25 08:57:38 -05:00
Jean-Paul Calderone
f1a7dcf309 A fair bit more Eliot conversion 2019-02-25 08:35:57 -05:00
Jean-Paul Calderone
3020fb979d Hoist some of this to a shared module 2019-02-25 08:35:57 -05:00
Jean-Paul Calderone
6ca328ba94 Fix comparison of NummedObj against non-NummedObj
Previously this would explode with AttributeError.
2019-02-25 08:35:57 -05:00
Jean-Paul Calderone
19f06809e2 That was all nonsense. Throw it out. 2019-02-22 09:11:44 -05:00
Jean-Paul Calderone
e55ee7d044 Move generator context state to a generator context manager 2019-02-22 08:40:16 -05:00
Jean-Paul Calderone
05807ace40 Split the implementation helpers out of the test suite 2019-02-21 12:54:10 -05:00
meejah
136495e405 basedir fixup 2018-09-10 21:58:27 -06:00
meejah
7685fb34cd pull 'basedir' entirely into _Config
Put all config-related methods into _Config; change
code to ask config for paths instead of using basedir;
add some better docstrings
2018-09-10 21:58:27 -06:00
meejah
739aaa3ef9 put create() methods in i2p_, tor_provider
Also Provider -> _Provider, improve docs and update tests
2018-09-10 21:58:27 -06:00
meejah
35810a5692 pull 'basedir' entirely into _Config
Put all config-related methods into _Config; change
code to ask config for paths instead of using basedir;
add some better docstrings
2018-08-25 02:10:44 -06:00
meejah
c9e00a988a
Merge pull request #475 from meejah/2909.backup-behavior.0
#2909 fix .backup file behavior and (some of) the incorrect .conflict cases (#2911)
2018-05-01 15:52:10 -06:00
meejah
b54d5d88f8 unused var 2018-04-30 14:55:52 -06:00
meejah
c71116d45e keep original exception behavior 2018-04-30 14:55:51 -06:00
meejah
c2aec93209 fix windows move_into_place logic 2018-04-30 14:55:51 -06:00
meejah
99c3d7f23f this shouldn't have gotten removed 2018-04-30 14:55:51 -06:00
meejah
cc93a12a85 Test fixups 2018-04-30 14:55:51 -06:00
meejah
affb80e39e test fixups 2018-04-30 14:55:51 -06:00
meejah
47b1787633 stop writing .backup files for normal overwrites 2018-04-30 14:55:51 -06:00
Jean-Paul Calderone
b623a4a199 Remove dead Tor TCP control port setup code.
If someone wants this I bet they can figure it out.
2018-04-26 15:32:27 -04:00
Jean-Paul Calderone
8d4d000132 Fix pre-release matching regex character class
Previously matched any single character from `abc|r` (with duplicate
specification of `c`).  Now matches any single character from `abc` or
the two character sequence `rc`.

I guess this was the intent, anyway.
2018-04-26 15:20:27 -04:00
Jean-Paul Calderone
6b16afaa2e Avoid using the list comprehension loop variable
It works fine but it relies on leaky scopes.
2018-04-26 15:16:00 -04:00
Brian Warner
5e62118e14 hush somewhat-real problems found by static analysis
These would have caused problems if a certain unlikely code path was taken:
mostly error-handling pathways.
2017-12-31 00:23:43 +01:00
Brian Warner
da4e7dcfbe hush a bunch of not-really-problems caught by the lgtm.com static analyzer
This is all minor stuff: unreachable debug code (that should be commented-out
instead of in an 'if False:' block), unnecessary 'pass' and 'global'
statements, redundantly-initialized variables. No behavior changes. Nothing
here was actually broken, it just looked suspicious to the static analysis at
https://lgtm.com/projects/g/tahoe-lafs/tahoe-lafs/alerts/?mode=list .
2017-12-31 00:19:41 +01:00
meejah
87ad3cb8e9 fix 2880
- remember upload URI so we don't re-download
- account for empty files in "should_download"
- fix 'conflicted' asserts in tests
2017-12-05 12:02:16 -07:00
meejah
1b6f477549 Pull "config" handling out of Node and hide node-derivitives
This includes:

 - refactor to create_client and _Client
 - refactor to _IntroducerNode and create_introducer
2017-11-20 12:57:20 -07:00
Brian Warner
1f1afe65ec tor_provider.get_listener(): return a real endpoint, instead of a descriptor
get_listener() is allowed to return either, and the Tor provider is currently
simple enough to not really need more than a basic descriptor, but have it
return a full Endpoint for use as an example of what I2P can do later.
2017-11-03 00:24:46 -07:00
Brian Warner
097abb42fa tahoe.cfg: add tub.port=listen:i2p (and/or listen:tor)
This delegates the construction of the server Endpoint object to the i2p/tor
Provider, which can use the i2p/tor section of the config file to add options
which would be awkward to express as text in an endpoint descriptor string.

refs ticket:2889 (but note this merely makes room for a function to be
written that can process I2CP options, it does not actually handle such
options, so it does not close this ticket yet)
2017-11-03 00:24:46 -07:00
Brian Warner
d1fd43aa4f tor/i2p: rename create_onion/create_dest to create_config 2017-11-03 00:24:46 -07:00
Brian Warner
8dd0a5f772 dictutil: remove unused move, subtract, del_if_present 2017-08-12 21:48:05 -07:00
Brian Warner
61b85dbf78 dictutil: remove unused ValueOrderedDict 2017-08-12 21:34:24 -07:00
Brian Warner
4f493b1a03 remove unused UtilDict 2017-08-12 21:31:47 -07:00
Brian Warner
a76d9e084f dictutil: remove unused NumDict 2017-08-12 21:21:04 -07:00
Brian Warner
3afa38adb4 NummedObj: stop using dictutil.NumDict 2017-08-12 21:02:32 -07:00
Brian Warner
3f2f7dfb05 dictutil: fix bug in str(ValueOrderedDict), and improve test coverage
It looks like str() was meant to truncate the dict, but a missing i+=1 meant
that it never actually did. I also changed the format to include a clear
"..." in case we truncate it, to avoid confusion with a non-truncated dict of
the same size.

This also improves test coverage in subtract() and
NumDict.item_with_largest_value().

refs ticket:2891
2017-08-10 10:27:02 -07:00
meejah
05f48c3601 Various cleanups, fixes and improvements
Squashed all commits that were meejah's between
30d68fb499f300a393fa0ced5980229f4bb6efda
and
33c268ed3a8c63a809f4403e307ecc13d848b1ab
On the branch meejah:1382.markberger-rewrite-rebase.6 as
per review
2017-06-05 16:31:41 -06:00
David Stainton
e68b331bb1 Add servers of happiness hypothesis testing
Fix happiness test var names

Remove unused imports

Get rid of trailing whitespace
2017-06-05 16:26:46 -06:00
meejah
ef17ef2c62 fix happiness calculation
unit-test for happiness calculation

unused function

put old servers_of_happiness() calculation back for now

test for calculate_happiness

remove some redundant functions
2017-06-05 16:26:46 -06:00
David Stainton
42011e775d Make correction to docstring for Tahoe2ServerSelector's _handle_existing_response
Add comments 10 and 8 from the servers of happiness spec

Fix bug in _filter_g3 for servers of happiness

Remove usage of HappinessUpload class

here we modifying the PeerSelector class.
we make sure to correctly calculate the happiness value
by ignoring keys who's value are None...

Remove HappinessUpload and tests

Replace helper servers_of_happiness

we replace it's previous implementation with a new
wrapper function that uses share_placement
2017-06-05 16:26:46 -06:00
Mark Berger
17cff7a176 Implements 'Servers of Happiness' algorithm for upload
This is Mark Berger's original commits, from ticket #1382
2017-06-05 16:26:46 -06:00
meejah
49ae2c5eb7 use @implementer instead of implements
Also, derive some more things from 'object'
2017-02-27 11:01:30 -07:00
str4d
998af5c26e Pass I2P keyfile to foolscap
If no session management is performed, txi2p starts a process-wide session the
first time a connection (client or server) is opened; all subsequent connections
use that session and its configuration properties.

This commit ensures that the same properties are passed to both client and
server endpoints, so that the correct I2P Destination is started regardless of
whether the first connection made by Tahoe-LAFS is for a client or server.

Closes #2858.
2017-01-17 20:12:32 -05:00
Brian Warner
5cbe580d90 improve ConnectionStatus and welcome-page display
* replace "last_details" with "non_connected_statuses" dict
* rename "last_connection_summary" to just "summary"
* for connected servers, show other hints in a tooltip
* for not-yet-connected servers, show all hints in a list
* build the list (in STAN) on the server side, not using IContainer
2016-12-09 18:41:38 -08:00
meejah
74209e643e Show the connection-status of all hints until we're connected to one 2016-12-09 18:41:38 -08:00
Brian Warner
48fc14bd30 add IConnectionStatus and implementation
also a function to build one from a foolscap.Reconnector
2016-12-08 15:41:37 -08:00
Brian Warner
8d008967e7 tor_provider: use new Foolscap API to provide better status 2016-12-08 15:41:37 -08:00
str4d
6071c2b6f8
Implement i2p_provider and --listen=i2p
Closes ticket:2838
2016-10-23 20:30:01 -05:00
Lukas Pirl
1491e9fab1 added /sbin/ip as possible binary to detect local IP address
fixes IP detection for RHEL flavors

Signed-off-by: Lukas Pirl <git@lukas-pirl.de>
2016-10-12 12:46:12 +02:00
meejah
2ecb8171eb flip remote/local around for Onion-service 'ports' lines 2016-10-09 01:21:52 -04:00
meejah
eb6c6bee9e tor's output, for debugging 2016-10-09 01:21:52 -04:00
meejah
646fad69ed ControlPort needs "unix:" prefix for Tor, and allocate SOCKSPort 2016-10-09 01:21:52 -04:00
Brian Warner
a1741ce4dc rewrite tor_provider
This uses a unix-domain control port, and includes test coverage.

create_onion() displays pacifier messages, since the allocate-onion step
takes around 35 seconds
2016-10-09 01:21:47 -04:00
Brian Warner
8f9ad009bb move torutil.py to util/tor_provider.py 2016-10-09 00:57:02 -04:00
Brian Warner
5ef04ef59e add hashutil.permute_server_hash
which uses SHA1 to combine the file's storage index (known as "peer
selection index" in this context) and each server's "server permutation
seed". This is the only thing in tahoe that uses SHA1.

With this change, we stop importing sha1 from anywhere else.
2016-09-26 20:42:42 -07:00
meejah
2732c37941 Check for unknown config options
The list of valid sections + config-items came from
grep'ing the source for `.get_config`
2016-09-06 15:05:46 -06:00
Brian Warner
57e7f7bb7c import/delegate-to foolscap's allocate_tcp_port
(instead of using a copy). Foolscap-0.12.3 fixes a problem with
allocate_tcp_port() that was causing intermittent test failures. I think
it makes more sense to use Foolscap's copy (and fixes) than to keep
re-copying it into Tahoe each time it changes.

If/when we manage to stop depending upon foolscap for server RPC, we can
re-copy this back into tahoe's source tree.

refs ticket:2795
2016-09-01 22:39:03 -07:00
meejah
87acfe968e pyflakes warnings 2016-07-21 12:35:59 -07:00
meejah
304da362f7 remove prints 2016-07-21 12:35:59 -07:00
Daira Hopwood
504e0d02f3 refactor to use context managers 2016-07-21 12:35:59 -07:00
Daira Hopwood
b949082f8e Change type of mtime and ctime stored in magicfolderdb to integer nanoseconds.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-07-21 12:35:59 -07:00
meejah
a50d0e5c8b WIP and debugging things
Conflicts:
	src/allmydata/test/test_magic_folder.py
2016-07-21 12:35:59 -07:00
meejah
e6104cd1a2 Replace _lazy_tail with iterative loop, cleanup tests (#2412)
* uses @inlineCallbacks to turn the _lazy_tail recursion into
   a "real" looking loop;
 * remove the need for "immediate" vs delayed iteration of said loop;
 * make it easier for the unit-tests to control the behavior of the
   uploader/downloader;
 * consolidates (some) setup/teardown code into the setUp and tearDown
   hooks provided by unittest so unit-tests aren't doing that themselves
 * re-factors some of the unit-tests to use an @inlineCallbacks style
   so they're easier to follow and debug

This doesn't tackle the "how to know when our inotify events have arrived"
problem the unit-tests still have, nor does it eliminate the myriad bits
of state that get added to tests via all the MixIns.
2016-07-21 12:35:59 -07:00
David Stainton
d2ae4255c8 Add daira's implementation of make_dirs_with_absolute_mode 2016-07-21 12:35:59 -07:00
David Stainton
cd390b90e4 remove superfluous trailing comma from make_dirs_with_absolute_mode def 2016-07-21 12:35:59 -07:00
David Stainton
ba35d7262b Add unit test and make corrections to make_dirs_with_absolute_mode 2016-07-21 12:35:59 -07:00
David Stainton
ee44732d03 Break out our chmod while loop into fielutils.py 2016-07-21 12:35:59 -07:00
Daira Hopwood
b53efdbf8b Fix a corner case for to_filepath on Windows to make it consistent with Unix.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-07-21 12:35:59 -07:00
Daira Hopwood
b6b638efd1 Depend on FilePath.asTextMode().
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-07-21 12:35:59 -07:00
Daira Hopwood
903c6f1d6c Fix check for initial '-' in argv_to_abspath.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-07-21 12:35:59 -07:00
David Stainton
16275cb8f6 Teach magic-folder join to use argv_to_abspath
- also we modify argv_to_abspath to through a usage error
if the name starts with a '-'

- add a test
currently the tests fail
2016-07-21 12:35:59 -07:00
Daira Hopwood
e03a1e3361 WIP.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-07-21 12:35:59 -07:00
Daira Hopwood
7b4067850f Unicode path fixes for drop-upload.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-07-21 12:35:59 -07:00
Brian Warner
2c5f7ed425 factor out yamlutil.py
The yaml.SafeLoader.add_constructor() should probably only be done once,
and moving this all into a module gives us an opportunity to test it
directly.
2016-07-19 17:31:41 -07:00
Brian Warner
3402f75454 copy fixed allocate_tcp_port() from foolscap
The old copy had a bug which occasionally returns a port that was
actually in use, causing intermittent test failures (when large numbers
of ports were allocated). I finally figured out how to fix it in
Foolscap, so this is just a copy of the updated function.

closes ticket:2795
2016-07-06 19:20:32 -07:00
Daira Hopwood
dd02f31a3b Fix missing import 2016-05-04 07:54:42 -07:00
David Stainton
3524729fb1 Add make_dirs_with_absolute_mode to fileutils 2016-05-04 07:54:42 -07:00
Daira Hopwood
50747e2ca0 Strip any long path marker in the input to flush_volume.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-05-04 07:54:42 -07:00
Daira Hopwood
d07c45bdf4 Improve all of the Windows-specific error reporting.
Also make the Windows function declarations more readable and consistent.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-05-04 07:54:42 -07:00
Daira Hopwood
b517e200aa replace_file should allow the replaced file not to exist on Windows.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-05-04 07:54:42 -07:00
Daira Hopwood
7015f5d826 Fix fileutil tests.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-05-04 07:54:42 -07:00
Daira Hopwood
07389571c2 Add long_path=False option to abspath_expanduser_unicode.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-05-04 07:54:42 -07:00
Daira Hopwood
62dc2f7ac1 Re-order some imports in fileutil
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-05-04 07:54:42 -07:00
Daira Hopwood
9bfcb72400 Windows fileutil improvements
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-05-04 07:54:42 -07:00
Brian Warner
bdf1f8460d iputil.py: expose get_local_addresses_sync 2016-04-26 18:39:47 -07:00
Brian Warner
6dfcd52b06 iputil.py: copy allocate_tcp_port() from foolscap
refs ticket:2491
2016-04-26 13:55:51 -07:00
Daira Hopwood
5186e68f6b Add ignore_count to deferredutil 2016-04-19 15:16:13 -06:00
meejah
6e17328eb5 teach abbreviate_time to understand timedelta
original review wanted to do this instead of depending on "humanize"
2016-04-12 18:42:53 -06:00
Brian Warner
04a9c51e3a download_to_data: fix comment 2016-04-12 00:38:08 -07:00
meejah
db517e8edd Progress API
- IProgress interface + PercentProgress implementation
 - progress= args to many upload/download APIs
 - ultimately for magic-folder
2016-04-12 00:30:50 -07:00
Brian Warner
ce548687f8 iputil.py: avoid DirtyReactorError when running tests offline
The udpprot.transport.connect() fails if we don't have a network
connection, but the port is still listening, so trial gives us a
DirtyReactorError. The fix is a "finally:" which does
port.stopListening() even in this case.

Closes ticket:2769
2016-04-03 19:26:57 -10:00
Leif Ryge
5b33f1fdf3 wui: improved columns in welcome page server list
As discussed at https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1973 and in
previous pull request #129.

 - replace lengthy timestamps with human-readable deltas (eg 1h 2m 3s)
 - replace "announced" column with "Last RX" column
 - remove service column (it always said the same thing, "storage")
 - fix colspan on 'You are not presently connected' message

Previous versions, some with github comments: 3fe9053134 , 486dbfc7bd , and c89ea62580, 9fabb92486, bbd8b42a25

Unlike previous attempts, the tests on this one should pass in any timezone.
(But like current master, will fail with Nevow >=0.12...)

Thanks to an anonymous contributor who wrote some of the tests.
2016-01-23 14:28:09 +00:00
Leif Ryge
c86388b79c revert previous commit to fix attribution (vanity)
This reverts the previous commit (modulo the one line that daira added) so that
my contributions will be properly attributed by git blame etc.
2016-01-23 14:27:11 +00:00
Daira Hopwood
a2d724aab7 wui: improved columns in welcome page server list
As discussed at https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1973 and in
previous pull request #129.

 - replace lengthy timestamps with human-readable deltas (eg 1h 2m 3s)
 - replace "announced" column with "Last RX" column
 - remove service column (it always said the same thing, "storage")
 - fix colspan on 'You are not presently connected' message

Previous versions, some with github comments: 3fe9053134 , 486dbfc7bd , and c89ea62580, 9fabb92486, bbd8b42a25

Unlike previous attempts, the tests on this one should pass in any timezone.
(But like current master, will fail with Nevow >=0.12...)

Thanks to an anonymous contributor who wrote some of the tests.
2016-01-15 20:02:19 +00:00
Leif Ryge
6226f6b497 wui: use standard time format (#1077) 2016-01-15 19:46:39 +00:00
Daira Hopwood
f2824459eb Reject path arguments that start with '-' with a usage error.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2015-12-28 21:27:15 +00:00
Daira Hopwood
e88e07a278 Improved error handling and cosmetics for ctypes calls on Windows.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2015-12-28 21:17:32 +00:00
Daira Hopwood
58841cab38 Refactor tahoe.cfg handling to configutil.
Author: David Stainton <david@leastauthority.com>
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2015-12-28 19:36:19 +00:00
Brian Warner
67dcedd8df introweb: fix connection hints for server announcements
A long time ago, the introducer's status web page would show the
advertised IP addresses for all published services, by parsing their
FURL's connection hints. This hasn't worked since about 12-Aug-2014 when
foolscap-0.6.5 changed the internal format of these hints (the column
has been empty this whole time).

This removes the "Advertised IPs" column from the Service Announcements
table. Instead, the service's full connection hints (not just the IP
address) is displayed in a tooltip/popup on the "Announced" timestamp
column.

The code that pulls these connection hints is now tolerant of all three
foolscap styles:

* foolscap<=0.6.4 : tuples of ("ipv4",host,port)
* 0.6.5 .. 0.8.0  : tuples of ("tcp",host,port)
* foolscap>=0.9.0 : strings

fixes ticket:2510
2015-09-22 17:19:52 -07:00