Commit Graph

344 Commits

Author SHA1 Message Date
Brian Warner
3402f75454 copy fixed allocate_tcp_port() from foolscap
The old copy had a bug which occasionally returns a port that was
actually in use, causing intermittent test failures (when large numbers
of ports were allocated). I finally figured out how to fix it in
Foolscap, so this is just a copy of the updated function.

closes ticket:2795
2016-07-06 19:20:32 -07:00
Daira Hopwood
dd02f31a3b Fix missing import 2016-05-04 07:54:42 -07:00
David Stainton
3524729fb1 Add make_dirs_with_absolute_mode to fileutils 2016-05-04 07:54:42 -07:00
Daira Hopwood
50747e2ca0 Strip any long path marker in the input to flush_volume.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-05-04 07:54:42 -07:00
Daira Hopwood
d07c45bdf4 Improve all of the Windows-specific error reporting.
Also make the Windows function declarations more readable and consistent.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-05-04 07:54:42 -07:00
Daira Hopwood
b517e200aa replace_file should allow the replaced file not to exist on Windows.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-05-04 07:54:42 -07:00
Daira Hopwood
7015f5d826 Fix fileutil tests.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-05-04 07:54:42 -07:00
Daira Hopwood
07389571c2 Add long_path=False option to abspath_expanduser_unicode.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-05-04 07:54:42 -07:00
Daira Hopwood
62dc2f7ac1 Re-order some imports in fileutil
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-05-04 07:54:42 -07:00
Daira Hopwood
9bfcb72400 Windows fileutil improvements
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-05-04 07:54:42 -07:00
Brian Warner
bdf1f8460d iputil.py: expose get_local_addresses_sync 2016-04-26 18:39:47 -07:00
Brian Warner
6dfcd52b06 iputil.py: copy allocate_tcp_port() from foolscap
refs ticket:2491
2016-04-26 13:55:51 -07:00
Daira Hopwood
5186e68f6b Add ignore_count to deferredutil 2016-04-19 15:16:13 -06:00
meejah
6e17328eb5 teach abbreviate_time to understand timedelta
original review wanted to do this instead of depending on "humanize"
2016-04-12 18:42:53 -06:00
Brian Warner
04a9c51e3a download_to_data: fix comment 2016-04-12 00:38:08 -07:00
meejah
db517e8edd Progress API
- IProgress interface + PercentProgress implementation
 - progress= args to many upload/download APIs
 - ultimately for magic-folder
2016-04-12 00:30:50 -07:00
Brian Warner
ce548687f8 iputil.py: avoid DirtyReactorError when running tests offline
The udpprot.transport.connect() fails if we don't have a network
connection, but the port is still listening, so trial gives us a
DirtyReactorError. The fix is a "finally:" which does
port.stopListening() even in this case.

Closes ticket:2769
2016-04-03 19:26:57 -10:00
Leif Ryge
5b33f1fdf3 wui: improved columns in welcome page server list
As discussed at https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1973 and in
previous pull request #129.

 - replace lengthy timestamps with human-readable deltas (eg 1h 2m 3s)
 - replace "announced" column with "Last RX" column
 - remove service column (it always said the same thing, "storage")
 - fix colspan on 'You are not presently connected' message

Previous versions, some with github comments: 3fe9053134 , 486dbfc7bd , and c89ea62580, 9fabb92486, bbd8b42a25

Unlike previous attempts, the tests on this one should pass in any timezone.
(But like current master, will fail with Nevow >=0.12...)

Thanks to an anonymous contributor who wrote some of the tests.
2016-01-23 14:28:09 +00:00
Leif Ryge
c86388b79c revert previous commit to fix attribution (vanity)
This reverts the previous commit (modulo the one line that daira added) so that
my contributions will be properly attributed by git blame etc.
2016-01-23 14:27:11 +00:00
Daira Hopwood
a2d724aab7 wui: improved columns in welcome page server list
As discussed at https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1973 and in
previous pull request #129.

 - replace lengthy timestamps with human-readable deltas (eg 1h 2m 3s)
 - replace "announced" column with "Last RX" column
 - remove service column (it always said the same thing, "storage")
 - fix colspan on 'You are not presently connected' message

Previous versions, some with github comments: 3fe9053134 , 486dbfc7bd , and c89ea62580, 9fabb92486, bbd8b42a25

Unlike previous attempts, the tests on this one should pass in any timezone.
(But like current master, will fail with Nevow >=0.12...)

Thanks to an anonymous contributor who wrote some of the tests.
2016-01-15 20:02:19 +00:00
Leif Ryge
6226f6b497 wui: use standard time format (#1077) 2016-01-15 19:46:39 +00:00
Daira Hopwood
f2824459eb Reject path arguments that start with '-' with a usage error.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2015-12-28 21:27:15 +00:00
Daira Hopwood
e88e07a278 Improved error handling and cosmetics for ctypes calls on Windows.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2015-12-28 21:17:32 +00:00
Daira Hopwood
58841cab38 Refactor tahoe.cfg handling to configutil.
Author: David Stainton <david@leastauthority.com>
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2015-12-28 19:36:19 +00:00
Brian Warner
67dcedd8df introweb: fix connection hints for server announcements
A long time ago, the introducer's status web page would show the
advertised IP addresses for all published services, by parsing their
FURL's connection hints. This hasn't worked since about 12-Aug-2014 when
foolscap-0.6.5 changed the internal format of these hints (the column
has been empty this whole time).

This removes the "Advertised IPs" column from the Service Announcements
table. Instead, the service's full connection hints (not just the IP
address) is displayed in a tooltip/popup on the "Announced" timestamp
column.

The code that pulls these connection hints is now tolerant of all three
foolscap styles:

* foolscap<=0.6.4 : tuples of ("ipv4",host,port)
* 0.6.5 .. 0.8.0  : tuples of ("tcp",host,port)
* foolscap>=0.9.0 : strings

fixes ticket:2510
2015-09-22 17:19:52 -07:00
Brian Warner
71323fbd8f introducer: stop tracking hints for subscribed clients
A long time ago, the introducer's status web page would show the
advertised IP addresses for all subscribers, by parsing their
RemoteReference's FURL's connection hints. This hasn't worked since
about 12-Aug-2014 when foolscap-0.6.5 changed the internal format of
these hints.

This removes the feature: we no longer attempt to show advertised IP
addresses of subscribed clients. It also removes the code that looked
inside foolscap internals for this information.
2015-09-22 17:19:43 -07:00
Daira Hopwood
3ae6ceb6a8 Add Namespace utility class. (This is already used in the leasedb/cloud backend branch.)
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2015-07-17 22:12:25 +01:00
Zooko
a9b152780e remove dead code: NamedTemporaryDirectory
fixes #2433
2015-05-26 17:20:11 +00:00
Daira Hopwood
143af61518 fileutil.py: use ctypes.get_last_error() instead of GetLastError(). refs #1531
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2015-05-18 01:42:01 +01:00
Daira Hopwood
4794666df6 On Windows, the user's home directory may be either %USERPROFILE% or %HOMEDRIVE%%HOMEPATH% depending on the Windows version. fixes ticket:2417
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2015-05-13 14:42:31 +01:00
Daira Hopwood
eb34e0d625 Ensure the 'base' argument to abspath_expanduser_unicode takes effect on Windows. fixes #2398
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2015-04-21 21:04:20 +01:00
Daira Hopwood
769de61a90 Add util/dbutil.py: open/create/update sqlite databases given some schema.
Author: Brian Warner <warner@lothar.com>
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2015-04-17 18:11:10 +01:00
Daira Hopwood
4de4e0e65e Some useful Deferred utilities, originally from the cloud backend branch.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2015-04-17 18:10:21 +01:00
Daira Hopwood
21226cbb82 Add from_utf8_or_none and tests.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2015-03-24 16:46:40 +00:00
Daira Hopwood
5a37ff0d0e Discourage future readers of iputil.py from using Win32 APIs in preference to route.exe.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2015-02-19 17:41:09 +00:00
Daira Hopwood
fc90607eec Further refine error handling in windows_getenv to correctly handle zero-length environment values. refs #1674
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2015-02-04 17:02:02 +00:00
Brian Warner
30987c1e7d remove stray trailing whitespace 2015-02-04 01:35:44 -08:00
Daira Hopwood
d756ef1765 More robust error handling in windows_getenv. refs #1674
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2015-02-03 23:47:53 +00:00
Daira Hopwood
c1d5c4f07a Fix user-path-expansion on Windows for non-ASCII home directories. refs #1674
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2015-01-30 00:50:24 +00:00
Daira Hopwood
14f783086f Change uses of os.path.expanduser and os.path.abspath. refs #2235
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2015-01-30 00:50:18 +00:00
Daira Hopwood
21b477f235 Add support in abspath_expanduser_unicode for expanding relative to a base path. refs #2235
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2015-01-30 00:47:09 +00:00
Daira Hopwood
95f98e1aae Quote local paths correctly. refs #2235
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2015-01-30 00:05:20 +00:00
Daira Hopwood
c20a3525b7 Use "long" paths prefixed with \\?\ on Windows. refs #2235
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2015-01-30 00:05:14 +00:00
Daira Hopwood
9a401b760e fileutil cleanup: delete the unused open_or_create function.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2015-01-29 18:19:57 +00:00
Matt Hazinski
4a18f03960 Move UDP listen inside try block 2014-12-29 20:21:24 -05:00
Brian Warner
d27a57cb49 Avoid Popen() of executables that don't exist
The stdlib 'subprocess' module in python-2.7.4 through 2.7.7 suffers
from http://bugs.python.org/issue18851 which causes unrelated file
descriptors to be closed when `subprocess.call()` fails the `exec()`,
such as when the executable being invoked does not actually exist. There
appears to be some randomness involved. This was fixed in python-2.7.8.

Tahoe's iputil.py uses subprocess.call on many different "ifconfig"-type
executables, most of which don't exist on any given platform (added in
git commit 8e31d66cd0). This results in a lot of file-descriptor
closing, which (at least during unit tests) tends to clobber important
things like Tub TCP sockets. This seems to be the root cause behind
ticket:2121, in which normal code tries to close already-closed sockets,
crashing the unit tests. Since different platforms have different
ifconfigs, some platforms will experience more failed execs than others,
so this bug could easily behave differently on linux vs freebsd, as well
as working normally on python-2.7.8 or 2.7.4.

This patch inserts a guard to make sure that os.path.isfile() is true
before allowing Popen.call() to try executing the target. This ought to
be enough to avoid the bug. It changes both iputil.py and
allmydata.__init__ (which uses Popen for calling "lsb_release"), which
are all the places where 'subprocess' is used outside of unit tests.

Other potential fixes: use the 'subprocess32' module from PyPI (which is
a bug-free backport of the Python3 stdlib subprocess module, but would
introduce a new dependency), or require python >= 2.7.8 (but this would
rule out development/deployment on the current OS-X 10.9 release, which
ships with 2.7.5, as well as other distributions like Ubuntu 14.04 LTS).

I believe this closes ticket:2121, and given the apparent relationship
between 2121 and 2023, I think it also closes ticket:2023 (although
since 2023 doesn't have copies of the failing log files, it's hard to
tell). I'm hoping that this will tide us over until 1.11 is released, at
which point we can execute on the plan to remove iputil.py entirely by
changing the way that nodes learn their externally-facing IP address.
2014-09-12 13:01:56 -07:00
Daira Hopwood
c85060c436 Revert [603f5eba32] which was applied unintentionally.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2014-05-05 22:09:33 +01:00
Daira Hopwood
603f5eba32 Use "long" paths prefixed with \\?\ on Windows. refs #2235
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2014-05-04 17:05:48 +01:00
Daira Hopwood
6ca5f5fc39 Use bigger random one-time keys in timing_safe_compare. fixes #2165
Follows this advice from Marsh Ray and Solar Designer: https://twitter.com/zooko/status/431105294777597952

Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2014-02-24 20:43:23 +00:00
Daira Hopwood
d5651a0d0e Rename 'constant_time_compare' to 'timing_safe_compare'. refs #2165
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2014-02-24 20:43:23 +00:00