ExternalVendorCode/tahoe-lafs
1
0
Fork 0
mirror of https://github.com/tahoe-lafs/tahoe-lafs.git synced 2025-01-31 08:25:35 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity

Merge branch '3584.integration-tests-sftp' into 3579.ftp-python-3

Browse Source
This commit is contained in:
Itamar Turner-Trauring 2021-01-11 14:05:07 -05:00
commit 1b1d87f9f9
27 changed files with 320 additions and 712 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

7
CREDITS
View File

@ -206,4 +206,9 @@ D: various bug-fixes and features
N: Viktoriia Savchuk
W: https://twitter.com/viktoriiasvchk
D: Developer community focused improvements on the README file.
D: Developer community focused improvements on the README file.
N: Lukas Pirl
E: tahoe@lukas-pirl.de
W: http://lukas-pirl.de
D: Buildslaves (Debian, Fedora, CentOS; 2016-2021)

4
docs/about.rst
View File

@ -67,12 +67,12 @@ Here's how it works:
A "storage grid" is made up of a number of storage servers. A storage server
has direct attached storage (typically one or more hard disks). A "gateway"
communicates with storage nodes, and uses them to provide access to the
grid over protocols such as HTTP(S), SFTP or FTP.
grid over protocols such as HTTP(S) and SFTP.
Note that you can find "client" used to refer to gateway nodes (which act as
a client to storage servers), and also to processes or programs connecting to
a gateway node and performing operations on the grid -- for example, a CLI
command, Web browser, SFTP client, or FTP client.
command, Web browser, or SFTP client.
Users do not rely on storage servers to provide *confidentiality* nor
*integrity* for their data -- instead all of the data is encrypted and

9
docs/configuration.rst
View File

@ -81,7 +81,6 @@ Client/server nodes provide one or more of the following services:
* web-API service
* SFTP service
* FTP service
* helper service
* storage service.
@ -708,12 +707,12 @@ CLI
file store, uploading/downloading files, and creating/running Tahoe
nodes. See :doc:`frontends/CLI` for details.
SFTP, FTP
SFTP
Tahoe can also run both SFTP and FTP servers, and map a username/password
Tahoe can also run SFTP servers, and map a username/password
pair to a top-level Tahoe directory. See :doc:`frontends/FTP-and-SFTP`
for instructions on configuring these services, and the ``[sftpd]`` and
``[ftpd]`` sections of ``tahoe.cfg``.
for instructions on configuring this service, and the ``[sftpd]``
section of ``tahoe.cfg``.
Storage Server Configuration

81
docs/frontends/FTP-and-SFTP.rst
View File

@ -1,22 +1,21 @@
.. -*- coding: utf-8-with-signature -*-
=================================
Tahoe-LAFS SFTP and FTP Frontends
=================================
========================
Tahoe-LAFS SFTP Frontend
========================
1. `SFTP/FTP Background`_
1. `SFTP Background`_
2. `Tahoe-LAFS Support`_
3. `Creating an Account File`_
4. `Running An Account Server (accounts.url)`_
5. `Configuring SFTP Access`_
6. `Configuring FTP Access`_
7. `Dependencies`_
8. `Immutable and Mutable Files`_
9. `Known Issues`_
6. `Dependencies`_
7. `Immutable and Mutable Files`_
8. `Known Issues`_
SFTP/FTP Background
===================
SFTP Background
===============
FTP is the venerable internet file-transfer protocol, first developed in
1971. The FTP server usually listens on port 21. A separate connection is
@ -33,20 +32,18 @@ Both FTP and SFTP were developed assuming a UNIX-like server, with accounts
and passwords, octal file modes (user/group/other, read/write/execute), and
ctime/mtime timestamps.
We recommend SFTP over FTP, because the protocol is better, and the server
implementation in Tahoe-LAFS is more complete. See `Known Issues`_, below,
for details.
Previous versions of Tahoe-LAFS supported FTP, but now only the superior SFTP
frontend is supported. See `Known Issues`_, below, for details on the
limitations of SFTP.
Tahoe-LAFS Support
==================
All Tahoe-LAFS client nodes can run a frontend SFTP server, allowing regular
SFTP clients (like ``/usr/bin/sftp``, the ``sshfs`` FUSE plugin, and many
others) to access the file store. They can also run an FTP server, so FTP
clients (like ``/usr/bin/ftp``, ``ncftp``, and others) can too. These
frontends sit at the same level as the web-API interface.
others) to access the file store.
Since Tahoe-LAFS does not use user accounts or passwords, the SFTP/FTP
Since Tahoe-LAFS does not use user accounts or passwords, the SFTP
servers must be configured with a way to first authenticate a user (confirm
that a prospective client has a legitimate claim to whatever authorities we
might grant a particular user), and second to decide what directory cap
@ -173,39 +170,6 @@ clients and with the sshfs filesystem, see wiki:SftpFrontend_
.. _wiki:SftpFrontend: https://tahoe-lafs.org/trac/tahoe-lafs/wiki/SftpFrontend
Configuring FTP Access
======================
To enable the FTP server with an accounts file, add the following lines to
the BASEDIR/tahoe.cfg file::
[ftpd]
enabled = true
port = tcp:8021:interface=127.0.0.1
accounts.file = private/accounts
The FTP server will listen on the given port number and on the loopback
interface only. The "accounts.file" pathname will be interpreted relative to
the node's BASEDIR.
To enable the FTP server with an account server instead, provide the URL of
that server in an "accounts.url" directive::
[ftpd]
enabled = true
port = tcp:8021:interface=127.0.0.1
accounts.url = https://example.com/login
You can provide both accounts.file and accounts.url, although it probably
isn't very useful except for testing.
FTP provides no security, and so your password or caps could be eavesdropped
if you connect to the FTP server remotely. The examples above include
":interface=127.0.0.1" in the "port" option, which causes the server to only
accept connections from localhost.
Public key authentication is not supported for FTP.
Dependencies
============
@ -216,7 +180,7 @@ separately: debian puts it in the "python-twisted-conch" package.
Immutable and Mutable Files
===========================
All files created via SFTP (and FTP) are immutable files. However, files can
All files created via SFTP are immutable files. However, files can
only be created in writeable directories, which allows the directory entry to
be relinked to a different file. Normally, when the path of an immutable file
is opened for writing by SFTP, the directory entry is relinked to another
@ -256,18 +220,3 @@ See also wiki:SftpFrontend_.
.. _ticket #1059: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1059
.. _ticket #1089: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1089
Known Issues in the FTP Frontend
--------------------------------
Mutable files are not supported by the FTP frontend (`ticket #680`_).
Non-ASCII filenames are not supported by FTP (`ticket #682`_).
The FTP frontend sometimes fails to report errors, for example if an upload
fails because it does meet the "servers of happiness" threshold (`ticket
#1081`_).
.. _ticket #680: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/680
.. _ticket #682: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/682
.. _ticket #1081: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1081

2
docs/frontends/webapi.rst
View File

@ -2157,7 +2157,7 @@ When modifying the file, be careful to update it atomically, otherwise a
request may arrive while the file is only halfway written, and the partial
file may be incorrectly parsed.
The blacklist is applied to all access paths (including SFTP, FTP, and CLI
The blacklist is applied to all access paths (including SFTP and CLI
operations), not just the web-API. The blacklist also applies to directories.
If a directory is blacklisted, the gateway will refuse access to both that
directory and any child files/directories underneath it, when accessed via

2
docs/helper.rst
View File

@ -122,7 +122,7 @@ Who should consider using a Helper?
* clients who experience problems with TCP connection fairness: if other
programs or machines in the same home are getting less than their fair
share of upload bandwidth. If the connection is being shared fairly, then
a Tahoe upload that is happening at the same time as a single FTP upload
a Tahoe upload that is happening at the same time as a single SFTP upload
should get half the bandwidth.
* clients who have been given the helper.furl by someone who is running a
Helper and is willing to let them use it

6
docs/known_issues.rst
View File

@ -23,7 +23,7 @@ Known Issues in Tahoe-LAFS v1.10.3, released 30-Mar-2016
* `Disclosure of file through embedded hyperlinks or JavaScript in that file`_
* `Command-line arguments are leaked to other local users`_
* `Capabilities may be leaked to web browser phishing filter / "safe browsing" servers`_
* `Known issues in the FTP and SFTP frontends`_
* `Known issues in the SFTP frontend`_
* `Traffic analysis based on sizes of files/directories, storage indices, and timing`_
* `Privacy leak via Google Chart API link in map-update timing web page`_
@ -213,8 +213,8 @@ To disable the filter in Chrome:
----
Known issues in the FTP and SFTP frontends
------------------------------------------
Known issues in the SFTP frontend
---------------------------------
These are documented in :doc:`frontends/FTP-and-SFTP` and on `the
SftpFrontend page`_ on the wiki.

7
docs/running.rst
View File

@ -207,10 +207,10 @@ create a new directory and lose the capability to it, then you cannot
access that directory ever again.
The SFTP and FTP frontends
--------------------------
The SFTP frontend
-----------------
You can access your Tahoe-LAFS grid via any SFTP_ or FTP_ client. See
You can access your Tahoe-LAFS grid via any SFTP_ client. See
:doc:`frontends/FTP-and-SFTP` for how to set this up. On most Unix
platforms, you can also use SFTP to plug Tahoe-LAFS into your computer's
local filesystem via ``sshfs``, but see the `FAQ about performance
@ -220,7 +220,6 @@ The SftpFrontend_ page on the wiki has more information about using SFTP with
Tahoe-LAFS.
.. _SFTP: https://en.wikipedia.org/wiki/SSH_file_transfer_protocol
.. _FTP: https://en.wikipedia.org/wiki/File_Transfer_Protocol
.. _FAQ about performance problems: https://tahoe-lafs.org/trac/tahoe-lafs/wiki/FAQ#Q23_FUSE
.. _SftpFrontend: https://tahoe-lafs.org/trac/tahoe-lafs/wiki/SftpFrontend

43
integration/conftest.py
View File

@ -7,6 +7,7 @@ from os import mkdir, listdir, environ
from os.path import join, exists
from tempfile import mkdtemp, mktemp
from functools import partial
from json import loads
from foolscap.furl import (
decode_furl,
@ -37,6 +38,9 @@ from util import (
_tahoe_runner_optional_coverage,
await_client_ready,
TahoeProcess,
cli,
_run_node,
generate_ssh_key
)
@ -349,6 +353,45 @@ def alice(reactor, temp_dir, introducer_furl, flog_gatherer, storage_nodes, requ
storage=False,
)
)
await_client_ready(process)
# 1. Create a new RW directory cap:
cli(process, "create-alias", "test")
rwcap = loads(cli(process, "list-aliases", "--json"))["test"]["readwrite"]
# 2. Enable SFTP on the node:
host_ssh_key_path = join(process.node_dir, "private", "ssh_host_rsa_key")
accounts_path = join(process.node_dir, "private", "accounts")
with open(join(process.node_dir, "tahoe.cfg"), "a") as f:
f.write("""\
[sftpd]
enabled = true
port = tcp:8022:interface=127.0.0.1
host_pubkey_file = {ssh_key_path}.pub
host_privkey_file = {ssh_key_path}
accounts.file = {accounts_path}
""".format(ssh_key_path=host_ssh_key_path, accounts_path=accounts_path))
generate_ssh_key(host_ssh_key_path)
# 3. Add a SFTP access file with username/password and SSH key auth.
# The client SSH key path is typically going to be somewhere else (~/.ssh,
# typically), but for convenience sake for testing we'll put it inside node.
client_ssh_key_path = join(process.node_dir, "private", "ssh_client_rsa_key")
generate_ssh_key(client_ssh_key_path)
# Pub key format is "ssh-rsa <thekey> <username>". We want the key.
ssh_public_key = open(client_ssh_key_path + ".pub").read().strip().split()[1]
with open(accounts_path, "w") as f:
f.write("""\
alice password {rwcap}
alice2 ssh-rsa {ssh_public_key} {rwcap}
""".format(rwcap=rwcap, ssh_public_key=ssh_public_key))
# 4. Restart the node with new SFTP config.
process.kill()
pytest_twisted.blockon(_run_node(reactor, process.node_dir, request, None))
await_client_ready(process)
return process

151
integration/test_sftp.py Normal file
View File

@ -0,0 +1,151 @@
"""
It's possible to create/rename/delete files and directories in Tahoe-LAFS using
SFTP.
"""
from __future__ import unicode_literals
from __future__ import absolute_import
from __future__ import division
from __future__ import print_function
from future.utils import PY2
if PY2:
from future.builtins import filter, map, zip, ascii, chr, hex, input, next, oct, open, pow, round, super, bytes, dict, list, object, range, str, max, min # noqa: F401
from posixpath import join
from stat import S_ISDIR
from paramiko import SSHClient
from paramiko.client import AutoAddPolicy
from paramiko.sftp_client import SFTPClient
from paramiko.ssh_exception import AuthenticationException
from paramiko.rsakey import RSAKey
import pytest
from .util import generate_ssh_key
def connect_sftp(connect_args={"username": "alice", "password": "password"}):
"""Create an SFTP client."""
client = SSHClient()
client.set_missing_host_key_policy(AutoAddPolicy)
client.connect("localhost", port=8022, look_for_keys=False,
allow_agent=False, **connect_args)
sftp = SFTPClient.from_transport(client.get_transport())
def rmdir(path, delete_root=True):
for f in sftp.listdir_attr(path=path):
childpath = join(path, f.filename)
if S_ISDIR(f.st_mode):
rmdir(childpath)
else:
sftp.remove(childpath)
if delete_root:
sftp.rmdir(path)
# Delete any files left over from previous tests :(
rmdir("/", delete_root=False)
return sftp
def test_bad_account_password_ssh_key(alice, tmpdir):
"""
Can't login with unknown username, wrong password, or wrong SSH pub key.
"""
# Wrong password, wrong username:
for u, p in [("alice", "wrong"), ("someuser", "password")]:
with pytest.raises(AuthenticationException):
connect_sftp(connect_args={
"username": u, "password": p,
})
another_key = join(str(tmpdir), "ssh_key")
generate_ssh_key(another_key)
good_key = RSAKey(filename=join(alice.node_dir, "private", "ssh_client_rsa_key"))
bad_key = RSAKey(filename=another_key)
# Wrong key:
with pytest.raises(AuthenticationException):
connect_sftp(connect_args={
"username": "alice2", "pkey": bad_key,
})
# Wrong username:
with pytest.raises(AuthenticationException):
connect_sftp(connect_args={
"username": "someoneelse", "pkey": good_key,
})
def test_ssh_key_auth(alice):
"""It's possible to login authenticating with SSH public key."""
key = RSAKey(filename=join(alice.node_dir, "private", "ssh_client_rsa_key"))
sftp = connect_sftp(connect_args={
"username": "alice2", "pkey": key
})
assert sftp.listdir() == []
def test_read_write_files(alice):
"""It's possible to upload and download files."""
sftp = connect_sftp()
f = sftp.file("myfile", "wb")
f.write(b"abc")
f.write(b"def")
f.close()
f = sftp.file("myfile", "rb")
assert f.read(4) == b"abcd"
assert f.read(2) == b"ef"
assert f.read(1) == b""
f.close()
def test_directories(alice):
"""
It's possible to create, list directories, and create and remove files in
them.
"""
sftp = connect_sftp()
assert sftp.listdir() == []
sftp.mkdir("childdir")
assert sftp.listdir() == ["childdir"]
with sftp.file("myfile", "wb") as f:
f.write(b"abc")
assert sorted(sftp.listdir()) == ["childdir", "myfile"]
sftp.chdir("childdir")
assert sftp.listdir() == []
with sftp.file("myfile2", "wb") as f:
f.write(b"def")
assert sftp.listdir() == ["myfile2"]
sftp.chdir(None) # root
with sftp.file("childdir/myfile2", "rb") as f:
assert f.read() == b"def"
sftp.remove("myfile")
assert sftp.listdir() == ["childdir"]
sftp.rmdir("childdir")
assert sftp.listdir() == []
def test_rename(alice):
"""Directories and files can be renamed."""
sftp = connect_sftp()
sftp.mkdir("dir")
filepath = join("dir", "file")
with sftp.file(filepath, "wb") as f:
f.write(b"abc")
sftp.rename(filepath, join("dir", "file2"))
sftp.rename("dir", "dir2")
with sftp.file(join("dir2", "file2"), "rb") as f:
assert f.read() == b"abc"

31
integration/util.py
View File

@ -5,6 +5,7 @@ from os import mkdir, environ
from os.path import exists, join
from six.moves import StringIO
from functools import partial
from subprocess import check_output, check_call
from twisted.python.filepath import (
FilePath,
@ -15,6 +16,8 @@ from twisted.internet.error import ProcessExitedAlready, ProcessDone
import requests
from paramiko.rsakey import RSAKey
from allmydata.util.configutil import (
get_config,
set_config,
@ -175,6 +178,10 @@ class TahoeProcess(object):
u"portnum",
)
def kill(self):
"""Kill the process, block until it's done."""
_cleanup_tahoe_process(self.transport, self.transport.exited)
def __str__(self):
return "<TahoeProcess in '{}'>".format(self._node_dir)
@ -249,7 +256,7 @@ def _create_node(reactor, request, temp_dir, introducer_furl, flog_gatherer, nam
'--helper',
]
if not storage:
args.append('--no-storage')
args.append('--no-storage')
args.append(node_dir)
_tahoe_runner_optional_coverage(done_proto, reactor, request, args)
@ -390,17 +397,13 @@ def await_file_vanishes(path, timeout=10):
raise FileShouldVanishException(path, timeout)
def cli(request, reactor, node_dir, *argv):
def cli(node, *argv):
"""
Run a tahoe CLI subcommand for a given node, optionally running
under coverage if '--coverage' was supplied.
Run a tahoe CLI subcommand for a given node in a blocking manner, returning
the output.
"""
proto = _CollectOutputProtocol()
_tahoe_runner_optional_coverage(
proto, reactor, request,
['--node-directory', node_dir] + list(argv),
)
return proto.done
arguments = ["tahoe", '--node-directory', node.node_dir]
return check_output(arguments + list(argv))
def node_url(node_dir, uri_fragment):
@ -505,3 +508,11 @@ def await_client_ready(tahoe, timeout=10, liveness=60*2):
tahoe,
)
)
def generate_ssh_key(path):
"""Create a new SSH private/public key pair."""
key = RSAKey.generate(2048)
key.write_private_key_file(path)
with open(path + ".pub", "wb") as f:
f.write(b"%s %s" % (key.get_name(), key.get_base64()))

0
newsfragments/3577.minor Normal file
View File

0
newsfragments/3582.minor Normal file
View File

1
newsfragments/3583.removed Normal file
View File

@ -0,0 +1 @@
FTP is no longer supported by Tahoe-LAFS. Please use the SFTP support instead.

1
newsfragments/3584.bugfix Normal file
View File

@ -0,0 +1 @@
SFTP public key auth likely works better, and SFTP in general was broken in the prerelease.

1
newsfragments/3587.minor Normal file
View File

@ -0,0 +1 @@

5
setup.py
View File

@ -63,12 +63,8 @@ install_requires = [
# version of cryptography will *really* be installed.
"cryptography >= 2.6",
# * We need Twisted 10.1.0 for the FTP frontend in order for
# Twisted's FTP server to support asynchronous close.
# * The SFTP frontend depends on Twisted 11.0.0 to fix the SSH server
# rekeying bug <https://twistedmatrix.com/trac/ticket/4395>
# * The FTP frontend depends on Twisted >= 11.1.0 for
# filepath.Permissions
# * The SFTP frontend and manhole depend on the conch extra. However, we
# can't explicitly declare that without an undesirable dependency on gmpy,
# as explained in ticket #2740.
@ -399,6 +395,7 @@ setup(name="tahoe-lafs", # also set in __init__.py
"html5lib",
"junitxml",
"tenacity",
"paramiko",
] + tor_requires + i2p_requires,
"tor": tor_requires,
"i2p": i2p_requires,

11
src/allmydata/blacklist.py
View File

@ -1,3 +1,14 @@
"""
Ported to Python 3.
"""
from __future__ import absolute_import
from __future__ import division
from __future__ import print_function
from __future__ import unicode_literals
from future.utils import PY2
if PY2:
from future.builtins import filter, map, zip, ascii, chr, hex, input, next, oct, open, pow, round, super, bytes, dict, list, object, range, str, max, min # noqa: F401
import os

19
src/allmydata/client.py
View File

@ -86,12 +86,6 @@ _client_config = configutil.ValidConfiguration(
"shares.total",
"storage.plugins",
),
"ftpd": (
"accounts.file",
"accounts.url",
"enabled",
"port",
),
"storage": (
"debug_discard",
"enabled",
@ -656,7 +650,6 @@ class _Client(node.Node, pollmixin.PollMixin):
raise ValueError("config error: helper is enabled, but tub "
"is not listening ('tub.port=' is empty)")
self.init_helper()
self.init_ftp_server()
self.init_sftp_server()
# If the node sees an exit_trigger file, it will poll every second to see
@ -1032,18 +1025,6 @@ class _Client(node.Node, pollmixin.PollMixin):
)
ws.setServiceParent(self)
def init_ftp_server(self):
if self.config.get_config("ftpd", "enabled", False, boolean=True):
accountfile = self.config.get_config("ftpd", "accounts.file", None)
if accountfile:
accountfile = self.config.get_config_path(accountfile)
accounturl = self.config.get_config("ftpd", "accounts.url", None)
ftp_portstr = self.config.get_config("ftpd", "port", "8021")
from allmydata.frontends import ftpd
s = ftpd.FTPServer(self, accountfile, accounturl, ftp_portstr)
s.setServiceParent(self)
def init_sftp_server(self):
if self.config.get_config("sftpd", "enabled", False, boolean=True):
accountfile = self.config.get_config("sftpd", "accounts.file", None)

38
src/allmydata/frontends/auth.py
View File

@ -4,8 +4,8 @@ from zope.interface import implementer
from twisted.web.client import getPage
from twisted.internet import defer
from twisted.cred import error, checkers, credentials
from twisted.conch import error as conch_error
from twisted.conch.ssh import keys
from twisted.conch.checkers import SSHPublicKeyChecker, InMemorySSHKeyDB
from allmydata.util import base32
from allmydata.util.fileutil import abspath_expanduser_unicode
@ -29,7 +29,7 @@ class AccountFileChecker(object):
def __init__(self, client, accountfile):
self.client = client
self.passwords = {}
self.pubkeys = {}
pubkeys = {}
self.rootcaps = {}
with open(abspath_expanduser_unicode(accountfile), "r") as f:
for line in f:
@ -40,12 +40,14 @@ class AccountFileChecker(object):
if passwd.startswith("ssh-"):
bits = rest.split()
keystring = " ".join([passwd] + bits[:-1])
key = keys.Key.fromString(keystring)
rootcap = bits[-1]
self.pubkeys[name] = keystring
pubkeys[name] = [key]
else:
self.passwords[name] = passwd
rootcap = rest
self.rootcaps[name] = rootcap
self._pubkeychecker = SSHPublicKeyChecker(InMemorySSHKeyDB(pubkeys))
def _avatarId(self, username):
return FTPAvatarID(username, self.rootcaps[username])
@ -57,11 +59,9 @@ class AccountFileChecker(object):
def requestAvatarId(self, creds):
if credentials.ISSHPrivateKey.providedBy(creds):
# Re-using twisted.conch.checkers.SSHPublicKeyChecker here, rather
# than re-implementing all of the ISSHPrivateKey checking logic,
# would be better. That would require Twisted 14.1.0 or newer,
# though.
return self._checkKey(creds)
d = defer.maybeDeferred(self._pubkeychecker.requestAvatarId, creds)
d.addCallback(self._avatarId)
return d
elif credentials.IUsernameHashedPassword.providedBy(creds):
return self._checkPassword(creds)
elif credentials.IUsernamePassword.providedBy(creds):
@ -86,28 +86,6 @@ class AccountFileChecker(object):
d.addCallback(self._cbPasswordMatch, str(creds.username))
return d
def _checkKey(self, creds):
"""
Determine whether some key-based credentials correctly authenticates a
user.
Returns a Deferred that fires with the username if so or with an
UnauthorizedLogin failure otherwise.
"""
# Is the public key indicated by the given credentials allowed to
# authenticate the username in those credentials?
if creds.blob == self.pubkeys.get(creds.username):
if creds.signature is None:
return defer.fail(conch_error.ValidPublicKey())
# Is the signature in the given credentials the correct
# signature for the data in those credentials?
key = keys.Key.fromString(creds.blob)
if key.verify(creds.signature, creds.sigData):
return defer.succeed(self._avatarId(creds.username))
return defer.fail(error.UnauthorizedLogin())
@implementer(checkers.ICredentialsChecker)
class AccountURLChecker(object):

350
src/allmydata/frontends/ftpd.py
View File

@ -1,350 +0,0 @@
"""
Ported to Python 3.
"""
from __future__ import absolute_import
from __future__ import division
from __future__ import print_function
from __future__ import unicode_literals
from future.utils import PY2
if PY2:
from future.builtins import filter, map, zip, ascii, chr, hex, input, next, oct, open, pow, round, super, bytes, dict, list, object, range, str, max, min # noqa: F401
from six import ensure_str
from zope.interface import implementer
from twisted.application import service, strports
from twisted.internet import defer
from twisted.internet.interfaces import IConsumer
from twisted.cred import portal
from twisted.python import filepath
from twisted.protocols import ftp
from allmydata.interfaces import IDirectoryNode, ExistingChildError, \
NoSuchChildError
from allmydata.immutable.upload import FileHandle
from allmydata.util.fileutil import EncryptedTemporaryFile
from allmydata.util.assertutil import precondition
@implementer(ftp.IReadFile)
class ReadFile(object):
def __init__(self, node):
self.node = node
def send(self, consumer):
d = self.node.read(consumer)
return d # when consumed
@implementer(IConsumer)
class FileWriter(object):
def registerProducer(self, producer, streaming):
if not streaming:
raise NotImplementedError("Non-streaming producer not supported.")
# we write the data to a temporary file, since Tahoe can't do
# streaming upload yet.
self.f = EncryptedTemporaryFile()
return None
def unregisterProducer(self):
# the upload actually happens in WriteFile.close()
pass
def write(self, data):
self.f.write(data)
@implementer(ftp.IWriteFile)
class WriteFile(object):
def __init__(self, parent, childname, convergence):
self.parent = parent
self.childname = childname
self.convergence = convergence
def receive(self):
self.c = FileWriter()
return defer.succeed(self.c)
def close(self):
u = FileHandle(self.c.f, self.convergence)
d = self.parent.add_file(self.childname, u)
return d
class NoParentError(Exception):
pass
# filepath.Permissions was added in Twisted-11.1.0, which we require. Twisted
# <15.0.0 expected an int, and only does '&' on it. Twisted >=15.0.0 expects
# a filepath.Permissions. This satisfies both.
class IntishPermissions(filepath.Permissions):
def __init__(self, statModeInt):
self._tahoe_statModeInt = statModeInt
filepath.Permissions.__init__(self, statModeInt)
def __and__(self, other):
return self._tahoe_statModeInt & other
@implementer(ftp.IFTPShell)
class Handler(object):
def __init__(self, client, rootnode, username, convergence):
self.client = client
self.root = rootnode
self.username = username
self.convergence = convergence
def makeDirectory(self, path):
d = self._get_root(path)
d.addCallback(lambda root_and_path:
self._get_or_create_directories(root_and_path[0], root_and_path[1]))
return d
def _get_or_create_directories(self, node, path):
if not IDirectoryNode.providedBy(node):
# unfortunately it is too late to provide the name of the
# blocking directory in the error message.
raise ftp.FileExistsError("cannot create directory because there "
"is a file in the way")
if not path:
return defer.succeed(node)
d = node.get(path[0])
def _maybe_create(f):
f.trap(NoSuchChildError)
return node.create_subdirectory(path[0])
d.addErrback(_maybe_create)
d.addCallback(self._get_or_create_directories, path[1:])
return d
def _get_parent(self, path):
# fire with (parentnode, childname)
path = [str(p) for p in path]
if not path:
raise NoParentError
childname = path[-1]
d = self._get_root(path)
def _got_root(root_and_path):
(root, path) = root_and_path
if not path:
raise NoParentError
return root.get_child_at_path(path[:-1])
d.addCallback(_got_root)
def _got_parent(parent):
return (parent, childname)
d.addCallback(_got_parent)
return d
def _remove_thing(self, path, must_be_directory=False, must_be_file=False):
d = defer.maybeDeferred(self._get_parent, path)
def _convert_error(f):
f.trap(NoParentError)
raise ftp.PermissionDeniedError("cannot delete root directory")
d.addErrback(_convert_error)
def _got_parent(parent_and_childname):
(parent, childname) = parent_and_childname
d = parent.get(childname)
def _got_child(child):
if must_be_directory and not IDirectoryNode.providedBy(child):
raise ftp.IsNotADirectoryError("rmdir called on a file")
if must_be_file and IDirectoryNode.providedBy(child):
raise ftp.IsADirectoryError("rmfile called on a directory")
return parent.delete(childname)
d.addCallback(_got_child)
d.addErrback(self._convert_error)
return d
d.addCallback(_got_parent)
return d
def removeDirectory(self, path):
return self._remove_thing(path, must_be_directory=True)
def removeFile(self, path):
return self._remove_thing(path, must_be_file=True)
def rename(self, fromPath, toPath):
# the target directory must already exist
d = self._get_parent(fromPath)
def _got_from_parent(fromparent_and_childname):
(fromparent, childname) = fromparent_and_childname
d = self._get_parent(toPath)
d.addCallback(lambda toparent_and_tochildname:
fromparent.move_child_to(childname,
toparent_and_tochildname[0], toparent_and_tochildname[1],
overwrite=False))
return d
d.addCallback(_got_from_parent)
d.addErrback(self._convert_error)
return d
def access(self, path):
# we allow access to everything that exists. We are required to raise
# an error for paths that don't exist: FTP clients (at least ncftp)
# uses this to decide whether to mkdir or not.
d = self._get_node_and_metadata_for_path(path)
d.addErrback(self._convert_error)
d.addCallback(lambda res: None)
return d
def _convert_error(self, f):
if f.check(NoSuchChildError):
childname = f.value.args[0].encode("utf-8")
msg = "'%s' doesn't exist" % childname
raise ftp.FileNotFoundError(msg)
if f.check(ExistingChildError):
msg = f.value.args[0].encode("utf-8")
raise ftp.FileExistsError(msg)
return f
def _get_root(self, path):
# return (root, remaining_path)
path = [str(p) for p in path]
if path and path[0] == "uri":
d = defer.maybeDeferred(self.client.create_node_from_uri,
str(path[1]))
d.addCallback(lambda root: (root, path[2:]))
else:
d = defer.succeed((self.root,path))
return d
def _get_node_and_metadata_for_path(self, path):
d = self._get_root(path)
def _got_root(root_and_path):
(root,path) = root_and_path
if path:
return root.get_child_and_metadata_at_path(path)
else:
return (root,{})
d.addCallback(_got_root)
return d
def _populate_row(self, keys, childnode_and_metadata):
(childnode, metadata) = childnode_and_metadata
values = []
isdir = bool(IDirectoryNode.providedBy(childnode))
for key in keys:
if key == "size":
if isdir:
value = 0
else:
value = childnode.get_size() or 0
elif key == "directory":
value = isdir
elif key == "permissions":
# Twisted-14.0.2 (and earlier) expected an int, and used it
# in a rendering function that did (mode & NUMBER).
# Twisted-15.0.0 expects a
# twisted.python.filepath.Permissions , and calls its
# .shorthand() method. This provides both.
value = IntishPermissions(0o600)
elif key == "hardlinks":
value = 1
elif key == "modified":
# follow sftpd convention (i.e. linkmotime in preference to mtime)
if "linkmotime" in metadata.get("tahoe", {}):
value = metadata["tahoe"]["linkmotime"]
else:
value = metadata.get("mtime", 0)
elif key == "owner":
value = self.username
elif key == "group":
value = self.username
else:
value = "??"
values.append(value)
return values
def stat(self, path, keys=()):
# for files only, I think
d = self._get_node_and_metadata_for_path(path)
def _render(node_and_metadata):
(node, metadata) = node_and_metadata
assert not IDirectoryNode.providedBy(node)
return self._populate_row(keys, (node,metadata))
d.addCallback(_render)
d.addErrback(self._convert_error)
return d
def list(self, path, keys=()):
# the interface claims that path is a list of unicodes, but in
# practice it is not
d = self._get_node_and_metadata_for_path(path)
def _list(node_and_metadata):
(node, metadata) = node_and_metadata
if IDirectoryNode.providedBy(node):
return node.list()
return { path[-1]: (node, metadata) } # need last-edge metadata
d.addCallback(_list)
def _render(children):
results = []
for (name, childnode) in list(children.items()):
# the interface claims that the result should have a unicode
# object as the name, but it fails unless you give it a
# bytestring
results.append( (name.encode("utf-8"),
self._populate_row(keys, childnode) ) )
return results
d.addCallback(_render)
d.addErrback(self._convert_error)
return d
def openForReading(self, path):
d = self._get_node_and_metadata_for_path(path)
d.addCallback(lambda node_and_metadata: ReadFile(node_and_metadata[0]))
d.addErrback(self._convert_error)
return d
def openForWriting(self, path):
path = [str(p) for p in path]
if not path:
raise ftp.PermissionDeniedError("cannot STOR to root directory")
childname = path[-1]
d = self._get_root(path)
def _got_root(root_and_path):
(root, path) = root_and_path
if not path:
raise ftp.PermissionDeniedError("cannot STOR to root directory")
return root.get_child_at_path(path[:-1])
d.addCallback(_got_root)
def _got_parent(parent):
return WriteFile(parent, childname, self.convergence)
d.addCallback(_got_parent)
return d
from allmydata.frontends.auth import AccountURLChecker, AccountFileChecker, NeedRootcapLookupScheme
@implementer(portal.IRealm)
class Dispatcher(object):
def __init__(self, client):
self.client = client
def requestAvatar(self, avatarID, mind, interface):
assert interface == ftp.IFTPShell
rootnode = self.client.create_node_from_uri(avatarID.rootcap)
convergence = self.client.convergence
s = Handler(self.client, rootnode, avatarID.username, convergence)
def logout(): pass
return (interface, s, None)
class FTPServer(service.MultiService):
def __init__(self, client, accountfile, accounturl, ftp_portstr):
precondition(isinstance(accountfile, (str, type(None))), accountfile)
service.MultiService.__init__(self)
r = Dispatcher(client)
p = portal.Portal(r)
if accountfile:
c = AccountFileChecker(self, accountfile)
p.registerChecker(c)
if accounturl:
c = AccountURLChecker(self, accounturl)
p.registerChecker(c)
if not accountfile and not accounturl:
# we could leave this anonymous, with just the /uri/CAP form
raise NeedRootcapLookupScheme("must provide some translation")
f = ftp.FTPFactory(p)
# strports requires a native string.
ftp_portstr = ensure_str(ftp_portstr)
s = strports.service(ftp_portstr, f)
s.setServiceParent(self)

2
src/allmydata/frontends/sftpd.py
View File

@ -2021,5 +2021,5 @@ class SFTPServer(service.MultiService):
f = SSHFactory()
f.portal = p
s = strports.service(sftp_portstr, f)
s = strports.service(six.ensure_str(sftp_portstr), f)
s.setServiceParent(self)

85
src/allmydata/test/test_client.py
View File

@ -51,7 +51,6 @@ from allmydata.nodemaker import (
NodeMaker,
)
from allmydata.node import OldConfigError, UnescapedHashError, create_node_dir
from allmydata.frontends.auth import NeedRootcapLookupScheme
from allmydata import client
from allmydata.storage_client import (
StorageClientConfig,
@ -424,88 +423,8 @@ class Basic(testutil.ReallyEqualMixin, unittest.TestCase):
expected = fileutil.abspath_expanduser_unicode(u"relative", abs_basedir)
self.failUnlessReallyEqual(w.staticdir, expected)
# TODO: also test config options for SFTP.
@defer.inlineCallbacks
def test_ftp_create(self):
"""
configuration for sftpd results in it being started
"""
root = FilePath(self.mktemp())
root.makedirs()
accounts = root.child(b"sftp-accounts")
accounts.touch()
data = FilePath(__file__).sibling(b"data")
privkey = data.child(b"openssh-rsa-2048.txt")
pubkey = data.child(b"openssh-rsa-2048.pub.txt")
basedir = u"client.Basic.test_ftp_create"
create_node_dir(basedir, "testing")
with open(os.path.join(basedir, "tahoe.cfg"), "w") as f:
f.write((
'[sftpd]\n'
'enabled = true\n'
'accounts.file = {}\n'
'host_pubkey_file = {}\n'
'host_privkey_file = {}\n'
).format(accounts.path, pubkey.path, privkey.path))
client_node = yield client.create_client(
basedir,
)
sftp = client_node.getServiceNamed("frontend:sftp")
self.assertIs(sftp.parent, client_node)
@defer.inlineCallbacks
def test_ftp_auth_keyfile(self):
"""
ftpd accounts.file is parsed properly
"""
basedir = u"client.Basic.test_ftp_auth_keyfile"
os.mkdir(basedir)
fileutil.write(os.path.join(basedir, "tahoe.cfg"),
(BASECONFIG +
"[ftpd]\n"
"enabled = true\n"
"port = tcp:0:interface=127.0.0.1\n"
"accounts.file = private/accounts\n"))
os.mkdir(os.path.join(basedir, "private"))
fileutil.write(os.path.join(basedir, "private", "accounts"), "\n")
c = yield client.create_client(basedir) # just make sure it can be instantiated
del c
@defer.inlineCallbacks
def test_ftp_auth_url(self):
"""
ftpd accounts.url is parsed properly
"""
basedir = u"client.Basic.test_ftp_auth_url"
os.mkdir(basedir)
fileutil.write(os.path.join(basedir, "tahoe.cfg"),
(BASECONFIG +
"[ftpd]\n"
"enabled = true\n"
"port = tcp:0:interface=127.0.0.1\n"
"accounts.url = http://0.0.0.0/\n"))
c = yield client.create_client(basedir) # just make sure it can be instantiated
del c
@defer.inlineCallbacks
def test_ftp_auth_no_accountfile_or_url(self):
"""
ftpd requires some way to look up accounts
"""
basedir = u"client.Basic.test_ftp_auth_no_accountfile_or_url"
os.mkdir(basedir)
fileutil.write(os.path.join(basedir, "tahoe.cfg"),
(BASECONFIG +
"[ftpd]\n"
"enabled = true\n"
"port = tcp:0:interface=127.0.0.1\n"))
with self.assertRaises(NeedRootcapLookupScheme):
yield client.create_client(basedir)
# TODO: also test config options for SFTP. See Git history for deleted FTP
# tests that could be used as basis for these tests.
@defer.inlineCallbacks
def _storage_dir_test(self, basedir, storage_path, expected_path):

117
src/allmydata/test/test_ftp.py
View File

@ -1,117 +0,0 @@
"""
Ported to Python 3.
"""
from __future__ import absolute_import
from __future__ import division
from __future__ import print_function
from __future__ import unicode_literals
from future.utils import PY2
if PY2:
from future.builtins import filter, map, zip, ascii, chr, hex, input, next, oct, open, pow, round, super, bytes, dict, list, object, range, str, max, min # noqa: F401
from twisted.trial import unittest
from allmydata.frontends import ftpd
from allmydata.immutable import upload
from allmydata.mutable import publish
from allmydata.test.no_network import GridTestMixin
from allmydata.test.common_util import ReallyEqualMixin
class Handler(GridTestMixin, ReallyEqualMixin, unittest.TestCase):
"""
This is a no-network unit test of ftpd.Handler and the abstractions
it uses.
"""
FALL_OF_BERLIN_WALL = 626644800
TURN_OF_MILLENIUM = 946684800
def _set_up(self, basedir, num_clients=1, num_servers=10):
self.basedir = "ftp/" + basedir
self.set_up_grid(num_clients=num_clients, num_servers=num_servers,
oneshare=True)
self.client = self.g.clients[0]
self.username = "alice"
self.convergence = ""
d = self.client.create_dirnode()
def _created_root(node):
self.root = node
self.root_uri = node.get_uri()
self.handler = ftpd.Handler(self.client, self.root, self.username,
self.convergence)
d.addCallback(_created_root)
return d
def _set_metadata(self, name, metadata):
"""Set metadata for `name', avoiding MetadataSetter's timestamp reset
behavior."""
def _modifier(old_contents, servermap, first_time):
children = self.root._unpack_contents(old_contents)
children[name] = (children[name][0], metadata)
return self.root._pack_contents(children)
return self.root._node.modify(_modifier)
def _set_up_tree(self):
# add immutable file at root
immutable = upload.Data(b"immutable file contents", None)
d = self.root.add_file(u"immutable", immutable)
# `mtime' and `linkmotime' both set
md_both = {'mtime': self.FALL_OF_BERLIN_WALL,
'tahoe': {'linkmotime': self.TURN_OF_MILLENIUM}}
d.addCallback(lambda _: self._set_metadata(u"immutable", md_both))
# add link to root from root
d.addCallback(lambda _: self.root.set_node(u"loop", self.root))
# `mtime' set, but no `linkmotime'
md_just_mtime = {'mtime': self.FALL_OF_BERLIN_WALL, 'tahoe': {}}
d.addCallback(lambda _: self._set_metadata(u"loop", md_just_mtime))
# add mutable file at root
mutable = publish.MutableData(b"mutable file contents")
d.addCallback(lambda _: self.client.create_mutable_file(mutable))
d.addCallback(lambda node: self.root.set_node(u"mutable", node))
# neither `mtime' nor `linkmotime' set
d.addCallback(lambda _: self._set_metadata(u"mutable", {}))
return d
def _compareDirLists(self, actual, expected):
actual_list = sorted(actual)
expected_list = sorted(expected)
self.failUnlessReallyEqual(len(actual_list), len(expected_list),
"%r is wrong length, expecting %r" % (
actual_list, expected_list))
for (a, b) in zip(actual_list, expected_list):
(name, meta) = a
(expected_name, expected_meta) = b
self.failUnlessReallyEqual(name, expected_name)
self.failUnlessReallyEqual(meta, expected_meta)
def test_list(self):
keys = ("size", "directory", "permissions", "hardlinks", "modified",
"owner", "group", "unexpected")
d = self._set_up("list")
d.addCallback(lambda _: self._set_up_tree())
d.addCallback(lambda _: self.handler.list("", keys=keys))
expected_root = [
(b'loop',
[0, True, ftpd.IntishPermissions(0o600), 1, self.FALL_OF_BERLIN_WALL, 'alice', 'alice', '??']),
(b'immutable',
[23, False, ftpd.IntishPermissions(0o600), 1, self.TURN_OF_MILLENIUM, 'alice', 'alice', '??']),
(b'mutable',
# timestamp should be 0 if no timestamp metadata is present
[0, False, ftpd.IntishPermissions(0o600), 1, 0, 'alice', 'alice', '??'])]
d.addCallback(lambda root: self._compareDirLists(root, expected_root))
return d

16
src/allmydata/test/web/test_webish.py
View File

@ -1,6 +1,16 @@
"""
Tests for ``allmydata.webish``.
Ported to Python 3.
"""
from __future__ import absolute_import
from __future__ import division
from __future__ import print_function
from __future__ import unicode_literals
from future.utils import PY2
if PY2:
from future.builtins import filter, map, zip, ascii, chr, hex, input, next, oct, open, pow, round, super, bytes, dict, list, object, range, str, max, min # noqa: F401
from uuid import (
uuid4,
@ -96,7 +106,7 @@ class TahoeLAFSRequestTests(SyncTestCase):
])
self._fields_test(
b"POST",
{b"content-type": b"multipart/form-data; boundary={}".format(boundary)},
{b"content-type": b"multipart/form-data; boundary=" + bytes(boundary, 'ascii')},
form_data.encode("ascii"),
AfterPreprocessing(
lambda fs: {
@ -105,8 +115,8 @@ class TahoeLAFSRequestTests(SyncTestCase):
in fs.keys()
},
Equals({
b"foo": b"bar",
b"baz": b"some file contents",
"foo": "bar",
"baz": b"some file contents",
}),
),
)

7
src/allmydata/util/_python3.py
View File

@ -27,6 +27,7 @@ PORTED_MODULES = [
"allmydata.__main__",
"allmydata._auto_deps",
"allmydata._monkeypatch",
"allmydata.blacklist",
"allmydata.codec",
"allmydata.crypto",
"allmydata.crypto.aes",
@ -36,7 +37,6 @@ PORTED_MODULES = [
"allmydata.crypto.util",
"allmydata.deep_stats",
"allmydata.dirnode",
"allmydata.frontends.ftpd",
"allmydata.frontends.sftpd",
"allmydata.hashtree",
"allmydata.immutable.checker",
@ -114,6 +114,7 @@ PORTED_MODULES = [
"allmydata.util.spans",
"allmydata.util.statistics",
"allmydata.util.time_format",
"allmydata.webish",
]
PORTED_TEST_MODULES = [
@ -150,7 +151,6 @@ PORTED_TEST_MODULES = [
"allmydata.test.test_encode",
"allmydata.test.test_encodingutil",
"allmydata.test.test_filenode",
"allmydata.test.test_ftp",
"allmydata.test.test_happiness",
"allmydata.test.test_hashtree",
"allmydata.test.test_hashutil",
@ -188,6 +188,7 @@ PORTED_TEST_MODULES = [
"allmydata.test.test_util",
"allmydata.test.web.test_common",
"allmydata.test.web.test_grid",
"allmydata.test.web.test_util",
"allmydata.test.web.test_status",
"allmydata.test.web.test_util",
"allmydata.test.web.test_webish",
]

36
src/allmydata/webish.py
View File

@ -1,3 +1,15 @@
"""
Ported to Python 3.
"""
from __future__ import absolute_import
from __future__ import division
from __future__ import print_function
from __future__ import unicode_literals
from future.utils import PY2
if PY2:
from future.builtins import filter, map, zip, ascii, chr, hex, input, next, oct, open, pow, round, super, bytes, dict, list, object, range, str, max, min # noqa: F401
from six import ensure_str
import re, time, tempfile
@ -65,18 +77,24 @@ class TahoeLAFSRequest(Request, object):
self.path, argstring = x
self.args = parse_qs(argstring, 1)
if self.method == 'POST':
if self.method == b'POST':
# We use FieldStorage here because it performs better than
# cgi.parse_multipart(self.content, pdict) which is what
# twisted.web.http.Request uses.
self.fields = FieldStorage(
self.content,
{
name.lower(): value[-1]
for (name, value)
in self.requestHeaders.getAllRawHeaders()
},
environ={'REQUEST_METHOD': 'POST'})
headers = {
ensure_str(name.lower()): ensure_str(value[-1])
for (name, value)
in self.requestHeaders.getAllRawHeaders()
}
if 'content-length' not in headers:
# Python 3's cgi module would really, really like us to set Content-Length.
self.content.seek(0, 2)
headers['content-length'] = str(self.content.tell())
self.content.seek(0)
self.fields = FieldStorage(self.content, headers, environ={'REQUEST_METHOD': 'POST'})
self.content.seek(0)
self._tahoeLAFSSecurityPolicy()