mirror of
https://github.com/tahoe-lafs/tahoe-lafs.git
synced 2025-02-27 11:46:44 +00:00
extirpate all references the "queen" and "metatracker"
This is a potentially disruptive and potentially ugly change to the code base, because I renamed the object that serves in both roles from "Queen" to "IntroducerAndVdrive", which is a bit of an ugly name. However, I think that clarity is important enough in this release to make this change. All unit tests pass. I'm now darcs recording this patch in order to pull it to other machines for more testing.
This commit is contained in:
parent
ced1d8189b
commit
d19d1058e0
@ -158,7 +158,7 @@ clean: clean-zfec clean-Crypto clean-foolscap
|
|||||||
rm -rf instdir
|
rm -rf instdir
|
||||||
|
|
||||||
create_dirs:
|
create_dirs:
|
||||||
mkdir -p queen-basedir
|
mkdir -p introducer_and_vdrive-basedir
|
||||||
mkdir -p client-basedir
|
mkdir -p client-basedir
|
||||||
mkdir -p client-basedir2
|
mkdir -p client-basedir2
|
||||||
mkdir -p client-basedir/storage
|
mkdir -p client-basedir/storage
|
||||||
|
14
README
14
README
@ -157,14 +157,14 @@ RUNNING:
|
|||||||
directory, inside of which you can add files to configure and control the
|
directory, inside of which you can add files to configure and control the
|
||||||
node. Nodes also read and write files within that directory.
|
node. Nodes also read and write files within that directory.
|
||||||
|
|
||||||
A mesh consists of a single central 'introducer' node and a large number of
|
A mesh consists of a single central 'introducer and vdrive' node and a large
|
||||||
'client' nodes. If you are joining an existing mesh, the introducer node
|
number of 'client' nodes. If you are joining an existing mesh, the
|
||||||
will already be running, and you'll just need to create a client node. If
|
introducer-and-vdrive node will already be running, and you'll just need to
|
||||||
you're creating a brand new mesh, you'll need to create both an introducer
|
create a client node. If you're creating a brand new mesh, you'll need to
|
||||||
and a client (and then invite other people to create their own client nodes
|
create both an introducer-and-vdrive and a client (and then invite other
|
||||||
and join your mesh).
|
people to create their own client nodes and join your mesh).
|
||||||
|
|
||||||
The introducer node is constructed by running 'allmydata-tahoe
|
The introducer (-and-vdrive) node is constructed by running 'allmydata-tahoe
|
||||||
create-introducer --basedir $HERE'. Once constructed, you can start the
|
create-introducer --basedir $HERE'. Once constructed, you can start the
|
||||||
introducer by running 'allmydata-tahoe start --basedir $HERE' (or, if you
|
introducer by running 'allmydata-tahoe start --basedir $HERE' (or, if you
|
||||||
are already in the introducer's base directory, just type 'allmydata-tahoe
|
are already in the introducer's base directory, just type 'allmydata-tahoe
|
||||||
|
@ -6,11 +6,11 @@ OVERVIEW
|
|||||||
The high-level view of this system consists of three layers: the mesh, the
|
The high-level view of this system consists of three layers: the mesh, the
|
||||||
virtual drive, and the application that sits on top.
|
virtual drive, and the application that sits on top.
|
||||||
|
|
||||||
The lowest layer is the "mesh" or "cloud", basically a DHT (Distributed Hash
|
The lowest layer is the "mesh", basically a DHT (Distributed Hash Table)
|
||||||
Table) which maps URIs to data. The URIs are relatively short ascii strings
|
which maps URIs to data. The URIs are relatively short ascii strings
|
||||||
(currently about 140 bytes), and each is used as references to an immutable
|
(currently about 140 bytes), and each is used as references to an immutable
|
||||||
arbitrary-length sequence of data bytes. This data is distributed around the
|
arbitrary-length sequence of data bytes. This data is distributed around the
|
||||||
cloud in a large number of nodes, such that a statistically unlikely number
|
mesh in a large number of nodes, such that a statistically unlikely number
|
||||||
of nodes would have to be unavailable for the data to become unavailable.
|
of nodes would have to be unavailable for the data to become unavailable.
|
||||||
|
|
||||||
The middle layer is the virtual drive: a tree-shaped data structure in which
|
The middle layer is the virtual drive: a tree-shaped data structure in which
|
||||||
@ -32,14 +32,13 @@ actual code present in the current release. Please take a look at roadmap.txt
|
|||||||
to get an idea of how much of this has been implemented so far.
|
to get an idea of how much of this has been implemented so far.
|
||||||
|
|
||||||
|
|
||||||
THE BIG CLOUD OF PEERS
|
THE BIG MESH OF PEERS
|
||||||
|
|
||||||
Underlying the mesh/cloud is a large collection of peer nodes. These are
|
Underlying the mesh is a large collection of peer nodes. These are processes
|
||||||
processes running on a wide variety of computers, all of which know about
|
running on a wide variety of computers, all of which know about each other in
|
||||||
each other in some way or another. They establish TCP connections to one
|
some way or another. They establish TCP connections to one another using
|
||||||
another using Foolscap, an encrypted+authenticated remote message passing
|
Foolscap, an encrypted+authenticated remote message passing library (using
|
||||||
library (using TLS connections and self-authenticating identifiers called
|
TLS connections and self-authenticating identifiers called "FURLs").
|
||||||
"FURLs").
|
|
||||||
|
|
||||||
Each peer offers certain services to the others. The primary service is the
|
Each peer offers certain services to the others. The primary service is the
|
||||||
StorageServer, which offers to hold data for a limited period of time (a
|
StorageServer, which offers to hold data for a limited period of time (a
|
||||||
@ -64,12 +63,12 @@ safety measure.
|
|||||||
In this release, peers learn about each other through the "introducer". Each
|
In this release, peers learn about each other through the "introducer". Each
|
||||||
peer connects to this central introducer at startup, and receives a list of
|
peer connects to this central introducer at startup, and receives a list of
|
||||||
all other peers from it. Each peer then connects to all other peers, creating
|
all other peers from it. Each peer then connects to all other peers, creating
|
||||||
a full-mesh topology. Future versions will reduce the number of connections
|
a fully-connected topology. Future versions will reduce the number of
|
||||||
considerably, to enable the mesh to scale to larger sizes: the design target
|
connections considerably, to enable the mesh to scale to larger sizes: the
|
||||||
is one million nodes. In addition, future versions will offer relay and
|
design target is one million nodes. In addition, future versions will offer
|
||||||
NAT-traversal services to allow nodes without full internet connectivity to
|
relay and NAT-traversal services to allow nodes without full internet
|
||||||
participate. In the current release, only one node may be behind a NAT box
|
connectivity to participate. In the current release, only one node may be
|
||||||
and still permit the cloud to achieve full-mesh connectivity.
|
behind a NAT box and still permit the mesh to achieve full connectivity.
|
||||||
|
|
||||||
|
|
||||||
FILE ENCODING
|
FILE ENCODING
|
||||||
@ -166,7 +165,7 @@ which ones? The "peer selection" algorithm is used to make this choice.
|
|||||||
In the current version, the verifierid is used to consistently-permute the
|
In the current version, the verifierid is used to consistently-permute the
|
||||||
set of all peers (by sorting the peers by HASH(verifierid+peerid)). Each file
|
set of all peers (by sorting the peers by HASH(verifierid+peerid)). Each file
|
||||||
gets a different permutation, which (on average) will evenly distribute
|
gets a different permutation, which (on average) will evenly distribute
|
||||||
shares among the cloud and avoid hotspots.
|
shares among the mesh and avoid hotspots.
|
||||||
|
|
||||||
This permutation places the peers around a 2^256-sized ring, like the rim of
|
This permutation places the peers around a 2^256-sized ring, like the rim of
|
||||||
a big clock. The 100-or-so shares are then placed around the same ring (at 0,
|
a big clock. The 100-or-so shares are then placed around the same ring (at 0,
|
||||||
|
@ -23,11 +23,11 @@ Within src/allmydata/ :
|
|||||||
node.py: the base Node, which handles connection establishment and
|
node.py: the base Node, which handles connection establishment and
|
||||||
application startup
|
application startup
|
||||||
|
|
||||||
client.py, queen.py: two specialized subclasses of Node, for users
|
client.py, introducer_and_vdrive.py: two specialized subclasses of Node, for users
|
||||||
and the central introducer/vdrive handler, respectively
|
and the central introducer/vdrive handler, respectively
|
||||||
|
|
||||||
introducer.py: node introduction handlers, client is used by client.py,
|
introducer.py: node introduction handlers, client is used by client.py,
|
||||||
server is used by queen.py
|
server is used by introducer_and_vdrive.py
|
||||||
|
|
||||||
storageserver.py: provides storage services to other nodes
|
storageserver.py: provides storage services to other nodes
|
||||||
|
|
||||||
@ -40,7 +40,7 @@ Within src/allmydata/ :
|
|||||||
|
|
||||||
filetable.py, vdrive.py: implements the current one-global-vdrive layer,
|
filetable.py, vdrive.py: implements the current one-global-vdrive layer,
|
||||||
part runs on client nodes, part runs on the
|
part runs on client nodes, part runs on the
|
||||||
central vdrive handler (aka the 'queen')
|
central vdrive handler
|
||||||
|
|
||||||
webish.py, web/*.xhtml: provides the web frontend, using a Nevow server
|
webish.py, web/*.xhtml: provides the web frontend, using a Nevow server
|
||||||
|
|
||||||
@ -57,11 +57,11 @@ Within src/allmydata/ :
|
|||||||
test/*.py: unit tests
|
test/*.py: unit tests
|
||||||
|
|
||||||
|
|
||||||
Both the client and the central-queen node runs as a tree of
|
Both the client and the central introducer-and-vdrive node runs as a tree of
|
||||||
(twisted.application.service) Service instances. The Foolscap "Tub" is one of
|
(twisted.application.service) Service instances. The Foolscap "Tub" is one of
|
||||||
these. Client nodes have an Uploader service and a Downloader service that
|
these. Client nodes have an Uploader service and a Downloader service that turn
|
||||||
turn data into URIs and back again. They also have a VDrive service which
|
data into URIs and back again. They also have a VDrive service which provides
|
||||||
provides access to the single global shared filesystem.
|
access to the single global shared filesystem.
|
||||||
|
|
||||||
The Uploader is given an "upload source" (which could be an open filehandle,
|
The Uploader is given an "upload source" (which could be an open filehandle,
|
||||||
a filename on local disk, or even a string), and returns a Deferred that
|
a filename on local disk, or even a string), and returns a Deferred that
|
||||||
|
@ -144,7 +144,7 @@ Properties of this approach:
|
|||||||
|
|
||||||
each node must maintain open (keep-alived) connections to something like
|
each node must maintain open (keep-alived) connections to something like
|
||||||
2*log2(N) peers. In tahoe2, this number is 0 (well, probably 1 for the
|
2*log2(N) peers. In tahoe2, this number is 0 (well, probably 1 for the
|
||||||
queen).
|
introducer).
|
||||||
|
|
||||||
during upload, each node must actively use 100 connections to a random set
|
during upload, each node must actively use 100 connections to a random set
|
||||||
of peers to push data (just like tahoe2).
|
of peers to push data (just like tahoe2).
|
||||||
|
@ -16,9 +16,9 @@ def randomid():
|
|||||||
return os.urandom(20)
|
return os.urandom(20)
|
||||||
|
|
||||||
class Node:
|
class Node:
|
||||||
def __init__(self, nid, queen, simulator):
|
def __init__(self, nid, introducer_and_vdrive, simulator):
|
||||||
self.nid = nid
|
self.nid = nid
|
||||||
self.queen = queen
|
self.introducer_and_vdrive = introducer_and_vdrive
|
||||||
self.simulator = simulator
|
self.simulator = simulator
|
||||||
self.shares = {}
|
self.shares = {}
|
||||||
self.capacity = random.randrange(1000)
|
self.capacity = random.randrange(1000)
|
||||||
@ -27,7 +27,7 @@ class Node:
|
|||||||
|
|
||||||
def permute_peers(self, fileid):
|
def permute_peers(self, fileid):
|
||||||
permuted = [(sha(fileid+n.nid),n)
|
permuted = [(sha(fileid+n.nid),n)
|
||||||
for n in self.queen.get_all_nodes()]
|
for n in self.introducer_and_vdrive.get_all_nodes()]
|
||||||
permuted.sort()
|
permuted.sort()
|
||||||
return permuted
|
return permuted
|
||||||
|
|
||||||
@ -50,7 +50,7 @@ class Node:
|
|||||||
node.delete_share(fileid)
|
node.delete_share(fileid)
|
||||||
return False
|
return False
|
||||||
self.files.append((fileid, numshares))
|
self.files.append((fileid, numshares))
|
||||||
self.queen.please_preserve(fileid, size, tried, last_givento)
|
self.introducer_and_vdrive.please_preserve(fileid, size, tried, last_givento)
|
||||||
return (True, tried)
|
return (True, tried)
|
||||||
|
|
||||||
def accept_share(self, fileid, sharesize):
|
def accept_share(self, fileid, sharesize):
|
||||||
@ -111,10 +111,10 @@ class Node:
|
|||||||
which = random.choice(self.files)
|
which = random.choice(self.files)
|
||||||
self.files.remove(which)
|
self.files.remove(which)
|
||||||
fileid,numshares = which
|
fileid,numshares = which
|
||||||
self.queen.delete(fileid)
|
self.introducer_and_vdrive.delete(fileid)
|
||||||
return True
|
return True
|
||||||
|
|
||||||
class Queen:
|
class IntroducerAndVdrive:
|
||||||
def __init__(self, simulator):
|
def __init__(self, simulator):
|
||||||
self.living_files = {}
|
self.living_files = {}
|
||||||
self.utilization = 0 # total size of all active files
|
self.utilization = 0 # total size of all active files
|
||||||
@ -169,7 +169,7 @@ class Simulator:
|
|||||||
self.rrd = RRD("/tmp/utilization.rrd", ds=[ds], rra=[rra], start=self.time)
|
self.rrd = RRD("/tmp/utilization.rrd", ds=[ds], rra=[rra], start=self.time)
|
||||||
self.rrd.create()
|
self.rrd.create()
|
||||||
|
|
||||||
self.queen = q = Queen(self)
|
self.introducer_and_vdrive = q = IntroducerAndVdrive(self)
|
||||||
self.all_nodes = [Node(randomid(), q, self)
|
self.all_nodes = [Node(randomid(), q, self)
|
||||||
for i in range(self.NUM_NODES)]
|
for i in range(self.NUM_NODES)]
|
||||||
q.all_nodes = self.all_nodes
|
q.all_nodes = self.all_nodes
|
||||||
@ -267,7 +267,7 @@ class Simulator:
|
|||||||
avg_tried = "NONE"
|
avg_tried = "NONE"
|
||||||
else:
|
else:
|
||||||
avg_tried = sum(self.published_files) / len(self.published_files)
|
avg_tried = sum(self.published_files) / len(self.published_files)
|
||||||
print time, etype, self.added_data, self.failed_files, self.lost_data_bytes, avg_tried, len(self.queen.living_files), self.queen.utilization
|
print time, etype, self.added_data, self.failed_files, self.lost_data_bytes, avg_tried, len(self.introducer_and_vdrive.living_files), self.introducer_and_vdrive.utilization
|
||||||
|
|
||||||
global s
|
global s
|
||||||
s = None
|
s = None
|
||||||
|
@ -40,7 +40,7 @@ there, with download links, and forms to upload new files.
|
|||||||
|
|
||||||
Other ways to access the filesystem are planned, as well as other structures
|
Other ways to access the filesystem are planned, as well as other structures
|
||||||
than the single globally-shared namespace implemented by this release: please
|
than the single globally-shared namespace implemented by this release: please
|
||||||
see the roadmap.txt for some rough details.
|
see the roadmap.txt [5] for some rough details.
|
||||||
|
|
||||||
|
|
||||||
HACKING AND COMMUNITY
|
HACKING AND COMMUNITY
|
||||||
|
21
roadmap.txt
21
roadmap.txt
@ -2,8 +2,8 @@
|
|||||||
['*' means complete]
|
['*' means complete]
|
||||||
|
|
||||||
Connection Management:
|
Connection Management:
|
||||||
*v1: foolscap, no relay, live=connected-to-queen, broadcast updates, full mesh
|
*v1: foolscap, no relay, live=connected-to-introducer, broadcast updates, fully connected topology
|
||||||
v2: live != connected-to-queen, connect on demand
|
v2: live != connected-to-introducer, connect on demand
|
||||||
v3: relay?
|
v3: relay?
|
||||||
|
|
||||||
File Encoding:
|
File Encoding:
|
||||||
@ -39,10 +39,10 @@ Download Peer Selection:
|
|||||||
v3: denver airport?
|
v3: denver airport?
|
||||||
|
|
||||||
Filetable Maintenance:
|
Filetable Maintenance:
|
||||||
*v1: queen-based tree of MutableDirectoryNodes, persisted to queen's disk
|
*v1: vdrive-based tree of MutableDirectoryNodes, persisted to vdrive's disk
|
||||||
no accounts
|
no accounts
|
||||||
v2: move tree to client side, serialize to a file, upload,
|
v2: move tree to client side, serialize to a file, upload,
|
||||||
queen.set_filetable_uri (still no accounts, just one global tree)
|
vdrive.set_filetable_uri (still no accounts, just one global tree)
|
||||||
v3: break world up into accounts, separate mutable spaces. Maybe
|
v3: break world up into accounts, separate mutable spaces. Maybe
|
||||||
implement SSKs
|
implement SSKs
|
||||||
v4: filetree
|
v4: filetree
|
||||||
@ -68,18 +68,13 @@ UI:
|
|||||||
v4: FUSE
|
v4: FUSE
|
||||||
|
|
||||||
Operations/Deployment/Doc/Free Software/Community:
|
Operations/Deployment/Doc/Free Software/Community:
|
||||||
- tweak licence as per zfec's recent tweak
|
- move this file into the wiki ?
|
||||||
- move this file into the wiki
|
|
||||||
- Windows port and testing
|
|
||||||
- Trac instance
|
|
||||||
- extirpate all references to "queen" and "metatracker"
|
|
||||||
- set up public trac, buildbot reports, mailing list, download page
|
|
||||||
|
|
||||||
back pocket ideas:
|
back pocket ideas:
|
||||||
when nodes are unable to reach storage servers, make a note of it, inform
|
when nodes are unable to reach storage servers, make a note of it, inform
|
||||||
queen eventually. queen then puts server under observation or otherwise
|
verifier/checker eventually. verifier/checker then puts server under
|
||||||
looks for differences between their self-reported availability and the
|
observation or otherwise looks for differences between their self-reported
|
||||||
experiences of others
|
availability and the experiences of others
|
||||||
|
|
||||||
store filetable URI in the first 10 peers that appear after your own nodeid
|
store filetable URI in the first 10 peers that appear after your own nodeid
|
||||||
each entry has a sequence number, maybe a timestamp
|
each entry has a sequence number, maybe a timestamp
|
||||||
|
@ -106,8 +106,8 @@ class ISubTree(Interface):
|
|||||||
|
|
||||||
Each subtree's populate_from_node() method is expected to use the
|
Each subtree's populate_from_node() method is expected to use the
|
||||||
downloader to obtain a file with the subtree's serialized contents
|
downloader to obtain a file with the subtree's serialized contents
|
||||||
(probably by pulling data from some source, like the mesh, the queen,
|
(probably by pulling data from some source, like the mesh, the vdrive
|
||||||
an HTTP server, or somewhere on the local filesystem), then
|
server, an HTTP server, or somewhere on the local filesystem), then
|
||||||
unserialize them and populate the subtree's state.
|
unserialize them and populate the subtree's state.
|
||||||
|
|
||||||
Return a Deferred that will fire (with self) when this subtree is
|
Return a Deferred that will fire (with self) when this subtree is
|
||||||
@ -250,8 +250,8 @@ class ISubTreeMaker(Interface):
|
|||||||
I accept an INode-providing specification of a subtree, and return a
|
I accept an INode-providing specification of a subtree, and return a
|
||||||
Deferred that fires with an ISubTree-providing instance. I will
|
Deferred that fires with an ISubTree-providing instance. I will
|
||||||
perform network IO and download the serialized data that the INode
|
perform network IO and download the serialized data that the INode
|
||||||
references, if necessary, or ask the queen (or other provider) for a
|
references, if necessary, or ask the vdrive server (or other provider)
|
||||||
pointer, or read it from local disk.
|
for a pointer, or read it from local disk.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
|
|
||||||
|
@ -13,9 +13,9 @@ all_node_types = [
|
|||||||
file.CHKFileNode,
|
file.CHKFileNode,
|
||||||
file.SSKFileNode,
|
file.SSKFileNode,
|
||||||
redirect.LocalFileRedirectionNode,
|
redirect.LocalFileRedirectionNode,
|
||||||
redirect.QueenRedirectionNode,
|
redirect.VdriveRedirectionNode,
|
||||||
redirect.HTTPRedirectionNode,
|
redirect.HTTPRedirectionNode,
|
||||||
redirect.QueenOrLocalFileRedirectionNode,
|
redirect.VdriveOrLocalFileRedirectionNode,
|
||||||
]
|
]
|
||||||
|
|
||||||
class NodeMaker(object):
|
class NodeMaker(object):
|
||||||
|
@ -102,24 +102,24 @@ class LocalFileRedirection(_BaseRedirection):
|
|||||||
self._update()
|
self._update()
|
||||||
return None
|
return None
|
||||||
|
|
||||||
class QueenRedirectionNode(LocalFileRedirectionNode):
|
class VdriveRedirectionNode(LocalFileRedirectionNode):
|
||||||
prefix = "QueenRedirection"
|
prefix = "VdriveRedirection"
|
||||||
|
|
||||||
class QueenRedirection(_BaseRedirection):
|
class VdriveRedirection(_BaseRedirection):
|
||||||
node_class = QueenRedirectionNode
|
node_class = VdriveRedirectionNode
|
||||||
|
|
||||||
def new(self, handle):
|
def new(self, handle):
|
||||||
self.handle = handle
|
self.handle = handle
|
||||||
return self
|
return self
|
||||||
|
|
||||||
def populate_from_node(self, node, parent_is_mutable, node_maker, downloader):
|
def populate_from_node(self, node, parent_is_mutable, node_maker, downloader):
|
||||||
# this specifies a handle for which the Queen maintains a serialized
|
# this specifies a handle for which the Vdrive maintains a serialized
|
||||||
# subtree specification.
|
# subtree specification.
|
||||||
assert isinstance(node, QueenRedirectionNode)
|
assert isinstance(node, VdriveRedirectionNode)
|
||||||
self.handle = node.handle
|
self.handle = node.handle
|
||||||
|
|
||||||
# TODO: queen?
|
# TODO: vdrive?
|
||||||
d = self._queen.callRemote("lookup_handle", self.handle)
|
d = self._vdrive.callRemote("lookup_handle", self.handle)
|
||||||
d.addCallback(self._populate_from_data, node_maker)
|
d.addCallback(self._populate_from_data, node_maker)
|
||||||
return d
|
return d
|
||||||
|
|
||||||
@ -127,30 +127,30 @@ class QueenRedirection(_BaseRedirection):
|
|||||||
return True # TODO: maybe, maybe not
|
return True # TODO: maybe, maybe not
|
||||||
|
|
||||||
def create_node_now(self):
|
def create_node_now(self):
|
||||||
return QueenRedirectionNode().new(self.handle)
|
return VdriveRedirectionNode().new(self.handle)
|
||||||
|
|
||||||
def update_now(self, uploader):
|
def update_now(self, uploader):
|
||||||
f = StringIO()
|
f = StringIO()
|
||||||
self.serialize_subtree_to_file(f)
|
self.serialize_subtree_to_file(f)
|
||||||
d = self._queen.callRemote("set_handle", self.handle, f.getvalue())
|
d = self._vdrive.callRemote("set_handle", self.handle, f.getvalue())
|
||||||
def _done(res):
|
def _done(res):
|
||||||
return self.create_node_now()
|
return self.create_node_now()
|
||||||
d.addCallback(_done)
|
d.addCallback(_done)
|
||||||
return d
|
return d
|
||||||
|
|
||||||
def update(self, workqueue):
|
def update(self, workqueue):
|
||||||
f, filename = workqueue.create_tempfile(".toqueen")
|
f, filename = workqueue.create_tempfile(".tovdrive")
|
||||||
self.serialize_subtree_to_file(f)
|
self.serialize_subtree_to_file(f)
|
||||||
f.close()
|
f.close()
|
||||||
workqueue.add_queen_update_handle(self.handle, filename)
|
workqueue.add_vdrive_update_handle(self.handle, filename)
|
||||||
workqueue.add_delete_tempfile(filename)
|
workqueue.add_delete_tempfile(filename)
|
||||||
return None
|
return None
|
||||||
|
|
||||||
class QueenOrLocalFileRedirectionNode(LocalFileRedirectionNode):
|
class VdriveOrLocalFileRedirectionNode(LocalFileRedirectionNode):
|
||||||
prefix = "QueenOrLocalFileRedirection"
|
prefix = "VdriveOrLocalFileRedirection"
|
||||||
|
|
||||||
class QueenOrLocalFileRedirection(_BaseRedirection):
|
class VdriveOrLocalFileRedirection(_BaseRedirection):
|
||||||
node_class = QueenOrLocalFileRedirectionNode
|
node_class = VdriveOrLocalFileRedirectionNode
|
||||||
|
|
||||||
def new(self, handle, child_node):
|
def new(self, handle, child_node):
|
||||||
self.handle = handle
|
self.handle = handle
|
||||||
@ -161,9 +161,9 @@ class QueenOrLocalFileRedirection(_BaseRedirection):
|
|||||||
|
|
||||||
def populate_from_node(self, node, parent_is_mutable, node_maker, downloader):
|
def populate_from_node(self, node, parent_is_mutable, node_maker, downloader):
|
||||||
# there is a local file which contains a bencoded serialized
|
# there is a local file which contains a bencoded serialized
|
||||||
# subtree specification. The queen also has a copy. Whomever has
|
# subtree specification. The vdrive also has a copy. Whomever has
|
||||||
# the higher version number wins.
|
# the higher version number wins.
|
||||||
assert isinstance(node, QueenOrLocalFileRedirectionNode)
|
assert isinstance(node, VdriveOrLocalFileRedirectionNode)
|
||||||
self.filename = self.handle = node.handle
|
self.filename = self.handle = node.handle
|
||||||
|
|
||||||
f = open(self.filename, "rb")
|
f = open(self.filename, "rb")
|
||||||
@ -171,19 +171,19 @@ class QueenOrLocalFileRedirection(_BaseRedirection):
|
|||||||
local_version_and_data = f.read()
|
local_version_and_data = f.read()
|
||||||
f.close()
|
f.close()
|
||||||
|
|
||||||
# TODO: queen?
|
# TODO: vdrive?
|
||||||
# TODO: pubsub so we can cache the queen's results
|
# TODO: pubsub so we can cache the vdrive's results
|
||||||
d = self._queen.callRemote("lookup_handle", self.handle)
|
d = self._vdrive.callRemote("lookup_handle", self.handle)
|
||||||
d.addCallback(self._choose_winner, local_version_and_data)
|
d.addCallback(self._choose_winner, local_version_and_data)
|
||||||
d.addCallback(self._populate_from_data, node_maker)
|
d.addCallback(self._populate_from_data, node_maker)
|
||||||
return d
|
return d
|
||||||
|
|
||||||
def _choose_winner(self, queen_version_and_data, local_version_and_data):
|
def _choose_winner(self, vdrive_version_and_data, local_version_and_data):
|
||||||
queen_version, queen_data = bencode.bdecode(queen_version_and_data)
|
vdrive_version, vdrive_data = bencode.bdecode(vdrive_version_and_data)
|
||||||
local_version, local_data = bencode.bdecode(local_version_and_data)
|
local_version, local_data = bencode.bdecode(local_version_and_data)
|
||||||
if queen_version > local_version:
|
if vdrive_version > local_version:
|
||||||
data = queen_data
|
data = vdrive_data
|
||||||
self.version = queen_version
|
self.version = vdrive_version
|
||||||
else:
|
else:
|
||||||
data = local_data
|
data = local_data
|
||||||
self.version = local_version
|
self.version = local_version
|
||||||
@ -194,7 +194,7 @@ class QueenOrLocalFileRedirection(_BaseRedirection):
|
|||||||
return True
|
return True
|
||||||
|
|
||||||
def create_node_now(self):
|
def create_node_now(self):
|
||||||
return QueenOrLocalFileRedirectionNode().new(self.handle)
|
return VdriveOrLocalFileRedirectionNode().new(self.handle)
|
||||||
|
|
||||||
def _update(self):
|
def _update(self):
|
||||||
self.version += 1
|
self.version += 1
|
||||||
@ -209,7 +209,7 @@ class QueenOrLocalFileRedirection(_BaseRedirection):
|
|||||||
f.write(version_and_data)
|
f.write(version_and_data)
|
||||||
f.close()
|
f.close()
|
||||||
|
|
||||||
d = self._queen.callRemote("set_handle", self.handle, version_and_data)
|
d = self._vdrive.callRemote("set_handle", self.handle, version_and_data)
|
||||||
def _done(res):
|
def _done(res):
|
||||||
return self.create_node_now()
|
return self.create_node_now()
|
||||||
d.addCallback(_done)
|
d.addCallback(_done)
|
||||||
@ -222,10 +222,10 @@ class QueenOrLocalFileRedirection(_BaseRedirection):
|
|||||||
f.write(version_and_data)
|
f.write(version_and_data)
|
||||||
f.close()
|
f.close()
|
||||||
|
|
||||||
f, filename = workqueue.create_tempfile(".toqueen")
|
f, filename = workqueue.create_tempfile(".tovdrive")
|
||||||
self.serialize_subtree_to_file(f)
|
self.serialize_subtree_to_file(f)
|
||||||
f.close()
|
f.close()
|
||||||
workqueue.add_queen_update_handle(self.handle, filename)
|
workqueue.add_vdrive_update_handle(self.handle, filename)
|
||||||
workqueue.add_delete_tempfile(filename)
|
workqueue.add_delete_tempfile(filename)
|
||||||
return None
|
return None
|
||||||
|
|
||||||
|
@ -18,19 +18,18 @@ all_openable_subtree_types = [
|
|||||||
directory.CHKDirectorySubTree,
|
directory.CHKDirectorySubTree,
|
||||||
directory.SSKDirectorySubTree,
|
directory.SSKDirectorySubTree,
|
||||||
redirect.LocalFileRedirection,
|
redirect.LocalFileRedirection,
|
||||||
redirect.QueenRedirection,
|
redirect.VdriveRedirection,
|
||||||
redirect.QueenOrLocalFileRedirection,
|
redirect.VdriveOrLocalFileRedirection,
|
||||||
redirect.HTTPRedirection,
|
redirect.HTTPRedirection,
|
||||||
]
|
]
|
||||||
|
|
||||||
class SubTreeMaker(object):
|
class SubTreeMaker(object):
|
||||||
implements(ISubTreeMaker)
|
implements(ISubTreeMaker)
|
||||||
|
|
||||||
def __init__(self, queen, downloader):
|
def __init__(self, downloader):
|
||||||
# this is created with everything it might need to download and
|
# this is created with everything it might need to download and
|
||||||
# create subtrees. That means a Downloader and a reference to the
|
# create subtrees. That means a Downloader and in the future (?) a
|
||||||
# queen.
|
# reference to the vdrive.
|
||||||
self._queen = queen
|
|
||||||
assert IDownloader(downloader)
|
assert IDownloader(downloader)
|
||||||
self._downloader = downloader
|
self._downloader = downloader
|
||||||
self._node_maker = NodeMaker()
|
self._node_maker = NodeMaker()
|
||||||
@ -85,10 +84,8 @@ class VirtualDrive(object):
|
|||||||
workqueue.set_uploader(uploader)
|
workqueue.set_uploader(uploader)
|
||||||
self._downloader = downloader
|
self._downloader = downloader
|
||||||
self._uploader = uploader
|
self._uploader = uploader
|
||||||
# TODO: queen?
|
|
||||||
self.queen = None
|
|
||||||
self.root_node = root_node
|
self.root_node = root_node
|
||||||
self.subtree_maker = SubTreeMaker(self.queen, downloader)
|
self.subtree_maker = SubTreeMaker(downloader)
|
||||||
|
|
||||||
# these methods are used to walk through our subtrees
|
# these methods are used to walk through our subtrees
|
||||||
|
|
||||||
@ -276,7 +273,7 @@ class VirtualDrive(object):
|
|||||||
# really say whether they're mutable or not. But we're pretty sure
|
# really say whether they're mutable or not. But we're pretty sure
|
||||||
# that the new subtree is supposed to be mutable, because we asserted
|
# that the new subtree is supposed to be mutable, because we asserted
|
||||||
# that earlier (although I suppose perhaps someone could change a
|
# that earlier (although I suppose perhaps someone could change a
|
||||||
# QueenRedirection or an SSK file while we're offline in the middle
|
# VdriveRedirection or an SSK file while we're offline in the middle
|
||||||
# of our workqueue..). Tell the new subtree that their parent is
|
# of our workqueue..). Tell the new subtree that their parent is
|
||||||
# mutable so we can be sure it will believe that it itself is
|
# mutable so we can be sure it will believe that it itself is
|
||||||
# mutable.
|
# mutable.
|
||||||
|
@ -531,10 +531,10 @@ class IWorkQueue(Interface):
|
|||||||
by the parent.
|
by the parent.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
def add_queen_update_handle(handle, source_filename):
|
def add_vdrive_update_handle(handle, source_filename):
|
||||||
"""Arrange for a central queen to be notified that the given handle
|
"""Arrange for a vdrive server to be notified that the given handle
|
||||||
has been updated with the contents of the given tempfile. This will
|
has been updated with the contents of the given tempfile. This will
|
||||||
send a set_handle() message to the queen."""
|
send a set_handle() message to the vdrive."""
|
||||||
|
|
||||||
def add_retain_ssk(read_capability):
|
def add_retain_ssk(read_capability):
|
||||||
"""Arrange for the given SSK to be kept alive."""
|
"""Arrange for the given SSK to be kept alive."""
|
||||||
@ -582,7 +582,7 @@ class IWorkQueue(Interface):
|
|||||||
subtree with the contents.
|
subtree with the contents.
|
||||||
|
|
||||||
If 'subtree_node' refers to a redirection subtree like
|
If 'subtree_node' refers to a redirection subtree like
|
||||||
LocalFileRedirection or QueenRedirection, then 'localpath' is
|
LocalFileRedirection or VdriveRedirection, then 'localpath' is
|
||||||
ignored, because redirection subtrees don't consume path components
|
ignored, because redirection subtrees don't consume path components
|
||||||
and have no internal directory structure (they just have the one
|
and have no internal directory structure (they just have the one
|
||||||
redirection target). Redirection subtrees generally retain a constant
|
redirection target). Redirection subtrees generally retain a constant
|
||||||
|
@ -5,7 +5,7 @@ from allmydata.filetable import GlobalVirtualDrive
|
|||||||
from allmydata.introducer import Introducer
|
from allmydata.introducer import Introducer
|
||||||
|
|
||||||
|
|
||||||
class Queen(node.Node):
|
class IntroducerAndVdrive(node.Node):
|
||||||
CERTFILE = "introducer.pem"
|
CERTFILE = "introducer.pem"
|
||||||
PORTNUMFILE = "introducer.port"
|
PORTNUMFILE = "introducer.port"
|
||||||
NODETYPE = "introducer"
|
NODETYPE = "introducer"
|
@ -141,7 +141,7 @@ class _BaseManhole(service.MultiService):
|
|||||||
def makeNamespace():
|
def makeNamespace():
|
||||||
# close over 'self' so we can get access to .parent later
|
# close over 'self' so we can get access to .parent later
|
||||||
from allmydata import debugshell
|
from allmydata import debugshell
|
||||||
debugshell.app = self.parent # make client/queen accesible via 'app'
|
debugshell.app = self.parent # make node accessible via 'app'
|
||||||
namespace = {}
|
namespace = {}
|
||||||
for sym in dir(debugshell):
|
for sym in dir(debugshell):
|
||||||
if sym.startswith('__') and sym.endswith('__'):
|
if sym.startswith('__') and sym.endswith('__'):
|
||||||
|
@ -12,8 +12,8 @@ import zfec
|
|||||||
import foolscap
|
import foolscap
|
||||||
|
|
||||||
class Node(service.MultiService):
|
class Node(service.MultiService):
|
||||||
# this implements common functionality of both Client nodes and the Queen
|
# this implements common functionality of both Client nodes, Introducer
|
||||||
# node.
|
# nodes, and Vdrive nodes
|
||||||
NODETYPE = "unknown NODETYPE"
|
NODETYPE = "unknown NODETYPE"
|
||||||
PORTNUMFILE = None
|
PORTNUMFILE = None
|
||||||
CERTFILE = None
|
CERTFILE = None
|
||||||
|
@ -110,10 +110,10 @@ c.setServiceParent(application)
|
|||||||
introducer_tac = """
|
introducer_tac = """
|
||||||
# -*- python -*-
|
# -*- python -*-
|
||||||
|
|
||||||
from allmydata import queen
|
from allmydata import introducer_and_vdrive
|
||||||
from twisted.application import service
|
from twisted.application import service
|
||||||
|
|
||||||
c = queen.Queen()
|
c = introducer_and_vdrive.IntroducerAndVdrive()
|
||||||
|
|
||||||
application = service.Application("allmydata_introducer")
|
application = service.Application("allmydata_introducer")
|
||||||
c.setServiceParent(application)
|
c.setServiceParent(application)
|
||||||
|
@ -4,7 +4,7 @@ import os, shutil
|
|||||||
|
|
||||||
from twisted.internet import defer, reactor, protocol, error
|
from twisted.internet import defer, reactor, protocol, error
|
||||||
from twisted.application import service
|
from twisted.application import service
|
||||||
from allmydata import client, queen
|
from allmydata import client, introducer_and_vdrive
|
||||||
from allmydata.scripts import runner
|
from allmydata.scripts import runner
|
||||||
from foolscap.eventual import eventually, flushEventualQueue
|
from foolscap.eventual import eventually, flushEventualQueue
|
||||||
from twisted.python import log
|
from twisted.python import log
|
||||||
@ -32,7 +32,7 @@ class SystemFramework:
|
|||||||
|
|
||||||
def start(self):
|
def start(self):
|
||||||
print "STARTING"
|
print "STARTING"
|
||||||
d = self.make_queen()
|
d = self.make_introducer_and_vdrive()
|
||||||
def _more(res):
|
def _more(res):
|
||||||
self.make_nodes()
|
self.make_nodes()
|
||||||
self.start_client()
|
self.start_client()
|
||||||
@ -56,23 +56,23 @@ class SystemFramework:
|
|||||||
s.setServiceParent(self.sparent)
|
s.setServiceParent(self.sparent)
|
||||||
return s
|
return s
|
||||||
|
|
||||||
def make_queen(self):
|
def make_introducer_and_vdrive(self):
|
||||||
queendir = os.path.join(self.basedir, "queen")
|
introducer_and_vdrive_dir = os.path.join(self.basedir, "introducer_and_vdrive")
|
||||||
os.mkdir(queendir)
|
os.mkdir(introducer_and_vdrive_dir)
|
||||||
self.queen = self.add_service(queen.Queen(basedir=queendir))
|
self.introducer_and_vdrive = self.add_service(introducer_and_vdrive.IntroducerAndVdrive(basedir=introducer_and_vdrive_dir))
|
||||||
d = self.queen.when_tub_ready()
|
d = self.introducer_and_vdrive.when_tub_ready()
|
||||||
return d
|
return d
|
||||||
|
|
||||||
def make_nodes(self):
|
def make_nodes(self):
|
||||||
q = self.queen
|
q = self.introducer_and_vdrive
|
||||||
self.queen_pburl = q.urls["introducer"]
|
self.introducer_furl = q.urls["introducer"]
|
||||||
vdrive_furl = q.urls["vdrive"]
|
vdrive_furl = q.urls["vdrive"]
|
||||||
self.nodes = []
|
self.nodes = []
|
||||||
for i in range(self.numnodes):
|
for i in range(self.numnodes):
|
||||||
nodedir = os.path.join(self.basedir, "node%d" % i)
|
nodedir = os.path.join(self.basedir, "node%d" % i)
|
||||||
os.mkdir(nodedir)
|
os.mkdir(nodedir)
|
||||||
f = open(os.path.join(nodedir, "introducer.furl"), "w")
|
f = open(os.path.join(nodedir, "introducer.furl"), "w")
|
||||||
f.write(self.queen_pburl)
|
f.write(self.introducer_furl)
|
||||||
f.close()
|
f.close()
|
||||||
f = open(os.path.join(nodedir, "vdrive.furl"), "w")
|
f = open(os.path.join(nodedir, "vdrive.furl"), "w")
|
||||||
f.write(vdrive_furl)
|
f.write(vdrive_furl)
|
||||||
@ -80,7 +80,7 @@ class SystemFramework:
|
|||||||
c = self.add_service(client.Client(basedir=nodedir))
|
c = self.add_service(client.Client(basedir=nodedir))
|
||||||
self.nodes.append(c)
|
self.nodes.append(c)
|
||||||
# the peers will start running, eventually they will connect to each
|
# the peers will start running, eventually they will connect to each
|
||||||
# other and the queen
|
# other and the introducer_and_vdrive
|
||||||
|
|
||||||
def touch_keepalive(self):
|
def touch_keepalive(self):
|
||||||
f = open(self.keepalive_file, "w")
|
f = open(self.keepalive_file, "w")
|
||||||
@ -98,7 +98,7 @@ class SystemFramework:
|
|||||||
runner.create_client(config)
|
runner.create_client(config)
|
||||||
log.msg("DONE MAKING CLIENT")
|
log.msg("DONE MAKING CLIENT")
|
||||||
f = open(os.path.join(clientdir, "introducer.furl"), "w")
|
f = open(os.path.join(clientdir, "introducer.furl"), "w")
|
||||||
f.write(self.queen_pburl + "\n")
|
f.write(self.introducer_furl + "\n")
|
||||||
f.close()
|
f.close()
|
||||||
self.keepalive_file = os.path.join(clientdir, "suicide_prevention_hotline")
|
self.keepalive_file = os.path.join(clientdir, "suicide_prevention_hotline")
|
||||||
self.touch_keepalive()
|
self.touch_keepalive()
|
||||||
|
@ -425,7 +425,7 @@ class VDrive(unittest.TestCase):
|
|||||||
self.failUnlessEqual(c1a, c2a)
|
self.failUnlessEqual(c1a, c2a)
|
||||||
|
|
||||||
def testDirectory(self):
|
def testDirectory(self):
|
||||||
stm = vdrive.SubTreeMaker(None, FakeMesh())
|
stm = vdrive.SubTreeMaker(FakeMesh())
|
||||||
|
|
||||||
# create an empty directory (stored locally)
|
# create an empty directory (stored locally)
|
||||||
subtree = directory.LocalFileSubTree()
|
subtree = directory.LocalFileSubTree()
|
||||||
|
@ -2,12 +2,12 @@
|
|||||||
from twisted.trial import unittest
|
from twisted.trial import unittest
|
||||||
from foolscap.eventual import flushEventualQueue
|
from foolscap.eventual import flushEventualQueue
|
||||||
|
|
||||||
from allmydata import queen
|
from allmydata import introducer_and_vdrive
|
||||||
from allmydata.util import testutil
|
from allmydata.util import testutil
|
||||||
|
|
||||||
class Basic(testutil.SignalMixin, unittest.TestCase):
|
class Basic(testutil.SignalMixin, unittest.TestCase):
|
||||||
def test_loadable(self):
|
def test_loadable(self):
|
||||||
q = queen.Queen()
|
q = introducer_and_vdrive.IntroducerAndVdrive()
|
||||||
d = q.startService()
|
d = q.startService()
|
||||||
d.addCallback(lambda res: q.stopService())
|
d.addCallback(lambda res: q.stopService())
|
||||||
d.addCallback(flushEventualQueue)
|
d.addCallback(flushEventualQueue)
|
@ -3,7 +3,7 @@ import os
|
|||||||
from twisted.trial import unittest
|
from twisted.trial import unittest
|
||||||
from twisted.internet import defer, reactor
|
from twisted.internet import defer, reactor
|
||||||
from twisted.application import service
|
from twisted.application import service
|
||||||
from allmydata import client, queen, uri, download
|
from allmydata import client, introducer_and_vdrive, uri, download
|
||||||
from allmydata.util import idlib, fileutil, testutil
|
from allmydata.util import idlib, fileutil, testutil
|
||||||
from foolscap.eventual import flushEventualQueue
|
from foolscap.eventual import flushEventualQueue
|
||||||
from twisted.python import log
|
from twisted.python import log
|
||||||
@ -38,17 +38,17 @@ class SystemTest(testutil.SignalMixin, unittest.TestCase):
|
|||||||
|
|
||||||
def set_up_nodes(self, NUMCLIENTS=5):
|
def set_up_nodes(self, NUMCLIENTS=5):
|
||||||
self.numclients = NUMCLIENTS
|
self.numclients = NUMCLIENTS
|
||||||
queendir = self.getdir("queen")
|
introducer_and_vdrive_dir = self.getdir("introducer_and_vdrive")
|
||||||
if not os.path.isdir(queendir):
|
if not os.path.isdir(introducer_and_vdrive_dir):
|
||||||
fileutil.make_dirs(queendir)
|
fileutil.make_dirs(introducer_and_vdrive_dir)
|
||||||
self.queen = self.add_service(queen.Queen(basedir=queendir))
|
self.introducer_and_vdrive = self.add_service(introducer_and_vdrive.IntroducerAndVdrive(basedir=introducer_and_vdrive_dir))
|
||||||
d = self.queen.when_tub_ready()
|
d = self.introducer_and_vdrive.when_tub_ready()
|
||||||
d.addCallback(self._set_up_nodes_2)
|
d.addCallback(self._set_up_nodes_2)
|
||||||
return d
|
return d
|
||||||
|
|
||||||
def _set_up_nodes_2(self, res):
|
def _set_up_nodes_2(self, res):
|
||||||
q = self.queen
|
q = self.introducer_and_vdrive
|
||||||
self.queen_furl = q.urls["introducer"]
|
self.introducer_furl = q.urls["introducer"]
|
||||||
self.vdrive_furl = q.urls["vdrive"]
|
self.vdrive_furl = q.urls["vdrive"]
|
||||||
self.clients = []
|
self.clients = []
|
||||||
for i in range(self.numclients):
|
for i in range(self.numclients):
|
||||||
@ -57,7 +57,7 @@ class SystemTest(testutil.SignalMixin, unittest.TestCase):
|
|||||||
fileutil.make_dirs(basedir)
|
fileutil.make_dirs(basedir)
|
||||||
if i == 0:
|
if i == 0:
|
||||||
open(os.path.join(basedir, "webport"), "w").write("tcp:0:interface=127.0.0.1")
|
open(os.path.join(basedir, "webport"), "w").write("tcp:0:interface=127.0.0.1")
|
||||||
open(os.path.join(basedir, "introducer.furl"), "w").write(self.queen_furl)
|
open(os.path.join(basedir, "introducer.furl"), "w").write(self.introducer_furl)
|
||||||
open(os.path.join(basedir, "vdrive.furl"), "w").write(self.vdrive_furl)
|
open(os.path.join(basedir, "vdrive.furl"), "w").write(self.vdrive_furl)
|
||||||
c = self.add_service(client.Client(basedir=basedir))
|
c = self.add_service(client.Client(basedir=basedir))
|
||||||
self.clients.append(c)
|
self.clients.append(c)
|
||||||
@ -77,7 +77,7 @@ class SystemTest(testutil.SignalMixin, unittest.TestCase):
|
|||||||
basedir = self.getdir("client%d" % client_num)
|
basedir = self.getdir("client%d" % client_num)
|
||||||
if not os.path.isdir(basedir):
|
if not os.path.isdir(basedir):
|
||||||
fileutil.make_dirs(basedir)
|
fileutil.make_dirs(basedir)
|
||||||
open(os.path.join(basedir, "introducer.furl"), "w").write(self.queen_furl)
|
open(os.path.join(basedir, "introducer.furl"), "w").write(self.introducer_furl)
|
||||||
open(os.path.join(basedir, "vdrive.furl"), "w").write(self.vdrive_furl)
|
open(os.path.join(basedir, "vdrive.furl"), "w").write(self.vdrive_furl)
|
||||||
|
|
||||||
c = client.Client(basedir=basedir)
|
c = client.Client(basedir=basedir)
|
||||||
|
Loading…
x
Reference in New Issue
Block a user