update foolscap to foolscap-0.1.5, the latest release

src/foolscap/ChangeLog

@@ -1,3 +1,129 @@
+2007-08-07  Brian Warner  <warner@lothar.com>
+
+	* foolscap/__init__.py: release Foolscap-0.1.5
+	* misc/{sid|sarge|dapper|edgy|feisty}/debian/changelog: same
+
+2007-08-07  Brian Warner  <warner@lothar.com>
+
+	* NEWS: update for the upcoming release
+
+	* foolscap/pb.py (Tub.registerNameLookupHandler): new function to
+	augment Tub.registerReference(). This allows names to be looked up
+	at request time, rather than requiring all Referenceables be
+	pre-registered with registerReference(). The chief use of this
+	would be for FURLs which point at objects that live on disk in
+	some persistent state until they are needed. Closes #6.
+	(Tub.unregisterNameLookupHandler): allow handlers to be removed
+	(Tub.getReferenceForName): use the handler during lookup
+	* foolscap/test/test_tub.py (NameLookup): test it
+
+2007-07-27  Brian Warner  <warner@lothar.com>
+
+	* foolscap/referenceable.py (LocalReferenceable): implement an
+	adapter that allows code to do IRemoteReference(t).callRemote(...)
+	and have it work for both RemoteReferences and local
+	Referenceables. You might want to do this if you're getting back
+	introductions to a variety of remote Referenceables, some of which
+	might actually be on your local system, and you want to treat all
+	of the, the same way. Local Referenceables will be wrapped with a
+	class that implements callRemote() and makes it behave like an
+	actual remote callRemote() would. Closes ticket #1.
+	* foolscap/test/test_reference.py (LocalReference): test it
+
+2007-07-26  Brian Warner  <warner@lothar.com>
+
+	* foolscap/call.py (AnswerUnslicer.receiveChild): accept a
+	ready_deferred, to accomodate Gifts in return values. Closes #5.
+	(AnswerUnslicer.receiveClose): .. and don't fire the response
+	until any such Gifts resolve
+	* foolscap/test/test_gifts.py (Gifts.testReturn): test it
+	(Gifts.testReturnInContainer): same
+	(Bad.testReturn_swissnum): and test the failure case too
+
+	* foolscap/test/test_pb.py (TestAnswer.testAccept1): fix a test
+	which wasn't calling start() properly and was broken by that change
+	(TestAnswer.testAccept2): same
+
+	* foolscap/test/test_gifts.py (Bad.setUp): disable these tests when
+	we don't have crypto, since TubIDs are not mangleable in the same
+	way without crypto.
+
+	* foolscap/slicer.py (BaseUnslicer.receiveChild): new convention:
+	Unslicers should accumulate their children's ready_deferreds into
+	an AsyncAND, and pass it to the parent. If something goes wrong,
+	the ready_deferred should errback, which will abandon the method
+	call that contains it.
+	* foolscap/slicers/dict.py (DictUnslicer.receiveClose): same
+	* foolscap/slicers/tuple.py (TupleUnslicer.receiveClose): same
+	(TupleUnslicer.complete): same
+	* foolscap/slicers/set.py (SetUnslicer.receiveClose): same
+	* foolscap/slicers/list.py (ListUnslicer.receiveClose): same
+	* foolscap/call.py (CallUnslicer.receiveClose): same
+
+	* foolscap/referenceable.py (TheirReferenceUnslicer.receiveClose):
+	use our ready_deferred to signal whether the gift resolves
+	correctly or not. If it fails, errback ready_deferred (to prevent
+	the message from being delivered without the resolved gift), but
+	callback obj_deferred with a placeholder to avoid causing too much
+	distress to the container.
+
+	* foolscap/broker.py (PBRootUnslicer.receiveChild): accept
+	ready_deferred in the InboundDelivery, stash both of them in the
+	broker.
+	(Broker.scheduleCall): rewrite inbound delivery handling: use a
+	self._call_is_running flag to prevent concurrent deliveries, and
+	wait for the ready_deferred before delivering the top-most
+	message. If the ready_deferred errbacks, that gets routed to
+	self.callFailed so the caller hears about the problem. This closes
+	ticket #2.
+
+	* foolscap/call.py (InboundDelivery): remove whenRunnable, relying
+	upon the ready_deferred to let the Broker know when the message
+	can be delivered.
+	(ArgumentUnslicer): significant cleanup, using ready_deferred.
+	Remove isReady and whenReady.
+
+	* foolscap/test/test_gifts.py (Base): factor setup code out
+	(Base.createCharacters): registerReference(tubname), for debugging
+	(Bad): add a bunch of tests to make sure that gifts which fail to
+	resolve (for various reasons) will inform the caller about the
+	problem, via an errback on the original callRemote()'s Deferred.
+
+2007-07-25  Brian Warner  <warner@lothar.com>
+
+	* foolscap/util.py (AsyncAND): new utility class, which is like
+	DeferredList but is specifically for control flow rather than data
+	flow.
+	* foolscap/test/test_util.py: test it
+
+	* foolscap/call.py (CopiedFailure.setCopyableState): set .type to
+	a class that behaves (as least as far as reflect.qual() is
+	concerned) just like the original exception class. This improves
+	the behavior of derived Failure objects, as well as trial's
+	handling of CopiedFailures that get handed to log.err().
+	CopiedFailures are now a bit more like actual Failures. See ticket
+	#4 (http://foolscap.lothar.com/trac/ticket/4) for more details.
+	(CopiedFailureSlicer): make sure that CopiedFailures can be
+	serialized, so that A-calls-B-calls-C can return a failure all
+	the way back.
+	* foolscap/test/test_call.py (TestCall.testCopiedFailure): test it
+	* foolscap/test/test_copyable.py: update to match, now we must
+	compare reflect.qual(f.type) against some extension classname,
+	rather than just f.type.
+	* foolscap/test/test_pb.py: same
+	* foolscap/test/common.py: same
+
+2007-07-15  Brian Warner  <warner@lothar.com>
+
+	* foolscap/test/test_interfaces.py (TestInterface.testStack):
+	don't look for a '/' in the stacktrace, since it won't be there
+	under windows. Thanks to 'strank'. Closes Twisted#2731.
+
+2007-06-29  Brian Warner  <warner@lothar.com>
+
+	* foolscap/__init__.py: bump revision to 0.1.4+ while between releases
+	* misc/{sid|sarge|dapper|edgy|feisty}/debian/changelog: same
+
 2007-05-14  Brian Warner  <warner@lothar.com>
 
 	* foolscap/__init__.py: release Foolscap-0.1.4

src/foolscap/Makefile

@@ -53,3 +53,4 @@ docs:
 	lore -p --config template=$(DOC_TEMPLATE) --config ext=.html \
 	`find doc -name '*.xhtml'`
 
+

src/foolscap/NEWS

@@ -1,5 +1,66 @@
 User visible changes in Foolscap (aka newpb/pb2).           -*- outline -*-
 
+* Release 0.1.5 (07 Aug 2007)
+
+** Compatibility
+
+This release is fully compatible with 0.1.4 and 0.1.3 .
+
+** CopiedFailure improvements
+
+When a remote method call fails, the calling side gets back a CopiedFailure
+instance. These instances now behave slightly more like the (local) Failure
+objects that they are intended to mirror, in that .type now behaves much like
+the original class. This should allow trial tests which result in a
+CopiedFailure to be logged without exploding. In addition, chained failures
+(where A calls B, and B calls C, and C fails, so C's Failure is eventually
+returned back to A) should work correctly now.
+
+** Gift improvements
+
+Gifts inside return values should properly stall the delivery of the response
+until the gift is resolved. Gifts in all sorts of containers should work
+properly now. Gifts which cannot be resolved successfully (either because the
+hosting Tub cannot be reached, or because the name cannot be found) will now
+cause a proper error rather than hanging forever. Unresolvable gifts in
+method arguments will cause the message to not be delivered and an error to
+be returned to the caller. Unresolvable gifts in method return values will
+cause the caller to receive an error.
+
+** IRemoteReference() adapter
+
+The IRemoteReference() interface now has an adapter from Referenceable which
+creates a wrapper that enables the use of callRemote() and other
+IRemoteReference methods on a local object.
+
+The situation where this might be useful is when you have a central
+introducer and a bunch of clients, and the clients are introducing themselves
+to each other (to create a fully-connected mesh), and the introductions are
+using live references (i.e. Gifts), then when a specific client learns about
+itself from the introducer, that client will receive a local object instead
+of a RemoteReference. Each client will wind up with n-1 RemoteReferences and
+a single local object.
+
+This adapter allows the client to treat all these introductions as equal. A
+client that wishes to send a message to everyone it's been introduced to
+(including itself) can use:
+
+  for i in introductions:
+    IRemoteReference(i).callRemote("hello", args)
+
+In the future, if we implement coercing Guards (instead of
+compliance-asserting Constraints), then IRemoteReference will be useful as a
+guard on methods that want to insure that they can do callRemote (and
+notifyOnDisconnect, etc) on their argument.
+
+** Tub.registerNameLookupHandler
+
+This method allows a one-argument name-lookup callable to be attached to the
+Tub. This augments the table maintained by Tub.registerReference, allowing
+Referenceables to be created on the fly, or persisted/retrieved on disk
+instead of requiring all of them to be generated and registered at startup.
+
+
 * Release 0.1.4 (14 May 2007)
 
 ** Compatibility

src/foolscap/foolscap/__init__.py

@@ -1,6 +1,6 @@
 """Foolscap"""
 
-__version__ = "0.1.4"
+__version__ = "0.1.5"
 
 # here are the primary entry points
 from foolscap.pb import Tub, UnauthenticatedTub, getRemoteURL_TCP

src/foolscap/foolscap/broker.py

@@ -111,8 +111,7 @@ class PBRootUnslicer(RootUnslicer):
 
     def receiveChild(self, token, ready_deferred):
         if isinstance(token, call.InboundDelivery):
-            assert ready_deferred is None
+            self.broker.scheduleCall(token, ready_deferred)
-            self.broker.scheduleCall(token)
 
 
 
@@ -215,6 +214,7 @@ class Broker(banana.Banana, referenceable.Referenceable):
         self.disconnectWatchers = []
         # receiving side uses these
         self.inboundDeliveryQueue = []
+        self._call_is_running = False
         self.activeLocalCalls = {} # the other side wants an answer from us
 
     def setTub(self, tub):
@@ -506,33 +506,31 @@ class Broker(banana.Banana, referenceable.Referenceable):
                 return m
         return None
 
-    def scheduleCall(self, delivery):
+    def scheduleCall(self, delivery, ready_deferred):
-        self.inboundDeliveryQueue.append(delivery)
+        self.inboundDeliveryQueue.append( (delivery,ready_deferred) )
         eventually(self.doNextCall)
 
-    def doNextCall(self, ignored=None):
+    def doNextCall(self):
+        if self._call_is_running:
+            return
         if not self.inboundDeliveryQueue:
             return
-        nextCall = self.inboundDeliveryQueue[0]
+        delivery, ready_deferred = self.inboundDeliveryQueue.pop(0)
-        if nextCall.isRunnable():
+        self._call_is_running = True
-            # remove it and arrange to run again soon
+        if not ready_deferred:
-            self.inboundDeliveryQueue.pop(0)
+            ready_deferred = defer.succeed(None)
-            delivery = nextCall
+        d = ready_deferred
-            if self.inboundDeliveryQueue:
+        d.addCallback(lambda res: self._doCall(delivery))
-                eventually(self.doNextCall)
+        d.addCallback(self._callFinished, delivery)
-
+        d.addErrback(self.callFailed, delivery.reqID, delivery)
-            # now perform the actual delivery
+        def _done(res):
-            d = defer.maybeDeferred(self._doCall, delivery)
+            self._call_is_running = False
-            d.addCallback(self._callFinished, delivery)
+            eventually(self.doNextCall)
-            d.addErrback(self.callFailed, delivery.reqID, delivery)
+        d.addBoth(_done)
-            return
+        return None
-        # arrange to wake up when the next call becomes runnable
-        d = nextCall.whenRunnable()
-        d.addCallback(self.doNextCall)
 
     def _doCall(self, delivery):
         obj = delivery.obj
-        assert delivery.allargs.isReady()
         args = delivery.allargs.args
         kwargs = delivery.allargs.kwargs
         for i in args + kwargs.values():

src/foolscap/foolscap/call.py

@@ -3,11 +3,11 @@ from twisted.python import failure, log, reflect
 from twisted.internet import defer
 
 from foolscap import copyable, slicer, tokens
-from foolscap.eventual import eventually
 from foolscap.copyable import AttributeDictConstraint
 from foolscap.constraint import ByteStringConstraint
 from foolscap.slicers.list import ListConstraint
 from tokens import BananaError, Violation
+from foolscap.util import AsyncAND
 
 
 class FailureConstraint(AttributeDictConstraint):
@@ -162,21 +162,6 @@ class InboundDelivery:
         self.methodname = methodname
         self.methodSchema = methodSchema
         self.allargs = allargs
-        if allargs.isReady():
-            self.runnable = True
-        self.runnable = False
-
-    def isRunnable(self):
-        if self.allargs.isReady():
-            return True
-        return False
-
-    def whenRunnable(self):
-        if self.allargs.isReady():
-            return defer.succeed(self)
-        d = self.allargs.whenReady()
-        d.addCallback(lambda res: self)
-        return d
 
     def logFailure(self, f):
         # called if tub.logLocalFailures is True
@@ -211,7 +196,8 @@ class ArgumentUnslicer(slicer.ScopedUnslicer):
         self.argname = None
         self.argConstraint = None
         self.num_unreferenceable_children = 0
-        self.num_unready_children = 0
+        self._all_children_are_referenceable_d = None
+        self._ready_deferreds = []
         self.closed = False
 
     def checkToken(self, typebyte, size):
@@ -248,7 +234,7 @@ class ArgumentUnslicer(slicer.ScopedUnslicer):
         if self.debug:
             log.msg("%s.receiveChild: %s %s %s %s %s args=%s kwargs=%s" %
                     (self, self.closed, self.num_unreferenceable_children,
-                     self.num_unready_children, token, ready_deferred,
+                     len(self._ready_deferreds), token, ready_deferred,
                      self.args, self.kwargs))
         if self.numargs is None:
             # this token is the number of positional arguments
@@ -273,12 +259,10 @@ class ArgumentUnslicer(slicer.ScopedUnslicer):
                 # resolved yet.
                 self.num_unreferenceable_children += 1
                 argvalue.addCallback(self.updateChild, argpos)
-                argvalue.addErrback(self.explode)
             if ready_deferred:
                 if self.debug:
                     log.msg("%s.receiveChild got an unready posarg" % self)
-                self.num_unready_children += 1
+                self._ready_deferreds.append(ready_deferred)
-                ready_deferred.addCallback(self.childReady)
             if len(self.args) < self.numargs:
                 # more to come
                 ms = self.methodSchema
@@ -291,6 +275,7 @@ class ArgumentUnslicer(slicer.ScopedUnslicer):
 
         if self.argname is None:
             # this token is the name of a keyword argument
+            assert ready_deferred is None
             self.argname = token
             # if the argname is invalid, this may raise Violation
             ms = self.methodSchema
@@ -308,12 +293,10 @@ class ArgumentUnslicer(slicer.ScopedUnslicer):
         if isinstance(argvalue, defer.Deferred):
             self.num_unreferenceable_children += 1
             argvalue.addCallback(self.updateChild, self.argname)
-            argvalue.addErrback(self.explode)
         if ready_deferred:
             if self.debug:
                 log.msg("%s.receiveChild got an unready kwarg" % self)
-            self.num_unready_children += 1
+            self._ready_deferreds.append(ready_deferred)
-            ready_deferred.addCallback(self.childReady)
         self.argname = None
         return
 
@@ -333,70 +316,31 @@ class ArgumentUnslicer(slicer.ScopedUnslicer):
         else:
             self.kwargs[which] = obj
         self.num_unreferenceable_children -= 1
-        self.checkComplete()
+        if self.num_unreferenceable_children == 0:
+            if self._all_children_are_referenceable_d:
+                self._all_children_are_referenceable_d.callback(None)
         return obj
 
-    def childReady(self, obj):
-        self.num_unready_children -= 1
-        if self.debug:
-            log.msg("%s.childReady, now %d left" %
-                    (self, self.num_unready_children))
-            log.msg(" obj=%s, args=%s, kwargs=%s" %
-                    (obj, self.args, self.kwargs))
-        self.checkComplete()
-        return obj
-
-    def checkComplete(self):
-        # this is called each time one of our children gets updated or
-        # becomes ready (like when a Gift is finally resolved)
-        if self.debug:
-            log.msg("%s.checkComplete: %s %s %s args=%s kwargs=%s" %
-                    (self, self.closed, self.num_unreferenceable_children,
-                     self.num_unready_children, self.args, self.kwargs))
-
-        if not self.closed:
-            return
-        if self.num_unreferenceable_children:
-            return
-        if self.num_unready_children:
-            return
-        # yup, we're done. Notify anyone who is still waiting
-        if self.debug:
-            log.msg(" we are ready")
-        for d in self.watchers:
-            eventually(d.callback, self)
-        del self.watchers
 
     def receiveClose(self):
         if self.debug:
             log.msg("%s.receiveClose: %s %s %s" %
                     (self, self.closed, self.num_unreferenceable_children,
-                     self.num_unready_children))
+                     len(self._ready_deferreds)))
         if (self.numargs is None or
             len(self.args) < self.numargs or
             self.argname is not None):
             raise BananaError("'arguments' sequence ended too early")
         self.closed = True
-        self.watchers = []
+        dl = []
-        # we don't return a ready_deferred. Instead, the InboundDelivery
-        # object queries our isReady() method directly.
-        return self, None
-
-    def isReady(self):
-        assert self.closed
         if self.num_unreferenceable_children:
-            return False
+            d = self._all_children_are_referenceable_d = defer.Deferred()
-        if self.num_unready_children:
+            dl.append(d)
-            return False
+        dl.extend(self._ready_deferreds)
-        return True
+        ready_deferred = None
-
+        if dl:
-    def whenReady(self):
+            ready_deferred = AsyncAND(dl)
-        assert self.closed
+        return self, ready_deferred
-        if self.isReady():
-            return defer.succeed(self)
-        d = defer.Deferred()
-        self.watchers.append(d)
-        return d
 
     def describe(self):
         s = "<arguments"
@@ -409,11 +353,9 @@ class ArgumentUnslicer(slicer.ScopedUnslicer):
                 else:
                     s += " arg[?]"
         if self.closed:
-            if self.isReady():
+            s += " closed"
-                # waiting to be delivered
+            # TODO: it would be nice to indicate if we still have unready
-                s += " ready"
+            # children
-            else:
-                s += " waiting"
         s += ">"
         return s
 
@@ -430,6 +372,7 @@ class CallUnslicer(slicer.ScopedUnslicer):
         self.interface = None
         self.methodname = None
         self.methodSchema = None # will be a MethodArgumentsConstraint
+        self._ready_deferreds = []
 
     def checkToken(self, typebyte, size):
         # TODO: limit strings by returning a number instead of None
@@ -472,13 +415,13 @@ class CallUnslicer(slicer.ScopedUnslicer):
 
     def receiveChild(self, token, ready_deferred=None):
         assert not isinstance(token, defer.Deferred)
-        assert ready_deferred is None
         if self.debug:
             log.msg("%s.receiveChild [s%d]: %s" %
                     (self, self.stage, repr(token)))
 
         if self.stage == 0: # reqID
             # we don't yet know which reqID to send any failure to
+            assert ready_deferred is None
             self.reqID = token
             self.stage = 1
             if self.reqID != 0:
@@ -488,6 +431,7 @@ class CallUnslicer(slicer.ScopedUnslicer):
 
         if self.stage == 1: # objID
             # this might raise an exception if objID is invalid
+            assert ready_deferred is None
             self.objID = token
             self.obj = self.broker.getMyReferenceByCLID(token)
             #iface = self.broker.getRemoteInterfaceByName(token)
@@ -517,6 +461,7 @@ class CallUnslicer(slicer.ScopedUnslicer):
             # class). If this expectation were to go away, a quick
             # obj.__class__ -> RemoteReferenceSchema cache could be built.
 
+            assert ready_deferred is None
             self.stage = 3
 
             if self.objID < 0:
@@ -548,6 +493,8 @@ class CallUnslicer(slicer.ScopedUnslicer):
             # queue the message. It will not be executed until all the
             # arguments are ready. The .args list and .kwargs dict may change
             # before then.
+            if ready_deferred:
+                self._ready_deferreds.append(ready_deferred)
             self.stage = 4
             return
 
@@ -559,7 +506,10 @@ class CallUnslicer(slicer.ScopedUnslicer):
                                    self.interface, self.methodname,
                                    self.methodSchema,
                                    self.allargs)
-        return delivery, None
+        ready_deferred = None
+        if self._ready_deferreds:
+            ready_deferred = AsyncAND(self._ready_deferreds)
+        return delivery, ready_deferred
 
     def describe(self):
         s = "<methodcall"
@@ -600,6 +550,11 @@ class AnswerUnslicer(slicer.ScopedUnslicer):
     resultConstraint = None
     haveResults = False
 
+    def start(self, count):
+        slicer.ScopedUnslicer.start(self, count)
+        self._ready_deferreds = []
+        self._child_deferred = None
+
     def checkToken(self, typebyte, size):
         if self.request is None:
             if typebyte != tokens.INT:
@@ -633,15 +588,20 @@ class AnswerUnslicer(slicer.ScopedUnslicer):
         return unslicer
 
     def receiveChild(self, token, ready_deferred=None):
-        assert not isinstance(token, defer.Deferred)
-        assert ready_deferred is None
         if self.request == None:
+            assert not isinstance(token, defer.Deferred)
+            assert ready_deferred is None
             reqID = token
             # may raise Violation for bad reqIDs
             self.request = self.broker.getRequest(reqID)
             self.resultConstraint = self.request.constraint
         else:
-            self.results = token
+            if isinstance(token, defer.Deferred):
+                self._child_deferred = token
+            else:
+                self._child_deferred = defer.succeed(token)
+            if ready_deferred:
+                self._ready_deferreds.append(ready_deferred)
             self.haveResults = True
 
     def reportViolation(self, f):
@@ -652,7 +612,32 @@ class AnswerUnslicer(slicer.ScopedUnslicer):
         return f # give up our sequence
 
     def receiveClose(self):
-        self.request.complete(self.results)
+        # three things must happen before our request is complete:
+        #   receiveClose has occurred
+        #   the receiveChild object deferred (if any) has fired
+        #   ready_deferred has finished
+        # If ready_deferred errbacks, provide its failure object to the
+        # request. If not, provide the request with whatever receiveChild
+        # got.
+
+        if not self._child_deferred:
+            raise BananaError("Answer didn't include an answer")
+
+        if self._ready_deferreds:
+            d = AsyncAND(self._ready_deferreds)
+        else:
+            d = defer.succeed(None)
+
+        def _ready(res):
+            return self._child_deferred
+        d.addCallback(_ready)
+
+        def _done(res):
+            self.request.complete(res)
+        def _fail(f):
+            self.request.fail(f)
+        d.addCallbacks(_done, _fail)
+
         return None, None
 
     def describe(self):
@@ -818,6 +803,30 @@ class CopiedFailure(failure.Failure, copyable.RemoteCopyOldStyle):
         self.frames = []
         self.stack = []
 
+        # MAYBE: for native exception types, be willing to wire up a
+        # reference to the real exception class. For other exception types,
+        # our .type attribute will be a string, which (from a Failure's point
+        # of view) looks as if someone raised an old-style string exception.
+        # This is here so that trial will properly render a CopiedFailure
+        # that comes out of a test case (since it unconditionally does
+        # reflect.qual(f.type)
+
+        # ACTUALLY: replace self.type with a class that looks a lot like the
+        # original exception class (meaning that reflect.qual() will return
+        # the same string for this as for the original). If someone calls our
+        # .trap method, resulting in a new Failure with contents copied from
+        # this one, then the new Failure.printTraceback will attempt to use
+        # reflect.qual() on our self.type, so it needs to be a class instead
+        # of a string.
+
+        assert isinstance(self.type, str)
+        typepieces = self.type.split(".")
+        class ExceptionLikeString:
+            pass
+        self.type = ExceptionLikeString
+        self.type.__module__ = ".".join(typepieces[:-1])
+        self.type.__name__ = typepieces[-1]
+
     def __str__(self):
         return "[CopiedFailure instance: %s]" % self.getBriefTraceback()
 
@@ -829,3 +838,21 @@ class CopiedFailure(failure.Failure, copyable.RemoteCopyOldStyle):
         file.write(self.traceback)
 
 copyable.registerRemoteCopy(FailureSlicer.classname, CopiedFailure)
+
+class CopiedFailureSlicer(FailureSlicer):
+    # A calls B. B calls C. C fails and sends a Failure to B. B gets a
+    # CopiedFailure and sends it to A. A should get a CopiedFailure too. This
+    # class lives on B and slicers the CopiedFailure as it is sent to A.
+    slices = CopiedFailure
+
+    def getStateToCopy(self, obj, broker):
+        state = {}
+        for k in ('value', 'type', 'parents'):
+            state[k] = getattr(obj, k)
+        if broker.unsafeTracebacks:
+            state['traceback'] = obj.traceback
+        else:
+            state['traceback'] = "Traceback unavailable\n"
+        if not isinstance(state['type'], str):
+            state['type'] = reflect.qual(state['type']) # Exception class
+        return state

src/foolscap/foolscap/ipb.py

@@ -62,6 +62,10 @@ class IRemoteReference(Interface):
         notifyOnDisconnect handlers are cancelled.
         """
 
+    def dontNotifyOnDisconnect(cookie):
+        """Deregister a callback that was registered with notifyOnDisconnect.
+        """
+
     def callRemote(name, *args, **kwargs):
         """Invoke a method on the remote object with which I am associated.
 

src/foolscap/foolscap/pb.py

@@ -230,6 +230,8 @@ class Tub(service.MultiService):
         self.nameToReference = weakref.WeakValueDictionary()
         self.referenceToName = weakref.WeakKeyDictionary()
         self.strongReferences = []
+        self.nameLookupHandlers = []
+
         # remote stuff. Most of these use a TubRef (or NoAuthTubRef) as a
         # dictionary key
         self.tubConnectors = {} # maps TubRef to a TubConnector
@@ -487,7 +489,15 @@ class Tub(service.MultiService):
         return name
 
     def getReferenceForName(self, name):
-        return self.nameToReference[name]
+        if name in self.nameToReference:
+            return self.nameToReference[name]
+        for lookup in self.nameLookupHandlers:
+            ref = lookup(name)
+            if ref:
+                if ref not in self.referenceToName:
+                    self.referenceToName[ref] = name
+                return ref
+        raise KeyError("unable to find reference for name '%s'" % (name,))
 
     def getReferenceForURL(self, url):
         # TODO: who should this be used by?
@@ -526,6 +536,46 @@ class Tub(service.MultiService):
             self.strongReferences.remove(ref)
         self.revokeReference(ref)
 
+    def registerNameLookupHandler(self, lookup):
+        """Add a function to help convert names to Referenceables.
+
+        When remote systems pass a FURL to their Tub.getReference(), our Tub
+        will be asked to locate a Referenceable for the name inside that
+        furl. The normal mechanism for this is to look at the table
+        maintained by registerReference() and unregisterReference(). If the
+        name does not exist in that table, other 'lookup handler' functions
+        are given a chance. Each lookup handler is asked in turn, and the
+        first which returns a non-None value wins.
+
+        This may be useful for cases where the furl represents an object that
+        lives on disk, or is generated on demand: rather than creating all
+        possible Referenceables at startup, the lookup handler can create or
+        retrieve the objects only when someone asks for them.
+
+        Note that constructing the FURLs of these objects may be non-trivial.
+        It is safe to create an object, use tub.registerReference in one
+        invocation of a program to obtain (and publish) the furl, parse the
+        furl to extract the name, save the contents of the object on disk,
+        then in a later invocation of the program use a lookup handler to
+        retrieve the object from disk. This approach means the objects that
+        are created in a given invocation stick around (inside
+        tub.strongReferences) for the rest of that invocation. An alternatve
+        approach is to create the object but *not* use tub.registerReference,
+        but in that case you have to construct the FURL yourself, and the Tub
+        does not currently provide any support for doing this robustly.
+
+        @param lookup: a callable which accepts a name (as a string) and
+                       returns either a Referenceable or None. Note that
+                       these strings should not contain a slash, a question
+                       mark, or an ampersand, as these are reserved in the
+                       FURL for later expansion (to add parameters beyond the
+                       object name)
+        """
+        self.nameLookupHandlers.append(lookup)
+
+    def unregisterNameLookupHandler(self, lookup):
+        self.nameLookupHandlers.remove(lookup)
+
     def getReference(self, sturdyOrURL):
         """Acquire a RemoteReference for the given SturdyRef/URL.
 

src/foolscap/foolscap/referenceable.py

@@ -11,7 +11,7 @@ from zope.interface import implements
 from twisted.python.components import registerAdapter
 Interface = interface.Interface
 from twisted.internet import defer
-from twisted.python import failure
+from twisted.python import failure, log
 
 from foolscap import ipb, slicer, tokens, call
 BananaError = tokens.BananaError
@@ -21,7 +21,7 @@ from foolscap.remoteinterface import getRemoteInterface, \
      getRemoteInterfaceByName, RemoteInterfaceConstraint
 from foolscap.schema import constraintMap
 from foolscap.copyable import Copyable, RemoteCopy
-from foolscap.eventual import eventually
+from foolscap.eventual import eventually, fireEventually
 
 class OnlyReferenceable(object):
     implements(ipb.IReferenceable)
@@ -538,6 +538,33 @@ class RemoteMethodReference(RemoteReference):
         methodSchema = None
         return interfaceName, methodName, methodSchema
 
+class LocalReferenceable:
+    implements(ipb.IRemoteReference)
+    def __init__(self, original):
+        self.original = original
+
+    def notifyOnDisconnect(self, callback, *args, **kwargs):
+        # local objects never disconnect
+        return None
+    def dontNotifyOnDisconnect(self, marker):
+        pass
+
+    def callRemote(self, methname, *args, **kwargs):
+        def _try(ignored):
+            meth = getattr(self.original, "remote_" + methname)
+            return meth(*args, **kwargs)
+        d = fireEventually()
+        d.addCallback(_try)
+        return d
+
+    def callRemoteOnly(self, methname, *args, **kwargs):
+        d = self.callRemote(methname, *args, **kwargs)
+        d.addErrback(lambda f: None)
+        return None
+
+registerAdapter(LocalReferenceable, ipb.IReferenceable, ipb.IRemoteReference)
+    
+
 
 class YourReferenceSlicer(slicer.BaseSlicer):
     """I handle pb.RemoteReference objects (being sent back home to the
@@ -635,11 +662,26 @@ class TheirReferenceUnslicer(slicer.LeafUnslicer):
         # but the message delivery must still wait for the getReference to
         # complete. See to it that we fire the object deferred before we fire
         # the ready_deferred.
-        obj_deferred, ready_deferred = defer.Deferred(), defer.Deferred()
+
+        obj_deferred = defer.Deferred()
+        ready_deferred = defer.Deferred()
+
         def _ready(rref):
             obj_deferred.callback(rref)
             ready_deferred.callback(rref)
-        d.addCallback(_ready)
+        def _failed(f):
+            # if an error in getReference() occurs, log it locally (with
+            # priority UNUSUAL), because this end might need to diagnose some
+            # connection or networking problems.
+            log.msg("gift (%s) failed to resolve: %s" % (self.url, f))
+            # deliver a placeholder object to the container, but signal the
+            # ready_deferred that we've failed. This will bubble up to the
+            # enclosing InboundDelivery, and when it gets to the top of the
+            # queue, it will be flunked.
+            obj_deferred.callback("Place holder for a Gift which failed to "
+                                  "resolve: %s" % f)
+            ready_deferred.errback(f)
+        d.addCallbacks(_ready, _failed)
 
         return obj_deferred, ready_deferred
 

src/foolscap/foolscap/slicer.py

@@ -1,6 +1,7 @@
 # -*- test-case-name: foolscap.test.test_banana -*-
 
 from twisted.python.components import registerAdapter
+from twisted.python import log
 from zope.interface import implements
 from twisted.internet.defer import Deferred
 import tokens
@@ -190,6 +191,11 @@ class BaseUnslicer:
         return self.open(opentype)
 
     def receiveChild(self, obj, ready_deferred=None):
+        """Unslicers for containers should accumulate their children's
+        ready_deferreds, then combine them in an AsyncAND when receiveClose()
+        happens, and return the AsyncAND as the ready_deferreds half of the
+        receiveClose() return value.
+        """
         pass
 
     def reportViolation(self, why):
@@ -221,16 +227,20 @@ class BaseUnslicer:
         return None
 
     def explode(self, failure):
-        """If something goes wrong in a Deferred callback, it may be too
+        """If something goes wrong in a Deferred callback, it may be too late
-        late to reject the token and to normal error handling. I haven't
+        to reject the token and to normal error handling. I haven't figured
-        figured out how to do sensible error-handling in this situation.
+        out how to do sensible error-handling in this situation. This method
-        This method exists to make sure that the exception shows up
+        exists to make sure that the exception shows up *somewhere*. If this
-        *somewhere*. If this is called, it is also likely that a placeholder
+        is called, it is also likely that a placeholder (probably a Deferred)
-        (probably a Deferred) will be left in the unserialized object about
+        will be left in the unserialized object graph about to be handed to
-        to be handed to the RootUnslicer.
+        the RootUnslicer.
         """
-        print "KABOOM"
+
-        print failure
+        # RootUnslicer pays attention to this .exploded attribute and refuses
+        # to deliver anything if it is set. But PBRootUnslicer ignores it.
+        # TODO: clean this up, and write some unit tests to trigger it (by
+        # violating schemas?)
+        log.msg("BaseUnslicer.explode: %s" % failure)
         self.protocol.exploded = failure
 
 class ScopedUnslicer(BaseUnslicer):

src/foolscap/foolscap/slicers/dict.py

@@ -1,10 +1,11 @@
 # -*- test-case-name: foolscap.test.test_banana -*-
 
 from twisted.python import log
-from twisted.internet.defer import Deferred, DeferredList
+from twisted.internet.defer import Deferred
 from foolscap.tokens import Violation, BananaError
 from foolscap.slicer import BaseSlicer, BaseUnslicer
 from foolscap.constraint import OpenerConstraint, Any, UnboundedSchema, IConstraint
+from foolscap.util import AsyncAND
 
 class DictSlicer(BaseSlicer):
     opentype = ('dict',)
@@ -105,7 +106,7 @@ class DictUnslicer(BaseUnslicer):
     def receiveClose(self):
         ready_deferred = None
         if self._ready_deferreds:
-            ready_deferred = DeferredList(self._ready_deferreds)
+            ready_deferred = AsyncAND(self._ready_deferreds)
         return self.d, ready_deferred
 
     def describe(self):

src/foolscap/foolscap/slicers/list.py

@@ -1,10 +1,11 @@
 # -*- test-case-name: foolscap.test.test_banana -*-
 
 from twisted.python import log
-from twisted.internet.defer import Deferred, DeferredList
+from twisted.internet.defer import Deferred
 from foolscap.tokens import Violation
 from foolscap.slicer import BaseSlicer, BaseUnslicer
 from foolscap.constraint import OpenerConstraint, Any, UnboundedSchema, IConstraint
+from foolscap.util import AsyncAND
 
 
 class ListSlicer(BaseSlicer):
@@ -105,7 +106,7 @@ class ListUnslicer(BaseUnslicer):
     def receiveClose(self):
         ready_deferred = None
         if self._ready_deferreds:
-            ready_deferred = DeferredList(self._ready_deferreds)
+            ready_deferred = AsyncAND(self._ready_deferreds)
         return self.list, ready_deferred
 
     def describe(self):

src/foolscap/foolscap/slicers/set.py

@@ -9,6 +9,7 @@ from foolscap.slicer import BaseUnslicer
 from foolscap.tokens import Violation
 from foolscap.constraint import OpenerConstraint, UnboundedSchema, Any, \
      IConstraint
+from foolscap.util import AsyncAND
 
 class SetSlicer(ListSlicer):
     opentype = ("set",)
@@ -136,7 +137,7 @@ class SetUnslicer(BaseUnslicer):
     def receiveClose(self):
         ready_deferred = None
         if self._ready_deferreds:
-            ready_deferred = defer.DeferredList(self._ready_deferreds)
+            ready_deferred = AsyncAND(self._ready_deferreds)
         return self.set, ready_deferred
 
 class FrozenSetUnslicer(TupleUnslicer):

src/foolscap/foolscap/slicers/tuple.py

@@ -1,10 +1,11 @@
 # -*- test-case-name: foolscap.test.test_banana -*-
 
-from twisted.internet.defer import Deferred, DeferredList
+from twisted.internet.defer import Deferred
 from foolscap.tokens import Violation
 from foolscap.slicer import BaseUnslicer
 from foolscap.slicers.list import ListSlicer
 from foolscap.constraint import OpenerConstraint, Any, UnboundedSchema, IConstraint
+from foolscap.util import AsyncAND
 
 
 class TupleSlicer(ListSlicer):
@@ -91,7 +92,7 @@ class TupleUnslicer(BaseUnslicer):
     def complete(self):
         ready_deferred = None
         if self._ready_deferreds:
-            ready_deferred = DeferredList(self._ready_deferreds)
+            ready_deferred = AsyncAND(self._ready_deferreds)
 
         t = tuple(self.list)
         if self.debug:
@@ -111,7 +112,7 @@ class TupleUnslicer(BaseUnslicer):
                 print " not finished yet"
             ready_deferred = None
             if self._ready_deferreds:
-                ready_deferred = DeferredList(self._ready_deferreds)
+                ready_deferred = AsyncAND(self._ready_deferreds)
             return self.deferred, ready_deferred
 
         # the list is already complete

src/foolscap/foolscap/test/common.py

@@ -266,6 +266,9 @@ class Target(Referenceable):
         return 24
     def remote_fail(self):
         raise ValueError("you asked me to fail")
+    def remote_fail_remotely(self, target):
+        return target.callRemote("fail")
+
     def remote_failstring(self):
         raise "string exceptions are annoying"
 

src/foolscap/foolscap/test/test_call.py

@@ -143,6 +143,20 @@ class TestCall(TargetMixin, unittest.TestCase):
         self.failUnless(f.check("string exceptions are annoying"),
                         "wrong exception type: %s" % f)
 
+    def testCopiedFailure(self):
+        # A calls B, who calls C. C fails. B gets a CopiedFailure and reports
+        # it back to A. What does a get?
+        rr, target = self.setupTarget(TargetWithoutInterfaces())
+        d = rr.callRemote("fail_remotely", target)
+        def _check(f):
+            # f should be a CopiedFailure
+            self.failUnless(isinstance(f, failure.Failure),
+                            "Hey, we didn't fail: %s" % f)
+            self.failUnless(f.check(ValueError),
+                            "wrong exception type: %s" % f)
+            self.failUnlessSubstring("you asked me to fail", f.value)
+        d.addBoth(_check)
+        return d
 
     def testCall2(self):
         # server end uses an interface this time, but not the client end

src/foolscap/foolscap/test/test_copyable.py

@@ -1,6 +1,6 @@
 
 from twisted.trial import unittest
-from twisted.python import components, failure
+from twisted.python import components, failure, reflect
 from foolscap.test.common import TargetMixin, HelperTarget
 
 from foolscap import copyable, tokens
@@ -121,13 +121,15 @@ class Copyable(TargetMixin, unittest.TestCase):
     def _testFailure1_1(self, (f,)):
         #print "CopiedFailure is:", f
         #print f.__dict__
-        self.failUnlessEqual(f.type, "exceptions.RuntimeError")
+        self.failUnlessEqual(reflect.qual(f.type), "exceptions.RuntimeError")
+        self.failUnless(f.check, RuntimeError)
         self.failUnlessEqual(f.value, "message here")
         self.failUnlessEqual(f.frames, [])
         self.failUnlessEqual(f.tb, None)
         self.failUnlessEqual(f.stack, [])
         # there should be a traceback
-        self.failUnless(f.traceback.find("raise RuntimeError") != -1)
+        self.failUnless(f.traceback.find("raise RuntimeError") != -1,
+                        "no 'raise RuntimeError' in '%s'" % (f.traceback,))
 
     def testFailure2(self):
         self.callingBroker.unsafeTracebacks = False
@@ -141,7 +143,8 @@ class Copyable(TargetMixin, unittest.TestCase):
     def _testFailure2_1(self, (f,)):
         #print "CopiedFailure is:", f
         #print f.__dict__
-        self.failUnlessEqual(f.type, "exceptions.RuntimeError")
+        self.failUnlessEqual(reflect.qual(f.type), "exceptions.RuntimeError")
+        self.failUnless(f.check, RuntimeError)
         self.failUnlessEqual(f.value, "message here")
         self.failUnlessEqual(f.frames, [])
         self.failUnlessEqual(f.tb, None)

src/foolscap/foolscap/test/test_gifts.py

@@ -1,12 +1,15 @@
 
 from zope.interface import implements
 from twisted.trial import unittest
-from twisted.internet import defer
+from twisted.internet import defer, protocol, reactor
-from twisted.internet.error import ConnectionDone, ConnectionLost
+from twisted.internet.error import ConnectionDone, ConnectionLost, \
+     ConnectionRefusedError
+from twisted.python import failure
 from foolscap import Tub, UnauthenticatedTub, RemoteInterface, Referenceable
-from foolscap.referenceable import RemoteReference
+from foolscap.referenceable import RemoteReference, SturdyRef
 from foolscap.test.common import HelperTarget, RIHelper
 from foolscap.eventual import flushEventualQueue
+from foolscap.tokens import BananaError, NegotiationError
 
 crypto_available = False
 try:
@@ -38,18 +41,13 @@ class ConstrainedHelper(Referenceable):
     def remote_set(self, obj):
         self.obj = obj
 
-class Gifts(unittest.TestCase):
+class Base:
-    # Here we test the three-party introduction process as depicted in the
-    # classic Granovetter diagram. Alice has a reference to Bob and another
-    # one to Carol. Alice wants to give her Carol-reference to Bob, by
-    # including it as the argument to a method she invokes on her
-    # Bob-reference.
 
     debug = False
 
     def setUp(self):
-        self.services = [GoodEnoughTub(), GoodEnoughTub(), GoodEnoughTub()]
+        self.services = [GoodEnoughTub() for i in range(4)]
-        self.tubA, self.tubB, self.tubC = self.services
+        self.tubA, self.tubB, self.tubC, self.tubD = self.services
         for s in self.services:
             s.startService()
             l = s.listenOn("tcp:0:interface=127.0.0.1")
@@ -63,9 +61,9 @@ class Gifts(unittest.TestCase):
     def createCharacters(self):
         self.alice = HelperTarget("alice")
         self.bob = HelperTarget("bob")
-        self.bob_url = self.tubB.registerReference(self.bob)
+        self.bob_url = self.tubB.registerReference(self.bob, "bob")
         self.carol = HelperTarget("carol")
-        self.carol_url = self.tubC.registerReference(self.carol)
+        self.carol_url = self.tubC.registerReference(self.carol, "carol")
         # cindy is Carol's little sister. She doesn't have a phone, but
         # Carol might talk about her anyway.
         self.cindy = HelperTarget("cindy")
@@ -75,6 +73,8 @@ class Gifts(unittest.TestCase):
         self.clarisse = HelperTarget("clarisse")
         self.colette = HelperTarget("colette")
         self.courtney = HelperTarget("courtney")
+        self.dave = HelperTarget("dave")
+        self.dave_url = self.tubD.registerReference(self.dave, "dave")
 
     def createInitialReferences(self):
         # we must start by giving Alice a reference to both Bob and Carol.
@@ -90,49 +90,80 @@ class Gifts(unittest.TestCase):
         def _aliceGotCarol(acarol):
             if self.debug: print "Alice got carol"
             self.acarol = acarol # Alice's reference to Carol
+            d = self.tubB.getReference(self.dave_url)
+            return d
         d.addCallback(_aliceGotCarol)
+        def _bobGotDave(bdave):
+            self.bdave = bdave
+        d.addCallback(_bobGotDave)
         return d
 
     def createMoreReferences(self):
         # give Alice references to Carol's sisters
         dl = []
 
-        url = self.tubC.registerReference(self.charlene)
+        url = self.tubC.registerReference(self.charlene, "charlene")
         d = self.tubA.getReference(url)
         def _got_charlene(rref):
             self.acharlene = rref
         d.addCallback(_got_charlene)
         dl.append(d)
 
-        url = self.tubC.registerReference(self.christine)
+        url = self.tubC.registerReference(self.christine, "christine")
         d = self.tubA.getReference(url)
         def _got_christine(rref):
             self.achristine = rref
         d.addCallback(_got_christine)
         dl.append(d)
 
-        url = self.tubC.registerReference(self.clarisse)
+        url = self.tubC.registerReference(self.clarisse, "clarisse")
         d = self.tubA.getReference(url)
         def _got_clarisse(rref):
             self.aclarisse = rref
         d.addCallback(_got_clarisse)
         dl.append(d)
 
-        url = self.tubC.registerReference(self.colette)
+        url = self.tubC.registerReference(self.colette, "colette")
         d = self.tubA.getReference(url)
         def _got_colette(rref):
             self.acolette = rref
         d.addCallback(_got_colette)
         dl.append(d)
 
-        url = self.tubC.registerReference(self.courtney)
+        url = self.tubC.registerReference(self.courtney, "courtney")
         d = self.tubA.getReference(url)
         def _got_courtney(rref):
             self.acourtney = rref
         d.addCallback(_got_courtney)
         dl.append(d)
+
         return defer.DeferredList(dl)
 
+    def shouldFail(self, res, expected_failure, which, substring=None):
+        # attach this with:
+        #  d = something()
+        #  d.addBoth(self.shouldFail, IndexError, "something")
+        # the 'which' string helps to identify which call to shouldFail was
+        # triggered, since certain versions of Twisted don't display this
+        # very well.
+
+        if isinstance(res, failure.Failure):
+            res.trap(expected_failure)
+            if substring:
+                self.failUnless(substring in str(res),
+                                "substring '%s' not in '%s'"
+                                % (substring, str(res)))
+        else:
+            self.fail("%s was supposed to raise %s, not get '%s'" %
+                      (which, expected_failure, res))
+
+class Gifts(Base, unittest.TestCase):
+    # Here we test the three-party introduction process as depicted in the
+    # classic Granovetter diagram. Alice has a reference to Bob and another
+    # one to Carol. Alice wants to give her Carol-reference to Bob, by
+    # including it as the argument to a method she invokes on her
+    # Bob-reference.
+
     def testGift(self):
         #defer.setDebugging(True)
         self.createCharacters()
@@ -164,7 +195,6 @@ class Gifts(unittest.TestCase):
         d.addCallback(_carolCalled)
         return d
 
-
     def testImplicitGift(self):
         # in this test, Carol was registered in her Tub (using
         # registerReference), but Cindy was not. Alice is given a reference
@@ -226,6 +256,42 @@ class Gifts(unittest.TestCase):
         d.addCallback(_carolAndCindyCalled)
         return d
 
+    # test gifts in return values too
+
+    def testReturn(self):
+        self.createCharacters()
+        d = self.createInitialReferences()
+        def _introduce(res):
+            self.bob.obj = self.bdave
+            return self.abob.callRemote("get")
+        d.addCallback(_introduce)
+        def _check(adave):
+            # this ought to be a RemoteReference to dave, usable by alice
+            self.failUnless(isinstance(adave, RemoteReference))
+            return adave.callRemote("set", 12)
+        d.addCallback(_check)
+        def _check2(res):
+            self.failUnlessEqual(self.dave.obj, 12)
+        d.addCallback(_check2)
+        return d
+
+    def testReturnInContainer(self):
+        self.createCharacters()
+        d = self.createInitialReferences()
+        def _introduce(res):
+            self.bob.obj = {"foo": [(set([self.bdave]),)]}
+            return self.abob.callRemote("get")
+        d.addCallback(_introduce)
+        def _check(obj):
+            adave = list(obj["foo"][0][0])[0]
+            # this ought to be a RemoteReference to dave, usable by alice
+            self.failUnless(isinstance(adave, RemoteReference))
+            return adave.callRemote("set", 12)
+        d.addCallback(_check)
+        def _check2(res):
+            self.failUnlessEqual(self.dave.obj, 12)
+        d.addCallback(_check2)
+        return d
 
     def testOrdering(self):
         self.createCharacters()
@@ -303,9 +369,11 @@ class Gifts(unittest.TestCase):
     def create_constrained_characters(self):
         self.alice = HelperTarget("alice")
         self.bob = ConstrainedHelper("bob")
-        self.bob_url = self.tubB.registerReference(self.bob)
+        self.bob_url = self.tubB.registerReference(self.bob, "bob")
         self.carol = HelperTarget("carol")
-        self.carol_url = self.tubC.registerReference(self.carol)
+        self.carol_url = self.tubC.registerReference(self.carol, "carol")
+        self.dave = HelperTarget("dave")
+        self.dave_url = self.tubD.registerReference(self.dave, "dave")
 
     def test_constraint(self):
         self.create_constrained_characters()
@@ -319,6 +387,8 @@ class Gifts(unittest.TestCase):
         d.addCallback(_checkBob)
         return d
 
+
+
     # this was used to alice's reference to carol (self.acarol) appeared in
     # alice's gift table at the right time, to make sure that the
     # RemoteReference is kept alive while the gift is in transit. The whole
@@ -359,3 +429,127 @@ class Gifts(unittest.TestCase):
         d.addCallback(lambda res: d1)
         return d
 
+
+class Bad(Base, unittest.TestCase):
+
+    # if the recipient cannot claim their gift, the caller should see an
+    # errback.
+
+    def setUp(self):
+        if not crypto_available:
+            raise unittest.SkipTest("crypto not available")
+        Base.setUp(self)
+
+    def test_swissnum(self):
+        self.createCharacters()
+        d = self.createInitialReferences()
+        d.addCallback(lambda res: self.tubA.getReference(self.dave_url))
+        def _introduce(adave):
+            # now break the gift to insure that Bob is unable to claim it.
+            # The first way to do this is to simple mangle the swissnum,
+            # which will result in a failure in remote_getReferenceByName.
+            # NOTE: this will have to change when we modify the way gifts are
+            # referenced, since tracker.url is scheduled to go away.
+            r = SturdyRef(adave.tracker.url)
+            r.name += ".MANGLED"
+            adave.tracker.url = r.getURL()
+            return self.acarol.callRemote("set", adave)
+        d.addCallback(_introduce)
+        d.addBoth(self.shouldFail, KeyError, "Bad.test_swissnum")
+        # make sure we can still talk to Carol, though
+        d.addCallback(lambda res: self.acarol.callRemote("set", 14))
+        d.addCallback(lambda res: self.failUnlessEqual(self.carol.obj, 14))
+        return d
+    test_swissnum.timeout = 10
+
+    def test_tubid(self):
+        self.createCharacters()
+        d = self.createInitialReferences()
+        d.addCallback(lambda res: self.tubA.getReference(self.dave_url))
+        def _introduce(adave):
+            # The second way is to mangle the tubid, which will result in a
+            # failure during negotiation. NOTE: this will have to change when
+            # we modify the way gifts are referenced, since tracker.url is
+            # scheduled to go away.
+            r = SturdyRef(adave.tracker.url)
+            r.tubID += ".MANGLED"
+            adave.tracker.url = r.getURL()
+            return self.acarol.callRemote("set", adave)
+        d.addCallback(_introduce)
+        d.addBoth(self.shouldFail, BananaError, "Bad.test_tubid",
+                  "unknown TubID")
+        return d
+    test_tubid.timeout = 10
+
+    def test_location(self):
+        self.createCharacters()
+        d = self.createInitialReferences()
+        d.addCallback(lambda res: self.tubA.getReference(self.dave_url))
+        def _introduce(adave):
+            # The third way is to mangle the location hints, which will
+            # result in a failure during negotiation as it attempts to
+            # establish a TCP connection.
+            r = SturdyRef(adave.tracker.url)
+            # highly unlikely that there's anything listening on this port
+            r.locationHints = ["127.0.0.47:1"]
+            adave.tracker.url = r.getURL()
+            return self.acarol.callRemote("set", adave)
+        d.addCallback(_introduce)
+        d.addBoth(self.shouldFail, ConnectionRefusedError, "Bad.test_location")
+        return d
+    test_location.timeout = 10
+
+    def test_hang(self):
+        f = protocol.Factory()
+        f.protocol = protocol.Protocol # ignores all input
+        p = reactor.listenTCP(0, f, interface="127.0.0.1")
+        self.createCharacters()
+        d = self.createInitialReferences()
+        d.addCallback(lambda res: self.tubA.getReference(self.dave_url))
+        def _introduce(adave):
+            # The next form of mangling is to connect to a port which never
+            # responds, which could happen if a firewall were silently
+            # dropping the TCP packets. We can't accurately simulate this
+            # case, but we can connect to a port which accepts the connection
+            # and then stays silent. This should trigger the overall
+            # connection timeout.
+            r = SturdyRef(adave.tracker.url)
+            r.locationHints = ["127.0.0.1:%d" % p.getHost().port]
+            adave.tracker.url = r.getURL()
+            self.tubD.options['connect_timeout'] = 2
+            return self.acarol.callRemote("set", adave)
+        d.addCallback(_introduce)
+        d.addBoth(self.shouldFail, NegotiationError, "Bad.test_hang",
+                  "no connection established within client timeout")
+        def _stop_listening(res):
+            d1 = p.stopListening()
+            def _done_listening(x):
+                return res
+            d1.addCallback(_done_listening)
+            return d1
+        d.addBoth(_stop_listening)
+        return d
+    test_hang.timeout = 10
+    
+
+    def testReturn_swissnum(self):
+        self.createCharacters()
+        d = self.createInitialReferences()
+        def _introduce(res):
+            # now break the gift to insure that Alice is unable to claim it.
+            # The first way to do this is to simple mangle the swissnum,
+            # which will result in a failure in remote_getReferenceByName.
+            # NOTE: this will have to change when we modify the way gifts are
+            # referenced, since tracker.url is scheduled to go away.
+            r = SturdyRef(self.bdave.tracker.url)
+            r.name += ".MANGLED"
+            self.bdave.tracker.url = r.getURL()
+            self.bob.obj = self.bdave
+            return self.abob.callRemote("get")
+        d.addCallback(_introduce)
+        d.addBoth(self.shouldFail, KeyError, "Bad.testReturn_swissnum")
+        # make sure we can still talk to Bob, though
+        d.addCallback(lambda res: self.abob.callRemote("set", 14))
+        d.addCallback(lambda res: self.failUnlessEqual(self.bob.obj, 14))
+        return d
+    testReturn_swissnum.timeout = 10

src/foolscap/foolscap/test/test_interfaces.py

@@ -121,7 +121,7 @@ class TestInterface(TargetMixin, unittest.TestCase):
             for i in range(len(s)):
                 line = s[i]
                 #print line
-                if ("test/test_interfaces.py" in line
+                if ("test_interfaces.py" in line
                     and i+1 < len(s)
                     and "rr.callRemote" in s[i+1]):
                     return # all good

src/foolscap/foolscap/test/test_pb.py

@@ -7,7 +7,7 @@ if False:
     from twisted.python import log
     log.startLogging(sys.stderr)
 
-from twisted.python import failure, log
+from twisted.python import failure, log, reflect
 from twisted.internet import defer
 from twisted.trial import unittest
 
@@ -117,6 +117,7 @@ class TestAnswer(unittest.TestCase):
         req = TestRequest(12)
         self.broker.addRequest(req)
         u = self.newUnslicer()
+        u.start(0)
         u.checkToken(INT, 0)
         u.receiveChild(12) # causes broker.getRequest
         u.checkToken(STRING, 8)
@@ -130,6 +131,7 @@ class TestAnswer(unittest.TestCase):
         req.setConstraint(IConstraint(str))
         self.broker.addRequest(req)
         u = self.newUnslicer()
+        u.start(0)
         u.checkToken(INT, 0)
         u.receiveChild(12) # causes broker.getRequest
         u.checkToken(STRING, 15)
@@ -617,7 +619,7 @@ class TestService(unittest.TestCase):
         return d
     testBadMethod2.timeout = 5
     def _testBadMethod2_eb(self, f):
-        self.failUnlessEqual(f.type, 'exceptions.AttributeError')
+        self.failUnlessEqual(reflect.qual(f.type), 'exceptions.AttributeError')
         self.failUnlessSubstring("TargetWithoutInterfaces", f.value)
         self.failUnlessSubstring(" has no attribute 'remote_missing'", f.value)
 

src/foolscap/foolscap/test/test_reference.py

@@ -0,0 +1,71 @@
+
+from zope.interface import implements
+from twisted.trial import unittest
+from twisted.python import failure
+from foolscap.ipb import IRemoteReference
+from foolscap.test.common import HelperTarget, Target
+from foolscap.eventual import flushEventualQueue
+
+class Remote:
+    implements(IRemoteReference)
+    pass
+
+
+class LocalReference(unittest.TestCase):
+    def tearDown(self):
+        return flushEventualQueue()
+
+    def ignored(self):
+        pass
+
+    def test_remoteReference(self):
+        r = Remote()
+        rref = IRemoteReference(r)
+        self.failUnlessIdentical(r, rref)
+
+    def test_callRemote(self):
+        t = HelperTarget()
+        t.obj = None
+        rref = IRemoteReference(t)
+        marker = rref.notifyOnDisconnect(self.ignored, "args", kwargs="foo")
+        rref.dontNotifyOnDisconnect(marker)
+        d = rref.callRemote("set", 12)
+        # the callRemote should be put behind an eventual-send
+        self.failUnlessEqual(t.obj, None)
+        def _check(res):
+            self.failUnlessEqual(t.obj, 12)
+            self.failUnlessEqual(res, True)
+        d.addCallback(_check)
+        return d
+
+    def test_callRemoteOnly(self):
+        t = HelperTarget()
+        t.obj = None
+        rref = IRemoteReference(t)
+        rc = rref.callRemoteOnly("set", 12)
+        self.failUnlessEqual(rc, None)
+
+    def shouldFail(self, res, expected_failure, which, substring=None):
+        # attach this with:
+        #  d = something()
+        #  d.addBoth(self.shouldFail, IndexError, "something")
+        # the 'which' string helps to identify which call to shouldFail was
+        # triggered, since certain versions of Twisted don't display this
+        # very well.
+
+        if isinstance(res, failure.Failure):
+            res.trap(expected_failure)
+            if substring:
+                self.failUnless(substring in str(res),
+                                "substring '%s' not in '%s'"
+                                % (substring, str(res)))
+        else:
+            self.fail("%s was supposed to raise %s, not get '%s'" %
+                      (which, expected_failure, res))
+
+    def test_fail(self):
+        t = Target()
+        d = IRemoteReference(t).callRemote("fail")
+        d.addBoth(self.shouldFail, ValueError, "test_fail",
+                  "you asked me to fail")
+        return d

src/foolscap/foolscap/test/test_tub.py

@@ -11,7 +11,7 @@ try:
 except ImportError:
     pass
 
-from foolscap import Tub, UnauthenticatedTub
+from foolscap import Tub, UnauthenticatedTub, SturdyRef, Referenceable
 from foolscap.referenceable import RemoteReference
 from foolscap.eventual import eventually, flushEventualQueue
 from foolscap.test.common import HelperTarget, TargetMixin
@@ -117,3 +117,93 @@ class QueuedStartup(TargetMixin, unittest.TestCase):
         eventually(t1.startService)
         return d
 
+
+class NameLookup(TargetMixin, unittest.TestCase):
+
+    # test registerNameLookupHandler
+
+    def setUp(self):
+        TargetMixin.setUp(self)
+        self.tubA, self.tubB = [GoodEnoughTub(), GoodEnoughTub()]
+        self.services = [self.tubA, self.tubB]
+        self.tubA.startService()
+        self.tubB.startService()
+        l = self.tubB.listenOn("tcp:0:interface=127.0.0.1")
+        self.tubB.setLocation("127.0.0.1:%d" % l.getPortnum())
+        self.url_on_b = self.tubB.registerReference(Referenceable())
+        self.lookups = []
+        self.lookups2 = []
+        self.names = {}
+        self.names2 = {}
+
+    def tearDown(self):
+        d = TargetMixin.tearDown(self)
+        def _more(res):
+            return defer.DeferredList([s.stopService() for s in self.services])
+        d.addCallback(_more)
+        d.addCallback(flushEventualQueue)
+        return d
+
+    def lookup(self, name):
+        self.lookups.append(name)
+        return self.names.get(name, None)
+
+    def lookup2(self, name):
+        self.lookups2.append(name)
+        return self.names2.get(name, None)
+
+    def testNameLookup(self):
+        t1 = HelperTarget()
+        t2 = HelperTarget()
+        self.names["foo"] = t1
+        self.names2["bar"] = t2
+        self.names2["baz"] = t2
+        self.tubB.registerNameLookupHandler(self.lookup)
+        self.tubB.registerNameLookupHandler(self.lookup2)
+        # hack up a new furl pointing at the same tub but with a name that
+        # hasn't been registered.
+        s = SturdyRef(self.url_on_b)
+        s.name = "foo"
+
+        d = self.tubA.getReference(s)
+
+        def _check(res):
+            self.failUnless(isinstance(res, RemoteReference))
+            self.failUnlessEqual(self.lookups, ["foo"])
+            # the first lookup should short-circuit the process
+            self.failUnlessEqual(self.lookups2, [])
+            self.lookups = []; self.lookups2 = []
+            s.name = "bar"
+            return self.tubA.getReference(s)
+        d.addCallback(_check)
+
+        def _check2(res):
+            self.failUnless(isinstance(res, RemoteReference))
+            # if the first lookup fails, the second handler should be asked
+            self.failUnlessEqual(self.lookups, ["bar"])
+            self.failUnlessEqual(self.lookups2, ["bar"])
+            self.lookups = []; self.lookups2 = []
+            # make sure that loopbacks use this too
+            return self.tubB.getReference(s)
+        d.addCallback(_check2)
+
+        def _check3(res):
+            self.failUnless(isinstance(res, RemoteReference))
+            self.failUnlessEqual(self.lookups, ["bar"])
+            self.failUnlessEqual(self.lookups2, ["bar"])
+            self.lookups = []; self.lookups2 = []
+            # and make sure we can de-register handlers
+            self.tubB.unregisterNameLookupHandler(self.lookup)
+            s.name = "baz"
+            return self.tubA.getReference(s)
+        d.addCallback(_check3)
+
+        def _check4(res):
+            self.failUnless(isinstance(res, RemoteReference))
+            self.failUnlessEqual(self.lookups, [])
+            self.failUnlessEqual(self.lookups2, ["baz"])
+            self.lookups = []; self.lookups2 = []
+        d.addCallback(_check4)
+
+        return d
+

src/foolscap/foolscap/test/test_util.py

@@ -0,0 +1,92 @@
+
+from twisted.trial import unittest
+from twisted.internet import defer
+from twisted.python import failure
+from foolscap import util, eventual
+
+
+class AsyncAND(unittest.TestCase):
+    def setUp(self):
+        self.fired = False
+        self.failed = False
+
+    def callback(self, res):
+        self.fired = True
+    def errback(self, res):
+        self.failed = True
+
+    def attach(self, d):
+        d.addCallbacks(self.callback, self.errback)
+        return d
+
+    def shouldNotFire(self, ignored=None):
+        self.failIf(self.fired)
+        self.failIf(self.failed)
+    def shouldFire(self, ignored=None):
+        self.failUnless(self.fired)
+        self.failIf(self.failed)
+    def shouldFail(self, ignored=None):
+        self.failUnless(self.failed)
+        self.failIf(self.fired)
+
+    def tearDown(self):
+        return eventual.flushEventualQueue()
+
+    def test_empty(self):
+        self.attach(util.AsyncAND([]))
+        self.shouldFire()
+
+    def test_simple(self):
+        d1 = eventual.fireEventually(None)
+        a = util.AsyncAND([d1])
+        self.attach(a)
+        a.addBoth(self.shouldFire)
+        return a
+
+    def test_two(self):
+        d1 = defer.Deferred()
+        d2 = defer.Deferred()
+        self.attach(util.AsyncAND([d1, d2]))
+        self.shouldNotFire()
+        d1.callback(1)
+        self.shouldNotFire()
+        d2.callback(2)
+        self.shouldFire()
+
+    def test_one_failure_1(self):
+        d1 = defer.Deferred()
+        d2 = defer.Deferred()
+        self.attach(util.AsyncAND([d1, d2]))
+        self.shouldNotFire()
+        d1.callback(1)
+        self.shouldNotFire()
+        d2.errback(RuntimeError())
+        self.shouldFail()
+
+    def test_one_failure_2(self):
+        d1 = defer.Deferred()
+        d2 = defer.Deferred()
+        self.attach(util.AsyncAND([d1, d2]))
+        self.shouldNotFire()
+        d1.errback(RuntimeError())
+        self.shouldFail()
+        d2.callback(1)
+        self.shouldFail()
+
+    def test_two_failure(self):
+        d1 = defer.Deferred()
+        d2 = defer.Deferred()
+        self.attach(util.AsyncAND([d1, d2]))
+        def _should_fire(res):
+            self.failIf(isinstance(res, failure.Failure))
+        def _should_fail(f):
+            self.failUnless(isinstance(f, failure.Failure))
+        d1.addBoth(_should_fire)
+        d2.addBoth(_should_fail)
+        self.shouldNotFire()
+        d1.errback(RuntimeError())
+        self.shouldFail()
+        d2.errback(RuntimeError())
+        self.shouldFail()
+
+        

src/foolscap/foolscap/util.py

@@ -0,0 +1,52 @@
+
+from twisted.internet import defer
+
+
+class AsyncAND(defer.Deferred):
+    """Like DeferredList, but results are discarded and failures handled
+    in a more convenient fashion.
+
+    Create me with a list of Deferreds. I will fire my callback (with None)
+    if and when all of my component Deferreds fire successfully. I will fire
+    my errback when and if any of my component Deferreds errbacks, in which
+    case I will absorb the failure. If a second Deferred errbacks, I will not
+    absorb that failure.
+
+    This means that you can put a bunch of Deferreds together into an
+    AsyncAND and then forget about them. If all succeed, the AsyncAND will
+    fire. If one fails, that Failure will be propagated to the AsyncAND. If
+    multiple ones fail, the first Failure will go to the AsyncAND and the
+    rest will be left unhandled (and therefore logged).
+    """
+
+    def __init__(self, deferredList):
+        defer.Deferred.__init__(self)
+
+        if not deferredList:
+            self.callback(None)
+            return
+
+        self.remaining = len(deferredList)
+        self._fired = False
+
+        for d in deferredList:
+            d.addCallbacks(self._cbDeferred, self._cbDeferred,
+                           callbackArgs=(True,), errbackArgs=(False,))
+
+    def _cbDeferred(self, result, succeeded):
+        self.remaining -= 1
+        if succeeded:
+            if not self._fired and self.remaining == 0:
+                # the last input has fired. We fire.
+                self._fired = True
+                self.callback(None)
+                return
+        else:
+            if not self._fired:
+                # the first Failure is carried into our output
+                self._fired = True
+                self.errback(result)
+                return None
+            else:
+                # second and later Failures are not absorbed
+                return result

src/foolscap/misc/dapper/debian/changelog

@@ -1,3 +1,9 @@
+foolscap (0.1.5) unstable; urgency=low
+
+  * new release
+
+ -- Brian Warner <warner@lothar.com>  Tue, 07 Aug 2007 17:47:53 -0700
+
 foolscap (0.1.4) unstable; urgency=low
 
   * new release

src/foolscap/misc/edgy/debian/changelog

@@ -1,3 +1,9 @@
+foolscap (0.1.5) unstable; urgency=low
+
+  * new release
+
+ -- Brian Warner <warner@lothar.com>  Tue, 07 Aug 2007 17:47:53 -0700
+
 foolscap (0.1.4) unstable; urgency=low
 
   * new release

src/foolscap/misc/feisty/debian/changelog

@@ -1,3 +1,9 @@
+foolscap (0.1.5) unstable; urgency=low
+
+  * new release
+
+ -- Brian Warner <warner@lothar.com>  Tue, 07 Aug 2007 17:47:53 -0700
+
 foolscap (0.1.4) unstable; urgency=low
 
   * new release

src/foolscap/misc/sarge/debian/changelog

@@ -1,3 +1,9 @@
+foolscap (0.1.5) unstable; urgency=low
+
+  * new release
+
+ -- Brian Warner <warner@lothar.com>  Tue, 07 Aug 2007 17:47:53 -0700
+
 foolscap (0.1.4) unstable; urgency=low
 
   * new release

src/foolscap/misc/sid/debian/changelog

@@ -1,3 +1,9 @@
+foolscap (0.1.5) unstable; urgency=low
+
+  * new release
+
+ -- Brian Warner <warner@lothar.com>  Tue, 07 Aug 2007 17:47:53 -0700
+
 foolscap (0.1.4) unstable; urgency=low
 
   * new release