0.3.0: BREAKS PROTOCOL BACKWARD COMPATIBILITY

This version is not compatible with versions prior to 0.3.0, so
'git pull' and restart if you are following the alpha.

Changes from 0.2.5:

 - All multicast frames are now signed by the original sender. This
   will permit very efficient and fault tolerant rate limitation
   across networks, and imposes a kind of "hash cash" cost on those
   who wish to flood the network by forcing them to keep regenerating
   new identities.

 - Simplified peer last unicast / last multicast accounting.

 - Improvements to multicast propagation algorithm to achieve better
   coverage with less redundant messages.

 - The bloated Switch class went on a diet, having packet decoding
   broken out into PacketDecoder and multicast propagation algorithm
   broken out into Multicaster.

 - Multicaster is implemented as a template mockable class to permit
   future simulations of huge scale multicast using the actual code
   instead of mockups in another language.

 - Introduced a faster non-cryptographic random source for things
   like multicast propagation and address choosing.

 - Some code cleanup, removal of outdated comments, etc.

AUTHORS.txt

@@ -1,6 +1,6 @@
 This file will track authors and contributors to ZeroTier's code base.
-People who contribute code that is accepted into the master (trunk) branch
-will be credited here.
+People who contribute new code or bug fixes that are accepted into
+the master (trunk) branch will be credited here.
 
 --
 

BUILDING.txt

@@ -2,6 +2,10 @@ Building ZeroTier One on different platforms:
 
 (See RUNNING.txt for what to do next.)
 
+Developers note: there is currently no management of dependencies on *nix
+platforms, so you should make clean ; make if you change a header. Will
+do this eventually.
+
 -- MacOS
 
 make -f Makefile.mac

Makefile.linux

@@ -6,7 +6,7 @@ ARCH=$(shell uname -m)
 DEFS=-DZT_ARCH="$(ARCH)" -DZT_OSNAME="linux" -DZT_TRACE
 
 # Uncomment for a release optimized build
-CFLAGS=-Wall -O6 -fno-unroll-loops -pthread $(INCLUDES) -DNDEBUG $(DEFS)
+CFLAGS=-Wall -O3 -fno-unroll-loops -fstack-protector -pthread $(INCLUDES) -DNDEBUG $(DEFS)
 STRIP=strip --strip-all
 
 # Uncomment for a debug build

Makefile.mac

@@ -5,7 +5,7 @@ INCLUDES=-Iext/bin/libcrypto/include -Iext/jsoncpp/include
 DEFS=-DZT_ARCH="x86_combined" -DZT_OSNAME="mac" -DZT_TRACE
 
 # Uncomment for a release optimized universal binary build
-CFLAGS=-arch i386 -arch x86_64 -Wall -O6 -ftree-vectorize -pthread -mmacosx-version-min=10.6 -DNDEBUG $(INCLUDES) $(DEFS)
+CFLAGS=-arch i386 -arch x86_64 -Wall -O3 -ftree-vectorize -fstack-protector -pthread -mmacosx-version-min=10.6 -DNDEBUG $(INCLUDES) $(DEFS)
 STRIP=strip
 
 # Uncomment for a debug build

README.md

@@ -5,7 +5,7 @@ ZeroTier One creates flat virtual Ethernet networks of almost unlimited size. [V
 
 This code is presently in **ALPHA** testing. That means that the protocol spec may change in incompatible ways, and it certainly has bugs. Testers should "git pull," rebuild, and restart fairly often. If things mysteriously stop working, do that.
 
-See BUILDING.txt and RUNNING.txt for instructions. It currently builds on Mac and Linux. A Windows port is coming soon. Nice packages/installers and auto-update is also coming when alpha transitions to beta.
+See BUILDING.txt and RUNNING.txt for instructions. It currently builds on Mac and Linux. A Windows port is coming soon. Nice packages/installers and auto-update are also coming when alpha transitions to beta.
 
 Note that this won't work if your firewall does not allow outbound UDP. It must allow UDP conversations on port 8993 at a minimum.
 
@@ -13,7 +13,7 @@ At present there is only one virtual LAN and you are dumped there by default. It
 
 **Security warning:** You read that right. ZeroTier One places your computer on an absolutely open global Ethernet party line. Please ensure that you are up to date on your OS patches and we recommend turning off unnecessary services. Also be sure that anything else you are sharing is password protected provided you don't want to share it: printers, iPhoto and iTunes shares, etc.
 
-ZeroTier One is licensed under the GNU General Public License version 3. Anyone wishing to embed this in a commercial product or create a derivative product should contact [ZeroTier Networks LLC](https://www.zerotier.com/) to obtain a commercial license.
+ZeroTier One is licensed under the GNU General Public License version 3. You are free to use, modify, or redistribute it under the terms of that license. If you would like to embed ZeroTier One in a closed source product or create a closed source derivative product, contact ZeroTier Networks LLC.
 
 Check out the [blog](http://blog.zerotier.com/) for announcements, in-depth articles, and related thoughts. There is also a [Google group](https://groups.google.com/forum/#!forum/zerotier-one-users) for questions and discussion.
 
@@ -21,16 +21,19 @@ Check out the [blog](http://blog.zerotier.com/) for announcements, in-depth arti
 **FAQ**
 
 **Q:** What can I do with this?  
-**A:** For starters, try opening iTunes if you have it installed. If others are also online and sharing their collections, you might see them. If you have any games that run over a LAN (except those that require IPX), try those. What else can you think of to do on a completely flat, open network? Collaborative software development? Remote debugging? Transferring files using simple drive shares? Sharing your desktop printer to someone on another continent? Use your imagination.
+**A:** For starters, try opening iTunes if you have it installed. If others are also online and sharing their collections, you might see them. If you have any games that run over a LAN (except those that require IPX), try those. What else can you think of to do on a completely flat, open network? Games? Collaborative software development? Remote debugging? Transferring files using simple drive shares? Sharing your desktop printer to someone on another continent? Use your imagination.
 
 **Q:** Why do I get an IP address in the 27.0.0.0 or 28.0.0.0 range? And why does a lookup claim these addresses belong to the U.S. Department of Defense?  
-**A:** Short answer: because IPv4 needs to die. Long answer: the Earth network assigns IPv4 IPs from these ranges. They do in fact belong to the DOD, but they are *not* routed to the open Internet. The DOD owns them but uses them internally for private networks. As a result, there is nothing *technically* wrong with "bogarting" these for our own private network. It's considered bad practice, but if you want a private address space in IPv4 that is unlikely to overlap other private address spaces (like 10/8 and 192.168/16), it's the only way. [Cellular carriers](http://www.androidcentral.com/sprint-internet-dept-defense-and-you)  and [cable companies](http://www.dslreports/forum/r25679029-Why-is-my-first-hop-to-a-DoD-assigned-IP-address-) frequently do the same thing.
+**A:** Short answer: because IPv4 needs to die. Long answer: the Earth network assigns IPv4 IPs from these ranges. They do in fact belong to the DOD, but they are *not* routed to the open Internet. The DOD owns them but uses them internally for private networks. As a result, there is nothing *technically* wrong with "bogarting" these for our own private network. It's considered bad practice, but if you want a private address space in IPv4 that is unlikely to overlap other private address spaces (like 10/8 and 192.168/16), it's the only way. [Cellular carriers](http://www.androidcentral.com/sprint-internet-dept-defense-and-you)  and [cable companies](http://www.dslreports.com/forum/r25679029-Why-is-my-first-hop-to-a-DoD-assigned-IP-address-) frequently do the same thing.
 
 **Q:** Is IPv6 supported?  
 **A:** Yes. IPv6 link-local addresses (those in the fe80::/10 block) are auto-assigned and should work fine. No other IPv6 addresses are assigned *yet*, but there are plans to do interesting things in this area in the future.
 
+**Q:** I don't want a giant Ethernet party line. Can I leave it and create private LANs instead?  
+**A:** Yes, soon. A GUI to configure such things is in development. But for now there's only Earth.
+
 **Q:** Are you going to charge for this?  
-**A:** Public networks will remain free, but we intend to charge for private virtual LANs. ZeroTier has other ideas too, but they're top secret for the moment.
+**A:** Public virtual LANs will remain free. We intend to charge for private networks in some way, but the exact model is TBD. Other cloud-supported paid features are also TBD.
 
 **Q:** What's a supernode?  
 **A:** Supernodes are nodes run by ZeroTier Networks that orindary users use to find one another and communicate until/unless they can perform NAT traversal and connect directly. They run the exact same software as everyone else. The only thing that really makes a supernode special is that it's designated as such.
@@ -38,14 +41,20 @@ Check out the [blog](http://blog.zerotier.com/) for announcements, in-depth arti
 **Q:** Can I run a supernode?  
 **A:** No, not at the moment, and there would be no benefit to doing so.
 
+**Q:** Will my local firewall rules apply to ZeroTier One traffic?  
+**A:** ZeroTier creates a virtual Ethernet tap device (zt# on Mac and Linux) that emulates a wired Ethernet port. If your firewall applies to *all* network ports, it will filter traffic through this port as well. If it applies only to the primary interface, it may not. See your OS's firewall documentation, as different OSes and flavors thereof have slightly different configurations in this regard.
+
 **Q:** Can you see my traffic? What about other users? Can you sniff the LAN?  
 **A:** No. All unicast (direct computer to computer) traffic is encrypted end-to-end (even if it's being relayed), and the ZeroTier virtual LAN behaves like a LAN with a secure enterprise-grade switch that does not allow unicast conversations to be sniffed. Multicast and broadcast traffic will of course be seen by many recipients, but that's the idea.
 
-**Q:** You say "almost unlimited size." Isn't multicast and broadcast traffic eventually going to be too much? What happens then?  
-**A:** ZeroTier One uses an algorithm called *implicit social switching*. The overall maximum number of recipients for a multicast is limited, so if there are too many listeners to a given multicast address then obviously not everyone will receive every message. So who does? Social switching causes multicasts to propagate in the direction of peers to whom you have recently communicated. As a result, multicasts tend to propagate along lines of association. The people most likely to get your service announcements are those with whom you frequently connect.
+**Q:** What about privacy? Does this hide my location on the network?  
+**A:** ZeroTier is not a connection anonymizer. Other than encryption, it doesn't do anything special to hide your identity or network location. If you want strong privacy protection there are already very advanced tools like [Tor](https://www.torproject.org) for that, and this isn't trying to duplicate their functionality. At the same time, ZT does not do anything special to harm your privacy either. It's not spyware or snoop-ware.
 
-**Q:** I don't see broadcasts.  
-**A:** At the moment only Ethernet multicast is propagated, not broadcast (ff:ff:ff:ff:ff:ff). This may change in the future. IPv4 ARP uses broadcast but is handled by special code that subscribes to a multicast group using the broadcast address combined with the IPv4 address to make it address-specific. See comments in MulticastGroup.hpp for deeper technical details.
+**Q:** Is this designed to replace IP, BGP, IPv6, routers, etc.?  
+**A:** No. Its purpose is to act as a collaboration tool, a VPN alternative, a network mobility tool, a testbed for the development of software that takes advantage of fully open networking, a virtual LAN party for gamers, and so on, but it's not intended (or able) to replace the core of the Internet.
+
+**Q:** Can I bridge this to a physical port and plug in an Xbox, PlayStation, etc.?  
+**A:** Not currently, as foreign Ethernet frames are not forwarded. This may be possible in a future version.
 
 ----
 **Status**
@@ -62,9 +71,8 @@ Check out the [blog](http://blog.zerotier.com/) for announcements, in-depth arti
  * Changes in your local network configuration are generally detected and will cause peers to be re-acquired.
 
 *Known immediate issues:*
- * Multiple network support is in but there is no interface to configure it, hence it is useless. But in alpha it'll be nice to shove everyone onto "Earth" in order to stress test that little "almost unlimited size" boast. LAN party!
-   * Multiple networks would currently all have the same MAC, which some OSes may not like. There is a possible strategy for dealing with this, but it needs to be explored.
- * There is no multiple-launch protection yet. If you launch more than one instance on the same working directory everything breaks. Take care that zerotier-one is not running before launching it again. The command "sudo killall zerotier-one" is helpful.
+ * Multiple network support is in but there is no interface to configure it, hence it is useless. But in alpha it'll be nice to shove everyone onto "Earth" in order to stress test that little "almost unlimited size" boast.
+ * There is no multiple-launch protection yet and multiple instances on the same system do not work well due to route conflicts. Take care that zerotier-one is not running before launching it again. The command "sudo killall zerotier-one" is helpful.
  * Sometimes ZeroTier One doesn't like to terminate when asked nicely. This is related to issues with the tap device closing down properly. If it hangs around after a TERM signal, send it a KILL (9) signal.
  * The locally bound UDP port isn't configurable yet. It's 8993 by default.
  * Known security issues:
@@ -81,4 +89,7 @@ Check out the [blog](http://blog.zerotier.com/) for announcements, in-depth arti
 
 ----
 
+<a href="http://flattr.com/thing/1611614/ZeroTier-Networks" target="_blank"><img src="http://api.flattr.com/button/flattr-badge-large.png" alt="Flattr this" title="Flattr this" border="0" /></a>  
+[![githalytics.com alpha](https://cruel-carlota.pagodabox.com/59b2cbb9c154bf84bddb4b714402e548 "githalytics.com")](http://githalytics.com/zerotier/ZeroTierOne)
+
 (c)2012-2013 [ZeroTier Networks LLC](https://www.zerotier.com/)

RUNNING.txt

@@ -9,7 +9,7 @@ By convention, ZeroTier One will keep its state here on mac:
 ZeroTier ships with a kernel extension for its own tap device, which it
 stores in the above directory. To install this from source, type:
 
-sudo make install-mac-tap
+sudo make -f Makefile.mac install-mac-tap
 
 This will create the ZeroTier One home above if it does not exist and install
 the kext there. Note that the kext must be owned by root:wheel. The make

bump_revision.sh

@@ -1,32 +0,0 @@
-#!/bin/bash
-
-#
-# This probably won't be useful to anyone outside ZeroTier itself.
-#
-
-#
-# This bumps the revision in version.h, which triggers a build and deploy
-# to the now/ subfolder on update.zerotier.com. This allows nodes tracking
-# the bleeding edge to track the bleedingest of the bleeding edge.
-#
-
-cur_rev=`grep -F ZEROTIER_ONE_VERSION_REVISION version.h | cut -d ' ' -f 3`
-next_rev=`expr $cur_rev + 1`
-
-echo Current revision: $cur_rev
-echo Next revision: $next_rev
-
-rm -f version.h.new
-cat version.h | sed "s/ZEROTIER_ONE_VERSION_REVISION $cur_rev/ZEROTIER_ONE_VERSION_REVISION $next_rev/g" >>version.h.new
-
-new_cur_rev=`grep -F ZEROTIER_ONE_VERSION_REVISION version.h.new | cut -d ' ' -f 3`
-
-if [ "$new_cur_rev" = "$next_rev" ]; then
-	mv -f version.h.new version.h
-	echo Done.
-else
-	echo Error: version.h.new updated incorrectly, leaving in place.
-	exit 1
-fi
-
-exit 0

main.cpp

@@ -70,12 +70,18 @@ static void sighandlerQuit(int sig)
 static void sighandlerUsr(int sig)
 {
 }
+static void sighandlerHup(int sig)
+{
+	Node *n = node;
+	if (n)
+		n->updateStatusNow();
+}
 #endif
 
 int main(int argc,char **argv)
 {
 #ifndef _WIN32
-	signal(SIGHUP,SIG_IGN);
+	signal(SIGHUP,&sighandlerHup);
 	signal(SIGPIPE,SIG_IGN);
 	signal(SIGUSR1,&sighandlerUsr);
 	signal(SIGUSR2,&sighandlerUsr);

node/BloomFilter.hpp

@@ -0,0 +1,133 @@
+/*
+ * ZeroTier One - Global Peer to Peer Ethernet
+ * Copyright (C) 2012-2013  ZeroTier Networks LLC
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation, either version 3 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ *
+ * --
+ *
+ * ZeroTier may be used and distributed under the terms of the GPLv3, which
+ * are available at: http://www.gnu.org/licenses/gpl-3.0.html
+ *
+ * If you would like to embed ZeroTier into a commercial application or
+ * redistribute it in a modified binary form, please contact ZeroTier Networks
+ * LLC. Start here: http://www.zerotier.com/
+ */
+
+#ifndef _ZT_BLOOMFILTER_HPP
+#define _ZT_BLOOMFILTER_HPP
+
+#include <string.h>
+#include "Utils.hpp"
+
+namespace ZeroTier {
+
+/**
+ * A simple bit field bloom filter
+ *
+ * This actually isn't a total filter, in that it does not contain a hashing
+ * algorithm. It's up to the caller to hash/sum the items being remembered
+ * so that the distribution of 'n' is random.
+ *
+ * @tparam B Size in BITS, must be a multiple of 8
+ */
+template<unsigned int B>
+class BloomFilter
+{
+public:
+	/**
+	 * Construct an empty filter
+	 */
+	BloomFilter()
+		throw()
+	{
+		memset(_field,0,sizeof(_field));
+	}
+
+	/**
+	 * Construct from a raw filter
+	 *
+	 * @param b Raw filter bits, must be exactly bytes() in length, or NULL to construct empty
+	 */
+	BloomFilter(const void *b)
+		throw()
+	{
+		if (b)
+			memcpy(_field,b,sizeof(_field));
+		else memset(_field,0,sizeof(_field));
+	}
+
+	/**
+	 * @return Size of filter in bits
+	 */
+	static inline unsigned int bits() throw() { return B; }
+
+	/**
+	 * @return Size of filter in bytes
+	 */
+	static inline unsigned int bytes() throw() { return (B / 8); }
+
+	/**
+	 * @return Pointer to portable array of bytes of bytes() length representing filter
+	 */
+	inline const unsigned char *data() const throw() { return _field; }
+
+	/**
+	 * Clear all bits in filter
+	 */
+	inline void clear()
+		throw()
+	{
+		memset(_field,0,sizeof(_field));
+	}
+
+	/**
+	 * @param n Value to set
+	 */
+	inline void set(unsigned int n)
+		throw()
+	{
+		n %= B;
+		_field[n / 8] |= (1 << (n % 8));
+	}
+
+	/**
+	 * @param n Value to test
+	 * @return True if value is present is set
+	 */
+	inline bool contains(unsigned int n)
+		throw()
+	{
+		n %= B;
+		return (_field[n / 8] & (1 << (n % 8)));
+	}
+
+	/**
+	 * Clear a random bit in this bloom filter
+	 *
+	 * @param rn Random number
+	 */
+	inline void decay(unsigned int rn)
+		throw()
+	{
+		_field[(rn >> 3) % (B / 8)] &= ~((unsigned char)(1 << (rn & 7)));
+	}
+
+private:
+	unsigned char _field[B / 8];
+};
+
+} // namespace ZeroTier
+
+#endif

node/Buffer.hpp

@@ -240,6 +240,22 @@ public:
 		_l += sizeof(T);
 	}
 
+	/**
+	 * Append a run of bytes
+	 *
+	 * @param c Character value to append
+	 * @param n Number of times to append
+	 * @throws std::out_of_range Attempt to append beyond capacity
+	 */
+	inline void append(unsigned char c,unsigned int n)
+		throw(std::out_of_range)
+	{
+		if ((_l + n) > C)
+			throw std::out_of_range("Buffer: append beyond capacity");
+		for(unsigned int i=0;i<n;++i)
+			_b[_l++] = (char)c;
+	}
+
 	/**
 	 * Append a C-array of bytes
 	 * 
@@ -388,7 +404,7 @@ public:
 		return !(*this < b);
 	}
 
-protected:
+private:
 	unsigned int _l;
 	char ZT_VAR_MAY_ALIAS _b[C];
 };

node/CMWC4096.hpp

@@ -0,0 +1,91 @@
+/*
+ * ZeroTier One - Global Peer to Peer Ethernet
+ * Copyright (C) 2012-2013  ZeroTier Networks LLC
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation, either version 3 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ *
+ * --
+ *
+ * ZeroTier may be used and distributed under the terms of the GPLv3, which
+ * are available at: http://www.gnu.org/licenses/gpl-3.0.html
+ *
+ * If you would like to embed ZeroTier into a commercial application or
+ * redistribute it in a modified binary form, please contact ZeroTier Networks
+ * LLC. Start here: http://www.zerotier.com/
+ */
+
+#ifndef _ZT_CMWC4096_HPP
+#define _ZT_CMWC4096_HPP
+
+#include <stdint.h>
+#include "Utils.hpp"
+
+namespace ZeroTier {
+
+/** 
+ * Complement Multiply With Carry random number generator
+ *
+ * Based on original code posted to Usenet in the public domain by
+ * George Marsaglia. Period is approximately 2^131086.
+ *
+ * This is not used for cryptographic purposes but for a very fast
+ * and high-quality PRNG elsewhere in the code.
+ */
+class CMWC4096
+{
+public:
+	/**
+	 * Construct and initialize from secure random source
+	 */
+	CMWC4096()
+		throw()
+	{
+		Utils::getSecureRandom(Q,sizeof(Q));
+		Utils::getSecureRandom(&c,sizeof(c));
+		c %= 809430660;
+		i = 4095;
+	}
+
+	inline uint32_t next32()
+		throw()
+	{
+		uint32_t __i = ++i & 4095;
+		const uint64_t t = (18782ULL * (uint64_t)Q[__i]) + (uint64_t)c;
+		c = (uint32_t)(t >> 32);
+		uint32_t x = c + (uint32_t)t;
+		const uint32_t p = (uint32_t)(x < c); x += p; c += p;
+		return (Q[__i] = 0xfffffffe - x);
+	}
+
+	inline uint64_t next64()
+		throw()
+	{
+		return ((((uint64_t)next32()) << 32) ^ (uint64_t)next32());
+	}
+
+	inline double nextDouble()
+		throw()
+	{
+		return ((double)(next32()) / 4294967296.0);
+	}
+
+private:
+	uint32_t Q[4096];
+	uint32_t c;
+	uint32_t i;
+};
+
+} // namespace ZeroTier
+
+#endif

node/Constants.hpp

@@ -28,10 +28,55 @@
 #ifndef _ZT_CONSTANTS_HPP
 #define _ZT_CONSTANTS_HPP
 
-// Assume these are little-endian, since we don't support old PPC MACs
-// and all newer Mac or Windows systems are either x86_32, x86_64, or
-// ARM in little-endian mode.
-#if defined(__APPLE__) || defined(_WIN32)
+//
+// This include file also auto-detects and canonicalizes some environment
+// information defines:
+//
+// __LINUX__
+// __APPLE__
+// __UNIX_LIKE__ - any "unix like" OS (BSD, posix, etc.)
+// __WINDOWS__
+//
+// Also makes sure __BYTE_ORDER is defined reasonably.
+//
+
+// Canonicalize Linux... is this necessary? Do it anyway to be defensive.
+#if defined(__linux__) || defined(linux) || defined(__LINUX__) || defined(__linux)
+#ifndef __LINUX__
+#define __LINUX__
+#ifndef __UNIX_LIKE__
+#define __UNIX_LIKE__
+#endif
+#endif
+#endif
+
+// TODO: Android is what? Linux technically, but does it define it?
+
+// OSX and iOS are unix-like OSes far as we're concerned
+#ifdef __APPLE__
+#ifndef __UNIX_LIKE__
+#define __UNIX_LIKE__
+#endif
+#endif
+
+// Linux has endian.h
+#ifdef __LINUX__
+#include <endian.h>
+#endif
+
+#if defined(_WIN32) || defined(_WIN64)
+#ifndef __WINDOWS__
+#define __WINDOWS__
+#endif
+#undef __UNIX_LIKE__
+#define ZT_PATH_SEPARATOR '\\'
+#define ZT_PATH_SEPARATOR_S "\\"
+#define ZT_EOL_S "\r\n"
+#endif
+
+// Assume these are little-endian. PPC is not supported for OSX, and ARM
+// runs in little-endian mode for these OS families.
+#if defined(__APPLE__) || defined(__WINDOWS__)
 #undef __BYTE_ORDER
 #undef __LITTLE_ENDIAN
 #undef __BIG_ENDIAN
@@ -40,33 +85,23 @@
 #define __BYTE_ORDER 1234
 #endif
 
-// Linux has endian.h, which should tell us
-#if defined(__linux__) || defined(linux) || defined(__LINUX__) || defined(__linux)
-#include <endian.h>
-#endif
-
-#ifndef __BYTE_ORDER
-error_no_byte_order_defined
-#endif
-
-#ifndef ZT_OSNAME
-error_no_ZT_OSNAME
-#endif
-
-#ifndef ZT_ARCH
-error_no_ZT_ARCH
-#endif
-
-#ifdef _WIN32
-#define ZT_PATH_SEPARATOR '\\'
-#define ZT_PATH_SEPARATOR_S "\\"
-#define ZT_EOL_S "\r\n"
-#else
+#ifdef __UNIX_LIKE__
 #define ZT_PATH_SEPARATOR '/'
 #define ZT_PATH_SEPARATOR_S "/"
 #define ZT_EOL_S "\n"
 #endif
 
+// Error out if required symbols are missing
+#ifndef __BYTE_ORDER
+error_no_byte_order_defined;
+#endif
+#ifndef ZT_OSNAME
+error_no_ZT_OSNAME_defined;
+#endif
+#ifndef ZT_ARCH
+error_no_ZT_ARCH_defined;
+#endif
+
 /**
  * Length of a ZeroTier address in bytes
  */
@@ -200,12 +235,17 @@ error_no_ZT_ARCH
 /**
  * Length of circular ring buffer history of multicast packets
  */
-#define ZT_MULTICAST_DEDUP_HISTORY_LENGTH 4096
+#define ZT_MULTICAST_DEDUP_HISTORY_LENGTH 1024
 
 /**
  * Expiration time in ms for multicast history items
  */
-#define ZT_MULTICAST_DEDUP_HISTORY_EXPIRE 8000
+#define ZT_MULTICAST_DEDUP_HISTORY_EXPIRE 4000
+
+/**
+ * Number of bits to randomly "decay" in bloom filter per hop
+ */
+#define ZT_MULTICAST_BLOOM_FILTER_DECAY_RATE 2
 
 /**
  * Period between announcements of all multicast 'likes' in ms
@@ -253,6 +293,11 @@ error_no_ZT_ARCH
  */
 #define ZT_AUTOCONFIGURE_CHECK_DELAY 15000
 
+/**
+ * Delay between updates of status file in home directory
+ */
+#define ZT_STATUS_OUTPUT_PERIOD 120000
+
 /**
  * Minimum delay in Node service loop
  * 

node/Demarc.cpp

@@ -93,9 +93,9 @@ bool Demarc::bindLocalUdp(unsigned int localPort)
 	uint64_t v4p = ((uint64_t)PORT_TYPE_UDP_SOCKET_V4 << 60) | (uint64_t)localPort;
 	uint64_t v6p = ((uint64_t)PORT_TYPE_UDP_SOCKET_V6 << 60) | (uint64_t)localPort;
 	if ((_ports.count((Port)v4p))||(_ports.count((Port)v6p)))
-		return true;
+		return true; // port already bound
 
-	UdpSocket *v4;
+	UdpSocket *v4 = (UdpSocket *)0;
 	try {
 		DemarcPortObj *v4r = &(_ports[(Port)v4p]);
 		v4r->port = (Port)v4p;
@@ -104,10 +104,10 @@ bool Demarc::bindLocalUdp(unsigned int localPort)
 		v4r->type = PORT_TYPE_UDP_SOCKET_V4;
 	} catch ( ... ) {
 		_ports.erase((Port)v4p);
-		return false;
+		v4 = (UdpSocket *)0;
 	}
 
-	UdpSocket *v6;
+	UdpSocket *v6 = (UdpSocket *)0;
 	try {
 		DemarcPortObj *v6r = &(_ports[(Port)v6p]);
 		v6r->port = (Port)v6p;
@@ -115,13 +115,11 @@ bool Demarc::bindLocalUdp(unsigned int localPort)
 		v6r->obj = v6 = new UdpSocket(localPort,true,&Demarc::_CBudpSocketPacketHandler,v6r);
 		v6r->type = PORT_TYPE_UDP_SOCKET_V6;
 	} catch ( ... ) {
-		delete v4;
-		_ports.erase((Port)v4p);
 		_ports.erase((Port)v6p);
-		return false;
+		v6 = (UdpSocket *)0;
 	}
 
-	return true;
+	return ((v4)||(v6));
 }
 
 Demarc::Port Demarc::pick(const InetAddress &to) const
@@ -145,7 +143,7 @@ Demarc::Port Demarc::pick(const InetAddress &to) const
 			}
 		}
 		if (possibilities.size())
-			return possibilities[Utils::randomInt<unsigned int>() % possibilities.size()]->first;
+			return possibilities[_r->prng->next32() % possibilities.size()]->first;
 		else return NULL_PORT;
 	} catch ( ... ) {
 		return NULL_PORT;
@@ -176,7 +174,7 @@ Demarc::Port Demarc::send(Demarc::Port fromPort,const InetAddress &to,const void
 				}
 			}
 			if (possibilities.size())
-				pe = possibilities[Utils::randomInt<unsigned int>() % possibilities.size()];
+				pe = possibilities[_r->prng->next32() % possibilities.size()];
 			else {
 				_ports_m.unlock();
 				return NULL_PORT;

node/EthernetTap.cpp

@@ -30,11 +30,17 @@
 #include "EthernetTap.hpp"
 #include "Logger.hpp"
 #include "RuntimeEnvironment.hpp"
+#include "Utils.hpp"
 #include "Mutex.hpp"
 
-/* ======================================================================== */
-#if defined(__linux__) || defined(linux) || defined(__LINUX__) || defined(__linux)
-/* ======================================================================== */
+// ff:ff:ff:ff:ff:ff with no ADI
+static const ZeroTier::MulticastGroup _blindWildcardMulticastGroup(ZeroTier::MAC(0xff),0);
+
+//
+// TAP implementation for *nix OSes, with some specialization for different flavors
+//
+
+#ifdef __UNIX_LIKE__ /////////////////////////////////////////////////////////
 
 #include <stdint.h>
 #include <stdio.h>
@@ -48,10 +54,13 @@
 #include <sys/stat.h>
 #include <sys/ioctl.h>
 #include <sys/wait.h>
+#include <sys/select.h>
 #include <netinet/in.h>
 #include <net/if_arp.h>
 #include <arpa/inet.h>
 
+#ifdef __LINUX__
+
 #include <linux/if.h>
 #include <linux/if_tun.h>
 #include <linux/if_addr.h>
@@ -60,23 +69,49 @@
 #define ZT_ETHERTAP_IP_COMMAND "/sbin/ip"
 #define ZT_ETHERTAP_SYSCTL_COMMAND "/sbin/sysctl"
 
+#endif // __LINUX__
+
+#ifdef __APPLE__
+
+#include <sys/uio.h>
+#include <sys/param.h>
+#include <sys/sysctl.h>
+#include <net/route.h>
+#include <net/if_dl.h>
+#include <ifaddrs.h>
+
+#define ZT_ETHERTAP_IFCONFIG "/sbin/ifconfig"
+#define ZT_MAC_KEXTLOAD "/sbin/kextload"
+#define ZT_MAC_IPCONFIG "/usr/sbin/ipconfig"
+
+#endif // __APPLE__
+
 namespace ZeroTier {
 
+// Only permit one tap to be opened concurrently across the entire process
 static Mutex __tapCreateLock;
 
-EthernetTap::EthernetTap(const RuntimeEnvironment *renv,const MAC &mac,unsigned int mtu) 
+#ifdef __LINUX__
+EthernetTap::EthernetTap(
+	const RuntimeEnvironment *renv,
+	const MAC &mac,
+	unsigned int mtu,
+	void (*handler)(void *,const MAC &,const MAC &,unsigned int,const Buffer<4096> &),
+	void *arg)
 	throw(std::runtime_error) :
 	_mac(mac),
 	_mtu(mtu),
 	_r(renv),
-	_putBuf((unsigned char *)0),
-	_getBuf((unsigned char *)0),
-	_fd(0),
-	_isReading(false)
+	_handler(handler),
+	_arg(arg),
+	_fd(0)
 {
 	char procpath[128];
 	Mutex::Lock _l(__tapCreateLock); // create only one tap at a time, globally
 
+	if (mtu > 4096)
+		throw std::runtime_error("max tap MTU is 4096");
+
 	_fd = ::open("/dev/net/tun",O_RDWR);
 	if (_fd <= 0)
 		throw std::runtime_error("could not open TUN/TAP device");
@@ -148,259 +183,36 @@ EthernetTap::EthernetTap(const RuntimeEnvironment *renv,const MAC &mac,unsigned
 
 	::close(sock);
 
-	_putBuf = new unsigned char[((mtu + 16) * 2)];
-	_getBuf = _putBuf + (mtu + 16);
+	::pipe(_shutdownSignalPipe);
 
 	TRACE("tap %s created",_dev);
+
+	start();
 }
+#endif // __LINUX__
 
-EthernetTap::~EthernetTap()
-{
-	this->close();
-	delete [] _putBuf;
-}
-
-static bool ___removeIp(const char *_dev,std::set<InetAddress> &_ips,const InetAddress &ip)
-{
-	long cpid;
-	if ((cpid = (long)fork()) == 0) {
-		execl(ZT_ETHERTAP_IP_COMMAND,ZT_ETHERTAP_IP_COMMAND,"addr","del",ip.toString().c_str(),"dev",_dev,(const char *)0);
-		exit(1); /* not reached unless exec fails */
-	} else {
-		int exitcode = 1;
-		waitpid(cpid,&exitcode,0);
-		if (exitcode == 0) {
-			_ips.erase(ip);
-			return true;
-		} else return false;
-	}
-}
-
-bool EthernetTap::addIP(const InetAddress &ip)
-{
-	Mutex::Lock _l(_ips_m);
-
-	if (!ip)
-		return false;
-	if (_ips.count(ip) > 0)
-		return true;
-
-	// Remove and reconfigure if address is the same but netmask is different
-	for(std::set<InetAddress>::iterator i(_ips.begin());i!=_ips.end();++i) {
-		if (i->ipsEqual(ip)) {
-			___removeIp(_dev,_ips,*i);
-			break;
-		}
-	}
-
-	int cpid;
-	if ((cpid = (int)fork()) == 0) {
-		execl(ZT_ETHERTAP_IP_COMMAND,ZT_ETHERTAP_IP_COMMAND,"addr","add",ip.toString().c_str(),"dev",_dev,(const char *)0);
-		exit(-1);
-	} else {
-		int exitcode = -1;
-		waitpid(cpid,&exitcode,0);
-		if (exitcode == 0) {
-			_ips.insert(ip);
-			return true;
-		} else return false;
-	}
-
-	return false;
-}
-
-bool EthernetTap::removeIP(const InetAddress &ip)
-{
-	Mutex::Lock _l(_ips_m);
-	if (_ips.count(ip) > 0)
-		return ___removeIp(_dev,_ips,ip);
-	return false;
-}
-
-void EthernetTap::put(const MAC &from,const MAC &to,unsigned int etherType,const void *data,unsigned int len)
-{
-	if ((_fd > 0)&&(len <= _mtu)) {
-		for(int i=0;i<6;++i)
-			_putBuf[i] = to.data[i];
-		for(int i=0;i<6;++i)
-			_putBuf[i+6] = from.data[i];
-		*((uint16_t *)(_putBuf + 12)) = htons((uint16_t)etherType);
-		memcpy(_putBuf + 14,data,len);
-		::write(_fd,_putBuf,len + 14);
-	}
-}
-
-unsigned int EthernetTap::get(MAC &from,MAC &to,unsigned int &etherType,void *buf)
-{
-	for(;;) {
-		if (_fd > 0) {
-			_isReading_m.lock();
-			_isReading = true;
-			_isReadingThreadId = pthread_self();
-			_isReading_m.unlock();
-
-			int n = (int)::read(_fd,_getBuf,_mtu + 14);
-
-			_isReading_m.lock();
-			_isReading = false;
-			_isReading_m.unlock();
-
-			if (n > 14) {
-				for(int i=0;i<6;++i)
-					to.data[i] = _getBuf[i];
-				for(int i=0;i<6;++i)
-					from.data[i] = _getBuf[i + 6];
-				etherType = ntohs(((uint16_t *)_getBuf)[6]);
-				n -= 14;
-				memcpy(buf,_getBuf + 14,n);
-				return (unsigned int)n;
-			} else if (n < 0) {
-				if (_fd <= 0)
-					break;
-				else if ((errno == EINTR)||(errno == ETIMEDOUT))
-					continue;
-				else {
-					TRACE("unexpected error reading from tap: %s",strerror(errno));
-					::close(_fd);
-					_fd = 0;
-					break;
-				}
-			} else {
-				TRACE("incomplete read from tap: %d bytes",n);
-				continue;
-			}
-		}
-	}
-	return 0;
-}
-
-std::string EthernetTap::deviceName()
-{
-	return std::string(_dev);
-}
-
-bool EthernetTap::open() const
-{
-	return (_fd > 0);
-}
-
-void EthernetTap::close()
-{
-	Mutex::Lock _l(__tapCreateLock); // also prevent create during close()
-	if (_fd > 0) {
-		int f = _fd;
-		_fd = 0;
-		::close(f);
-
-		_isReading_m.lock();
-		if (_isReading)
-			pthread_kill(_isReadingThreadId,SIGUSR2);
-		_isReading_m.unlock();
-	}
-}
-
-bool EthernetTap::updateMulticastGroups(std::set<MulticastGroup> &groups)
-{
-	char *ptr,*ptr2;
-	unsigned char mac[6];
-	std::set<MulticastGroup> newGroups;
-
-	int fd = ::open("/proc/net/dev_mcast",O_RDONLY);
-	if (fd > 0) {
-		char buf[131072];
-		int n = (int)::read(fd,buf,sizeof(buf));
-		if ((n > 0)&&(n < (int)sizeof(buf))) {
-			buf[n] = (char)0;
-			for(char *l=strtok_r(buf,"\r\n",&ptr);(l);l=strtok_r((char *)0,"\r\n",&ptr)) {
-				int fno = 0;
-				char *devname = (char *)0;
-				char *mcastmac = (char *)0;
-				for(char *f=strtok_r(l," \t",&ptr2);(f);f=strtok_r((char *)0," \t",&ptr2)) {
-					if (fno == 1)
-						devname = f;
-					else if (fno == 4)
-						mcastmac = f;
-					++fno;
-				}
-				if ((devname)&&(!strcmp(devname,_dev))&&(mcastmac)&&(Utils::unhex(mcastmac,mac,6) == 6))
-					newGroups.insert(MulticastGroup(MAC(mac),0));
-			}
-		}
-		::close(fd);
-	}
-
-	{
-		Mutex::Lock _l(_ips_m);
-		for(std::set<InetAddress>::const_iterator i(_ips.begin());i!=_ips.end();++i)
-			newGroups.insert(MulticastGroup::deriveMulticastGroupForAddressResolution(*i));
-	}
-
-	bool changed = false;
-
-	for(std::set<MulticastGroup>::iterator mg(newGroups.begin());mg!=newGroups.end();++mg) {
-		if (!groups.count(*mg)) {
-			groups.insert(*mg);
-			changed = true;
-		}
-	}
-	for(std::set<MulticastGroup>::iterator mg(groups.begin());mg!=groups.end();) {
-		if (!newGroups.count(*mg)) {
-			groups.erase(mg++);
-			changed = true;
-		} else ++mg;
-	}
-
-	return changed;
-}
-
-} // namespace ZeroTier
-
-/* ======================================================================== */
-#elif defined(__APPLE__) /* ----------------------------------------------- */
-/* ======================================================================== */
-
-#include <stdint.h>
-#include <stdio.h>
-#include <stdlib.h>
-#include <string.h>
-#include <unistd.h>
-#include <signal.h>
-#include <fcntl.h>
-#include <errno.h>
-#include <sys/types.h>
-#include <sys/stat.h>
-#include <sys/ioctl.h>
-#include <sys/uio.h>
-#include <sys/param.h>
-#include <sys/sysctl.h>
-#include <netinet/in.h>
-#include <arpa/inet.h>
-#include <net/route.h>
-#include <net/if_dl.h>
-#include <ifaddrs.h>
-
-#define ZT_ETHERTAP_IFCONFIG "/sbin/ifconfig"
-#define ZT_MAC_KEXTLOAD "/sbin/kextload"
-#define ZT_MAC_IPCONFIG "/usr/sbin/ipconfig"
-
-namespace ZeroTier {
-
-static Mutex __tapCreateLock;
-
-EthernetTap::EthernetTap(const RuntimeEnvironment *renv,const MAC &mac,unsigned int mtu)
+#ifdef __APPLE__
+EthernetTap::EthernetTap(
+	const RuntimeEnvironment *renv,
+	const MAC &mac,
+	unsigned int mtu,
+	void (*handler)(void *,const MAC &,const MAC &,unsigned int,const Buffer<4096> &),
+	void *arg)
 	throw(std::runtime_error) :
 	_mac(mac),
 	_mtu(mtu),
 	_r(renv),
-	_putBuf((unsigned char *)0),
-	_getBuf((unsigned char *)0),
-	_fd(0),
-	_isReading(false)
+	_handler(handler),
+	_arg(arg),
+	_fd(0)
 {
 	char devpath[64],ethaddr[64],mtustr[16];
 	struct stat tmp;
 	Mutex::Lock _l(__tapCreateLock); // create only one tap at a time, globally
 
+	if (mtu > 4096)
+		throw std::runtime_error("max tap MTU is 4096");
+
 	// Check for existence of ZT tap devices, try to load module if not there
 	if (stat("/dev/zt0",&tmp)) {
 		int kextpid;
@@ -442,8 +254,8 @@ EthernetTap::EthernetTap(const RuntimeEnvironment *renv,const MAC &mac,unsigned
 	sprintf(mtustr,"%u",mtu);
 
 	// Configure MAC address and MTU, bring interface up
-	int cpid;
-	if ((cpid = (int)fork()) == 0) {
+	long cpid;
+	if ((cpid = (long)fork()) == 0) {
 		execl(ZT_ETHERTAP_IFCONFIG,ZT_ETHERTAP_IFCONFIG,_dev,"lladdr",ethaddr,"mtu",mtustr,"up",(const char *)0);
 		exit(-1);
 	} else {
@@ -455,42 +267,51 @@ EthernetTap::EthernetTap(const RuntimeEnvironment *renv,const MAC &mac,unsigned
 		}
 	}
 
-	// OSX seems to require that IPv6 be turned on on tap devices
-	if ((cpid = (int)fork()) == 0) {
+	whack(); // turns on IPv6 on OSX
+
+	::pipe(_shutdownSignalPipe);
+
+	start();
+}
+#endif // __APPLE__
+
+EthernetTap::~EthernetTap()
+{
+	::write(_shutdownSignalPipe[1],"\0",1); // causes thread to exit
+	join();
+	::close(_fd);
+}
+
+#ifdef __APPLE__
+void EthernetTap::whack()
+{
+	long cpid = (long)fork();
+	if (cpid == 0) {
 		execl(ZT_MAC_IPCONFIG,ZT_MAC_IPCONFIG,"set",_dev,"AUTOMATIC-V6",(const char *)0);
 		exit(-1);
 	} else {
 		int exitcode = -1;
 		waitpid(cpid,&exitcode,0);
 		if (exitcode) {
-			::close(_fd);
-			throw std::runtime_error("ifconfig failure setting link-layer address and activating tap interface");
+			LOG("%s: ipconfig set AUTOMATIC-V6 failed",_dev);
 		}
 	}
-
-	_putBuf = new unsigned char[((mtu + 14) * 2)];
-	_getBuf = _putBuf + (mtu + 14);
 }
+#else
+void EthernetTap::whack() {}
+#endif // __APPLE__ / !__APPLE__
 
-EthernetTap::~EthernetTap()
+#ifdef __LINUX__
+static bool ___removeIp(const char *_dev,const InetAddress &ip)
 {
-	this->close();
-	delete [] _putBuf;
-}
-
-static bool ___removeIp(const char *_dev,std::set<InetAddress> &_ips,const InetAddress &ip)
-{
-	int cpid;
-	if ((cpid = (int)fork()) == 0) {
-		execl(ZT_ETHERTAP_IFCONFIG,ZT_ETHERTAP_IFCONFIG,_dev,"inet",ip.toIpString().c_str(),"-alias",(const char *)0);
-		exit(-1);
+	long cpid = (long)fork();
+	if (cpid == 0) {
+		execl(ZT_ETHERTAP_IP_COMMAND,ZT_ETHERTAP_IP_COMMAND,"addr","del",ip.toString().c_str(),"dev",_dev,(const char *)0);
+		exit(1); /* not reached unless exec fails */
 	} else {
-		int exitcode = -1;
+		int exitcode = 1;
 		waitpid(cpid,&exitcode,0);
-		if (exitcode == 0) {
-			_ips.erase(ip);
-			return true;
-		} else return false;
+		return (exitcode == 0);
 	}
 }
 
@@ -506,8 +327,65 @@ bool EthernetTap::addIP(const InetAddress &ip)
 	// Remove and reconfigure if address is the same but netmask is different
 	for(std::set<InetAddress>::iterator i(_ips.begin());i!=_ips.end();++i) {
 		if (i->ipsEqual(ip)) {
-			___removeIp(_dev,_ips,*i);
-			break;
+			if (___removeIp(_dev,*i)) {
+				_ips.erase(i);
+				break;
+			} else {
+				LOG("WARNING: failed to remove old IP/netmask %s to replace with %s",i->toString().c_str(),ip.toString().c_str());
+			}
+		}
+	}
+
+	long cpid;
+	if ((cpid = (long)fork()) == 0) {
+		execl(ZT_ETHERTAP_IP_COMMAND,ZT_ETHERTAP_IP_COMMAND,"addr","add",ip.toString().c_str(),"dev",_dev,(const char *)0);
+		exit(-1);
+	} else {
+		int exitcode = -1;
+		waitpid(cpid,&exitcode,0);
+		if (exitcode == 0) {
+			_ips.insert(ip);
+			return true;
+		} else return false;
+	}
+
+	return false;
+}
+#endif // __LINUX__
+
+#ifdef __APPLE__
+static bool ___removeIp(const char *_dev,const InetAddress &ip)
+{
+	int cpid;
+	if ((cpid = (int)fork()) == 0) {
+		execl(ZT_ETHERTAP_IFCONFIG,ZT_ETHERTAP_IFCONFIG,_dev,"inet",ip.toIpString().c_str(),"-alias",(const char *)0);
+		exit(-1);
+	} else {
+		int exitcode = -1;
+		waitpid(cpid,&exitcode,0);
+		return (exitcode == 0);
+	}
+	return false; // never reached, make compiler shut up about return value
+}
+
+bool EthernetTap::addIP(const InetAddress &ip)
+{
+	Mutex::Lock _l(_ips_m);
+
+	if (!ip)
+		return false;
+	if (_ips.count(ip) > 0)
+		return true; // IP/netmask already assigned
+
+	// Remove and reconfigure if address is the same but netmask is different
+	for(std::set<InetAddress>::iterator i(_ips.begin());i!=_ips.end();++i) {
+		if ((i->ipsEqual(ip))&&(i->netmaskBits() != ip.netmaskBits())) {
+			if (___removeIp(_dev,*i)) {
+				_ips.erase(i);
+				break;
+			} else {
+				LOG("WARNING: failed to remove old IP/netmask %s to replace with %s",i->toString().c_str(),ip.toString().c_str());
+			}
 		}
 	}
 
@@ -526,105 +404,106 @@ bool EthernetTap::addIP(const InetAddress &ip)
 
 	return false;
 }
+#endif // __APPLE__
 
 bool EthernetTap::removeIP(const InetAddress &ip)
 {
 	Mutex::Lock _l(_ips_m);
-	if (_ips.count(ip) > 0)
-		return ___removeIp(_dev,_ips,ip);
+	if (_ips.count(ip) > 0) {
+		if (___removeIp(_dev,ip)) {
+			_ips.erase(ip);
+			return true;
+		}
+	}
 	return false;
 }
 
 void EthernetTap::put(const MAC &from,const MAC &to,unsigned int etherType,const void *data,unsigned int len)
 {
+	char putBuf[4096 + 14];
 	if ((_fd > 0)&&(len <= _mtu)) {
 		for(int i=0;i<6;++i)
-			_putBuf[i] = to.data[i];
+			putBuf[i] = to.data[i];
 		for(int i=0;i<6;++i)
-			_putBuf[i+6] = from.data[i];
-		*((uint16_t *)(_putBuf + 12)) = htons((uint16_t)etherType);
-		memcpy(_putBuf + 14,data,len);
+			putBuf[i+6] = from.data[i];
+		*((uint16_t *)(putBuf + 12)) = htons((uint16_t)etherType);
+		memcpy(putBuf + 14,data,len);
 		len += 14;
-		int n = (int)::write(_fd,_putBuf,len);
+		int n = ::write(_fd,putBuf,len);
 		if (n <= 0) {
 			LOG("error writing packet to Ethernet tap device: %s",strerror(errno));
 		} else if (n != (int)len) {
 			// Saw this gremlin once, so log it if we see it again... OSX tap
 			// or something seems to have goofy issues with certain MTUs.
-			LOG("WARNING: Apple gremlin: tap write() wrote %d of %u bytes of frame",n,len);
+			LOG("ERROR: write underrun: %s tap write() wrote %d of %u bytes of frame",_dev,n,len);
 		}
 	}
 }
 
-unsigned int EthernetTap::get(MAC &from,MAC &to,unsigned int &etherType,void *buf)
-{
-	for(;;) {
-		if (_fd > 0) {
-			_isReading_m.lock();
-			_isReading = true;
-			_isReadingThreadId = pthread_self();
-			_isReading_m.unlock();
-
-			int n = (int)::read(_fd,_getBuf,_mtu + 14);
-
-			_isReading_m.lock();
-			_isReading = false;
-			_isReading_m.unlock();
-
-			if (n > 14) {
-				for(int i=0;i<6;++i)
-					to.data[i] = _getBuf[i];
-				for(int i=0;i<6;++i)
-					from.data[i] = _getBuf[i + 6];
-				etherType = ntohs(((uint16_t *)_getBuf)[6]);
-				n -= 14;
-				memcpy(buf,_getBuf + 14,n);
-				return (unsigned int)n;
-			} else if (n < 0) {
-				if (_fd <= 0)
-					break;
-				else if ((errno == EINTR)||(errno == ETIMEDOUT))
-					continue;
-				else {
-					TRACE("unexpected error reading from tap: %s",strerror(errno));
-					::close(_fd);
-					_fd = 0;
-					break;
-				}
-			} else {
-				TRACE("incomplete read from tap: %d bytes",n);
-				continue;
-			}
-		}
-	}
-	return 0;
-}
-
-std::string EthernetTap::deviceName()
+std::string EthernetTap::deviceName() const
 {
 	return std::string(_dev);
 }
 
-bool EthernetTap::open() const
+#ifdef __LINUX__
+bool EthernetTap::updateMulticastGroups(std::set<MulticastGroup> &groups)
 {
-	return (_fd > 0);
-}
+	char *ptr,*ptr2;
+	unsigned char mac[6];
+	std::set<MulticastGroup> newGroups;
 
-void EthernetTap::close()
-{
-	Mutex::Lock _l(__tapCreateLock); // also prevent create during close()
-	if (_fd > 0) {
-		int f = _fd;
-		_fd = 0;
-		::close(f);
-
-		_isReading_m.lock();
-		if (_isReading)
-			pthread_kill(_isReadingThreadId,SIGUSR2);
-		_isReading_m.unlock();
+	int fd = ::open("/proc/net/dev_mcast",O_RDONLY);
+	if (fd > 0) {
+		char buf[131072];
+		int n = (int)::read(fd,buf,sizeof(buf));
+		if ((n > 0)&&(n < (int)sizeof(buf))) {
+			buf[n] = (char)0;
+			for(char *l=strtok_r(buf,"\r\n",&ptr);(l);l=strtok_r((char *)0,"\r\n",&ptr)) {
+				int fno = 0;
+				char *devname = (char *)0;
+				char *mcastmac = (char *)0;
+				for(char *f=strtok_r(l," \t",&ptr2);(f);f=strtok_r((char *)0," \t",&ptr2)) {
+					if (fno == 1)
+						devname = f;
+					else if (fno == 4)
+						mcastmac = f;
+					++fno;
+				}
+				if ((devname)&&(!strcmp(devname,_dev))&&(mcastmac)&&(Utils::unhex(mcastmac,mac,6) == 6))
+					newGroups.insert(MulticastGroup(MAC(mac),0));
+			}
+		}
+		::close(fd);
 	}
-}
 
+	{
+		Mutex::Lock _l(_ips_m);
+		for(std::set<InetAddress>::const_iterator i(_ips.begin());i!=_ips.end();++i)
+			newGroups.insert(MulticastGroup::deriveMulticastGroupForAddressResolution(*i));
+	}
+
+	bool changed = false;
+
+	newGroups.insert(_blindWildcardMulticastGroup); // always join this
+
+	for(std::set<MulticastGroup>::iterator mg(newGroups.begin());mg!=newGroups.end();++mg) {
+		if (!groups.count(*mg)) {
+			groups.insert(*mg);
+			changed = true;
+		}
+	}
+	for(std::set<MulticastGroup>::iterator mg(groups.begin());mg!=groups.end();) {
+		if (!newGroups.count(*mg)) {
+			groups.erase(mg++);
+			changed = true;
+		} else ++mg;
+	}
+
+	return changed;
+}
+#endif // __LINUX__
+
+#ifdef __APPLE__
 bool EthernetTap::updateMulticastGroups(std::set<MulticastGroup> &groups)
 {
 	std::set<MulticastGroup> newGroups;
@@ -651,6 +530,8 @@ bool EthernetTap::updateMulticastGroups(std::set<MulticastGroup> &groups)
 
 	bool changed = false;
 
+	newGroups.insert(_blindWildcardMulticastGroup); // always join this
+
 	for(std::set<MulticastGroup>::iterator mg(newGroups.begin());mg!=newGroups.end();++mg) {
 		if (!groups.count(*mg)) {
 			groups.insert(*mg);
@@ -666,13 +547,54 @@ bool EthernetTap::updateMulticastGroups(std::set<MulticastGroup> &groups)
 
 	return changed;
 }
+#endif // __APPLE__
+
+void EthernetTap::main()
+	throw()
+{
+	fd_set readfds,nullfds;
+	MAC to,from;
+	char getBuf[4096 + 14];
+	Buffer<4096> data;
+
+	FD_ZERO(&readfds);
+	FD_ZERO(&nullfds);
+	int nfds = (int)std::max(_shutdownSignalPipe[0],_fd) + 1;
+
+	for(;;) {
+		FD_SET(_shutdownSignalPipe[0],&readfds);
+		FD_SET(_fd,&readfds);
+		select(nfds,&readfds,&nullfds,&nullfds,(struct timeval *)0);
+
+		if (FD_ISSET(_shutdownSignalPipe[0],&readfds)) // writes to shutdown pipe terminate thread
+			break;
+
+		if (FD_ISSET(_fd,&readfds)) {
+			int n = (int)::read(_fd,getBuf,_mtu + 14);
+
+			if (n > 14) {
+				for(int i=0;i<6;++i)
+					to.data[i] = (unsigned char)getBuf[i];
+				for(int i=0;i<6;++i)
+					from.data[i] = (unsigned char)getBuf[i + 6];
+				data.copyFrom(getBuf + 14,(unsigned int)n - 14);
+				_handler(_arg,from,to,ntohs(((const uint16_t *)getBuf)[6]),data);
+			} else if (n < 0) {
+				if ((errno != EINTR)&&(errno != ETIMEDOUT)) {
+					TRACE("unexpected error reading from tap: %s",strerror(errno));
+					break;
+				}
+			}
+		}
+	}
+}
 
 } // namespace ZeroTier
 
-/* ======================================================================== */
-#elif defined(_WIN32) /* -------------------------------------------------- */
-/* ======================================================================== */
+#endif // __UNIX_LIKE__ //////////////////////////////////////////////////////
 
-/* ======================================================================== */
-#endif
-/* ======================================================================== */
+#ifdef __WINDOWS__
+
+// TODO
+
+#endif // __WINDOWS__

node/EthernetTap.hpp

@@ -36,14 +36,13 @@
 #include <set>
 #include <string>
 #include <stdexcept>
-#include "Array.hpp"
-#include "Utils.hpp"
-#include "InetAddress.hpp"
-#include "NonCopyable.hpp"
-#include "MAC.hpp"
 #include "Constants.hpp"
+#include "InetAddress.hpp"
+#include "MAC.hpp"
 #include "Mutex.hpp"
 #include "MulticastGroup.hpp"
+#include "Thread.hpp"
+#include "Buffer.hpp"
 
 namespace ZeroTier {
 
@@ -52,21 +51,40 @@ class RuntimeEnvironment;
 /**
  * System ethernet tap device
  */
-class EthernetTap : NonCopyable
+class EthernetTap : protected Thread
 {
 public:
 	/**
 	 * Construct a new TAP device
 	 *
+	 * Handler arguments: arg,from,to,etherType,data
+	 * 
 	 * @param renv Runtime environment
 	 * @param mac MAC address of device
 	 * @param mtu MTU of device
+	 * @param handler Handler function to be called when data is received from the tap
+	 * @param arg First argument to handler function
 	 * @throws std::runtime_error Unable to allocate device
 	 */
-	EthernetTap(const RuntimeEnvironment *renv,const MAC &mac,unsigned int mtu)
+	EthernetTap(
+		const RuntimeEnvironment *renv,
+		const MAC &mac,
+		unsigned int mtu,
+		void (*handler)(void *,const MAC &,const MAC &,unsigned int,const Buffer<4096> &),
+		void *arg)
 		throw(std::runtime_error);
 
-	~EthernetTap();
+	/**
+	 * Close tap and shut down thread
+	 *
+	 * This may block for a few seconds while thread exits.
+	 */
+	virtual ~EthernetTap();
+
+	/**
+	 * Perform OS dependent actions on network configuration change detection
+	 */
+	void whack();
 
 	/**
 	 * @return MAC address of this interface
@@ -132,31 +150,10 @@ public:
 	 */
 	void put(const MAC &from,const MAC &to,unsigned int etherType,const void *data,unsigned int len);
 
-	/**
-	 * Get the next packet from the interface, blocking if none is available.
-	 *
-	 * @param from Filled with MAC address of source (normally our own)
-	 * @param to Filled with MAC address of destination
-	 * @param etherType Filled with Ethernet frame type
-	 * @param buf Buffer to fill (must have room for MTU bytes)
-	 * @return Number of bytes read or 0 if none
-	 */
-	unsigned int get(MAC &from,MAC &to,unsigned int &etherType,void *buf);
-
 	/**
 	 * @return OS-specific device or connection name
 	 */
-	std::string deviceName();
-
-	/**
-	 * @return True if tap is open
-	 */
-	bool open() const;
-
-	/**
-	 * Close this tap, invalidating the object and causing get() to abort
-	 */
-	void close();
+	std::string deviceName() const;
 
 	/**
 	 * Fill or modify a set to contain multicast groups for this device
@@ -164,11 +161,18 @@ public:
 	 * This populates a set or, if already populated, modifies it to contain
 	 * only multicast groups in which this device is interested.
 	 *
+	 * This should always include the blind wildcard MulticastGroup (MAC of
+	 * ff:ff:ff:ff:ff:ff and 0 ADI field).
+	 *
 	 * @param groups Set to modify in place
 	 * @return True if set was changed since last call
 	 */
 	bool updateMulticastGroups(std::set<MulticastGroup> &groups);
 
+protected:
+	virtual void main()
+		throw();
+
 private:
 	const MAC _mac;
 	const unsigned int _mtu;
@@ -178,20 +182,14 @@ private:
 	std::set<InetAddress> _ips;
 	Mutex _ips_m;
 
-#if defined(__APPLE__) || defined(__linux__) || defined(linux) || defined(__LINUX__) || defined(__linux)
+	void (*_handler)(void *,const MAC &,const MAC &,unsigned int,const Buffer<4096> &);
+	void *_arg;
 
+#ifdef __UNIX_LIKE__
 	char _dev[16];
-	unsigned char *_putBuf;
-	unsigned char *_getBuf;
 	int _fd;
-
-	bool _isReading;
-	pthread_t _isReadingThreadId;
-	Mutex _isReading_m;
-
-#elif defined(_WIN32) /* -------------------------------------------------- */
-
-#endif /* ----------------------------------------------------------------- */
+	int _shutdownSignalPipe[2];
+#endif
 };
 
 } // namespace ZeroTier

node/Filter.cpp

@@ -0,0 +1,241 @@
+/*
+ * ZeroTier One - Global Peer to Peer Ethernet
+ * Copyright (C) 2012-2013  ZeroTier Networks LLC
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation, either version 3 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ *
+ * --
+ *
+ * ZeroTier may be used and distributed under the terms of the GPLv3, which
+ * are available at: http://www.gnu.org/licenses/gpl-3.0.html
+ *
+ * If you would like to embed ZeroTier into a commercial application or
+ * redistribute it in a modified binary form, please contact ZeroTier Networks
+ * LLC. Start here: http://www.zerotier.com/
+ */
+
+#include <stdint.h>
+
+#include "RuntimeEnvironment.hpp"
+#include "Logger.hpp"
+#include "Filter.hpp"
+#include "Utils.hpp"
+
+namespace ZeroTier {
+
+bool Filter::Rule::operator()(unsigned int etype,const void *data,unsigned int len) const
+{
+	if ((!_etherType)||(_etherType(etype))) { // ethertype is ANY, or matches
+		// Ethertype determines meaning of protocol and port
+		switch(etype) {
+			default:
+				if ((!_protocol)&&(!_port))
+					return true; // match other ethertypes if protocol and port are ANY, since we don't know what to do with them
+				break;
+
+			case ZT_ETHERTYPE_IPV4:
+				if (len > 20) {
+					if ((!_protocol)||(_protocol(((const uint8_t *)data)[9]))) { // IP protocol
+						if (!_port)
+							return true; // protocol matches or is ANY, port is ANY
+
+						// Don't match on fragments beyond fragment 0. If we've blocked
+						// fragment 0, further fragments will fall on deaf ears anyway.
+						if ((Utils::ntoh(((const uint16_t *)data)[3]) & 0x1fff))
+							return false;
+
+						// Internet header length determines where data begins, in multiples of 32 bits
+						unsigned int ihl = 4 * (((const uint8_t *)data)[0] & 0x0f);
+
+						switch(((const uint8_t *)data)[9]) { // port's meaning depends on IP protocol
+							case ZT_IPPROTO_ICMP:
+								return _port(((const uint8_t *)data)[ihl]); // port = ICMP type
+							case ZT_IPPROTO_TCP:
+							case ZT_IPPROTO_UDP:
+							case ZT_IPPROTO_SCTP:
+							case ZT_IPPROTO_UDPLITE:
+								return _port(((const uint16_t *)data)[(ihl / 2) + 1]); // destination port
+						}
+
+						return false; // no match on port
+					}
+				}
+				break;
+
+			case ZT_ETHERTYPE_IPV6:
+				if (len > 40) {
+					// see: http://stackoverflow.com/questions/17518951/is-the-ipv6-header-really-this-nutty
+					int nextHeader = ((const uint8_t *)data)[6];
+					unsigned int pos = 40;
+					while ((pos < len)&&(nextHeader >= 0)&&(nextHeader != 59)) { // 59 == no next header
+						fprintf(stderr,"[rule] V6: start header parse, header %.2x pos %d\n",nextHeader,pos);
+
+						switch(nextHeader) {
+							case 0: // hop-by-hop options
+							case 60: // destination options
+							case 43: // routing
+							case 135: // mobility (mobile IPv6 options)
+								if (_protocol((unsigned int)nextHeader))
+									return true; // match if our goal was to match any of these
+								nextHeader = ((const uint8_t *)data)[pos];
+								pos += 8 + (8 * ((const uint8_t *)data)[pos + 1]);
+								break;
+							case 44: // fragment
+								if (_protocol(44))
+									return true; // match if our goal was to match fragments
+								nextHeader = ((const uint8_t *)data)[pos];
+								pos += 8;
+								break;
+							case ZT_IPPROTO_AH: // AH
+								return _protocol(ZT_IPPROTO_AH); // true if AH is matched protocol, otherwise false since packet will be IPsec
+							case ZT_IPPROTO_ESP: // ESP
+								return _protocol(ZT_IPPROTO_ESP); // true if ESP is matched protocol, otherwise false since packet will be IPsec
+							case ZT_IPPROTO_ICMPV6:
+								if (_protocol(ZT_IPPROTO_ICMPV6)) { // only match ICMPv6 if specified
+									if ((!_port)||(_port(((const uint8_t *)data)[pos])))
+										return true; // protocol matches, port is ANY or matches ICMP type
+								}
+								break;
+							case ZT_IPPROTO_TCP:
+							case ZT_IPPROTO_UDP:
+							case ZT_IPPROTO_SCTP:
+							case ZT_IPPROTO_UDPLITE:
+								// If we encounter any of these, match if protocol matches or is wildcard as
+								// we'll consider these the "real payload" if present.
+								if ((!_protocol)||(_protocol(nextHeader))) {
+									if ((!_port)||(_port(((const uint16_t *)data)[(pos / 2) + 1])))
+										return true; // protocol matches or is ANY, port is ANY or matches
+								}
+								break;
+						}
+
+						fprintf(stderr,"[rule] V6: end header parse, next header %.2x, new pos %d\n",nextHeader,pos);
+					}
+				}
+				break;
+		}
+	}
+
+	return false;
+}
+
+Filter::Filter(const RuntimeEnvironment *renv) :
+	_r(renv)
+{
+}
+
+Filter::~Filter()
+{
+}
+
+void Filter::add(const Rule &r,const Action &a)
+{
+	Mutex::Lock _l(_chain_m);
+	for(std::vector<Entry>::iterator i(_chain.begin());i!=_chain.end();++i) {
+		if (i->rule == r) {
+			_chain.erase(i);
+			break;
+		}
+	}
+	_chain.push_back(Entry(r,a));
+}
+
+std::string Filter::toString(const char *sep) const
+{
+	char buf[256];
+
+	if (!sep)
+		sep = ",";
+
+	std::string s;
+
+	Mutex::Lock _l(_chain_m);
+	for(std::vector<Entry>::const_iterator i(_chain.begin());i!=_chain.end();++i) {
+		bool first = (i == _chain.begin());
+
+		s.push_back('[');
+
+		if (i->rule.etherType()) {
+			if (i->rule.etherType().magnitude() > 1)
+				sprintf(buf,"%u-%u",i->rule.etherType().start,i->rule.etherType().end);
+			else sprintf(buf,"%u",i->rule.etherType().start);
+			s.append(buf);
+		} else s.push_back('*');
+
+		s.push_back(';');
+
+		if (i->rule.protocol()) {
+			if (i->rule.protocol().magnitude() > 1)
+				sprintf(buf,"%u-%u",i->rule.protocol().start,i->rule.protocol().end);
+			else sprintf(buf,"%u",i->rule.protocol().start);
+			s.append(buf);
+		} else s.push_back('*');
+
+		s.push_back(';');
+
+		if (i->rule.port()) {
+			if (i->rule.port().magnitude() > 1)
+				sprintf(buf,"%u-%u",i->rule.port().start,i->rule.port().end);
+			else sprintf(buf,"%u",i->rule.port().start);
+			s.append(buf);
+		} else s.push_back('*');
+
+		s.append("]:");
+
+		switch(i->action) {
+			case ACTION_DENY:
+				s.append("DENY");
+				break;
+			case ACTION_ALLOW:
+				s.append("ALLOW");
+				break;
+			case ACTION_LOG:
+				s.append("LOG");
+				break;
+		}
+
+		if (!first)
+			s.append(sep);
+	}
+
+	return s;
+}
+
+const char *Filter::etherTypeName(const unsigned int etherType)
+	throw()
+{
+	static char tmp[6];
+	switch(etherType) {
+		case ZT_ETHERTYPE_IPV4:
+			return "IPV4";
+		case ZT_ETHERTYPE_ARP:
+			return "ARP";
+		case ZT_ETHERTYPE_RARP:
+			return "RARP";
+		case ZT_ETHERTYPE_ATALK:
+			return "ATALK";
+		case ZT_ETHERTYPE_AARP:
+			return "AARP";
+		case ZT_ETHERTYPE_IPX_A:
+			return "IPX_A";
+		case ZT_ETHERTYPE_IPX_B:
+			return "IPX_B";
+		case ZT_ETHERTYPE_IPV6:
+			return "IPV6";
+	}
+	sprintf(tmp,"%.4x",etherType);
+	return tmp; // technically not thread safe, but we're only going to see this in debugging if ever
+}
+
+} // namespace ZeroTier

node/Filter.hpp

@@ -0,0 +1,300 @@
+/*
+ * ZeroTier One - Global Peer to Peer Ethernet
+ * Copyright (C) 2012-2013  ZeroTier Networks LLC
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation, either version 3 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ *
+ * --
+ *
+ * ZeroTier may be used and distributed under the terms of the GPLv3, which
+ * are available at: http://www.gnu.org/licenses/gpl-3.0.html
+ *
+ * If you would like to embed ZeroTier into a commercial application or
+ * redistribute it in a modified binary form, please contact ZeroTier Networks
+ * LLC. Start here: http://www.zerotier.com/
+ */
+
+#ifndef _ZT_FILTER_HPP
+#define _ZT_FILTER_HPP
+
+#include <stdio.h>
+#include <string.h>
+#include <string>
+#include <vector>
+#include <utility>
+
+#include "Mutex.hpp"
+#include "Range.hpp"
+
+/* Ethernet frame types that might be relevant to us */
+#define ZT_ETHERTYPE_IPV4 0x0800
+#define ZT_ETHERTYPE_ARP 0x0806
+#define ZT_ETHERTYPE_RARP 0x8035
+#define ZT_ETHERTYPE_ATALK 0x809b
+#define ZT_ETHERTYPE_AARP 0x80f3
+#define ZT_ETHERTYPE_IPX_A 0x8137
+#define ZT_ETHERTYPE_IPX_B 0x8138
+#define ZT_ETHERTYPE_IPV6 0x86dd
+
+/* IP protocols we might care about */
+#define ZT_IPPROTO_ICMP 0x01
+#define ZT_IPPROTO_IGMP 0x02
+#define ZT_IPPROTO_TCP 0x06
+#define ZT_IPPROTO_UDP 0x11
+#define ZT_IPPROTO_GRE 0x2f
+#define ZT_IPPROTO_ESP 0x32
+#define ZT_IPPROTO_AH 0x33
+#define ZT_IPPROTO_ICMPV6 0x3a
+#define ZT_IPPROTO_OSPF 0x59
+#define ZT_IPPROTO_IPIP 0x5e
+#define ZT_IPPROTO_IPCOMP 0x6c
+#define ZT_IPPROTO_L2TP 0x73
+#define ZT_IPPROTO_SCTP 0x84
+#define ZT_IPPROTO_FC 0x85
+#define ZT_IPPROTO_UDPLITE 0x88
+#define ZT_IPPROTO_HIP 0x8b
+
+/* IPv4 ICMP types */
+#define ZT_ICMP_ECHO_REPLY 0
+#define ZT_ICMP_DESTINATION_UNREACHABLE 3
+#define ZT_ICMP_SOURCE_QUENCH 4
+#define ZT_ICMP_REDIRECT 5
+#define ZT_ICMP_ALTERNATE_HOST_ADDRESS 6
+#define ZT_ICMP_ECHO_REQUEST 8
+#define ZT_ICMP_ROUTER_ADVERTISEMENT 9
+#define ZT_ICMP_ROUTER_SOLICITATION 10
+#define ZT_ICMP_TIME_EXCEEDED 11
+#define ZT_ICMP_BAD_IP_HEADER 12
+#define ZT_ICMP_TIMESTAMP 13
+#define ZT_ICMP_TIMESTAMP_REPLY 14
+#define ZT_ICMP_INFORMATION_REQUEST 15
+#define ZT_ICMP_INFORMATION_REPLY 16
+#define ZT_ICMP_ADDRESS_MASK_REQUEST 17
+#define ZT_ICMP_ADDRESS_MASK_REPLY 18
+#define ZT_ICMP_TRACEROUTE 30
+#define ZT_ICMP_MOBILE_HOST_REDIRECT 32
+#define ZT_ICMP_MOBILE_REGISTRATION_REQUEST 35
+#define ZT_ICMP_MOBILE_REGISTRATION_REPLY 36
+
+/* IPv6 ICMP types */
+#define ZT_ICMP6_DESTINATION_UNREACHABLE 1
+#define ZT_ICMP6_PACKET_TOO_BIG 2
+#define ZT_ICMP6_TIME_EXCEEDED 3
+#define ZT_ICMP6_PARAMETER_PROBLEM 4
+#define ZT_ICMP6_ECHO_REQUEST 128
+#define ZT_ICMP6_ECHO_REPLY 129
+#define ZT_ICMP6_MULTICAST_LISTENER_QUERY 130
+#define ZT_ICMP6_MULTICAST_LISTENER_REPORT 131
+#define ZT_ICMP6_MULTICAST_LISTENER_DONE 132
+#define ZT_ICMP6_ROUTER_SOLICITATION 133
+#define ZT_ICMP6_ROUTER_ADVERTISEMENT 134
+#define ZT_ICMP6_NEIGHBOR_SOLICITATION 135
+#define ZT_ICMP6_NEIGHBOR_ADVERTISEMENT 136
+#define ZT_ICMP6_REDIRECT_MESSAGE 137
+#define ZT_ICMP6_ROUTER_RENUMBERING 138
+#define ZT_ICMP6_NODE_INFORMATION_QUERY 139
+#define ZT_ICMP6_NODE_INFORMATION_RESPONSE 140
+#define ZT_ICMP6_INV_NEIGHBOR_SOLICITATION 141
+#define ZT_ICMP6_INV_NEIGHBOR_ADVERTISEMENT 142
+#define ZT_ICMP6_MLDV2 143
+#define ZT_ICMP6_HOME_AGENT_ADDRESS_DISCOVERY_REQUEST 144
+#define ZT_ICMP6_HOME_AGENT_ADDRESS_DISCOVERY_REPLY 145
+#define ZT_ICMP6_MOBILE_PREFIX_SOLICITATION 146
+#define ZT_ICMP6_MOBILE_PREFIX_ADVERTISEMENT 147
+#define ZT_ICMP6_CERTIFICATION_PATH_SOLICITATION 148
+#define ZT_ICMP6_CERTIFICATION_PATH_ADVERTISEMENT 149
+#define ZT_ICMP6_MULTICAST_ROUTER_ADVERTISEMENT 151
+#define ZT_ICMP6_MULTICAST_ROUTER_SOLICITATION 152
+#define ZT_ICMP6_MULTICAST_ROUTER_TERMINATION 153
+#define ZT_ICMP6_RPL_CONTROL_MESSAGE 155
+
+namespace ZeroTier {
+
+class RuntimeEnvironment;
+
+/**
+ * A simple Ethernet frame level filter supporting basic IP port DENY
+ */
+class Filter
+{
+public:
+	/**
+	 * A filter rule
+	 *
+	 * This behaves as an immutable value object.
+	 */
+	class Rule
+	{
+	public:
+		Rule()
+			throw() :
+			_etherType(),
+			_protocol(),
+			_port()
+		{
+		}
+
+		/**
+		 * Construct a new rule
+		 *
+		 * @param etype Ethernet type or empty range for ANY
+		 * @param prot Protocol or empty range for ANY (meaning depends on ethertype, e.g. IP protocol numbers)
+		 * @param prt Port or empty range for ANY (only applies to some protocols)
+		 */
+		Rule(const Range<unsigned int> &etype,const Range<unsigned int> &prot,const Range<unsigned int> &prt)
+			throw() :
+			_etherType(etype),
+			_protocol(prot),
+			_port(prt)
+		{
+		}
+
+		inline const Range<unsigned int> &etherType() const throw() { return _etherType; }
+		inline const Range<unsigned int> &protocol() const throw() { return _protocol; }
+		inline const Range<unsigned int> &port() const throw() { return _port; }
+
+		/**
+		 * Test this rule against a frame
+		 *
+		 * @param etype Type of ethernet frame
+		 * @param data Ethernet frame data
+		 * @param len Length of ethernet frame
+		 * @return True if rule matches
+		 */
+		bool operator()(unsigned int etype,const void *data,unsigned int len) const;
+
+		inline bool operator==(const Rule &r) const throw() { return ((_etherType == r._etherType)&&(_protocol == r._protocol)&&(_port == r._port)); }
+		inline bool operator!=(const Rule &r) const throw() { return !(*this == r); }
+		inline bool operator<(const Rule &r) const
+			throw()
+		{
+			if (_etherType < r._etherType)
+				return true;
+			else if (_etherType == r._etherType) {
+				if (_protocol < r._protocol)
+					return true;
+				else if (_protocol == r._protocol) {
+					if (_port < r._port)
+						return true;
+				}
+			}
+			return false;
+		}
+		inline bool operator>(const Rule &r) const throw() { return (r < *this); }
+		inline bool operator<=(const Rule &r) const throw() { return !(r < *this); }
+		inline bool operator>=(const Rule &r) const throw() { return !(*this < r); }
+
+	private:
+		Range<unsigned int> _etherType;
+		Range<unsigned int> _protocol;
+		Range<unsigned int> _port;
+	};
+
+	/**
+	 * Action if a rule matches
+	 */
+	enum Action
+	{
+		ACTION_DENY = 0,
+		ACTION_ALLOW = 1,
+		ACTION_LOG = 2
+	};
+
+	/**
+	 * Entry in filter chain
+	 */
+	struct Entry
+	{
+		Entry() {}
+		Entry(const Rule &r,const Action &a) :
+			rule(r),
+			action(a)
+		{
+		}
+
+		Rule rule;
+		Action action;
+	};
+
+	Filter(const RuntimeEnvironment *renv);
+	~Filter();
+
+	/**
+	 * Remove all filter entries
+	 */
+	inline void clear()
+	{
+		Mutex::Lock _l(_chain_m);
+		_chain.clear();
+	}
+
+	/**
+	 * Append a rule/action pair to this chain
+	 *
+	 * If an identical rule already exists it is removed and a new entry is
+	 * added to the end with the new action. (Two identical rules with the
+	 * same action wouldn't make sense.)
+	 *
+	 * @param r Rule to add
+	 * @param a Action if rule matches
+	 */
+	void add(const Rule &r,const Action &a);
+
+	/**
+	 * @return Number of rules in filter chain
+	 */
+	inline unsigned int length() const
+		throw()
+	{
+		Mutex::Lock _l(_chain_m);
+		return _chain.size();
+	}
+
+	/**
+	 * @return Entry in filter chain or null entry if out of bounds
+	 */
+	inline Entry operator[](const unsigned int i) const
+		throw()
+	{
+		Mutex::Lock _l(_chain_m);
+		if (i < _chain.size())
+			return _chain[i];
+		return Entry();
+	}
+
+	/**
+	 * Get a string representation of this filter
+	 *
+	 * @param sep Separator between filter rules, or NULL for comma (default)
+	 * @return Human-readable string
+	 */
+	std::string toString(const char *sep = (const char *)0) const;
+
+	/**
+	 * @param etherType Ethernet type ID
+	 * @return Name of Ethernet protocol (e.g. ARP, IPV4)
+	 */
+	static const char *etherTypeName(const unsigned int etherType)
+		throw();
+
+private:
+	const RuntimeEnvironment *_r;
+
+	std::vector<Entry> _chain;
+	Mutex _chain_m;
+};
+
+} // namespace ZeroTier
+
+#endif

node/Http.cpp

@@ -173,7 +173,7 @@ void Http::Request::main()
 				addrList->sort();
 				addrList->unique();
 				unsigned int i = 0,k = 0;
-				k = Utils::randomInt<unsigned int>() % addrList->size();
+				k = rand() % addrList->size();
 				std::list<InetAddress>::iterator a(addrList->begin());
 				while (i++ != k) ++a;
 				addr = &(*a);

node/MulticastGroup.hpp

@@ -36,7 +36,7 @@
 namespace ZeroTier {
 
 /**
- * A multicast group composed of a multicast MAC and a 64-bit ADI field
+ * A multicast group composed of a multicast MAC and a 32-bit ADI field
  *
  * ADI stands for additional distinguishing information. ADI is primarily for
  * adding additional information to broadcast (ff:ff:ff:ff:ff:ff) memberships,

node/Multicaster.hpp

@@ -0,0 +1,325 @@
+/*
+ * ZeroTier One - Global Peer to Peer Ethernet
+ * Copyright (C) 2012-2013  ZeroTier Networks LLC
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation, either version 3 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ *
+ * --
+ *
+ * ZeroTier may be used and distributed under the terms of the GPLv3, which
+ * are available at: http://www.gnu.org/licenses/gpl-3.0.html
+ *
+ * If you would like to embed ZeroTier into a commercial application or
+ * redistribute it in a modified binary form, please contact ZeroTier Networks
+ * LLC. Start here: http://www.zerotier.com/
+ */
+
+#ifndef _ZT_MULTICASTER_HPP
+#define _ZT_MULTICASTER_HPP
+
+#include <stdint.h>
+#include <string.h>
+
+#include <openssl/sha.h>
+
+#include <utility>
+#include <algorithm>
+#include <map>
+#include <set>
+#include <vector>
+#include <string>
+
+#include "Constants.hpp"
+#include "Buffer.hpp"
+#include "Packet.hpp"
+#include "MulticastGroup.hpp"
+#include "Utils.hpp"
+#include "MAC.hpp"
+#include "Address.hpp"
+#include "SharedPtr.hpp"
+#include "BloomFilter.hpp"
+#include "Identity.hpp"
+#include "CMWC4096.hpp"
+
+// Maximum sample size to pick during choice of multicast propagation peers
+#define ZT_MULTICAST_PICK_MAX_SAMPLE_SIZE (ZT_MULTICAST_PROPAGATION_BREADTH * 8)
+
+namespace ZeroTier {
+
+/**
+ * Multicast propagation engine
+ *
+ * This is written as a generic class so that it can be mocked and tested
+ * in simulation. It also always takes 'now' as an argument, permitting
+ * running in simulated time.
+ */
+class Multicaster
+{
+public:
+	/**
+	 * 256-bit simple bloom filter included with multicast frame packets
+	 */
+	typedef BloomFilter<ZT_PROTO_VERB_MULTICAST_FRAME_BLOOM_FILTER_SIZE_BITS> MulticastBloomFilter;
+
+	Multicaster()
+		throw()
+	{
+		memset(_multicastHistory,0,sizeof(_multicastHistory));
+		_multicastHistoryPtr = 0;
+	}
+
+	/**
+	 * Generate a signature of a multicast packet using an identity
+	 *
+	 * @param id Identity to sign with (must have secret key portion)
+	 * @param nwid Network ID
+	 * @param from MAC address of sender
+	 * @param to Multicast group
+	 * @param etherType 16-bit ethernet type
+	 * @param data Ethernet frame data
+	 * @param len Length of frame
+	 * @return ECDSA signature
+	 */
+	static inline std::string signMulticastPacket(const Identity &id,uint64_t nwid,const MAC &from,const MulticastGroup &to,unsigned int etherType,const void *data,unsigned int len)
+	{
+		unsigned char digest[32];
+		_hashMulticastPacketForSig(nwid,from,to,etherType,data,len,digest);
+		return id.sign(digest);
+	}
+
+	/**
+	 * Verify a signature from a multicast packet
+	 *
+	 * @param id Identity of original signer
+	 * @param nwid Network ID
+	 * @param from MAC address of sender
+	 * @param to Multicast group
+	 * @param etherType 16-bit ethernet type
+	 * @param data Ethernet frame data
+	 * @param len Length of frame
+	 * @param signature ECDSA signature
+	 * @param siglen Length of signature in bytes
+	 * @return ECDSA signature
+	 */
+	static bool verifyMulticastPacket(const Identity &id,uint64_t nwid,const MAC &from,const MulticastGroup &to,unsigned int etherType,const void *data,unsigned int len,const void *signature,unsigned int siglen)
+	{
+		unsigned char digest[32];
+		_hashMulticastPacketForSig(nwid,from,to,etherType,data,len,digest);
+		return id.verifySignature(digest,signature,siglen);
+	}
+
+	/**
+	 * Compute the CRC64 code for multicast deduplication
+	 *
+	 * @param nwid Network ID
+	 * @param from Sender MAC
+	 * @param to Destination multicast group
+	 * @param etherType Ethernet frame type
+	 * @param payload Multicast frame data
+	 * @param len Length of frame
+	 */
+	static inline uint64_t computeMulticastDedupCrc(
+		uint64_t nwid,
+		const MAC &from,
+		const MulticastGroup &to,
+		unsigned int etherType,
+		const void *payload,
+		unsigned int len)
+		throw()
+	{
+		// This CRC is only used locally, so byte order issues and
+		// such don't matter. It can also be changed without protocol
+		// impact.
+		uint64_t crc = Utils::crc64(0,from.data,6);
+		crc = Utils::crc64(crc,to.mac().data,6);
+		crc ^= (uint64_t)to.adi();
+		crc ^= (uint64_t)etherType;
+		crc = Utils::crc64(crc,payload,len);
+		crc ^= nwid; // also include network ID in CRC
+		return crc;
+	}
+
+	/**
+	 * Check multicast history to see if this is a duplicate
+	 *
+	 * @param crc Multicast CRC
+	 * @param now Current time
+	 * @return True if this appears to be a duplicate to within history expiration time
+	 */
+	inline bool checkDuplicate(uint64_t crc,uint64_t now) const
+		throw()
+	{
+		for(unsigned int i=0;i<ZT_MULTICAST_DEDUP_HISTORY_LENGTH;++i) {
+			if ((_multicastHistory[i][0] == crc)&&((now - _multicastHistory[i][1]) < ZT_MULTICAST_DEDUP_HISTORY_EXPIRE))
+				return true;
+		}
+		return false;
+	}
+
+	/**
+	 * Add a multicast CRC to the multicast deduplication history
+	 *
+	 * @param crc Multicast CRC
+	 * @param now Current time
+	 */
+	inline void addToDedupHistory(uint64_t crc,uint64_t now)
+		throw()
+	{
+		unsigned int mhi = ++_multicastHistoryPtr % ZT_MULTICAST_DEDUP_HISTORY_LENGTH;
+		_multicastHistory[mhi][0] = crc;
+		_multicastHistory[mhi][1] = now;
+	}
+
+	/**
+	 * Update the most recent LIKE time for an address in a given multicast group on a given network
+	 *
+	 * @param nwid Network ID
+	 * @param mg Multicast group
+	 * @param addr Address that likes group on given network
+	 * @param now Current timestamp
+	 */
+	inline void likesMulticastGroup(const uint64_t nwid,const MulticastGroup &mg,const Address &addr,const uint64_t now)
+	{
+		Mutex::Lock _l(_multicastMemberships_m);
+		std::vector<MulticastMembership> &memberships = _multicastMemberships[MulticastChannel(nwid,mg)];
+		for(std::vector<MulticastMembership>::iterator mm(memberships.begin());mm!=memberships.end();++mm) {
+			if (mm->first == addr) {
+				mm->second = now;
+				return;
+			}
+		}
+		memberships.push_back(MulticastMembership(addr,now));
+	}
+
+	/**
+	 * Choose peers to send a propagating multicast to
+	 *
+	 * @param topology Topology object or mock thereof
+	 * @param nwid Network ID
+	 * @param mg Multicast group
+	 * @param originalSubmitter Original submitter of multicast message to network
+	 * @param upstream Address from which message originated, or null (0) address if none
+	 * @param bf Bloom filter, updated in place with sums of addresses in chosen peers and/or decay
+	 * @param max Maximum number of peers to pick
+	 * @param peers Array of objects of type P to fill with up to [max] peers
+	 * @param now Current timestamp
+	 * @return Number of peers actually stored in peers array
+	 * @tparam T Type of topology, which is Topology in running code or a mock in simulation
+	 * @tparam P Type of peers, which is SharedPtr<Peer> in running code or a mock in simulation (mock must behave like a pointer type)
+	 */
+	template<typename T,typename P>
+	inline unsigned int pickNextPropagationPeers(
+		CMWC4096 &prng,
+		T &topology,
+		uint64_t nwid,
+		const MulticastGroup &mg,
+		const Address &originalSubmitter,
+		const Address &upstream,
+		MulticastBloomFilter &bf,
+		unsigned int max,
+		P *peers,
+		uint64_t now)
+	{
+		typename std::set< P,_PeerPropagationPrioritySortOrder<P> > toConsider;
+
+		// Pick up to ZT_MULTICAST_PICK_MAX_SAMPLE_SIZE peers that have
+		// subscribed to this channel and that are not in bloom filter.
+		// Pick randomly from subscribers, but place into a set that is
+		// sorted in descending order of time of most recent unicast
+		// frame transfer. (Implicit social ordering.) Also ignore original
+		// submitter and upstream, since we know these have seen this
+		// message.
+		{
+			Mutex::Lock _l(_multicastMemberships_m);
+			std::map< MulticastChannel,std::vector<MulticastMembership> >::iterator mm(_multicastMemberships.find(MulticastChannel(nwid,mg)));
+			if ((mm != _multicastMemberships.end())&&(!mm->second.empty())) {
+				for(unsigned int stries=0;stries<ZT_MULTICAST_PICK_MAX_SAMPLE_SIZE;++stries) {
+					MulticastMembership &m = mm->second[prng.next32() % mm->second.size()];
+					if (((now - m.second) < ZT_MULTICAST_LIKE_EXPIRE)&&(!bf.contains(m.first.sum()))&&(m.first != originalSubmitter)&&(m.first != upstream)) {
+						P peer(topology.getPeer(m.first));
+						if (peer)
+							toConsider.insert(peer);
+					}
+				}
+			}
+		}
+
+		// The first peers in toConsider will be the 'best'
+		unsigned int chosen = 0;
+		for(typename std::set< P,_PeerPropagationPrioritySortOrder<P> >::iterator i(toConsider.begin());((i!=toConsider.end())&&(chosen < max));++i)
+			bf.set((peers[chosen++] = *i)->address().sum());
+
+		// Add a supernode if there are fewer than the desired
+		// number of recipients.
+		if (chosen < max) {
+			P peer = topology.getBestSupernode(&originalSubmitter,1,true);
+			if (peer)
+				peers[chosen++] = peer;
+		}
+
+		return chosen;
+	}
+
+private:
+	// Sort order for chosen propagation peers
+	template<typename P>
+	struct _PeerPropagationPrioritySortOrder
+	{
+		inline bool operator()(const P &p1,const P &p2) const
+		{
+			return (p1->lastUnicastFrame() > p2->lastUnicastFrame());
+		}
+	};
+
+	static inline void _hashMulticastPacketForSig(uint64_t nwid,const MAC &from,const MulticastGroup &to,unsigned int etherType,const void *data,unsigned int len,unsigned char *digest)
+		throw()
+	{
+		unsigned char zero = 0;
+		SHA256_CTX sha;
+		SHA256_Init(&sha);
+		uint64_t _nwid = Utils::hton(nwid);
+		SHA256_Update(&sha,(unsigned char *)&_nwid,sizeof(_nwid));
+		SHA256_Update(&sha,&zero,1);
+		SHA256_Update(&sha,(unsigned char *)from.data,6);
+		SHA256_Update(&sha,&zero,1);
+		SHA256_Update(&sha,(unsigned char *)to.mac().data,6);
+		SHA256_Update(&sha,&zero,1);
+		uint32_t _adi = Utils::hton(to.adi());
+		SHA256_Update(&sha,(unsigned char *)&_adi,sizeof(_adi));
+		SHA256_Update(&sha,&zero,1);
+		uint16_t _etype = Utils::hton((uint16_t)etherType);
+		SHA256_Update(&sha,(unsigned char *)&_etype,sizeof(_etype));
+		SHA256_Update(&sha,&zero,1);
+		SHA256_Update(&sha,(unsigned char *)data,len);
+		SHA256_Final(digest,&sha);
+	}
+
+	// ring buffer: [0] - CRC, [1] - timestamp
+	uint64_t _multicastHistory[ZT_MULTICAST_DEDUP_HISTORY_LENGTH][2];
+	volatile unsigned int _multicastHistoryPtr;
+
+	// A multicast channel, essentially a pub/sub channel. It consists of a
+	// network ID and a multicast group within that network.
+	typedef std::pair<uint64_t,MulticastGroup> MulticastChannel;
+
+	// Address and time of last LIKE
+	typedef std::pair<Address,uint64_t> MulticastMembership;
+
+	std::map< MulticastChannel,std::vector<MulticastMembership> > _multicastMemberships;
+	Mutex _multicastMemberships_m;
+};
+
+} // namespace ZeroTier
+
+#endif

node/Network.cpp

@@ -32,49 +32,29 @@ namespace ZeroTier {
 
 Network::Network(const RuntimeEnvironment *renv,uint64_t id)
 	throw(std::runtime_error) :
-	Thread(),
 	_r(renv),
 	_id(id),
-	_tap(renv,renv->identity.address().toMAC(),ZT_IF_MTU),
+	_tap(renv,renv->identity.address().toMAC(),ZT_IF_MTU,&_CBhandleTapData,this),
 	_members(),
 	_open(false),
 	_lock()
 {
-	TRACE("new network %llu created, TAP device: %s",id,_tap.deviceName().c_str());
-	start();
 }
 
 Network::~Network()
 {
-	_tap.close();
-	join();
-	TRACE("network %llu (%s) closed",_id,_tap.deviceName().c_str());
 }
 
-void Network::main()
-	throw()
+void Network::_CBhandleTapData(void *arg,const MAC &from,const MAC &to,unsigned int etherType,const Buffer<4096> &data)
 {
-	Buffer<4096> buf;
-	MAC from,to;
-	unsigned int etherType = 0;
-
-	while (_tap.open()) {
-		unsigned int len = _tap.get(from,to,etherType,buf.data());
-		if (len) {
-			buf.setSize(len);
-			try {
-				if (!*__refCount)
-					break; // sanity check
-				_r->sw->onLocalEthernet(SharedPtr<Network>(this),from,to,etherType,buf);
-			} catch (std::exception &exc) {
-				TRACE("unexpected exception handling local packet: %s",exc.what());
-			} catch ( ... ) {
-				TRACE("unexpected exception handling local packet");
-			}
-		} else break;
+	const RuntimeEnvironment *_r = ((Network *)arg)->_r;
+	try {
+		_r->sw->onLocalEthernet(SharedPtr<Network>((Network *)arg),from,to,etherType,data);
+	} catch (std::exception &exc) {
+		TRACE("unexpected exception handling local packet: %s",exc.what());
+	} catch ( ... ) {
+		TRACE("unexpected exception handling local packet");
 	}
-
-	TRACE("network %llu thread terminating",_id);
 }
 
 } // namespace ZeroTier

node/Network.hpp

@@ -40,8 +40,9 @@
 #include "SharedPtr.hpp"
 #include "AtomicCounter.hpp"
 #include "RuntimeEnvironment.hpp"
-#include "Thread.hpp"
 #include "MulticastGroup.hpp"
+#include "NonCopyable.hpp"
+#include "MAC.hpp"
 
 namespace ZeroTier {
 
@@ -50,17 +51,17 @@ class NodeConfig;
 /**
  * Local network endpoint
  */
-class Network : protected Thread
+class Network : NonCopyable
 {
 	friend class SharedPtr<Network>;
 	friend class NodeConfig;
 
 private:
-	virtual ~Network();
-
 	Network(const RuntimeEnvironment *renv,uint64_t id)
 		throw(std::runtime_error);
 
+	~Network();
+
 public:
 	/**
 	 * @return Network ID
@@ -141,11 +142,9 @@ public:
 		return _multicastGroups;
 	}
 
-protected:
-	virtual void main()
-		throw();
-
 private:
+	static void _CBhandleTapData(void *arg,const MAC &from,const MAC &to,unsigned int etherType,const Buffer<4096> &data);
+
 	const RuntimeEnvironment *_r;
 	uint64_t _id;
 	EthernetTap _tap;

node/Node.cpp

@@ -64,6 +64,8 @@
 #include "Network.hpp"
 #include "MulticastGroup.hpp"
 #include "Mutex.hpp"
+#include "Multicaster.hpp"
+#include "CMWC4096.hpp"
 
 #include "../version.h"
 
@@ -74,8 +76,9 @@ struct _NodeImpl
 	RuntimeEnvironment renv;
 	std::string reasonForTerminationStr;
 	Node::ReasonForTermination reasonForTermination;
-	bool started;
-	bool running;
+	volatile bool started;
+	volatile bool running;
+	volatile bool updateStatusNow;
 	volatile bool terminateNow;
 
 	// Helper used to rapidly terminate from run()
@@ -104,6 +107,7 @@ Node::Node(const char *hp,const char *urlPrefix,const char *configAuthorityIdent
 	impl->reasonForTermination = Node::NODE_RUNNING;
 	impl->started = false;
 	impl->running = false;
+	impl->updateStatusNow = false;
 	impl->terminateNow = false;
 }
 
@@ -114,8 +118,10 @@ Node::~Node()
 	delete impl->renv.sysEnv;
 	delete impl->renv.topology;
 	delete impl->renv.sw;
+	delete impl->renv.multicaster;
 	delete impl->renv.demarc;
 	delete impl->renv.nc;
+	delete impl->renv.prng;
 	delete impl->renv.log;
 
 	delete impl;
@@ -149,6 +155,8 @@ Node::ReasonForTermination Node::run()
 
 		TRACE("initializing...");
 
+		_r->prng = new CMWC4096();
+
 		if (!_r->configAuthority.fromString(_r->configAuthorityIdentityStr))
 			return impl->terminateBecause(Node::NODE_UNRECOVERABLE_ERROR,"configuration authority identity is not valid");
 
@@ -189,8 +197,11 @@ Node::ReasonForTermination Node::run()
 		std::string ovsPath(_r->homePath + ZT_PATH_SEPARATOR_S + "thisdeviceismine");
 		if (((Utils::now() - Utils::getLastModified(ovsPath.c_str())) >= ZT_OVS_GENERATE_NEW_IF_OLDER_THAN)||(!Utils::readFile(ovsPath.c_str(),_r->ownershipVerificationSecret))) {
 			_r->ownershipVerificationSecret = "";
-			for(unsigned int i=0;i<24;++i)
-				_r->ownershipVerificationSecret.push_back("abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789"[Utils::randomInt<unsigned int>() % 62]);
+			unsigned int securern = 0;
+			for(unsigned int i=0;i<24;++i) {
+				Utils::getSecureRandom(&securern,sizeof(securern));
+				_r->ownershipVerificationSecret.push_back("abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789"[securern % 62]);
+			}
 			_r->ownershipVerificationSecret.append(ZT_EOL_S);
 			if (!Utils::writeFile(ovsPath.c_str(),_r->ownershipVerificationSecret))
 				return impl->terminateBecause(Node::NODE_UNRECOVERABLE_ERROR,"could not write 'thisdeviceismine' (home path not writable?)");
@@ -209,6 +220,7 @@ Node::ReasonForTermination Node::run()
 		// watcher.
 		_r->nc = new NodeConfig(_r,_r->autoconfUrlPrefix + _r->identity.address().toString());
 		_r->demarc = new Demarc(_r);
+		_r->multicaster = new Multicaster();
 		_r->sw = new Switch(_r);
 		_r->topology = new Topology(_r,(_r->homePath + ZT_PATH_SEPARATOR_S + "peer.db").c_str());
 		_r->sysEnv = new SysEnv(_r);
@@ -236,6 +248,8 @@ Node::ReasonForTermination Node::run()
 	}
 
 	try {
+		std::string statusPath(_r->homePath + ZT_PATH_SEPARATOR_S + "status");
+
 		uint64_t lastPingCheck = 0;
 		uint64_t lastTopologyClean = Utils::now(); // don't need to do this immediately
 		uint64_t lastNetworkFingerprintCheck = 0;
@@ -243,6 +257,7 @@ Node::ReasonForTermination Node::run()
 		uint64_t networkConfigurationFingerprint = _r->sysEnv->getNetworkConfigurationFingerprint();
 		uint64_t lastMulticastCheck = 0;
 		uint64_t lastMulticastAnnounceAll = 0;
+		uint64_t lastStatusUpdate = 0;
 		long lastDelayDelta = 0;
 
 		LOG("%s starting version %s",_r->identity.address().toString().c_str(),versionString());
@@ -268,11 +283,12 @@ Node::ReasonForTermination Node::run()
 				lastNetworkFingerprintCheck = now;
 				uint64_t fp = _r->sysEnv->getNetworkConfigurationFingerprint();
 				if (fp != networkConfigurationFingerprint) {
-					LOG("netconf fingerprint change: %.16llx != %.16llx, pinging all peers",networkConfigurationFingerprint,fp);
+					LOG("netconf fingerprint change: %.16llx != %.16llx, resyncing with network",networkConfigurationFingerprint,fp);
 					networkConfigurationFingerprint = fp;
 					pingAll = true;
 					lastAutoconfigureCheck = 0; // check autoconf after network config change
 					lastMulticastCheck = 0; // check multicast group membership after network config change
+					_r->nc->whackAllTaps(); // call whack() on all tap devices
 				}
 			}
 
@@ -373,10 +389,24 @@ Node::ReasonForTermination Node::run()
 				_r->topology->clean(); // happens in background
 			}
 
+			if (((now - lastStatusUpdate) >= ZT_STATUS_OUTPUT_PERIOD)||(impl->updateStatusNow)) {
+				lastStatusUpdate = now;
+				impl->updateStatusNow = false;
+				FILE *statusf = ::fopen(statusPath.c_str(),"w");
+				if (statusf) {
+					try {
+						_r->topology->eachPeer(Topology::DumpPeerStatistics(statusf));
+					} catch ( ... ) {
+						TRACE("unexpected exception updating status dump");
+					}
+					::fclose(statusf);
+				}
+			}
+
 			try {
 				unsigned long delay = std::min((unsigned long)ZT_MIN_SERVICE_LOOP_INTERVAL,_r->sw->doTimerTasks());
 				uint64_t start = Utils::now();
-				Thread::sleep(delay);
+				_r->mainLoopWaitCondition.wait(delay);
 				lastDelayDelta = (long)(Utils::now() - start) - (long)delay;
 			} catch (std::exception &exc) {
 				LOG("unexpected exception running Switch doTimerTasks: %s",exc.what());
@@ -391,11 +421,6 @@ Node::ReasonForTermination Node::run()
 	return impl->terminateBecause(Node::NODE_NORMAL_TERMINATION,"normal termination");
 }
 
-/**
- * Obtain a human-readable reason for node termination
- *
- * @return Reason for node termination or NULL if run() has not returned
- */
 const char *Node::reasonForTermination() const
 	throw()
 {
@@ -404,17 +429,18 @@ const char *Node::reasonForTermination() const
 	return ((_NodeImpl *)_impl)->reasonForTerminationStr.c_str();
 }
 
-/**
- * Cause run() to return with NODE_NORMAL_TERMINATION
- *
- * This can be called from a signal handler or another thread to signal a
- * running node to shut down. Shutdown may take a few seconds, so run()
- * may not return instantly. Multiple calls are ignored.
- */
 void Node::terminate()
 	throw()
 {
 	((_NodeImpl *)_impl)->terminateNow = true;
+	((_NodeImpl *)_impl)->renv.mainLoopWaitCondition.signal();
+}
+
+void Node::updateStatusNow()
+	throw()
+{
+	((_NodeImpl *)_impl)->updateStatusNow = true;
+	((_NodeImpl *)_impl)->renv.mainLoopWaitCondition.signal();
 }
 
 class _VersionStringMaker

node/Node.hpp

@@ -98,6 +98,12 @@ public:
 	void terminate()
 		throw();
 
+	/**
+	 * Update the status file in the home directory on next service loop
+	 */
+	void updateStatusNow()
+		throw();
+
 	/**
 	 * Get the ZeroTier version in major.minor.revision string format
 	 * 

node/NodeConfig.cpp

@@ -152,7 +152,7 @@ void NodeConfig::__CBautoconfHandler(const std::string &lastModified,const std::
 									if (rawAddr.length() == ZT_ADDRESS_LENGTH) {
 										Address addr(rawAddr.data());
 										if ((addr)&&(!addr.isReserved())) {
-											TRACE("network %llu member: %s",nwid,addr.toString().c_str());
+											//TRACE("network %llu member: %s",nwid,addr.toString().c_str());
 											nw->_members.insert(addr);
 										}
 									}

node/NodeConfig.hpp

@@ -78,6 +78,17 @@ public:
 		return nwlist;
 	}
 
+	/**
+	 * Call whack() on all networks' tap devices
+	 */
+	inline void whackAllTaps()
+	{
+		std::vector< SharedPtr<Network> > nwlist;
+		Mutex::Lock _l(_networks_m);
+		for(std::map< uint64_t,SharedPtr<Network> >::const_iterator n(_networks.begin());n!=_networks.end();++n)
+			n->second->tap().whack();
+	}
+
 	/**
 	 * @param nwid Network ID
 	 * @return True if this network exists

node/Packet.hpp

@@ -45,8 +45,13 @@
 
 /**
  * Protocol version
+ *
+ * 1 - 0.2.0 ... 0.2.5
+ * 2 - 0.3.0 ...
+ *   * Added signature and originating peer to multicast frame
+ *   * Double size of multicast frame bloom filter
  */
-#define ZT_PROTO_VERSION 1
+#define ZT_PROTO_VERSION 2
 
 /**
  * Maximum hop count allowed by packet structure (3 bits, 0-7)
@@ -122,7 +127,9 @@
  */
 #define ZT_PROTO_MIN_FRAGMENT_LENGTH ZT_PACKET_FRAGMENT_IDX_PAYLOAD
 
-#define ZT_PROTO_VERB_MULTICAST_FRAME_BLOOM_FILTER_SIZE 32
+// Size of bloom filter used in multicast propagation
+#define ZT_PROTO_VERB_MULTICAST_FRAME_BLOOM_FILTER_SIZE_BITS 512
+#define ZT_PROTO_VERB_MULTICAST_FRAME_BLOOM_FILTER_SIZE_BYTES 64
 
 // Field incides for parsing verbs
 #define ZT_PROTO_VERB_HELLO_IDX_PROTOCOL_VERSION (ZT_PACKET_IDX_PAYLOAD)
@@ -146,15 +153,18 @@
 #define ZT_PROTO_VERB_FRAME_IDX_NETWORK_ID (ZT_PACKET_IDX_PAYLOAD)
 #define ZT_PROTO_VERB_FRAME_IDX_ETHERTYPE (ZT_PROTO_VERB_FRAME_IDX_NETWORK_ID + 8)
 #define ZT_PROTO_VERB_FRAME_IDX_PAYLOAD (ZT_PROTO_VERB_FRAME_IDX_ETHERTYPE + 2)
-#define ZT_PROTO_VERB_MULTICAST_FRAME_IDX_NETWORK_ID (ZT_PACKET_IDX_PAYLOAD)
-#define ZT_PROTO_VERB_MULTICAST_FRAME_IDX_MULTICAST_MAC (ZT_PROTO_VERB_MULTICAST_FRAME_IDX_NETWORK_ID + 8)
-#define ZT_PROTO_VERB_MULTICAST_FRAME_IDX_ADI (ZT_PROTO_VERB_MULTICAST_FRAME_IDX_MULTICAST_MAC + 6)
-#define ZT_PROTO_VERB_MULTICAST_FRAME_IDX_BLOOM (ZT_PROTO_VERB_MULTICAST_FRAME_IDX_ADI + 4)
-#define ZT_PROTO_VERB_MULTICAST_FRAME_IDX_HOPS (ZT_PROTO_VERB_MULTICAST_FRAME_IDX_BLOOM + ZT_PROTO_VERB_MULTICAST_FRAME_BLOOM_FILTER_SIZE)
-#define ZT_PROTO_VERB_MULTICAST_FRAME_IDX_LOAD_FACTOR (ZT_PROTO_VERB_MULTICAST_FRAME_IDX_HOPS + 1)
-#define ZT_PROTO_VERB_MULTICAST_FRAME_IDX_FROM_MAC (ZT_PROTO_VERB_MULTICAST_FRAME_IDX_LOAD_FACTOR + 2)
-#define ZT_PROTO_VERB_MULTICAST_FRAME_IDX_ETHERTYPE (ZT_PROTO_VERB_MULTICAST_FRAME_IDX_FROM_MAC + 6)
-#define ZT_PROTO_VERB_MULTICAST_FRAME_IDX_PAYLOAD (ZT_PROTO_VERB_MULTICAST_FRAME_IDX_ETHERTYPE + 2)
+#define ZT_PROTO_VERB_MULTICAST_FRAME_IDX_FLAGS (ZT_PACKET_IDX_PAYLOAD)
+#define ZT_PROTO_VERB_MULTICAST_FRAME_IDX_NETWORK_ID (ZT_PROTO_VERB_MULTICAST_FRAME_IDX_FLAGS + 1)
+#define ZT_PROTO_VERB_MULTICAST_FRAME_IDX_SUBMITTER_ADDRESS (ZT_PROTO_VERB_MULTICAST_FRAME_IDX_NETWORK_ID + 8)
+#define ZT_PROTO_VERB_MULTICAST_FRAME_IDX_SOURCE_MAC (ZT_PROTO_VERB_MULTICAST_FRAME_IDX_SUBMITTER_ADDRESS + 5)
+#define ZT_PROTO_VERB_MULTICAST_FRAME_IDX_DESTINATION_MAC (ZT_PROTO_VERB_MULTICAST_FRAME_IDX_SOURCE_MAC + 6)
+#define ZT_PROTO_VERB_MULTICAST_FRAME_IDX_ADI (ZT_PROTO_VERB_MULTICAST_FRAME_IDX_DESTINATION_MAC + 6)
+#define ZT_PROTO_VERB_MULTICAST_FRAME_IDX_BLOOM_FILTER (ZT_PROTO_VERB_MULTICAST_FRAME_IDX_ADI + 4)
+#define ZT_PROTO_VERB_MULTICAST_FRAME_IDX_HOP_COUNT (ZT_PROTO_VERB_MULTICAST_FRAME_IDX_BLOOM_FILTER + 64)
+#define ZT_PROTO_VERB_MULTICAST_FRAME_IDX_ETHERTYPE (ZT_PROTO_VERB_MULTICAST_FRAME_IDX_HOP_COUNT + 1)
+#define ZT_PROTO_VERB_MULTICAST_FRAME_IDX_PAYLOAD_LENGTH (ZT_PROTO_VERB_MULTICAST_FRAME_IDX_ETHERTYPE + 2)
+#define ZT_PROTO_VERB_MULTICAST_FRAME_IDX_SIGNATURE_LENGTH (ZT_PROTO_VERB_MULTICAST_FRAME_IDX_PAYLOAD_LENGTH + 2)
+#define ZT_PROTO_VERB_MULTICAST_FRAME_IDX_PAYLOAD (ZT_PROTO_VERB_MULTICAST_FRAME_IDX_SIGNATURE_LENGTH + 2)
 
 // Field indices for parsing OK and ERROR payloads of replies
 #define ZT_PROTO_VERB_HELLO__OK__IDX_TIMESTAMP (ZT_PROTO_VERB_OK_IDX_PAYLOAD)
@@ -263,13 +273,13 @@ public:
 			setSize(fragLen + ZT_PROTO_MIN_FRAGMENT_LENGTH);
 
 			// NOTE: this copies both the IV/packet ID and the destination address.
-			memcpy(_b + ZT_PACKET_FRAGMENT_IDX_PACKET_ID,p.data() + ZT_PACKET_IDX_IV,13);
+			memcpy(field(ZT_PACKET_FRAGMENT_IDX_PACKET_ID,13),p.data() + ZT_PACKET_IDX_IV,13);
 
-			_b[ZT_PACKET_FRAGMENT_IDX_FRAGMENT_INDICATOR] = ZT_PACKET_FRAGMENT_INDICATOR;
-			_b[ZT_PACKET_FRAGMENT_IDX_FRAGMENT_NO] = (char)(((fragTotal & 0xf) << 4) | (fragNo & 0xf));
-			_b[ZT_PACKET_FRAGMENT_IDX_HOPS] = 0;
+			(*this)[ZT_PACKET_FRAGMENT_IDX_FRAGMENT_INDICATOR] = ZT_PACKET_FRAGMENT_INDICATOR;
+			(*this)[ZT_PACKET_FRAGMENT_IDX_FRAGMENT_NO] = (char)(((fragTotal & 0xf) << 4) | (fragNo & 0xf));
+			(*this)[ZT_PACKET_FRAGMENT_IDX_HOPS] = 0;
 
-			memcpy(_b + ZT_PACKET_FRAGMENT_IDX_PAYLOAD,p.data() + fragStart,fragLen);
+			memcpy(field(ZT_PACKET_FRAGMENT_IDX_PAYLOAD,fragLen),p.data() + fragStart,fragLen);
 		}
 
 		/**
@@ -277,12 +287,12 @@ public:
 		 * 
 		 * @return Destination ZT address
 		 */
-		inline Address destination() const { return Address(_b + ZT_PACKET_FRAGMENT_IDX_DEST); }
+		inline Address destination() const { return Address(field(ZT_PACKET_FRAGMENT_IDX_DEST,ZT_ADDRESS_LENGTH)); }
 
 		/**
 		 * @return True if fragment is of a valid length
 		 */
-		inline bool lengthValid() const { return (_l >= ZT_PACKET_FRAGMENT_IDX_PAYLOAD); }
+		inline bool lengthValid() const { return (size() >= ZT_PACKET_FRAGMENT_IDX_PAYLOAD); }
 
 		/**
 		 * @return ID of packet this is a fragment of
@@ -292,36 +302,38 @@ public:
 		/**
 		 * @return Total number of fragments in packet
 		 */
-		inline unsigned int totalFragments() const { return (((unsigned int)_b[ZT_PACKET_FRAGMENT_IDX_FRAGMENT_NO] >> 4) & 0xf); }
+		inline unsigned int totalFragments() const { return (((unsigned int)((*this)[ZT_PACKET_FRAGMENT_IDX_FRAGMENT_NO]) >> 4) & 0xf); }
 
 		/**
 		 * @return Fragment number of this fragment
 		 */
-		inline unsigned int fragmentNumber() const { return ((unsigned int)_b[ZT_PACKET_FRAGMENT_IDX_FRAGMENT_NO] & 0xf); }
+		inline unsigned int fragmentNumber() const { return ((unsigned int)((*this)[ZT_PACKET_FRAGMENT_IDX_FRAGMENT_NO]) & 0xf); }
 
 		/**
 		 * @return Fragment ZT hop count
 		 */
-		inline unsigned int hops() const { return (unsigned int)_b[ZT_PACKET_FRAGMENT_IDX_HOPS]; }
+		inline unsigned int hops() const { return (unsigned int)((*this)[ZT_PACKET_FRAGMENT_IDX_HOPS]); }
 
 		/**
 		 * Increment this packet's hop count
 		 */
 		inline void incrementHops()
 		{
-			_b[ZT_PACKET_FRAGMENT_IDX_HOPS] = (_b[ZT_PACKET_FRAGMENT_IDX_HOPS] + 1) & ZT_PROTO_MAX_HOPS;
+			(*this)[ZT_PACKET_FRAGMENT_IDX_HOPS] = (((*this)[ZT_PACKET_FRAGMENT_IDX_HOPS]) + 1) & ZT_PROTO_MAX_HOPS;
 		}
 
-		/**
-		 * @return Fragment payload
-		 */
-		inline unsigned char *payload() { return (unsigned char *)(_b + ZT_PACKET_FRAGMENT_IDX_PAYLOAD); }
-		inline const unsigned char *payload() const { return (const unsigned char *)(_b + ZT_PACKET_FRAGMENT_IDX_PAYLOAD); }
-
 		/**
 		 * @return Length of payload in bytes
 		 */
-		inline unsigned int payloadLength() const { return ((_l > ZT_PACKET_FRAGMENT_IDX_PAYLOAD) ? (_l - ZT_PACKET_FRAGMENT_IDX_PAYLOAD) : 0); }
+		inline unsigned int payloadLength() const { return ((size() > ZT_PACKET_FRAGMENT_IDX_PAYLOAD) ? (size() - ZT_PACKET_FRAGMENT_IDX_PAYLOAD) : 0); }
+
+		/**
+		 * @return Raw packet payload
+		 */
+		inline const unsigned char *payload() const
+		{
+			return field(ZT_PACKET_FRAGMENT_IDX_PAYLOAD,size() - ZT_PACKET_FRAGMENT_IDX_PAYLOAD);
+		}
 	};
 
 	/**
@@ -333,40 +345,40 @@ public:
 		VERB_NOP = 0,
 
 		/* Announcement of a node's existence:
-		 *	 <[1] protocol version>
+		 *   <[1] protocol version>
 		 *   <[1] software major version>
 		 *   <[1] software minor version>
 		 *   <[2] software revision>
-		 *	 <[8] timestamp (ms since epoch)>
-		 *	 <[...] binary serialized identity (see Identity)>
+		 *   <[8] timestamp (ms since epoch)>
+		 *   <[...] binary serialized identity (see Identity)>
 		 *
 		 * OK payload:
-		 *	 <[8] timestamp (echoed from original HELLO)>
+		 *   <[8] timestamp (echoed from original HELLO)>
 		 *
 		 * ERROR has no payload.
 		 */
 		VERB_HELLO = 1,
 
 		/* Error response:
-		 *	 <[1] in-re verb>
-		 *	 <[8] in-re packet ID>
-		 *	 <[1] error code>
-		 *	 <[...] error-dependent payload>
+		 *   <[1] in-re verb>
+		 *   <[8] in-re packet ID>
+		 *   <[1] error code>
+		 *   <[...] error-dependent payload>
 		 */
 		VERB_ERROR = 2,
 
 		/* Success response:
-		 *	 <[1] in-re verb>
-		 *	 <[8] in-re packet ID>
-		 *	 <[...] request-specific payload>
+		 *   <[1] in-re verb>
+		 *   <[8] in-re packet ID>
+		 *   <[...] request-specific payload>
 		 */
 		VERB_OK = 3,
 
 		/* Query an identity by address:
-		 *	 <[5] address to look up>
+		 *   <[5] address to look up>
 		 *
 		 * OK response payload:
-		 *	 <[...] binary serialized identity>
+		 *   <[...] binary serialized identity>
 		 *
 		 * Error payload will be address queried.
 		 */
@@ -400,8 +412,8 @@ public:
 
 		/* A ZT-to-ZT unicast ethernet frame:
 		 *   <[8] 64-bit network ID>
-		 *	 <[2] 16-bit ethertype>
-		 *	 <[...] ethernet payload>
+		 *   <[2] 16-bit ethertype>
+		 *   <[...] ethernet payload>
 		 *
 		 * MAC addresses are derived from the packet's source and destination
 		 * ZeroTier addresses. ZeroTier does not support VLANs or other extensions
@@ -411,20 +423,8 @@ public:
 		 */
 		VERB_FRAME = 6,
 
-		/* A multicast frame:
-		 *   <[8] 64-bit network ID>
-		 *   <[6] destination multicast Ethernet address>
-		 *   <[4] multicast additional distinguishing information (ADI)>
-		 *   <[32] multicast propagation bloom filter>
-		 *   <[1] 8-bit strict propagation hop count>
-		 *   <[2] 16-bit average peer multicast bandwidth load>
-		 *   <[6] source Ethernet address>
-		 *   <[2] 16-bit ethertype>
-		 *   <[...] ethernet payload>
-		 *
-		 * No OK or ERROR is generated.
-		 */
-		VERB_MULTICAST_FRAME = 7,
+		/* 7 - old VERB_MULTICAST_FRAME, might be reused once all old 0.2
+		 * clients are off the net. */
 
 		/* Announce interest in multicast group(s):
 		 *   <[8] 64-bit network ID>
@@ -434,7 +434,36 @@ public:
 		 *
 		 * OK is generated on successful receipt.
 		 */
-		VERB_MULTICAST_LIKE = 8
+		VERB_MULTICAST_LIKE = 8,
+
+		/* A multicast frame:
+		 *   <[1] flags, currently unused and must be 0>
+		 *   <[8] 64-bit network ID>
+		 *   <[5] ZeroTier address of original submitter of this multicast>
+		 *   <[6] source MAC address>
+		 *   <[6] destination multicast Ethernet address>
+		 *   <[4] multicast additional distinguishing information (ADI)>
+		 *   <[64] multicast propagation bloom filter>
+		 *   <[1] 8-bit propagation hop count>
+		 *   <[2] 16-bit ethertype>
+		 *   <[2] 16-bit length of payload>
+		 *   <[2] 16-bit length of signature>
+		 *   <[...] ethernet payload>
+		 *   <[...] ECDSA signature>
+		 *
+		 * The signature is made using the key of the original submitter, and
+		 * can be used to authenticate the submitter for security and rate
+		 * control purposes. Fields in the signature are: network ID, source
+		 * MAC, destination MAC, multicast ADI, ethertype, and payload. All
+		 * integers are hashed in big-endian byte order. A zero byte is added
+		 * to the hash between each field.
+		 *
+		 * In the future flags could indicate additional fields appended to the
+		 * end or a different signature algorithm.
+		 *
+		 * No OK or ERROR is generated.
+		 */
+		VERB_MULTICAST_FRAME = 9
 	};
 
 	/**
@@ -495,8 +524,8 @@ public:
 	Packet() :
 		Buffer<ZT_PROTO_MAX_PACKET_LENGTH>(ZT_PROTO_MIN_PACKET_LENGTH)
 	{
-		Utils::getSecureRandom(_b + ZT_PACKET_IDX_IV,8);
-		_b[ZT_PACKET_IDX_FLAGS] = 0; // zero flags and hops
+		Utils::getSecureRandom(field(ZT_PACKET_IDX_IV,8),8);
+		(*this)[ZT_PACKET_IDX_FLAGS] = 0; // zero flags and hops
 	}
 
 	/**
@@ -509,10 +538,10 @@ public:
 	Packet(const Address &dest,const Address &source,const Verb v) :
 		Buffer<ZT_PROTO_MAX_PACKET_LENGTH>(ZT_PROTO_MIN_PACKET_LENGTH)
 	{
-		Utils::getSecureRandom(_b + ZT_PACKET_IDX_IV,8);
+		Utils::getSecureRandom(field(ZT_PACKET_IDX_IV,8),8);
 		setDestination(dest);
 		setSource(source);
-		_b[ZT_PACKET_IDX_FLAGS] = 0; // zero flags and hops
+		(*this)[ZT_PACKET_IDX_FLAGS] = 0; // zero flags and hops
 		setVerb(v);
 	}
 
@@ -526,13 +555,25 @@ public:
 	inline void reset(const Address &dest,const Address &source,const Verb v)
 	{
 		setSize(ZT_PROTO_MIN_PACKET_LENGTH);
-		Utils::getSecureRandom(_b + ZT_PACKET_IDX_IV,8);
+		Utils::getSecureRandom(field(ZT_PACKET_IDX_IV,8),8);
 		setDestination(dest);
 		setSource(source);
-		_b[ZT_PACKET_IDX_FLAGS] = 0; // zero flags and hops
+		(*this)[ZT_PACKET_IDX_FLAGS] = 0; // zero flags and hops
 		setVerb(v);
 	}
 
+	/**
+	 * Generate a new IV / packet ID in place
+	 *
+	 * This can be used to re-use a packet buffer multiple times to send
+	 * technically different but otherwise identical copies of the same
+	 * packet.
+	 */
+	inline void newInitializationVector()
+	{
+		Utils::getSecureRandom(field(ZT_PACKET_IDX_IV,8),8);
+	}
+
 	/**
 	 * Set this packet's destination
 	 * 
@@ -540,8 +581,9 @@ public:
 	 */
 	inline void setDestination(const Address &dest)
 	{
+		unsigned char *d = field(ZT_PACKET_IDX_DEST,ZT_ADDRESS_LENGTH);
 		for(unsigned int i=0;i<ZT_ADDRESS_LENGTH;++i)
-			_b[i + ZT_PACKET_IDX_DEST] = dest[i];
+			d[i] = dest[i];
 	}
 
 	/**
@@ -551,8 +593,9 @@ public:
 	 */
 	inline void setSource(const Address &source)
 	{
+		unsigned char *s = field(ZT_PACKET_IDX_SOURCE,ZT_ADDRESS_LENGTH);
 		for(unsigned int i=0;i<ZT_ADDRESS_LENGTH;++i)
-			_b[i + ZT_PACKET_IDX_SOURCE] = source[i];
+			s[i] = source[i];
 	}
 
 	/**
@@ -560,29 +603,29 @@ public:
 	 * 
 	 * @return Destination ZT address
 	 */
-	inline Address destination() const { return Address(_b + ZT_PACKET_IDX_DEST); }
+	inline Address destination() const { return Address(field(ZT_PACKET_IDX_DEST,ZT_ADDRESS_LENGTH)); }
 
 	/**
 	 * Get this packet's source
 	 * 
 	 * @return Source ZT address
 	 */
-	inline Address source() const { return Address(_b + ZT_PACKET_IDX_SOURCE); }
+	inline Address source() const { return Address(field(ZT_PACKET_IDX_SOURCE,ZT_ADDRESS_LENGTH)); }
 
 	/**
 	 * @return True if packet is of valid length
 	 */
-	inline bool lengthValid() const { return (_l >= ZT_PROTO_MIN_PACKET_LENGTH); }
+	inline bool lengthValid() const { return (size() >= ZT_PROTO_MIN_PACKET_LENGTH); }
 
 	/**
 	 * @return True if packet is encrypted
 	 */
-	inline bool encrypted() const { return (((unsigned char)_b[ZT_PACKET_IDX_FLAGS] & ZT_PROTO_FLAG_ENCRYPTED)); }
+	inline bool encrypted() const { return (((unsigned char)(*this)[ZT_PACKET_IDX_FLAGS] & ZT_PROTO_FLAG_ENCRYPTED)); }
 
 	/**
 	 * @return True if packet is fragmented (expect fragments)
 	 */
-	inline bool fragmented() const { return (((unsigned char)_b[ZT_PACKET_IDX_FLAGS] & ZT_PROTO_FLAG_FRAGMENTED)); }
+	inline bool fragmented() const { return (((unsigned char)(*this)[ZT_PACKET_IDX_FLAGS] & ZT_PROTO_FLAG_FRAGMENTED)); }
 
 	/**
 	 * Set this packet's fragmented flag
@@ -592,26 +635,26 @@ public:
 	inline void setFragmented(bool f)
 	{
 		if (f)
-			_b[ZT_PACKET_IDX_FLAGS] |= (char)ZT_PROTO_FLAG_FRAGMENTED;
-		else _b[ZT_PACKET_IDX_FLAGS] &= (char)(~ZT_PROTO_FLAG_FRAGMENTED);
+			(*this)[ZT_PACKET_IDX_FLAGS] |= (char)ZT_PROTO_FLAG_FRAGMENTED;
+		else (*this)[ZT_PACKET_IDX_FLAGS] &= (char)(~ZT_PROTO_FLAG_FRAGMENTED);
 	}
 
 	/**
 	 * @return True if compressed (result only valid if unencrypted)
 	 */
-	inline bool compressed() const { return (((unsigned char)_b[ZT_PACKET_IDX_VERB] & ZT_PROTO_VERB_FLAG_COMPRESSED)); }
+	inline bool compressed() const { return (((unsigned char)(*this)[ZT_PACKET_IDX_VERB] & ZT_PROTO_VERB_FLAG_COMPRESSED)); }
 
 	/**
 	 * @return ZeroTier forwarding hops (0 to 7)
 	 */
-	inline unsigned int hops() const { return ((unsigned int)_b[ZT_PACKET_IDX_FLAGS] & 0x07); }
+	inline unsigned int hops() const { return ((unsigned int)(*this)[ZT_PACKET_IDX_FLAGS] & 0x07); }
 
 	/**
 	 * Increment this packet's hop count
 	 */
 	inline void incrementHops()
 	{
-		_b[ZT_PACKET_IDX_FLAGS] = (char)((unsigned char)_b[ZT_PACKET_IDX_FLAGS] & 0xf8) | (((unsigned char)_b[ZT_PACKET_IDX_FLAGS] + 1) & 0x07);
+		(*this)[ZT_PACKET_IDX_FLAGS] = (char)((unsigned char)(*this)[ZT_PACKET_IDX_FLAGS] & 0xf8) | (((unsigned char)(*this)[ZT_PACKET_IDX_FLAGS] + 1) & 0x07);
 	}
 
 	/**
@@ -629,23 +672,25 @@ public:
 	 * 
 	 * @param v New packet verb
 	 */
-	inline void setVerb(Verb v) { _b[ZT_PACKET_IDX_VERB] = (char)v; }
+	inline void setVerb(Verb v) { (*this)[ZT_PACKET_IDX_VERB] = (char)v; }
 
 	/**
 	 * @return Packet verb (not including flag bits)
 	 */
-	inline Verb verb() const { return (Verb)(_b[ZT_PACKET_IDX_VERB] & 0x1f); }
+	inline Verb verb() const { return (Verb)((*this)[ZT_PACKET_IDX_VERB] & 0x1f); }
 
 	/**
 	 * @return Length of packet payload
 	 */
-	inline unsigned int payloadLength() const throw() { return ((_l < ZT_PROTO_MIN_PACKET_LENGTH) ? 0 : (_l - ZT_PROTO_MIN_PACKET_LENGTH)); }
+	inline unsigned int payloadLength() const { return ((size() < ZT_PROTO_MIN_PACKET_LENGTH) ? 0 : (size() - ZT_PROTO_MIN_PACKET_LENGTH)); }
 
 	/**
-	 * @return Packet payload
+	 * @return Raw packet payload
 	 */
-	inline unsigned char *payload() throw() { return (unsigned char *)(_b + ZT_PACKET_IDX_PAYLOAD); }
-	inline const unsigned char *payload() const throw() { return (const unsigned char *)(_b + ZT_PACKET_IDX_PAYLOAD); }
+	inline const unsigned char *payload() const
+	{
+		return field(ZT_PACKET_IDX_PAYLOAD,size() - ZT_PACKET_IDX_PAYLOAD);
+	}
 
 	/**
 	 * Compute the HMAC of this packet's payload and set HMAC field
@@ -655,13 +700,13 @@ public:
 	 * @param key 256-bit (32 byte) key
 	 */
 	inline void hmacSet(const void *key)
-		throw()
 	{
 		unsigned char mac[32];
 		unsigned char key2[32];
 		_mangleKey((const unsigned char *)key,key2);
-		HMAC::sha256(key2,sizeof(key2),_b + ZT_PACKET_IDX_VERB,(_l >= ZT_PACKET_IDX_VERB) ? (_l - ZT_PACKET_IDX_VERB) : 0,mac);
-		memcpy(_b + ZT_PACKET_IDX_HMAC,mac,8);
+		unsigned int hmacLen = (size() >= ZT_PACKET_IDX_VERB) ? (size() - ZT_PACKET_IDX_VERB) : 0;
+		HMAC::sha256(key2,sizeof(key2),field(ZT_PACKET_IDX_VERB,hmacLen),hmacLen,mac);
+		memcpy(field(ZT_PACKET_IDX_HMAC,8),mac,8);
 	}
 
 	/**
@@ -672,15 +717,15 @@ public:
 	 * @param key 256-bit (32 byte) key
 	 */
 	inline bool hmacVerify(const void *key) const
-		throw()
 	{
 		unsigned char mac[32];
 		unsigned char key2[32];
-		if (_l < ZT_PACKET_IDX_VERB)
+		if (size() < ZT_PACKET_IDX_VERB)
 			return false; // incomplete packets fail
 		_mangleKey((const unsigned char *)key,key2);
-		HMAC::sha256(key2,sizeof(key2),_b + ZT_PACKET_IDX_VERB,_l - ZT_PACKET_IDX_VERB,mac);
-		return (!memcmp(_b + ZT_PACKET_IDX_HMAC,mac,8));
+		unsigned int hmacLen = size() - ZT_PACKET_IDX_VERB;
+		HMAC::sha256(key2,sizeof(key2),field(ZT_PACKET_IDX_VERB,hmacLen),hmacLen,mac);
+		return (!memcmp(field(ZT_PACKET_IDX_HMAC,8),mac,8));
 	}
 
 	/**
@@ -689,13 +734,16 @@ public:
 	 * @param key 256-bit (32 byte) key
 	 */
 	inline void encrypt(const void *key)
-		throw()
 	{
-		_b[ZT_PACKET_IDX_FLAGS] |= ZT_PROTO_FLAG_ENCRYPTED;
+		(*this)[ZT_PACKET_IDX_FLAGS] |= ZT_PROTO_FLAG_ENCRYPTED;
 		unsigned char key2[32];
-		_mangleKey((const unsigned char *)key,key2);
-		Salsa20 s20(key2,256,_b + ZT_PACKET_IDX_IV);
-		s20.encrypt(_b + ZT_PACKET_IDX_VERB,_b + ZT_PACKET_IDX_VERB,(_l >= ZT_PACKET_IDX_VERB) ? (_l - ZT_PACKET_IDX_VERB) : 0);
+		if (size() >= ZT_PACKET_IDX_VERB) {
+			_mangleKey((const unsigned char *)key,key2);
+			Salsa20 s20(key2,256,field(ZT_PACKET_IDX_IV,8));
+			unsigned int encLen = size() - ZT_PACKET_IDX_VERB;
+			unsigned char *const encBuf = field(ZT_PACKET_IDX_VERB,encLen);
+			s20.encrypt(encBuf,encBuf,encLen);
+		}
 	}
 
 	/**
@@ -704,13 +752,16 @@ public:
 	 * @param key 256-bit (32 byte) key
 	 */
 	inline void decrypt(const void *key)
-		throw()
 	{
 		unsigned char key2[32];
-		_mangleKey((const unsigned char *)key,key2);
-		Salsa20 s20(key2,256,_b + ZT_PACKET_IDX_IV);
-		s20.decrypt(_b + ZT_PACKET_IDX_VERB,_b + ZT_PACKET_IDX_VERB,(_l >= ZT_PACKET_IDX_VERB) ? (_l - ZT_PACKET_IDX_VERB) : 0);
-		_b[ZT_PACKET_IDX_FLAGS] &= (char)(~ZT_PROTO_FLAG_ENCRYPTED);
+		if (size() >= ZT_PACKET_IDX_VERB) {
+			_mangleKey((const unsigned char *)key,key2);
+			Salsa20 s20(key2,256,field(ZT_PACKET_IDX_IV,8));
+			unsigned int decLen = size() - ZT_PACKET_IDX_VERB;
+			unsigned char *const decBuf = field(ZT_PACKET_IDX_VERB,decLen);
+			s20.decrypt(decBuf,decBuf,decLen);
+		}
+		(*this)[ZT_PACKET_IDX_FLAGS] &= (char)(~ZT_PROTO_FLAG_ENCRYPTED);
 	}
 
 	/**
@@ -724,20 +775,19 @@ public:
 	 * @return True if compression occurred
 	 */
 	inline bool compress()
-		throw()
 	{
 		unsigned char buf[ZT_PROTO_MAX_PACKET_LENGTH * 2];
-		if ((!compressed())&&(_l > (ZT_PACKET_IDX_PAYLOAD + 32))) {
-			int pl = (int)(_l - ZT_PACKET_IDX_PAYLOAD);
-			int cl = LZ4_compress((const char *)(_b + ZT_PACKET_IDX_PAYLOAD),(char *)buf,pl);
+		if ((!compressed())&&(size() > (ZT_PACKET_IDX_PAYLOAD + 32))) {
+			int pl = (int)(size() - ZT_PACKET_IDX_PAYLOAD);
+			int cl = LZ4_compress((const char *)field(ZT_PACKET_IDX_PAYLOAD,(unsigned int)pl),(char *)buf,pl);
 			if ((cl > 0)&&(cl < pl)) {
-				_b[ZT_PACKET_IDX_VERB] |= (char)ZT_PROTO_VERB_FLAG_COMPRESSED;
-				memcpy(_b + ZT_PACKET_IDX_PAYLOAD,buf,cl);
-				_l = (unsigned int)cl + ZT_PACKET_IDX_PAYLOAD;
+				(*this)[ZT_PACKET_IDX_VERB] |= (char)ZT_PROTO_VERB_FLAG_COMPRESSED;
+				setSize((unsigned int)cl + ZT_PACKET_IDX_PAYLOAD);
+				memcpy(field(ZT_PACKET_IDX_PAYLOAD,(unsigned int)cl),buf,cl);
 				return true;
 			}
 		}
-		_b[ZT_PACKET_IDX_VERB] &= (char)(~ZT_PROTO_VERB_FLAG_COMPRESSED);
+		(*this)[ZT_PACKET_IDX_VERB] &= (char)(~ZT_PROTO_VERB_FLAG_COMPRESSED);
 		return false;
 	}
 
@@ -750,18 +800,18 @@ public:
 	 * @return True if data is now decompressed and valid, false on error
 	 */
 	inline bool uncompress()
-		throw()
 	{
 		unsigned char buf[ZT_PROTO_MAX_PACKET_LENGTH];
-		if ((compressed())&&(_l >= ZT_PROTO_MIN_PACKET_LENGTH)) {
-			if (_l > ZT_PACKET_IDX_PAYLOAD) {
-				int ucl = LZ4_uncompress_unknownOutputSize((const char *)(_b + ZT_PACKET_IDX_PAYLOAD),(char *)buf,_l - ZT_PACKET_IDX_PAYLOAD,sizeof(buf));
+		if ((compressed())&&(size() >= ZT_PROTO_MIN_PACKET_LENGTH)) {
+			if (size() > ZT_PACKET_IDX_PAYLOAD) {
+				unsigned int compLen = size() - ZT_PACKET_IDX_PAYLOAD;
+				int ucl = LZ4_uncompress_unknownOutputSize((const char *)field(ZT_PACKET_IDX_PAYLOAD,compLen),(char *)buf,compLen,sizeof(buf));
 				if ((ucl > 0)&&(ucl <= (int)(capacity() - ZT_PACKET_IDX_PAYLOAD))) {
-					memcpy(_b + ZT_PACKET_IDX_PAYLOAD,buf,ucl);
-					_l = (unsigned int)ucl + ZT_PACKET_IDX_PAYLOAD;
+					setSize((unsigned int)ucl + ZT_PACKET_IDX_PAYLOAD);
+					memcpy(field(ZT_PACKET_IDX_PAYLOAD,(unsigned int)ucl),buf,ucl);
 				} else return false;
 			}
-			_b[ZT_PACKET_IDX_VERB] &= ~ZT_PROTO_VERB_FLAG_COMPRESSED;
+			(*this)[ZT_PACKET_IDX_VERB] &= ~ZT_PROTO_VERB_FLAG_COMPRESSED;
 		}
 		return true;
 	}
@@ -788,19 +838,18 @@ private:
 	 * @param out Output buffer (32 bytes)
 	 */
 	inline void _mangleKey(const unsigned char *in,unsigned char *out) const
-		throw()
 	{
 		// Random IV (Salsa20 also uses the IV natively, but HMAC doesn't), and
 		// destination and source addresses. Using dest and source addresses
 		// gives us a (likely) different key space for a->b vs b->a.
 		for(unsigned int i=0;i<18;++i) // 8 + (ZT_ADDRESS_LENGTH * 2) == 18
-			out[i] = in[i] ^ (unsigned char)_b[i];
+			out[i] = in[i] ^ (unsigned char)(*this)[i];
 		// Flags, but masking off hop count which is altered by forwarding nodes
-		out[18] = in[18] ^ ((unsigned char)_b[ZT_PACKET_IDX_FLAGS] & 0xf8);
+		out[18] = in[18] ^ ((unsigned char)(*this)[ZT_PACKET_IDX_FLAGS] & 0xf8);
 		// Raw packet size in bytes -- each raw packet size defines a possibly
 		// different space of keys.
-		out[19] = in[19] ^ (unsigned char)(_l & 0xff);
-		out[20] = in[20] ^ (unsigned char)((_l >> 8) & 0xff); // little endian
+		out[19] = in[19] ^ (unsigned char)(size() & 0xff);
+		out[20] = in[20] ^ (unsigned char)((size() >> 8) & 0xff); // little endian
 		// Rest of raw key is used unchanged
 		for(unsigned int i=21;i<32;++i)
 			out[i] = in[i];

node/PacketDecoder.cpp

@@ -0,0 +1,532 @@
+/*
+ * ZeroTier One - Global Peer to Peer Ethernet
+ * Copyright (C) 2012-2013  ZeroTier Networks LLC
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation, either version 3 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ *
+ * --
+ *
+ * ZeroTier may be used and distributed under the terms of the GPLv3, which
+ * are available at: http://www.gnu.org/licenses/gpl-3.0.html
+ *
+ * If you would like to embed ZeroTier into a commercial application or
+ * redistribute it in a modified binary form, please contact ZeroTier Networks
+ * LLC. Start here: http://www.zerotier.com/
+ */
+
+#include "RuntimeEnvironment.hpp"
+#include "Topology.hpp"
+#include "PacketDecoder.hpp"
+#include "Switch.hpp"
+#include "Peer.hpp"
+#include "NodeConfig.hpp"
+#include "Filter.hpp"
+
+namespace ZeroTier {
+
+bool PacketDecoder::tryDecode(const RuntimeEnvironment *_r)
+	throw(std::out_of_range,std::runtime_error)
+{
+	if ((!encrypted())&&(verb() == Packet::VERB_HELLO)) {
+		// Unencrypted HELLOs are handled here since they are used to
+		// populate our identity cache in the first place. Thus we might get
+		// a HELLO for someone for whom we don't have a Peer record.
+		TRACE("HELLO from %s(%s)",source().toString().c_str(),_remoteAddress.toString().c_str());
+		return _doHELLO(_r);
+	}
+
+	SharedPtr<Peer> peer = _r->topology->getPeer(source());
+	if (peer) {
+		if (_step == DECODE_STEP_WAITING_FOR_ORIGINAL_SUBMITTER_LOOKUP) {
+			// This means we've already decoded, decrypted, decompressed, and
+			// validated, and we're processing a MULTICAST_FRAME. We're waiting
+			// for a lookup on the frame's original submitter. So try again and
+			// see if we have it.
+			return _doMULTICAST_FRAME(_r,peer);
+		}
+
+		if (!hmacVerify(peer->macKey())) {
+			TRACE("dropped packet from %s(%s), HMAC authentication failed (size: %u)",source().toString().c_str(),_remoteAddress.toString().c_str(),size());
+			return true;
+		}
+
+		if (encrypted()) {
+			decrypt(peer->cryptKey());
+		} else {
+			// Unencrypted is tolerated in case we want to run this on
+			// devices where squeezing out cycles matters. HMAC is
+			// what's really important. But log it in debug to catch any
+			// packets being mistakenly sent in the clear.
+			TRACE("ODD: %s from %s(%s) wasn't encrypted",Packet::verbString(verb()),source().toString().c_str(),_remoteAddress.toString().c_str());
+		}
+
+		if (!uncompress()) {
+			TRACE("dropped packet from %s(%s), compressed data invalid",source().toString().c_str(),_remoteAddress.toString().c_str());
+			return true;
+		}
+
+		Packet::Verb v = verb();
+
+		// Validated packets that have passed HMAC can result in us learning a new
+		// path to this peer.
+		peer->onReceive(_r,_localPort,_remoteAddress,hops(),v,Utils::now());
+
+		switch(v) {
+			case Packet::VERB_NOP:
+				TRACE("NOP from %s(%s)",source().toString().c_str(),_remoteAddress.toString().c_str());
+				return true;
+			case Packet::VERB_HELLO:
+				return _doHELLO(_r);
+			case Packet::VERB_ERROR:
+				return _doERROR(_r,peer);
+			case Packet::VERB_OK:
+				return _doOK(_r,peer);
+			case Packet::VERB_WHOIS:
+				return _doWHOIS(_r,peer);
+			case Packet::VERB_RENDEZVOUS:
+				return _doRENDEZVOUS(_r,peer);
+			case Packet::VERB_FRAME:
+				return _doFRAME(_r,peer);
+			case Packet::VERB_MULTICAST_LIKE:
+				return _doMULTICAST_LIKE(_r,peer);
+			case Packet::VERB_MULTICAST_FRAME:
+				return _doMULTICAST_FRAME(_r,peer);
+			default:
+				// This might be something from a new or old version of the protocol.
+				// Technically it passed HMAC so the packet is still valid, but we
+				// ignore it.
+				TRACE("ignored unrecognized verb %.2x from %s(%s)",(unsigned int)v,source().toString().c_str(),_remoteAddress.toString().c_str());
+				return true;
+		}
+	} else {
+		_step = DECODE_STEP_WAITING_FOR_SENDER_LOOKUP;
+		_r->sw->requestWhois(source());
+		return false;
+	}
+}
+
+void PacketDecoder::_CBaddPeerFromHello(void *arg,const SharedPtr<Peer> &p,Topology::PeerVerifyResult result)
+{
+	_CBaddPeerFromHello_Data *req = (_CBaddPeerFromHello_Data *)arg;
+	const RuntimeEnvironment *_r = req->renv;
+
+	switch(result) {
+		case Topology::PEER_VERIFY_ACCEPTED_NEW:
+		case Topology::PEER_VERIFY_ACCEPTED_ALREADY_HAVE:
+		case Topology::PEER_VERIFY_ACCEPTED_DISPLACED_INVALID_ADDRESS: {
+			_r->sw->doAnythingWaitingForPeer(p);
+
+			Packet outp(req->source,_r->identity.address(),Packet::VERB_OK);
+			outp.append((unsigned char)Packet::VERB_HELLO);
+			outp.append(req->helloPacketId);
+			outp.append(req->helloTimestamp);
+			outp.encrypt(p->cryptKey());
+			outp.hmacSet(p->macKey());
+			_r->demarc->send(req->localPort,req->remoteAddress,outp.data(),outp.size(),-1);
+		}	break;
+
+		case Topology::PEER_VERIFY_REJECTED_INVALID_IDENTITY: {
+			Packet outp(req->source,_r->identity.address(),Packet::VERB_ERROR);
+			outp.append((unsigned char)Packet::VERB_HELLO);
+			outp.append(req->helloPacketId);
+			outp.append((unsigned char)Packet::ERROR_IDENTITY_INVALID);
+			outp.encrypt(p->cryptKey());
+			outp.hmacSet(p->macKey());
+			_r->demarc->send(req->localPort,req->remoteAddress,outp.data(),outp.size(),-1);
+		}	break;
+
+		case Topology::PEER_VERIFY_REJECTED_DUPLICATE:
+		case Topology::PEER_VERIFY_REJECTED_DUPLICATE_TRIAGED: {
+			Packet outp(req->source,_r->identity.address(),Packet::VERB_ERROR);
+			outp.append((unsigned char)Packet::VERB_HELLO);
+			outp.append(req->helloPacketId);
+			outp.append((unsigned char)Packet::ERROR_IDENTITY_COLLISION);
+			outp.encrypt(p->cryptKey());
+			outp.hmacSet(p->macKey());
+			_r->demarc->send(req->localPort,req->remoteAddress,outp.data(),outp.size(),-1);
+		}	break;
+	}
+
+	delete req;
+}
+
+void PacketDecoder::_CBaddPeerFromWhois(void *arg,const SharedPtr<Peer> &p,Topology::PeerVerifyResult result)
+{
+	switch(result) {
+		case Topology::PEER_VERIFY_ACCEPTED_NEW:
+		case Topology::PEER_VERIFY_ACCEPTED_ALREADY_HAVE:
+		case Topology::PEER_VERIFY_ACCEPTED_DISPLACED_INVALID_ADDRESS:
+			((const RuntimeEnvironment *)arg)->sw->doAnythingWaitingForPeer(p);
+			break;
+		default:
+			break;
+	}
+}
+
+bool PacketDecoder::_doERROR(const RuntimeEnvironment *_r,const SharedPtr<Peer> &peer)
+{
+	try {
+#ifdef ZT_TRACE
+		Packet::Verb inReVerb = (Packet::Verb)(*this)[ZT_PROTO_VERB_ERROR_IDX_IN_RE_VERB];
+		Packet::ErrorCode errorCode = (Packet::ErrorCode)(*this)[ZT_PROTO_VERB_ERROR_IDX_ERROR_CODE];
+		TRACE("ERROR %s from %s(%s) in-re %s",Packet::errorString(errorCode),source().toString().c_str(),_remoteAddress.toString().c_str(),Packet::verbString(inReVerb));
+#endif
+		// TODO (sorta):
+		// The fact is that the protocol works fine without error handling.
+		// The only error that really needs to be handled here is duplicate
+		// identity collision, which if it comes from a supernode should cause
+		// us to restart and regenerate a new identity.
+	} catch (std::exception &ex) {
+		TRACE("dropped ERROR from %s(%s): unexpected exception: %s",source().toString().c_str(),_remoteAddress.toString().c_str(),ex.what());
+	} catch ( ... ) {
+		TRACE("dropped ERROR from %s(%s): unexpected exception: (unknown)",source().toString().c_str(),_remoteAddress.toString().c_str());
+	}
+	return true;
+}
+
+bool PacketDecoder::_doHELLO(const RuntimeEnvironment *_r)
+{
+	try {
+		//unsigned int protoVersion = (*this)[ZT_PROTO_VERB_HELLO_IDX_PROTOCOL_VERSION];
+		unsigned int vMajor = (*this)[ZT_PROTO_VERB_HELLO_IDX_MAJOR_VERSION];
+		unsigned int vMinor = (*this)[ZT_PROTO_VERB_HELLO_IDX_MINOR_VERSION];
+		unsigned int vRevision = at<uint16_t>(ZT_PROTO_VERB_HELLO_IDX_REVISION);
+		uint64_t timestamp = at<uint64_t>(ZT_PROTO_VERB_HELLO_IDX_TIMESTAMP);
+		Identity id(*this,ZT_PROTO_VERB_HELLO_IDX_IDENTITY);
+
+		SharedPtr<Peer> candidate(new Peer(_r->identity,id));
+		candidate->setPathAddress(_remoteAddress,false);
+
+		// Initial sniff test
+		if (id.address().isReserved()) {
+			TRACE("rejected HELLO from %s(%s): identity has reserved address",source().toString().c_str(),_remoteAddress.toString().c_str());
+			Packet outp(source(),_r->identity.address(),Packet::VERB_ERROR);
+			outp.append((unsigned char)Packet::VERB_HELLO);
+			outp.append(packetId());
+			outp.append((unsigned char)Packet::ERROR_IDENTITY_INVALID);
+			outp.encrypt(candidate->cryptKey());
+			outp.hmacSet(candidate->macKey());
+			_r->demarc->send(_localPort,_remoteAddress,outp.data(),outp.size(),-1);
+			return true;
+		}
+		if (id.address() != source()) {
+			TRACE("rejected HELLO from %s(%s): identity is not for sender of packet (HELLO is a self-announcement)",source().toString().c_str(),_remoteAddress.toString().c_str());
+			Packet outp(source(),_r->identity.address(),Packet::VERB_ERROR);
+			outp.append((unsigned char)Packet::VERB_HELLO);
+			outp.append(packetId());
+			outp.append((unsigned char)Packet::ERROR_INVALID_REQUEST);
+			outp.encrypt(candidate->cryptKey());
+			outp.hmacSet(candidate->macKey());
+			_r->demarc->send(_localPort,_remoteAddress,outp.data(),outp.size(),-1);
+			return true;
+		}
+
+		// Is this a HELLO for a peer we already know? If so just update its
+		// packet receive stats and send an OK.
+		SharedPtr<Peer> existingPeer(_r->topology->getPeer(id.address()));
+		if ((existingPeer)&&(existingPeer->identity() == id)) {
+			existingPeer->onReceive(_r,_localPort,_remoteAddress,hops(),Packet::VERB_HELLO,Utils::now());
+			existingPeer->setRemoteVersion(vMajor,vMinor,vRevision);
+
+			Packet outp(source(),_r->identity.address(),Packet::VERB_OK);
+			outp.append((unsigned char)Packet::VERB_HELLO);
+			outp.append(packetId());
+			outp.append(timestamp);
+			outp.encrypt(existingPeer->cryptKey());
+			outp.hmacSet(existingPeer->macKey());
+			_r->demarc->send(_localPort,_remoteAddress,outp.data(),outp.size(),-1);
+			return true;
+		}
+
+		// Otherwise we call addPeer() and set up a callback to handle the verdict
+		_CBaddPeerFromHello_Data *arg = new _CBaddPeerFromHello_Data;
+		arg->renv = _r;
+		arg->source = source();
+		arg->remoteAddress = _remoteAddress;
+		arg->localPort = _localPort;
+		arg->vMajor = vMajor;
+		arg->vMinor = vMinor;
+		arg->vRevision = vRevision;
+		arg->helloPacketId = packetId();
+		arg->helloTimestamp = timestamp;
+		_r->topology->addPeer(candidate,&PacketDecoder::_CBaddPeerFromHello,arg);
+	} catch (std::exception &ex) {
+		TRACE("dropped HELLO from %s(%s): %s",source().toString().c_str(),_remoteAddress.toString().c_str(),ex.what());
+	} catch ( ... ) {
+		TRACE("dropped HELLO from %s(%s): unexpected exception",source().toString().c_str(),_remoteAddress.toString().c_str());
+	}
+	return true;
+}
+
+bool PacketDecoder::_doOK(const RuntimeEnvironment *_r,const SharedPtr<Peer> &peer)
+{
+	try {
+		Packet::Verb inReVerb = (Packet::Verb)(*this)[ZT_PROTO_VERB_OK_IDX_IN_RE_VERB];
+		switch(inReVerb) {
+			case Packet::VERB_HELLO: {
+				// OK from HELLO permits computation of latency.
+				unsigned int latency = std::min((unsigned int)(Utils::now() - at<uint64_t>(ZT_PROTO_VERB_HELLO__OK__IDX_TIMESTAMP)),(unsigned int)0xffff);
+				TRACE("%s(%s): OK(HELLO), latency: %u",source().toString().c_str(),_remoteAddress.toString().c_str(),latency);
+				peer->setLatency(_remoteAddress,latency);
+			}	break;
+			case Packet::VERB_WHOIS:
+				// Right now we only query supernodes for WHOIS and only accept
+				// OK back from them. If we query other nodes, we'll have to
+				// do something to prevent WHOIS cache poisoning such as
+				// using the packet ID field in the OK packet to match with the
+				// original query. Technically we should be doing this anyway.
+				TRACE("%s(%s): OK(%s)",source().toString().c_str(),_remoteAddress.toString().c_str(),Packet::verbString(inReVerb));
+				if (_r->topology->isSupernode(source()))
+					_r->topology->addPeer(SharedPtr<Peer>(new Peer(_r->identity,Identity(*this,ZT_PROTO_VERB_WHOIS__OK__IDX_IDENTITY))),&PacketDecoder::_CBaddPeerFromWhois,const_cast<void *>((const void *)_r));
+				break;
+			default:
+				//TRACE("%s(%s): OK(%s)",source().toString().c_str(),_remoteAddress.toString().c_str(),Packet::verbString(inReVerb));
+				break;
+		}
+	} catch (std::exception &ex) {
+		TRACE("dropped OK from %s(%s): unexpected exception: %s",source().toString().c_str(),_remoteAddress.toString().c_str(),ex.what());
+	} catch ( ... ) {
+		TRACE("dropped OK from %s(%s): unexpected exception: (unknown)",source().toString().c_str(),_remoteAddress.toString().c_str());
+	}
+	return true;
+}
+
+bool PacketDecoder::_doWHOIS(const RuntimeEnvironment *_r,const SharedPtr<Peer> &peer)
+{
+	if (payloadLength() == ZT_ADDRESS_LENGTH) {
+		SharedPtr<Peer> p(_r->topology->getPeer(Address(payload())));
+		if (p) {
+			Packet outp(source(),_r->identity.address(),Packet::VERB_OK);
+			outp.append((unsigned char)Packet::VERB_WHOIS);
+			outp.append(packetId());
+			p->identity().serialize(outp,false);
+			outp.encrypt(peer->cryptKey());
+			outp.hmacSet(peer->macKey());
+			_r->demarc->send(_localPort,_remoteAddress,outp.data(),outp.size(),-1);
+			TRACE("sent WHOIS response to %s for %s",source().toString().c_str(),Address(payload()).toString().c_str());
+		} else {
+			Packet outp(source(),_r->identity.address(),Packet::VERB_ERROR);
+			outp.append((unsigned char)Packet::VERB_WHOIS);
+			outp.append(packetId());
+			outp.append((unsigned char)Packet::ERROR_NOT_FOUND);
+			outp.append(payload(),ZT_ADDRESS_LENGTH);
+			outp.encrypt(peer->cryptKey());
+			outp.hmacSet(peer->macKey());
+			_r->demarc->send(_localPort,_remoteAddress,outp.data(),outp.size(),-1);
+			TRACE("sent WHOIS ERROR to %s for %s (not found)",source().toString().c_str(),Address(payload()).toString().c_str());
+		}
+	} else {
+		TRACE("dropped WHOIS from %s(%s): missing or invalid address",source().toString().c_str(),_remoteAddress.toString().c_str());
+	}
+	return true;
+}
+
+bool PacketDecoder::_doRENDEZVOUS(const RuntimeEnvironment *_r,const SharedPtr<Peer> &peer)
+{
+	try {
+		Address with(field(ZT_PROTO_VERB_RENDEZVOUS_IDX_ZTADDRESS,ZT_ADDRESS_LENGTH));
+		SharedPtr<Peer> withPeer(_r->topology->getPeer(with));
+		if (withPeer) {
+			unsigned int port = at<uint16_t>(ZT_PROTO_VERB_RENDEZVOUS_IDX_PORT);
+			unsigned int addrlen = (*this)[ZT_PROTO_VERB_RENDEZVOUS_IDX_ADDRLEN];
+			if ((port > 0)&&((addrlen == 4)||(addrlen == 16))) {
+				InetAddress atAddr(field(ZT_PROTO_VERB_RENDEZVOUS_IDX_ADDRESS,addrlen),addrlen,port);
+				TRACE("RENDEZVOUS from %s says %s might be at %s, starting NAT-t",source().toString().c_str(),with.toString().c_str(),atAddr.toString().c_str());
+				_r->sw->contact(withPeer,atAddr);
+			} else {
+				TRACE("dropped corrupt RENDEZVOUS from %s(%s) (bad address or port)",source().toString().c_str(),_remoteAddress.toString().c_str());
+			}
+		} else {
+			TRACE("ignored RENDEZVOUS from %s(%s) to meet unknown peer %s",source().toString().c_str(),_remoteAddress.toString().c_str(),with.toString().c_str());
+		}
+	} catch (std::exception &ex) {
+		TRACE("dropped RENDEZVOUS from %s(%s): %s",source().toString().c_str(),_remoteAddress.toString().c_str(),ex.what());
+	} catch ( ... ) {
+		TRACE("dropped RENDEZVOUS from %s(%s): unexpected exception",source().toString().c_str(),_remoteAddress.toString().c_str());
+	}
+	return true;
+}
+
+bool PacketDecoder::_doFRAME(const RuntimeEnvironment *_r,const SharedPtr<Peer> &peer)
+{
+	try {
+		SharedPtr<Network> network(_r->nc->network(at<uint64_t>(ZT_PROTO_VERB_FRAME_IDX_NETWORK_ID)));
+		if (network) {
+			if (network->isAllowed(source())) {
+				unsigned int etherType = at<uint16_t>(ZT_PROTO_VERB_FRAME_IDX_ETHERTYPE);
+				if ((etherType != ZT_ETHERTYPE_ARP)&&(etherType != ZT_ETHERTYPE_IPV4)&&(etherType != ZT_ETHERTYPE_IPV6)) {
+					TRACE("dropped FRAME from %s: unsupported ethertype",source().toString().c_str());
+				} else if (size() > ZT_PROTO_VERB_FRAME_IDX_PAYLOAD) {
+					network->tap().put(source().toMAC(),network->tap().mac(),etherType,data() + ZT_PROTO_VERB_FRAME_IDX_PAYLOAD,size() - ZT_PROTO_VERB_FRAME_IDX_PAYLOAD);
+				}
+			} else {
+				TRACE("dropped FRAME from %s(%s): not a member of closed network %llu",source().toString().c_str(),_remoteAddress.toString().c_str(),network->id());
+			}
+		} else {
+			TRACE("dropped FRAME from %s(%s): network %llu unknown",source().toString().c_str(),_remoteAddress.toString().c_str(),at<uint64_t>(ZT_PROTO_VERB_FRAME_IDX_NETWORK_ID));
+		}
+	} catch (std::exception &ex) {
+		TRACE("dropped FRAME from %s(%s): unexpected exception: %s",source().toString().c_str(),_remoteAddress.toString().c_str(),ex.what());
+	} catch ( ... ) {
+		TRACE("dropped FRAME from %s(%s): unexpected exception: (unknown)",source().toString().c_str(),_remoteAddress.toString().c_str());
+	}
+	return true;
+}
+
+bool PacketDecoder::_doMULTICAST_LIKE(const RuntimeEnvironment *_r,const SharedPtr<Peer> &peer)
+{
+	try {
+		unsigned int ptr = ZT_PACKET_IDX_PAYLOAD;
+		unsigned int numAccepted = 0;
+		uint64_t now = Utils::now();
+
+		// Iterate through 18-byte network,MAC,ADI tuples:
+		while ((ptr + 18) <= size()) {
+			uint64_t nwid = at<uint64_t>(ptr); ptr += 8;
+			SharedPtr<Network> network(_r->nc->network(nwid));
+			if (network) {
+				if (network->isAllowed(source())) {
+					MAC mac(field(ptr,6)); ptr += 6;
+					uint32_t adi = at<uint32_t>(ptr); ptr += 4;
+					//TRACE("peer %s likes multicast group %s:%.8lx on network %llu",source().toString().c_str(),mac.toString().c_str(),(unsigned long)adi,nwid);
+					_r->multicaster->likesMulticastGroup(nwid,MulticastGroup(mac,adi),source(),now);
+					++numAccepted;
+				} else {
+					TRACE("ignored MULTICAST_LIKE from %s(%s): not a member of closed network %llu",source().toString().c_str(),_remoteAddress.toString().c_str(),nwid);
+				}
+			} else {
+				TRACE("ignored MULTICAST_LIKE from %s(%s): network %llu unknown or we are not a member",source().toString().c_str(),_remoteAddress.toString().c_str(),nwid);
+			}
+		}
+
+		Packet outp(source(),_r->identity.address(),Packet::VERB_OK);
+		outp.append((unsigned char)Packet::VERB_MULTICAST_LIKE);
+		outp.append(packetId());
+		outp.append((uint16_t)numAccepted);
+		outp.encrypt(peer->cryptKey());
+		outp.hmacSet(peer->macKey());
+		_r->demarc->send(_localPort,_remoteAddress,outp.data(),outp.size(),-1);
+	} catch (std::exception &ex) {
+		TRACE("dropped MULTICAST_LIKE from %s(%s): unexpected exception: %s",source().toString().c_str(),_remoteAddress.toString().c_str(),ex.what());
+	} catch ( ... ) {
+		TRACE("dropped MULTICAST_LIKE from %s(%s): unexpected exception: (unknown)",source().toString().c_str(),_remoteAddress.toString().c_str());
+	}
+	return true;
+}
+
+bool PacketDecoder::_doMULTICAST_FRAME(const RuntimeEnvironment *_r,const SharedPtr<Peer> &peer)
+{
+	try {
+		SharedPtr<Network> network(_r->nc->network(at<uint64_t>(ZT_PROTO_VERB_MULTICAST_FRAME_IDX_NETWORK_ID)));
+		if (network) {
+			if (network->isAllowed(source())) {
+				if (size() > ZT_PROTO_VERB_MULTICAST_FRAME_IDX_PAYLOAD) {
+
+					Address originalSubmitterAddress(field(ZT_PROTO_VERB_MULTICAST_FRAME_IDX_SUBMITTER_ADDRESS,ZT_ADDRESS_LENGTH));
+					MAC fromMac(field(ZT_PROTO_VERB_MULTICAST_FRAME_IDX_SOURCE_MAC,6));
+					MulticastGroup mg(MAC(field(ZT_PROTO_VERB_MULTICAST_FRAME_IDX_DESTINATION_MAC,6)),at<uint32_t>(ZT_PROTO_VERB_MULTICAST_FRAME_IDX_ADI));
+					unsigned int hops = (*this)[ZT_PROTO_VERB_MULTICAST_FRAME_IDX_HOP_COUNT];
+					unsigned int etherType = at<uint16_t>(ZT_PROTO_VERB_MULTICAST_FRAME_IDX_ETHERTYPE);
+					unsigned int datalen = at<uint16_t>(ZT_PROTO_VERB_MULTICAST_FRAME_IDX_PAYLOAD_LENGTH);
+					unsigned int signaturelen = at<uint16_t>(ZT_PROTO_VERB_MULTICAST_FRAME_IDX_SIGNATURE_LENGTH);
+					unsigned char *dataAndSignature = field(ZT_PROTO_VERB_MULTICAST_FRAME_IDX_PAYLOAD,datalen + signaturelen);
+
+					uint64_t mccrc = Multicaster::computeMulticastDedupCrc(network->id(),fromMac,mg,etherType,dataAndSignature,datalen);
+					uint64_t now = Utils::now();
+					bool isDuplicate = _r->multicaster->checkDuplicate(mccrc,now);
+
+					if (originalSubmitterAddress == _r->identity.address()) {
+						// Technically should not happen, since the original submitter is
+						// excluded from consideration as a propagation recipient.
+						TRACE("dropped boomerang MULTICAST_FRAME received from %s(%s)",source().toString().c_str(),_remoteAddress.toString().c_str());
+					} else if ((!isDuplicate)||(_r->topology->isSupernode(_r->identity.address()))) {
+						// If I am a supernode, I will repeatedly propagate duplicates. That's
+						// because supernodes are used to bridge sparse multicast groups. Non-
+						// supernodes will ignore duplicates completely.
+						SharedPtr<Peer> originalSubmitter(_r->topology->getPeer(originalSubmitterAddress));
+						if (!originalSubmitter) {
+							TRACE("requesting WHOIS on original multicast frame submitter %s",originalSubmitterAddress.toString().c_str());
+							_r->sw->requestWhois(originalSubmitterAddress);
+							_step = DECODE_STEP_WAITING_FOR_ORIGINAL_SUBMITTER_LOOKUP;
+							return false;
+						} else if (Multicaster::verifyMulticastPacket(originalSubmitter->identity(),network->id(),fromMac,mg,etherType,dataAndSignature,datalen,dataAndSignature + datalen,signaturelen)) {
+							_r->multicaster->addToDedupHistory(mccrc,now);
+
+							if (!isDuplicate)
+								network->tap().put(fromMac,mg.mac(),etherType,dataAndSignature,datalen);
+
+							if (++hops < ZT_MULTICAST_PROPAGATION_DEPTH) {
+								Address upstream(source()); // save this since we mangle it
+
+								Multicaster::MulticastBloomFilter bloom(field(ZT_PROTO_VERB_MULTICAST_FRAME_IDX_BLOOM_FILTER,ZT_PROTO_VERB_MULTICAST_FRAME_BLOOM_FILTER_SIZE_BYTES));
+								SharedPtr<Peer> propPeers[ZT_MULTICAST_PROPAGATION_BREADTH];
+								unsigned int np = _r->multicaster->pickNextPropagationPeers(
+									*(_r->prng),
+									*(_r->topology),
+									network->id(),
+									mg,
+									originalSubmitterAddress,
+									upstream,
+									bloom,
+									ZT_MULTICAST_PROPAGATION_BREADTH,
+									propPeers,
+									now);
+
+								// In a bit of a hack, we re-use this packet to repeat it
+								// to our multicast propagation recipients. Afterwords we
+								// return true just to be sure this is the end of this
+								// packet's life cycle, since it is now mangled.
+
+								setSource(_r->identity.address());
+								(*this)[ZT_PROTO_VERB_MULTICAST_FRAME_IDX_HOP_COUNT] = hops;
+								memcpy(field(ZT_PROTO_VERB_MULTICAST_FRAME_IDX_BLOOM_FILTER,ZT_PROTO_VERB_MULTICAST_FRAME_BLOOM_FILTER_SIZE_BYTES),bloom.data(),ZT_PROTO_VERB_MULTICAST_FRAME_BLOOM_FILTER_SIZE_BYTES);
+								compress();
+
+								for(unsigned int i=0;i<np;++i) {
+									TRACE("propagating multicast from original node %s: %s -> %s",originalSubmitterAddress.toString().c_str(),upstream.toString().c_str(),propPeers[i]->address().toString().c_str());
+									// Re-use this packet to re-send multicast frame to everyone
+									// downstream from us.
+									newInitializationVector();
+									setDestination(propPeers[i]->address());
+									_r->sw->send(*this,true);
+								}
+
+								return true;
+							} else {
+								TRACE("terminating MULTICAST_FRAME propagation from %s(%s): max depth reached",source().toString().c_str(),_remoteAddress.toString().c_str());
+							}
+						} else {
+							LOG("rejected MULTICAST_FRAME from %s(%s) due to failed signature check (claims original sender %s)",source().toString().c_str(),_remoteAddress.toString().c_str(),originalSubmitterAddress.toString().c_str());
+						}
+					} else {
+						TRACE("dropped redundant MULTICAST_FRAME from %s(%s)",source().toString().c_str(),_remoteAddress.toString().c_str());
+					}
+				} else {
+					TRACE("dropped MULTICAST_FRAME from %s(%s): invalid short packet",source().toString().c_str(),_remoteAddress.toString().c_str());
+				}
+			} else {
+				TRACE("dropped MULTICAST_FRAME from %s(%s): not a member of closed network %llu",source().toString().c_str(),_remoteAddress.toString().c_str(),network->id());
+			}
+		} else {
+			TRACE("dropped MULTICAST_FRAME from %s(%s): network %llu unknown or we are not a member",source().toString().c_str(),_remoteAddress.toString().c_str(),at<uint64_t>(ZT_PROTO_VERB_MULTICAST_FRAME_IDX_NETWORK_ID));
+		}
+	} catch (std::exception &ex) {
+		TRACE("dropped MULTICAST_FRAME from %s(%s): unexpected exception: %s",source().toString().c_str(),_remoteAddress.toString().c_str(),ex.what());
+	} catch ( ... ) {
+		TRACE("dropped MULTICAST_FRAME from %s(%s): unexpected exception: (unknown)",source().toString().c_str(),_remoteAddress.toString().c_str());
+	}
+	return true;
+}
+
+} // namespace ZeroTier

node/PacketDecoder.hpp

@@ -0,0 +1,140 @@
+/*
+ * ZeroTier One - Global Peer to Peer Ethernet
+ * Copyright (C) 2012-2013  ZeroTier Networks LLC
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation, either version 3 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ *
+ * --
+ *
+ * ZeroTier may be used and distributed under the terms of the GPLv3, which
+ * are available at: http://www.gnu.org/licenses/gpl-3.0.html
+ *
+ * If you would like to embed ZeroTier into a commercial application or
+ * redistribute it in a modified binary form, please contact ZeroTier Networks
+ * LLC. Start here: http://www.zerotier.com/
+ */
+
+#ifndef _ZT_PACKETDECODER_HPP
+#define _ZT_PACKETDECODER_HPP
+
+#include <stdexcept>
+
+#include "Packet.hpp"
+#include "Demarc.hpp"
+#include "InetAddress.hpp"
+#include "Utils.hpp"
+#include "SharedPtr.hpp"
+#include "AtomicCounter.hpp"
+#include "Peer.hpp"
+
+namespace ZeroTier {
+
+class RuntimeEnvironment;
+
+/**
+ * Subclass of packet that handles the decoding of it
+ */
+class PacketDecoder : public Packet
+{
+	friend class SharedPtr<PacketDecoder>;
+
+public:
+	/**
+	 * Create a new packet-in-decode
+	 *
+	 * @param b Source buffer with raw packet data
+	 * @param localPort Local port on which packet was received
+	 * @param remoteAddress Address from which packet came
+	 * @throws std::out_of_range Range error processing packet
+	 */
+	template<unsigned int C2>
+	PacketDecoder(const Buffer<C2> &b,Demarc::Port localPort,const InetAddress &remoteAddress)
+ 		throw(std::out_of_range) :
+ 		Packet(b),
+ 		_receiveTime(Utils::now()),
+ 		_localPort(localPort),
+ 		_remoteAddress(remoteAddress),
+ 		_step(DECODE_STEP_WAITING_FOR_SENDER_LOOKUP),
+ 		__refCount()
+	{
+	}
+
+	/**
+	 * Attempt to decode this packet
+	 *
+	 * Note that this returns 'true' if processing is complete. This says nothing
+	 * about whether the packet was valid. A rejection is 'complete.'
+	 *
+	 * Once true is returned, this should not be called again.
+	 *
+	 * @param _r Runtime environment
+	 * @return True if decoding and processing is complete, false if caller should try again
+	 * @throws std::out_of_range Range error processing packet (should be discarded)
+	 * @throws std::runtime_error Other error processing packet (should be discarded)
+	 */
+	bool tryDecode(const RuntimeEnvironment *_r)
+		throw(std::out_of_range,std::runtime_error);
+
+	/**
+	 * @return Time of packet receipt
+	 */
+	inline uint64_t receiveTime() const throw() { return _receiveTime; }
+
+private:
+	struct _CBaddPeerFromHello_Data
+	{
+		const RuntimeEnvironment *renv;
+		Address source;
+		InetAddress remoteAddress;
+		int localPort;
+		unsigned int vMajor,vMinor,vRevision;
+		uint64_t helloPacketId;
+		uint64_t helloTimestamp;
+	};
+	static void _CBaddPeerFromHello(
+		void *arg, // _CBaddPeerFromHello_Data
+		const SharedPtr<Peer> &p,
+		Topology::PeerVerifyResult result);
+
+	static void _CBaddPeerFromWhois(
+		void *arg, // RuntimeEnvironment
+		const SharedPtr<Peer> &p,
+		Topology::PeerVerifyResult result);
+
+	// These are called internally to handle packet contents once it has
+	// been authenticated, decrypted, decompressed, and classified.
+	bool _doERROR(const RuntimeEnvironment *_r,const SharedPtr<Peer> &peer);
+	bool _doHELLO(const RuntimeEnvironment *_r);
+	bool _doOK(const RuntimeEnvironment *_r,const SharedPtr<Peer> &peer);
+	bool _doWHOIS(const RuntimeEnvironment *_r,const SharedPtr<Peer> &peer);
+	bool _doRENDEZVOUS(const RuntimeEnvironment *_r,const SharedPtr<Peer> &peer);
+	bool _doFRAME(const RuntimeEnvironment *_r,const SharedPtr<Peer> &peer);
+	bool _doMULTICAST_LIKE(const RuntimeEnvironment *_r,const SharedPtr<Peer> &peer);
+	bool _doMULTICAST_FRAME(const RuntimeEnvironment *_r,const SharedPtr<Peer> &peer);
+
+	uint64_t _receiveTime;
+	Demarc::Port _localPort;
+	InetAddress _remoteAddress;
+
+	enum {
+		DECODE_STEP_WAITING_FOR_SENDER_LOOKUP, // on initial receipt, we need peer's identity
+		DECODE_STEP_WAITING_FOR_ORIGINAL_SUBMITTER_LOOKUP // this only applies to MULTICAST_FRAME
+	} _step;
+
+	AtomicCounter __refCount;
+};
+
+} // namespace ZeroTier
+
+#endif

node/Peer.cpp

@@ -30,6 +30,14 @@
 namespace ZeroTier {
 
 Peer::Peer() :
+	_id(),
+	_ipv4p(),
+	_ipv6p(),
+	_lastUnicastFrame(0),
+	_lastMulticastFrame(0),
+	_vMajor(0),
+	_vMinor(0),
+	_vRevision(0),
 	_dirty(false)
 {
 }
@@ -37,37 +45,44 @@ Peer::Peer() :
 Peer::Peer(const Identity &myIdentity,const Identity &peerIdentity)
 	throw(std::runtime_error) :
 	_id(peerIdentity),
+	_ipv4p(),
+	_ipv6p(),
+	_lastUnicastFrame(0),
+	_lastMulticastFrame(0),
+	_vMajor(0),
+	_vMinor(0),
+	_vRevision(0),
 	_dirty(true)
 {
 	if (!myIdentity.agree(peerIdentity,_keys,sizeof(_keys)))
 		throw std::runtime_error("new peer identity key agreement failed");
 }
 
-void Peer::onReceive(const RuntimeEnvironment *_r,Demarc::Port localPort,const InetAddress &fromAddr,unsigned int latency,unsigned int hops,Packet::Verb verb,uint64_t now)
+void Peer::onReceive(const RuntimeEnvironment *_r,Demarc::Port localPort,const InetAddress &remoteAddr,unsigned int hops,Packet::Verb verb,uint64_t now)
 {
 	if (!hops) { // direct packet
-		WanPath *wp = (fromAddr.isV4() ? &_ipv4p : &_ipv6p);
-
+		WanPath *wp = (remoteAddr.isV4() ? &_ipv4p : &_ipv6p);
 		wp->lastReceive = now;
-		if (verb == Packet::VERB_FRAME)
-			wp->lastUnicastFrame = now;
-		if (latency)
-			wp->latency = latency;
 		wp->localPort = localPort;
 		if (!wp->fixed)
-			wp->addr = fromAddr;
+			wp->addr = remoteAddr;
+		_dirty = true;
+	}
 
+	if (verb == Packet::VERB_FRAME) {
+		_lastUnicastFrame = now;
+		_dirty = true;
+	} else if (verb == Packet::VERB_MULTICAST_FRAME) {
+		_lastMulticastFrame = now;
 		_dirty = true;
 	}
 }
 
-bool Peer::send(const RuntimeEnvironment *_r,const void *data,unsigned int len,bool relay,Packet::Verb verb,uint64_t now)
+bool Peer::send(const RuntimeEnvironment *_r,const void *data,unsigned int len,uint64_t now)
 {
 	if ((_ipv6p.isActive(now))||((!(_ipv4p.addr))&&(_ipv6p.addr))) {
 		if (_r->demarc->send(_ipv6p.localPort,_ipv6p.addr,data,len,-1)) {
 			_ipv6p.lastSend = now;
-			if (verb == Packet::VERB_FRAME)
-				_ipv6p.lastUnicastFrame = now;
 			_dirty = true;
 			return true;
 		}
@@ -76,8 +91,6 @@ bool Peer::send(const RuntimeEnvironment *_r,const void *data,unsigned int len,b
 	if (_ipv4p.addr) {
 		if (_r->demarc->send(_ipv4p.localPort,_ipv4p.addr,data,len,-1)) {
 			_ipv4p.lastSend = now;
-			if (verb == Packet::VERB_FRAME)
-				_ipv4p.lastUnicastFrame = now;
 			_dirty = true;
 			return true;
 		}
@@ -86,6 +99,17 @@ bool Peer::send(const RuntimeEnvironment *_r,const void *data,unsigned int len,b
 	return false;
 }
 
+void Peer::onSent(const RuntimeEnvironment *_r,bool relay,Packet::Verb verb,uint64_t now)
+{
+	if (verb == Packet::VERB_FRAME) {
+		_lastUnicastFrame = now;
+		_dirty = true;
+	} else if (verb == Packet::VERB_MULTICAST_FRAME) {
+		_lastMulticastFrame = now;
+		_dirty = true;
+	}
+}
+
 bool Peer::sendFirewallOpener(const RuntimeEnvironment *_r,uint64_t now)
 {
 	bool sent = false;

node/Peer.hpp

@@ -53,15 +53,16 @@
 #define ZT_PEER_MAX_SERIALIZED_LENGTH ( \
 	64 + \
 	IDENTITY_MAX_BINARY_SERIALIZED_LENGTH + \
-	(( \
-		(sizeof(uint64_t) * 5) + \
+	( ( \
+		(sizeof(uint64_t) * 4) + \
 		sizeof(uint16_t) + \
 		1 + \
 		sizeof(uint16_t) + \
 		16 + \
 		1 \
 	) * 2) + \
-	64 \
+	sizeof(uint64_t) + \
+	sizeof(uint64_t) \
 )
 
 namespace ZeroTier {
@@ -110,33 +111,42 @@ public:
 	/**
 	 * Must be called on authenticated packet receive from this peer
 	 * 
+	 * This must be called only after a packet has passed authentication
+	 * checking. Packets that fail are silently discarded.
+	 *
 	 * @param _r Runtime environment
 	 * @param localPort Local port on which packet was received
-	 * @param fromAddr Internet address of sender
-	 * @param latency Latency or 0 if unknown
+	 * @param remoteAddr Internet address of sender
 	 * @param hops ZeroTier (not IP) hops
 	 * @param verb Packet verb
 	 * @param now Current time
 	 */
-	void onReceive(const RuntimeEnvironment *_r,Demarc::Port localPort,const InetAddress &fromAddr,unsigned int latency,unsigned int hops,Packet::Verb verb,uint64_t now);
+	void onReceive(const RuntimeEnvironment *_r,Demarc::Port localPort,const InetAddress &remoteAddr,unsigned int hops,Packet::Verb verb,uint64_t now);
 
 	/**
-	 * Send a UDP packet to this peer
-	 * 
-	 * If the active link is timed out (no receives for ping timeout ms), then
-	 * the active link number is incremented after send. This causes sends to
-	 * cycle through links if there is no clear active link. This also happens
-	 * if the send fails for some reason.
+	 * Send a packet to this peer
 	 * 
 	 * @param _r Runtime environment
 	 * @param data Data to send
 	 * @param len Length of packet
-	 * @param relay This is a relay on behalf of another peer (verb is ignored)
-	 * @param verb Packet verb (if not relay)
 	 * @param now Current time
 	 * @return True if packet appears to have been sent, false on local failure
 	 */
-	bool send(const RuntimeEnvironment *_r,const void *data,unsigned int len,bool relay,Packet::Verb verb,uint64_t now);
+	bool send(const RuntimeEnvironment *_r,const void *data,unsigned int len,uint64_t now);
+
+	/**
+	 * Must be called after a packet is successfully sent to this peer
+	 *
+	 * Note that 'relay' means we've sent a packet *from* this node to this
+	 * peer by relaying it, not that we have relayed a packet from somewhere
+	 * else to this peer. In the latter case this is not called.
+	 *
+	 * @param _r Runtime environment
+	 * @param relay If true, packet was sent indirectly via a relay
+	 * @param verb Packet verb
+	 * @param now Current time
+	 */
+	void onSent(const RuntimeEnvironment *_r,bool relay,Packet::Verb verb,uint64_t now);
 
 	/**
 	 * Send firewall opener to active link
@@ -195,7 +205,25 @@ public:
 	uint64_t lastUnicastFrame() const
 		throw()
 	{
-		return std::max(_ipv4p.lastUnicastFrame,_ipv6p.lastUnicastFrame);
+		return _lastUnicastFrame;
+	}
+
+	/**
+	 * @return Time of most recent multicast frame
+	 */
+	uint64_t lastMulticastFrame() const
+		throw()
+	{
+		return _lastMulticastFrame;
+	}
+
+	/**
+	 * @return Time of most recent frame of any kind (unicast or multicast)
+	 */
+	uint64_t lastFrame() const
+		throw()
+	{
+		return std::max(_lastUnicastFrame,_lastMulticastFrame);
 	}
 
 	/**
@@ -213,6 +241,21 @@ public:
 		return 0;
 	}
 
+	/**
+	 * @param addr Remote address
+	 * @param latency Latency measurment
+	 */
+	void setLatency(const InetAddress &addr,unsigned int latency)
+	{
+		if (addr == _ipv4p.addr) {
+			_ipv4p.latency = latency;
+			_dirty = true;
+		} else if (addr == _ipv6p.addr) {
+			_ipv6p.latency = latency;
+			_dirty = true;
+		}
+	}
+
 	/**
 	 * @return True if this peer has at least one direct IP address path
 	 */
@@ -232,6 +275,46 @@ public:
 		return ((_ipv4p.isActive(now))||(_ipv6p.isActive(now)));
 	}
 
+	/**
+	 * @return IPv4 direct address or null InetAddress if none
+	 */
+	inline InetAddress ipv4Path() const
+		throw()
+	{
+		return _ipv4p.addr;
+	}
+
+	/**
+	 * @return IPv6 direct address or null InetAddress if none
+	 */
+	inline InetAddress ipv6Path() const
+		throw()
+	{
+		return _ipv4p.addr;
+	}
+
+	/**
+	 * @return IPv4 direct address or null InetAddress if none
+	 */
+	inline InetAddress ipv4ActivePath(uint64_t now) const
+		throw()
+	{
+		if (_ipv4p.isActive(now))
+			return _ipv4p.addr;
+		return InetAddress();
+	}
+
+	/**
+	 * @return IPv6 direct address or null InetAddress if none
+	 */
+	inline InetAddress ipv6ActivePath(uint64_t now) const
+		throw()
+	{
+		if (_ipv6p.isActive(now))
+			return _ipv6p.addr;
+		return InetAddress();
+	}
+
 	/**
 	 * @return 256-bit encryption key
 	 */
@@ -250,6 +333,20 @@ public:
 		return (_keys + 32); // mac key is second 32-byte key
 	}
 
+	/**
+	 * Set the remote version of the peer (not persisted)
+	 *
+	 * @param vmaj Major version
+	 * @param vmin Minor version
+	 * @param vrev Revision
+	 */
+	inline void setRemoteVersion(unsigned int vmaj,unsigned int vmin,unsigned int vrev)
+	{
+		_vMajor = vmaj;
+		_vMinor = vmin;
+		_vRevision = vrev;
+	}
+
 	/**
 	 * Get and reset dirty flag
 	 * 
@@ -272,11 +369,13 @@ public:
 	inline void serialize(Buffer<C> &b)
 		throw(std::out_of_range)
 	{
-		b.append((unsigned char)1); // version
+		b.append((unsigned char)2); // version
 		b.append(_keys,sizeof(_keys));
 		_id.serialize(b,false);
 		_ipv4p.serialize(b);
 		_ipv6p.serialize(b);
+		b.append(_lastUnicastFrame);
+		b.append(_lastMulticastFrame);
 	}
 
 	template<unsigned int C>
@@ -285,14 +384,19 @@ public:
 	{
 		unsigned int p = startAt;
 
-		if (b[p++] != 1)
+		if (b[p++] != 2)
 			throw std::invalid_argument("Peer: deserialize(): version mismatch");
 
 		memcpy(_keys,b.field(p,sizeof(_keys)),sizeof(_keys)); p += sizeof(_keys);
 		p += _id.deserialize(b,p);
 		p += _ipv4p.deserialize(b,p);
 		p += _ipv6p.deserialize(b,p);
+		_lastUnicastFrame = b.template at<uint64_t>(p); p += sizeof(uint64_t);
+		_lastMulticastFrame = b.template at<uint64_t>(p); p += sizeof(uint64_t);
 
+		_vMajor = 0;
+		_vMinor = 0;
+		_vRevision = 0;
 		_dirty = false;
 
 		return (p - startAt);
@@ -332,7 +436,6 @@ private:
 		WanPath() :
 			lastSend(0),
 			lastReceive(0),
-			lastUnicastFrame(0),
 			lastFirewallOpener(0),
 			localPort(Demarc::ANY_PORT),
 			latency(0),
@@ -353,7 +456,6 @@ private:
 		{
 			b.append(lastSend);
 			b.append(lastReceive);
-			b.append(lastUnicastFrame);
 			b.append(lastFirewallOpener);
 			b.append(Demarc::portToInt(localPort));
 			b.append((uint16_t)latency);
@@ -383,7 +485,6 @@ private:
 
 			lastSend = b.template at<uint64_t>(p); p += sizeof(uint64_t);
 			lastReceive = b.template at<uint64_t>(p); p += sizeof(uint64_t);
-			lastUnicastFrame = b.template at<uint64_t>(p); p += sizeof(uint64_t);
 			lastFirewallOpener = b.template at<uint64_t>(p); p += sizeof(uint64_t);
 			localPort = Demarc::intToPort(b.template at<uint64_t>(p)); p += sizeof(uint64_t);
 			latency = b.template at<uint16_t>(p); p += sizeof(uint16_t);
@@ -409,9 +510,8 @@ private:
 
 		uint64_t lastSend;
 		uint64_t lastReceive;
-		uint64_t lastUnicastFrame;
 		uint64_t lastFirewallOpener;
-		Demarc::Port localPort; // ANY_PORT if not defined
+		Demarc::Port localPort; // ANY_PORT if not defined (size: uint64_t)
 		unsigned int latency; // 0 if never determined
 		InetAddress addr; // null InetAddress if path is undefined
 		bool fixed; // do not learn address from received packets
@@ -423,8 +523,12 @@ private:
 	WanPath _ipv4p;
 	WanPath _ipv6p;
 
+	uint64_t _lastUnicastFrame;
+	uint64_t _lastMulticastFrame;
+
 	// Fields below this line are not persisted with serialize()
 
+	unsigned int _vMajor,_vMinor,_vRevision;
 	bool _dirty;
 
 	AtomicCounter __refCount;
@@ -432,4 +536,13 @@ private:
 
 } // namespace ZeroTier
 
+// Add a swap() for shared ptr's to peers to speed up peer sorts
+namespace std {
+	template<>
+	inline void swap(ZeroTier::SharedPtr<ZeroTier::Peer> &a,ZeroTier::SharedPtr<ZeroTier::Peer> &b)
+	{
+		a.swap(b);
+	}
+}
+
 #endif

node/Range.hpp

@@ -0,0 +1,123 @@
+/*
+ * ZeroTier One - Global Peer to Peer Ethernet
+ * Copyright (C) 2012-2013  ZeroTier Networks LLC
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation, either version 3 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ *
+ * --
+ *
+ * ZeroTier may be used and distributed under the terms of the GPLv3, which
+ * are available at: http://www.gnu.org/licenses/gpl-3.0.html
+ *
+ * If you would like to embed ZeroTier into a commercial application or
+ * redistribute it in a modified binary form, please contact ZeroTier Networks
+ * LLC. Start here: http://www.zerotier.com/
+ */
+
+#ifndef _ZT_RANGE_HPP
+#define _ZT_RANGE_HPP
+
+namespace ZeroTier {
+
+/**
+ * A range of numeric values
+ *
+ * @tparam T Type, can be any numeric value (int, float, double, etc.)
+ */
+template<typename T>
+class Range
+{
+public:
+	/**
+	 * Construct an empty range
+	 */
+	Range()
+		throw() :
+		start(0),
+		end(0)
+	{
+	}
+
+	/**
+	 * @param s Starting value (inclusive)
+	 * @param e Ending value (exclusive)
+	 */
+	Range(T s,T e)
+		throw() :
+		start(s),
+		end(e)
+	{
+	}
+
+	/**
+	 * Construct a range containing from n to n+1 (thus only n for integers)
+	 *
+	 * @param n Number to contain
+	 */
+	Range(T n)
+		throw() :
+		start(n),
+		end(n+1)
+	{
+	}
+
+	/**
+	 * @return end - start
+	 */
+	inline T magnitude() const
+		throw()
+	{
+		return (end - start);
+	}
+
+	/**
+	 * @return True if range contains something (magnitude is nonzero)
+	 */
+	inline operator bool() const
+		throw()
+	{
+		return (end > start);
+	}
+
+	/**
+	 * @param v Value to test
+	 * @return True if value is between start (inclusive) and end (exclusive)
+	 */
+	inline bool operator()(const T &v) const
+		throw()
+	{
+		return ((v >= start)&&(v < end));
+	}
+
+	inline bool operator==(const Range &r) const throw() { return ((start == r.start)&&(end == r.end)); }
+	inline bool operator!=(const Range &r) const throw() { return (!(*this == r)); }
+	inline bool operator<(const Range &r) const throw() { return ((start < r.start) ? true : ((start == r.start) ? (end < r.end) : false)); }
+	inline bool operator>(const Range &r) const throw() { return (r < *this); }
+	inline bool operator<=(const Range &r) const throw() { return !(r < *this); }
+	inline bool operator>=(const Range &r) const throw() { return !(*this < r); }
+
+	/**
+	 * Start of range (may be modified directly)
+	 */
+	T start;
+
+	/**
+	 * End of range (may be modified directly)
+	 */
+	T end;
+};
+
+} // namespace ZeroTier
+
+#endif

node/RuntimeEnvironment.hpp

@@ -30,6 +30,7 @@
 
 #include <string>
 #include "Identity.hpp"
+#include "Condition.hpp"
 
 namespace ZeroTier {
 
@@ -39,6 +40,8 @@ class Demarc;
 class Switch;
 class Topology;
 class SysEnv;
+class Multicaster;
+class CMWC4096;
 
 /**
  * Holds global state for an instance of ZeroTier::Node
@@ -58,8 +61,10 @@ public:
 	RuntimeEnvironment() :
 		identity(),
 		log((Logger *)0),
+		prng((CMWC4096 *)0),
 		nc((NodeConfig *)0),
 		demarc((Demarc *)0),
+		multicaster((Multicaster *)0),
 		sw((Switch *)0),
 		topology((Topology *)0)
 	{
@@ -71,12 +76,17 @@ public:
 	std::string ownershipVerificationSecret;
 	std::string ownershipVerificationSecretHash; // base64 of SHA-256 X16 rounds
 
+	// signal() to prematurely interrupt main loop wait
+	Condition mainLoopWaitCondition;
+
 	Identity configAuthority;
 	Identity identity;
 
 	Logger *log; // may be null
+	CMWC4096 *prng;
 	NodeConfig *nc;
 	Demarc *demarc;
+	Multicaster *multicaster;
 	Switch *sw;
 	Topology *topology;
 	SysEnv *sysEnv;

node/Salsa20.cpp

@@ -5,6 +5,7 @@
  */
 
 #include "Salsa20.hpp"
+#include "Constants.hpp"
 
 #define ROTATE(v,c) (((v) << (c)) | ((v) >> (32 - (c))))
 #define XOR(v,w) ((v) ^ (w))

node/SharedPtr.hpp

@@ -88,6 +88,14 @@ public:
 		return *this;
 	}
 
+	inline void swap(SharedPtr &with)
+		throw()
+	{
+		T *tmp = _ptr;
+		_ptr = with._ptr;
+		with._ptr = tmp;
+	}
+
 	inline operator bool() const throw() { return (_ptr); }
 	inline T &operator*() const throw() { return *_ptr; }
 	inline T *operator->() const throw() { return _ptr; }

node/Switch.hpp

@@ -31,6 +31,7 @@
 #include <map>
 #include <set>
 #include <vector>
+#include <list>
 
 #include "Mutex.hpp"
 #include "MAC.hpp"
@@ -44,6 +45,8 @@
 #include "Network.hpp"
 #include "SharedPtr.hpp"
 #include "Demarc.hpp"
+#include "Multicaster.hpp"
+#include "PacketDecoder.hpp"
 
 namespace ZeroTier {
 
@@ -106,6 +109,16 @@ public:
 	 */
 	void sendHELLO(const Address &dest);
 
+	/**
+	 * Send a HELLO announcement immediately to the indicated address
+	 *
+	 * @param localPort Originating local port or ANY_PORT to pick
+	 * @param remoteAddr IP address to send to
+	 * @param dest Destination peer
+	 * @return True if send appears successful
+	 */
+	bool sendHELLO(const SharedPtr<Peer> &dest,Demarc::Port localPort,const InetAddress &remoteAddr);
+
 	/**
 	 * Send RENDEZVOUS to two peers to permit them to directly connect
 	 *
@@ -122,6 +135,14 @@ public:
 	 */
 	bool unite(const Address &p1,const Address &p2,bool force);
 
+	/**
+	 * Send NAT traversal messages to peer at the given candidate address
+	 *
+	 * @param peer Peer to contact
+	 * @param atAddr Address of peer
+	 */
+	void contact(const SharedPtr<Peer> &peer,const InetAddress &atAddr);
+
 	/**
 	 * Perform retries and other periodic timer tasks
 	 * 
@@ -139,70 +160,44 @@ public:
 	 */
 	void announceMulticastGroups(const std::map< SharedPtr<Network>,std::set<MulticastGroup> > &allMemberships);
 
+	/**
+	 * Request WHOIS on a given address
+	 *
+	 * @param addr Address to look up
+	 */
+	void requestWhois(const Address &addr);
+
+	/**
+	 * Run any processes that are waiting for this peer
+	 *
+	 * Called when we learn of a peer's identity from HELLO, OK(WHOIS), etc.
+	 *
+	 * @param peer New peer
+	 */
+	void doAnythingWaitingForPeer(const SharedPtr<Peer> &peer);
+
 private:
-	// Returned by _send() and _processRemotePacket() to indicate what happened
-	enum PacketServiceAttemptResult
-	{
-		PACKET_SERVICE_ATTEMPT_OK,
-		PACKET_SERVICE_ATTEMPT_PEER_UNKNOWN,
-		PACKET_SERVICE_ATTEMPT_SEND_FAILED
-	};
+	void _handleRemotePacketFragment(
+		Demarc::Port localPort,
+		const InetAddress &fromAddr,
+		const Buffer<4096> &data);
 
-	struct _CBaddPeerFromHello_Data
-	{
-		Switch *parent;
-		Address source;
-		InetAddress fromAddr;
-		int localPort;
-		unsigned int vMajor,vMinor,vRevision;
-		uint64_t helloPacketId;
-		uint64_t helloTimestamp;
-	};
-	static void _CBaddPeerFromHello(void *arg,const SharedPtr<Peer> &p,Topology::PeerVerifyResult result);
-	static void _CBaddPeerFromWhois(void *arg,const SharedPtr<Peer> &p,Topology::PeerVerifyResult result); // arg == this
+	void _handleRemotePacketHead(
+		Demarc::Port localPort,
+		const InetAddress &fromAddr,
+		const Buffer<4096> &data);
 
-	void _propagateMulticast(const SharedPtr<Network> &network,unsigned char *bloom,const MulticastGroup &mg,unsigned int mcHops,unsigned int mcLoadFactor,const MAC &from,unsigned int etherType,const void *data,unsigned int len);
-	PacketServiceAttemptResult _tryHandleRemotePacket(Demarc::Port localPort,const InetAddress &fromAddr,Packet &packet);
-	void _doHELLO(Demarc::Port localPort,const InetAddress &fromAddr,Packet &packet);
-	void _requestWhois(const Address &addr);
-	Address _sendWhoisRequest(const Address &addr,const Address *peersAlreadyConsulted,unsigned int numPeersAlreadyConsulted);
-	PacketServiceAttemptResult _trySend(const Packet &packet,bool encrypt);
-	void _retryPendingFor(const Address &addr);
+	Address _sendWhoisRequest(
+		const Address &addr,
+		const Address *peersAlreadyConsulted,
+		unsigned int numPeersAlreadyConsulted);
 
-	// Updates entry for crc in multicast history, returns true if already
-	// present in history and not expired.
-	inline bool _checkAndUpdateMulticastHistory(const MAC &fromMac,const MAC &toMulticastMac,const void *payload,unsigned int len,const uint64_t nwid,const uint64_t now)
-	{
-		uint64_t crc = Utils::crc64(0,fromMac.data,6);
-		crc = Utils::crc64(crc,toMulticastMac.data,6);
-		crc = Utils::crc64(crc,payload,len);
-		crc += nwid; // also include network ID
-
-		uint64_t earliest = 0xffffffffffffffffULL;
-		unsigned long earliestIdx = 0;
-		for(unsigned int i=0;i<ZT_MULTICAST_DEDUP_HISTORY_LENGTH;++i) {
-			if (_multicastHistory[i][0] == crc) {
-				uint64_t then = _multicastHistory[i][1];
-				_multicastHistory[i][1] = now;
-				return ((now - then) < ZT_MULTICAST_DEDUP_HISTORY_EXPIRE);
-			} else if (_multicastHistory[i][1] < earliest) {
-				earliest = _multicastHistory[i][1];
-				earliestIdx = i;
-			}
-		}
-
-		_multicastHistory[earliestIdx][0] = crc; // replace oldest entry
-		_multicastHistory[earliestIdx][1] = now;
-
-		return false;
-	}
+	bool _trySend(
+		const Packet &packet,
+		bool encrypt);
 
 	const RuntimeEnvironment *const _r;
 
-	// Multicast packet CRC64's for packets we've received recently, to reject
-	// duplicates during propagation. [0] is CRC64, [1] is time.
-	uint64_t _multicastHistory[ZT_MULTICAST_DEDUP_HISTORY_LENGTH][2];
-
 	struct WhoisRequest
 	{
 		uint64_t lastSent;
@@ -212,29 +207,28 @@ private:
 	std::map< Address,WhoisRequest > _outstandingWhoisRequests;
 	Mutex _outstandingWhoisRequests_m;
 
+	std::list< SharedPtr<PacketDecoder> > _rxQueue;
+	Mutex _rxQueue_m;
+
 	struct TXQueueEntry
 	{
+		TXQueueEntry() {}
+		TXQueueEntry(uint64_t ct,const Packet &p,bool enc) :
+			creationTime(ct),
+			packet(p),
+			encrypt(enc) {}
+
 		uint64_t creationTime;
 		Packet packet; // unencrypted/untagged for TX queue
 		bool encrypt;
 	};
-	std::multimap< Address,TXQueueEntry > _txQueue; // by destination address
+	std::multimap< Address,TXQueueEntry > _txQueue;
 	Mutex _txQueue_m;
 
-	struct RXQueueEntry
-	{
-		uint64_t creationTime;
-		Demarc::Port localPort;
-		Packet packet; // encrypted/tagged
-		InetAddress fromAddr;
-	};
-	std::multimap< Address,RXQueueEntry > _rxQueue; // by source address
-	Mutex _rxQueue_m;
-
 	struct DefragQueueEntry
 	{
 		uint64_t creationTime;
-		Packet frag0;
+		SharedPtr<PacketDecoder> frag0;
 		Packet::Fragment frags[ZT_MAX_PACKET_FRAGMENTS - 1];
 		unsigned int totalFragments; // 0 if only frag0 received, waiting for frags
 		uint32_t haveFragments; // bit mask, LSB to MSB
@@ -245,14 +239,22 @@ private:
 	std::map< Array< Address,2 >,uint64_t > _lastUniteAttempt; // key is always sorted in ascending order, for set-like behavior
 	Mutex _lastUniteAttempt_m;
 
-	struct RendezvousQueueEntry
+	struct ContactQueueEntry
 	{
-		InetAddress inaddr;
+		ContactQueueEntry() {}
+		ContactQueueEntry(const SharedPtr<Peer> &p,uint64_t ft,Demarc::Port lp,const InetAddress &a) :
+			peer(p),
+			fireAtTime(ft),
+			localPort(lp),
+			inaddr(a) {}
+
+		SharedPtr<Peer> peer;
 		uint64_t fireAtTime;
 		Demarc::Port localPort;
+		InetAddress inaddr;
 	};
-	std::map< Address,RendezvousQueueEntry > _rendezvousQueue;
-	Mutex _rendezvousQueue_m;
+	std::list<ContactQueueEntry> _contactQueue;
+	Mutex _contactQueue_m;
 };
 
 } // namespace ZeroTier

node/Topology.cpp

@@ -25,8 +25,10 @@
  * LLC. Start here: http://www.zerotier.com/
  */
 
+#include <algorithm>
 #include "Topology.hpp"
 #include "NodeConfig.hpp"
+#include "CMWC4096.hpp"
 
 namespace ZeroTier {
 
@@ -145,23 +147,32 @@ SharedPtr<Peer> Topology::getPeer(const Address &zta)
 	return SharedPtr<Peer>();
 }
 
-SharedPtr<Peer> Topology::getBestSupernode(const Address *avoid,unsigned int avoidCount) const
+SharedPtr<Peer> Topology::getBestSupernode(const Address *avoid,unsigned int avoidCount,bool strictAvoid) const
 {
 	SharedPtr<Peer> bestSupernode;
-	unsigned long bestSupernodeLatency = 0xffff;
+	unsigned int bestSupernodeLatency = 0xffff;
 	uint64_t now = Utils::now();
 
 	Mutex::Lock _l(_supernodes_m);
 
+	if (_supernodePeers.empty())
+		return bestSupernode;
+
 	for(std::vector< SharedPtr<Peer> >::const_iterator sn=_supernodePeers.begin();sn!=_supernodePeers.end();) {
 		for(unsigned int i=0;i<avoidCount;++i) {
 			if (avoid[i] == (*sn)->address())
 				goto skip_and_try_next_supernode;
 		}
-		if ((*sn)->hasActiveDirectPath(now)) { // only consider those that responded to pings
+		if ((*sn)->hasActiveDirectPath(now)) {
 			unsigned int l = (*sn)->latency();
-			if ((l)&&(l <= bestSupernodeLatency)) {
-				bestSupernodeLatency = l;
+			if (bestSupernode) {
+				if ((l)&&(l < bestSupernodeLatency)) {
+					bestSupernodeLatency = l;
+					bestSupernode = *sn;
+				}
+			} else {
+				if (l)
+					bestSupernodeLatency = l;
 				bestSupernode = *sn;
 			}
 		}
@@ -169,14 +180,20 @@ skip_and_try_next_supernode:
 		++sn;
 	}
 
-	if (bestSupernode)
+	if ((bestSupernode)||(strictAvoid))
 		return bestSupernode;
 
 	for(std::vector< SharedPtr<Peer> >::const_iterator sn=_supernodePeers.begin();sn!=_supernodePeers.end();++sn) {
-		if ((*sn)->hasActiveDirectPath(now)) { // only consider those that responded to pings
+		if ((*sn)->hasActiveDirectPath(now)) {
 			unsigned int l = (*sn)->latency();
-			if ((l)&&(l <= bestSupernodeLatency)) {
-				bestSupernodeLatency = l;
+			if (bestSupernode) {
+				if ((l)&&(l < bestSupernodeLatency)) {
+					bestSupernodeLatency = l;
+					bestSupernode = *sn;
+				}
+			} else {
+				if (l)
+					bestSupernodeLatency = l;
 				bestSupernode = *sn;
 			}
 		}
@@ -185,16 +202,7 @@ skip_and_try_next_supernode:
 	if (bestSupernode)
 		return bestSupernode;
 
-	uint64_t bestSupernodeLastDirectReceive = 0;
-	for(std::vector< SharedPtr<Peer> >::const_iterator sn=_supernodePeers.begin();sn!=_supernodePeers.end();++sn) {
-		uint64_t l = (*sn)->lastDirectReceive();
-		if (l > bestSupernodeLastDirectReceive) {
-			bestSupernodeLastDirectReceive = l;
-			bestSupernode = *sn;
-		}
-	}
-
-	return bestSupernode;
+	return _supernodePeers[_r->prng->next32() % _supernodePeers.size()];
 }
 
 void Topology::clean()
@@ -207,104 +215,6 @@ void Topology::clean()
 	_peerDeepVerifyJobs_c.signal();
 }
 
-void Topology::likesMulticastGroup(uint64_t nwid,const MulticastGroup &mg,const Address &addr,uint64_t now)
-{
-	Mutex::Lock _l(_multicastGroupMembers_m);
-	_multicastGroupMembers[nwid][mg][addr] = now;
-}
-
-struct _PickMulticastPropagationPeersPeerPrioritySortOrder
-{
-	inline bool operator()(const SharedPtr<Peer> &p1,const SharedPtr<Peer> &p2) const
-	{
-		return (p1->lastUnicastFrame() >= p2->lastUnicastFrame());
-	}
-};
-#define _MAX_PEERS_TO_CONSIDER 256
-unsigned int Topology::pickMulticastPropagationPeers(uint64_t nwid,const Address &exclude,const void *propagationBloom,unsigned int propagationBloomSize,unsigned int count,const MulticastGroup &mg,SharedPtr<Peer> *peers)
-{
-	SharedPtr<Peer> possiblePeers[_MAX_PEERS_TO_CONSIDER];
-	unsigned int numPossiblePeers = 0;
-
-	if (count > _MAX_PEERS_TO_CONSIDER)
-		count = _MAX_PEERS_TO_CONSIDER;
-
-	Mutex::Lock _l1(_activePeers_m);
-	Mutex::Lock _l2(_supernodes_m);
-
-	// Grab known non-supernode peers in multicast group, excluding 'exclude'
-	// Also lazily clean up the _multicastGroupMembers structure
-	{
-		Mutex::Lock _l3(_multicastGroupMembers_m);
-		std::map< uint64_t,std::map< MulticastGroup,std::map< Address,uint64_t > > >::iterator mgm(_multicastGroupMembers.find(nwid));
-		if (mgm != _multicastGroupMembers.end()) {
-			std::map< MulticastGroup,std::map< Address,uint64_t > >::iterator g(mgm->second.find(mg));
-			if (g != mgm->second.end()) {
-				uint64_t now = Utils::now();
-				for(std::map< Address,uint64_t >::iterator m(g->second.begin());m!=g->second.end();) {
-					if ((now - m->second) < ZT_MULTICAST_LIKE_EXPIRE) {
-						std::map< Address,SharedPtr<Peer> >::const_iterator p(_activePeers.find(m->first));
-						if (p != _activePeers.end()) {
-							possiblePeers[numPossiblePeers++] = p->second;
-							if (numPossiblePeers > _MAX_PEERS_TO_CONSIDER)
-								break;
-						}
-						++m;
-					} else g->second.erase(m++);
-				}
-				if (!g->second.size())
-					mgm->second.erase(g);
-			}
-		}
-	}
-
-	// Sort non-supernode peers in descending order of most recent data
-	// exchange timestamp. This sorts by implicit social relationships -- who
-	// you are talking to are the people who get multicasts first.
-	std::sort(&(possiblePeers[0]),&(possiblePeers[numPossiblePeers]),_PickMulticastPropagationPeersPeerPrioritySortOrder());
-
-	// Tack on a supernode peer to the end if we don't have enough regular
-	// peers, using supernodes to bridge gaps in sparse multicast groups.
-	if (numPossiblePeers < count) {
-		SharedPtr<Peer> bestSupernode;
-		unsigned int bestSupernodeLatency = 0xffff;
-		for(std::vector< SharedPtr<Peer> >::const_iterator sn(_supernodePeers.begin());sn!=_supernodePeers.end();++sn) {
-			if (((*sn)->latency())&&((*sn)->latency() < bestSupernodeLatency)) {
-				bestSupernodeLatency = (*sn)->latency();
-				bestSupernode = *sn;
-			}
-		}
-		if (bestSupernode)
-			possiblePeers[numPossiblePeers++] = bestSupernode;
-	}
-
-	unsigned int num = 0;
-
-	// First, try to pick peers not in the propgation bloom filter
-	for(unsigned int i=0;i<numPossiblePeers;++i) {
-		if (!Utils::bloomContains(propagationBloom,propagationBloomSize,possiblePeers[i]->address().sum())) {
-			peers[num++] = possiblePeers[i];
-			if (num >= count)
-				return num;
-		}
-	}
-
-	// Next, pick other peers until full (without duplicates)
-	for(unsigned int i=0;i<numPossiblePeers;++i) {
-		for(unsigned int j=0;j<num;++j) {
-			if (peers[j] == possiblePeers[i])
-				goto check_next_peer;
-		}
-		peers[num++] = possiblePeers[i];
-		if (num >= count)
-			return num;
-check_next_peer:
-		continue;
-	}
-
-	return num;
-}
-
 void Topology::main()
 	throw()
 {
@@ -404,14 +314,6 @@ void Topology::main()
 						}
 					}
 				}
-				{
-					Mutex::Lock _l(_multicastGroupMembers_m);
-					for(std::map< uint64_t,std::map< MulticastGroup,std::map< Address,uint64_t > > >::iterator mgm(_multicastGroupMembers.begin());mgm!=_multicastGroupMembers.end();) {
-						if (_r->nc->hasNetwork(mgm->first))
-							++mgm;
-						else _multicastGroupMembers.erase(mgm++);
-					}
-				}
 				break;
 			case _PeerDeepVerifyJob::EXIT_THREAD:
 				TRACE("thread terminating...");

node/Topology.hpp

@@ -28,6 +28,8 @@
 #ifndef _ZT_TOPOLOGY_HPP
 #define _ZT_TOPOLOGY_HPP
 
+#include <stdio.h>
+#include <string.h>
 #include <map>
 #include <set>
 #include <list>
@@ -132,7 +134,7 @@ public:
 	 */
 	inline SharedPtr<Peer> getBestSupernode() const
 	{
-		return getBestSupernode((const Address *)0,0);
+		return getBestSupernode((const Address *)0,0,false);
 	}
 
 	/**
@@ -144,9 +146,10 @@ public:
 	 * 
 	 * @param avoid Nodes to avoid
 	 * @param avoidCount Number of nodes to avoid
+	 * @param strictAvoid If false, consider avoided supernodes anyway if no non-avoid supernodes are available
 	 * @return Supernode or NULL if none
 	 */
-	SharedPtr<Peer> getBestSupernode(const Address *avoid,unsigned int avoidCount) const;
+	SharedPtr<Peer> getBestSupernode(const Address *avoid,unsigned int avoidCount,bool strictAvoid) const;
 
 	/**
 	 * @param zta ZeroTier address
@@ -164,30 +167,6 @@ public:
 	 */
 	void clean();
 
-	/**
-	 * Pick peers for multicast propagation
-	 *
-	 * @param nwid Network ID
-	 * @param exclude Peer to exclude or zero address for none
-	 * @param propagationBloom Propgation bloom filter
-	 * @param propagationBloomSize Size of propagation bloom filter in BITS
-	 * @param count Number of peers desired (propagation breadth)
-	 * @param mg Multicast group
-	 * @param peers Array to receive peers (must be at least [count])
-	 * @return Number of peers actually picked
-	 */
-	unsigned int pickMulticastPropagationPeers(uint64_t nwid,const Address &exclude,const void *propagationBloom,unsigned int propagationBloomSize,unsigned int count,const MulticastGroup &mg,SharedPtr<Peer> *peers);
-
-	/**
-	 * Add or update last 'like' time for an address's membership in a multicast group
-	 *
-	 * @param nwid Network ID
-	 * @param mg Multicast group
-	 * @param addr ZeroTier address
-	 * @param now Current time
-	 */
-	void likesMulticastGroup(uint64_t nwid,const MulticastGroup &mg,const Address &addr,uint64_t now);
-
 	/**
 	 * Apply a function or function object to all peers
 	 *
@@ -292,6 +271,35 @@ public:
 		std::vector< SharedPtr<Peer> > &_v;
 	};
 
+	/**
+	 * Dump peer I/O statistics to an open FILE (for status reporting and debug)
+	 */
+	class DumpPeerStatistics
+	{
+	public:
+		DumpPeerStatistics(FILE *out) :
+			_out(out),
+			_now(Utils::now())
+		{
+			fprintf(_out,"Peer       Direct IPv4           Direct IPv6                                         Latency(ms)"ZT_EOL_S);
+		}
+
+		inline void operator()(Topology &t,const SharedPtr<Peer> &p)
+		{
+			InetAddress v4(p->ipv4ActivePath(_now));
+			InetAddress v6(p->ipv6ActivePath(_now));
+			fprintf(_out,"%-10s %-21s %-51s %u"ZT_EOL_S,
+				p->address().toString().c_str(),
+				((v4) ? v4.toString().c_str() : "(none)"),
+				((v6) ? v6.toString().c_str() : "(none)"),
+				p->latency());
+		}
+
+	private:
+		FILE *_out;
+		uint64_t _now;
+	};
+
 protected:
 	virtual void main()
 		throw();
@@ -328,10 +336,6 @@ private:
 
 	KISSDB _dbm;
 	Mutex _dbm_m;
-
-	// Multicast group members by network ID, then multicast group
-	std::map< uint64_t,std::map< MulticastGroup,std::map< Address,uint64_t > > > _multicastGroupMembers;
-	Mutex _multicastGroupMembers_m;
 };
 
 } // namespace ZeroTier

node/Utils.cpp

@@ -283,31 +283,6 @@ std::string Utils::base64Decode(const char *data,unsigned int len)
 	return out.substr(0,outLen);
 }
 
-const char *Utils::etherTypeName(const unsigned int etherType)
-{
-	static char tmp[6];
-	switch(etherType) {
-		case ZT_ETHERTYPE_IPV4:
-			return "IPV4";
-		case ZT_ETHERTYPE_ARP:
-			return "ARP";
-		case ZT_ETHERTYPE_RARP:
-			return "RARP";
-		case ZT_ETHERTYPE_ATALK:
-			return "ATALK";
-		case ZT_ETHERTYPE_AARP:
-			return "AARP";
-		case ZT_ETHERTYPE_IPX_A:
-			return "IPX_A";
-		case ZT_ETHERTYPE_IPX_B:
-			return "IPX_B";
-		case ZT_ETHERTYPE_IPV6:
-			return "IPV6";
-	}
-	sprintf(tmp,"%.4x",etherType);
-	return tmp; // technically not thread safe, but we're only going to see this in debugging if ever
-}
-
 std::string Utils::hex(const void *data,unsigned int len)
 {
 	std::string r;

node/Utils.hpp

@@ -44,16 +44,6 @@
 
 #include "Constants.hpp"
 
-/* Ethernet frame types that might be relevant to us */
-#define ZT_ETHERTYPE_IPV4 0x0800
-#define ZT_ETHERTYPE_ARP 0x0806
-#define ZT_ETHERTYPE_RARP 0x8035
-#define ZT_ETHERTYPE_ATALK 0x809b
-#define ZT_ETHERTYPE_AARP 0x80f3
-#define ZT_ETHERTYPE_IPX_A 0x8137
-#define ZT_ETHERTYPE_IPX_B 0x8138
-#define ZT_ETHERTYPE_IPV6 0x86dd
-
 /**
  * Maximum compression/decompression block size (do not change)
  */
@@ -67,12 +57,6 @@ namespace ZeroTier {
 class Utils
 {
 public:
-	/**
-	 * @param etherType Ethernet type ID
-	 * @return Name of Ethernet protocol (e.g. ARP, IPV4)
-	 */
-	static const char *etherTypeName(const unsigned int etherType);
-
 	/**
 	 * @param data Data to convert to hex
 	 * @param len Length of data
@@ -102,18 +86,6 @@ public:
 	 */
 	static void getSecureRandom(void *buf,unsigned int bytes);
 
-	/**
-	 * @tparam T Integer type to fill and return
-	 * @return Random int using secure random source
-	 */
-	template<typename T>
-	static inline T randomInt()
-	{
-		T foo = 0; // prevents valgrind warnings
-		getSecureRandom(&foo,sizeof(foo));
-		return foo;
-	}
-
 	/**
 	 * Set modes on a file to something secure
 	 * 
@@ -476,38 +448,6 @@ public:
 		return ((*aptr & mask) == (*aptr & mask));
 	}
 
-	/**
-	 * Add a value to a bloom filter
-	 *
-	 * Note that bloom filter methods depend on n being evenly distributed, so
-	 * it's the job of the caller to implement any hashing.
-	 *
-	 * @param bits Bloom filter data (must be filterSize / 8 bytes in length)
-	 * @param filterSize Size of bloom filter in BITS
-	 * @param n Number to add
-	 */
-	static inline void bloomAdd(void *bits,unsigned int filterSize,unsigned int n)
-		throw()
-	{
-		n %= filterSize;
-		((unsigned char *)bits)[n / 8] |= (0x80 >> (n % 8));
-	}
-
-	/**
-	 * Test for a value in a bloom filter
-	 *
-	 * @param bits Bloom filter data (must be filterSize / 8 bytes in length)
-	 * @param filterSize Size of bloom filter in BITS
-	 * @param n Number to test
-	 * @return True if number might be in filter
-	 */
-	static inline bool bloomContains(const void *bits,unsigned int filterSize,unsigned int n)
-		throw()
-	{
-		n %= filterSize;
-		return ((((const unsigned char *)bits)[n / 8] & (0x80 >> (n % 8))));
-	}
-
 	/**
 	 * Compute CRC64
 	 *

objects.mk

@@ -11,6 +11,7 @@ OBJS=\
 	node/Demarc.o \
 	node/EllipticCurveKeyPair.o \
 	node/EthernetTap.o \
+	node/Filter.o \
 	node/HMAC.o \
 	node/Http.o \
 	node/Identity.o \
@@ -20,6 +21,7 @@ OBJS=\
 	node/Node.o \
 	node/NodeConfig.o \
 	node/Packet.o \
+	node/PacketDecoder.o \
 	node/Pack.o \
 	node/Peer.o \
 	node/Salsa20.o \

selftest.cpp

@@ -291,6 +291,8 @@ static int testNet()
 	std::cout << "[net] GET http://www.google.com/" << std::endl;
 	new Http::Request(Http::HTTP_METHOD_GET,"http://www.google.com/",Http::EMPTY_HEADERS,std::string(),&testHttpHandler,(void *)0);
 	testHttpDoneCondition.wait();
+
+	return 0;
 }
 
 int main(int argc,char **argv)

version.h

@@ -36,7 +36,7 @@
 /**
  * Minor version: 8-bit (0-255)
  */
-#define ZEROTIER_ONE_VERSION_MINOR 2
+#define ZEROTIER_ONE_VERSION_MINOR 3
 
 /**
  * Revision: 16-bit (0-65535)