The largest new feature in 1.2.0, and the product of many months of work, is our advanced network rules engine. With this release we achieve traffic control, security monitoring, and micro-segmentation capability on par with many enterprise SDN solutions designed for use in advanced data centers and corporate networks.
Rules allow you to filter packets on your network and vector traffic to security observers. Security observation can be performed in-band using REDIRECT or out of band using TEE.
Tags and capabilites provide advanced methods for implementing fine grained permission structures and micro-segmentation schemes without bloating the size and complexity of your rules table.
See the [rules engine announcement blog post](https://www.zerotier.com/blog/?p=927) for an in-depth discussion of theory and implementation. The [manual](https://www.zerotier.com/manual.shtml) contains detailed information on rule, tag, and capability use, and the `rule-compiler/` subfolder of the ZeroTier source tree contains a JavaScript function to compile rules in our human-readable rule definition language into rules suitable for import into a network controller. (ZeroTier Central uses this same script to compile rules on [my.zerotier.com](https://my.zerotier.com/).)
It's now possible to create your own root servers and add them to the root server pool on your nodes. This is done by creating what's called a "moon," which is a signed enumeration of root servers and their stable points on the network. Refer to the [manual](https://www.zerotier.com/manual.shtml) for instructions.
Federated roots achieve a number of things:
* You can deploy your own infrastructure to reduce dependency on ours.
* You can deploy roots *inside your LAN* to ensure that network connectivity inside your facility still works if the Internet goes down. This is the first step toward making ZeroTier viable as an in-house SDN solution.
* Roots can be deployed inside national boundaries for countries with data residency laws or "great firewalls." (As of 1.2.0 there is still no way to force all traffic to use these roots, but that will be easy to do in a later version.)
* Last but not least this makes ZeroTier somewhat less centralized by eliminating any hard dependency on ZeroTier, Inc.'s infrastructure.
Our roots will of course remain and continue to provide zero-configuration instant-on deployment, a secure global authority for identities, and free traffic relaying for those who can't establish peer to peer connections.
An element of our design philosophy is "features are bugs." This isn't an absolute dogma but more of a guiding principle. We try as hard as we can to avoid adding features, especially "knobs" that must be tweaked by a user.
As of 1.2.0 we've decided that certain knobs are unavoidable, and so there is now a `local.conf` file that can be used to configure them. See the ZeroTier One documentation for these. They include:
* Blacklisting interfaces you want to make sure ZeroTier doesn't use for network traffic, such as VPNs, slow links, or backplanes designated for only certain kinds of traffic.
* Turning uPnP/NAT-PMP on or off.
* Configuring software updates on Windows and Mac platforms.
* Defining trusted paths (the old trusted paths file is now deprecated)
* Setting the ZeroTier main port so it doesn't have to be changed on the command line, which is very inconvenient in many cases.
A good software update system for Windows and Mac clients has been a missing feature in previous versions. It does exist but we've been shy about using it so far due to its fragility in some environments.
We've greatly improved this mechanism in 1.2.0. Not only does it now do a better job of actually invoking the update, but it also transfers updates in-band using the ZeroTier protocol. This means it can work in environments that do not allows http/https traffic or that force it through proxies. There's also now an update channel setting: `beta` or `release` (the default).
Software updates are authenticated three ways:
1. ZeroTier's own signing key is used to sign all updates and this signature is checked prior to installation. ZeroTier, Inc.'s signatures are performed on an air-gapped machine.
2. Updates for Mac and Windows are signed using Apple and Microsoft (DigiCert EV) keys and will not install unless these signatures are also valid.
3. The new in-band update mechanism also authenticates the source of the update via ZeroTier's built-in security features. This provides transport security, while 1 and 2 provide security of the update at rest.
Updates are now configurable via `local.conf`. There are three options: `disable`, `download`, and `apply`. The third (apply) is the default for official builds on Windows and Mac, making updates happen silently and automatically as they do for popular browsers like Chrome and Firefox. Updates are disabled by default on Linux and other Unix-type systems as these are typically updated through package managers.
Version 1.2.0 is now aware of the link quality of direct paths with other 1.2.0 nodes. This information isn't used yet but is visible through the JSON API. (Quality always shows as 100% with pre-1.2.0 nodes.) Quality is measured passively with no additional overhead using a counter based packet loss detection algorithm.
This information is visible from the command line via `listpeers`:
200 listpeers XXXXXXXXXX 199.XXX.XXX.XXX/9993;10574;15250;1.00 48 1.2.0 LEAF
200 listpeers XXXXXXXXXX 195.XXX.XXX.XXX/45584;467;7608;0.44 290 1.2.0 LEAF
The first peer's path is at 100% (1.00), while the second peer's path is suffering quite a bit of packet loss (0.44).
Link quality awareness is a precursor to intelligent multi-path and QoS support, which will in future versions bring us to feature parity with SD-WAN products like Cisco iWAN.
Version 1.2.0 adds anti-DOS (denial of service) rate limits and other hardening for improved resiliency against a number of denial of service attack scenarios.
It also adds a mechanism for instantaneous credential revocation. This can be used to revoke certificates of membership instantly to kick a node off a network (for private networks) and also to revoke capabilities and tags. The new controller sends revocations by default when a peer is de-authorized.
Revocations propagate using a "rumor mill" peer to peer algorithm. This means that a controller need only successfully send a revocation to at least one member of a network with connections to other active members. At this point the revocation will flood through the network peer to peer very quickly. This helps make revocations more robust in the face of poor connectivity with the controller or attempts to incapacitate the controller with denial of service attacks, as well as making revocations faster on huge networks.
The Mac has a whole new UI built natively in Objective-C. It provides a pulldown similar in appearance and operation to the Mac WiFi task bar menu.
The Windows UI has also been improved and now provides a task bar icon that can be right-clicked to manage networks. Both now expose managed route and IP permissions, allowing nodes to easily opt in to full tunnel operation if you have a router configured on your network.
A special kind of public network called an ad-hoc network may be accessed by joining a network ID with the format:
ffSSSSEEEE000000
| | | |
| | | Reserved for future use, must be 0
| | End of port range (hex)
| Start of port range (hex)
Reserved ZeroTier address prefix indicating a controller-less network
Ad-hoc networks are public (no access control) networks that have no network controller. Instead their configuration and other credentials are generated locally. Ad-hoc networks permit only IPv6 UDP and TCP unicast traffic (no multicast or broadcast) using 6plane format NDP-emulated IPv6 addresses. In addition an ad-hoc network ID encodes an IP port range. UDP packets and TCP SYN (connection open) packets are only allowed to desintation ports within the encoded range.
For example `ff00160016000000` is an ad-hoc network allowing only SSH, while `ff0000ffff000000` is an ad-hoc network allowing any UDP or TCP port.
Keep in mind that these networks are public and anyone in the entire world can join them. Care must be taken to avoid exposing vulnerable services or sharing unwanted files or other resources.
The network controller has been largely rewritten to use a simple in-filesystem JSON data store in place of SQLite, and it is now included by default in all Windows, Mac, Linux, and BSD builds. This means any desktop or server node running ZeroTier One can now be a controller with no recompilation needed.
If you have data in an old SQLite3 controller we've included a NodeJS script in `controller/migrate-sqlite` to migrate data to the new format. If you don't migrate, members will start getting `NOT_FOUND` when they attempt to query for updates.
* **The Windows HyperV 100% CPU bug is FINALLY DEAD**: This long-running problem turns out to have been an issue with Windows itself, but one we were triggering by placing invalid data into the Windows registry. Microsoft is aware of the issue but we've also fixed the triggering problem on our side. ZeroTier should now co-exist quite well with HyperV and should now be able to be bridged with a HyperV virtual switch.
* **Segmenation faults on musl-libc based Linux systems**: Alpine Linux and some embedded Linux systems that use musl libc (a minimal libc) experienced segmentation faults. These were due to a smaller default stack size. A work-around that sets the stack size for new threads has been added.
* **Windows firewall blocks local JSON API**: On some Windows systems the firewall likes to block 127.0.0.1:9993 for mysterious reasons. This is now fixed in the installer via the addition of another firewall exemption rule.
* **UI crash on embedded Windows due to missing fonts**: The MSI installer now ships fonts and will install them if they are not present, so this should be fixed.
* **Improved dead path detection**: ZeroTier is now more aggressive about expiring paths that do not seem to be active. If a path seems marginal it is re-confirmed before re-use.
* **Minor performance improvements**: We've reduced unnecessary memcpy's and made a few other performance improvements in the core.
* **Linux static binaries**: For our official packages (the ones in the download.zerotier.com apt and yum repositories) we now build Linux binaries with static linking. Hopefully this will stop all the bug reports relating to library inconsistencies, as well as allowing our deb packages to run on a wider variety of Debian-based distributions. (There are far too many of these to support officially!) The overhead for this is very small, especially since we built our static versions against musl-libc. Distribution maintainers are of course free to build dynamically linked versions for inclusion into distributions; this only affects our official binaries.
Merge branch 'master' of github.com:zerotier/ZeroTierOne
The largest new feature in 1.2.0, and the product of many months of work, is our advanced network rules engine. With this release we achieve traffic control, security monitoring, and micro-segmentation capability on par with many enterprise SDN solutions designed for use in advanced data centers and corporate networks.
Rules allow you to filter packets on your network and vector traffic to security observers. Security observation can be performed in-band using REDIRECT or out of band using TEE.
Tags and capabilites provide advanced methods for implementing fine grained permission structures and micro-segmentation schemes without bloating the size and complexity of your rules table.
See the [rules engine announcement blog post](https://www.zerotier.com/blog/?p=927) for an in-depth discussion of theory and implementation. The [manual](https://www.zerotier.com/manual.shtml) contains detailed information on rule, tag, and capability use, and the `rule-compiler/` subfolder of the ZeroTier source tree contains a JavaScript function to compile rules in our human-readable rule definition language into rules suitable for import into a network controller. (ZeroTier Central uses this same script to compile rules on [my.zerotier.com](https://my.zerotier.com/).)
It's now possible to create your own root servers and add them to the root server pool on your nodes. This is done by creating what's called a "moon," which is a signed enumeration of root servers and their stable points on the network. Refer to the [manual](https://www.zerotier.com/manual.shtml) for instructions.
Federated roots achieve a number of things:
* You can deploy your own infrastructure to reduce dependency on ours.
* You can deploy roots *inside your LAN* to ensure that network connectivity inside your facility still works if the Internet goes down. This is the first step toward making ZeroTier viable as an in-house SDN solution.
* Roots can be deployed inside national boundaries for countries with data residency laws or "great firewalls." (As of 1.2.0 there is still no way to force all traffic to use these roots, but that will be easy to do in a later version.)
* Last but not least this makes ZeroTier somewhat less centralized by eliminating any hard dependency on ZeroTier, Inc.'s infrastructure.
Our roots will of course remain and continue to provide zero-configuration instant-on deployment, a secure global authority for identities, and free traffic relaying for those who can't establish peer to peer connections.
An element of our design philosophy is "features are bugs." This isn't an absolute dogma but more of a guiding principle. We try as hard as we can to avoid adding features, especially "knobs" that must be tweaked by a user.
As of 1.2.0 we've decided that certain knobs are unavoidable, and so there is now a `local.conf` file that can be used to configure them. See the ZeroTier One documentation for these. They include:
* Blacklisting interfaces you want to make sure ZeroTier doesn't use for network traffic, such as VPNs, slow links, or backplanes designated for only certain kinds of traffic.
* Turning uPnP/NAT-PMP on or off.
* Configuring software updates on Windows and Mac platforms.
* Defining trusted paths (the old trusted paths file is now deprecated)
* Setting the ZeroTier main port so it doesn't have to be changed on the command line, which is very inconvenient in many cases.
A good software update system for Windows and Mac clients has been a missing feature in previous versions. It does exist but we've been shy about using it so far due to its fragility in some environments.
We've greatly improved this mechanism in 1.2.0. Not only does it now do a better job of actually invoking the update, but it also transfers updates in-band using the ZeroTier protocol. This means it can work in environments that do not allows http/https traffic or that force it through proxies. There's also now an update channel setting: `beta` or `release` (the default).
Software updates are authenticated three ways:
1. ZeroTier's own signing key is used to sign all updates and this signature is checked prior to installation. ZeroTier, Inc.'s signatures are performed on an air-gapped machine.
2. Updates for Mac and Windows are signed using Apple and Microsoft (DigiCert EV) keys and will not install unless these signatures are also valid.
3. The new in-band update mechanism also authenticates the source of the update via ZeroTier's built-in security features. This provides transport security, while 1 and 2 provide security of the update at rest.
Updates are now configurable via `local.conf`. There are three options: `disable`, `download`, and `apply`. The third (apply) is the default for official builds on Windows and Mac, making updates happen silently and automatically as they do for popular browsers like Chrome and Firefox. Updates are disabled by default on Linux and other Unix-type systems as these are typically updated through package managers.
Version 1.2.0 is now aware of the link quality of direct paths with other 1.2.0 nodes. This information isn't used yet but is visible through the JSON API. (Quality always shows as 100% with pre-1.2.0 nodes.) Quality is measured passively with no additional overhead using a counter based packet loss detection algorithm.
This information is visible from the command line via `listpeers`:
200 listpeers XXXXXXXXXX 199.XXX.XXX.XXX/9993;10574;15250;1.00 48 1.2.0 LEAF
200 listpeers XXXXXXXXXX 195.XXX.XXX.XXX/45584;467;7608;0.44 290 1.2.0 LEAF
The first peer's path is at 100% (1.00), while the second peer's path is suffering quite a bit of packet loss (0.44).
Link quality awareness is a precursor to intelligent multi-path and QoS support, which will in future versions bring us to feature parity with SD-WAN products like Cisco iWAN.
Version 1.2.0 adds anti-DOS (denial of service) rate limits and other hardening for improved resiliency against a number of denial of service attack scenarios.
It also adds a mechanism for instantaneous credential revocation. This can be used to revoke certificates of membership instantly to kick a node off a network (for private networks) and also to revoke capabilities and tags. The new controller sends revocations by default when a peer is de-authorized.
Revocations propagate using a "rumor mill" peer to peer algorithm. This means that a controller need only successfully send a revocation to at least one member of a network with connections to other active members. At this point the revocation will flood through the network peer to peer very quickly. This helps make revocations more robust in the face of poor connectivity with the controller or attempts to incapacitate the controller with denial of service attacks, as well as making revocations faster on huge networks.
The Mac has a whole new UI built natively in Objective-C. It provides a pulldown similar in appearance and operation to the Mac WiFi task bar menu.
The Windows UI has also been improved and now provides a task bar icon that can be right-clicked to manage networks. Both now expose managed route and IP permissions, allowing nodes to easily opt in to full tunnel operation if you have a router configured on your network.
A special kind of public network called an ad-hoc network may be accessed by joining a network ID with the format:
ffSSSSEEEE000000
| | | |
| | | Reserved for future use, must be 0
| | End of port range (hex)
| Start of port range (hex)
Reserved ZeroTier address prefix indicating a controller-less network
Ad-hoc networks are public (no access control) networks that have no network controller. Instead their configuration and other credentials are generated locally. Ad-hoc networks permit only IPv6 UDP and TCP unicast traffic (no multicast or broadcast) using 6plane format NDP-emulated IPv6 addresses. In addition an ad-hoc network ID encodes an IP port range. UDP packets and TCP SYN (connection open) packets are only allowed to desintation ports within the encoded range.
For example `ff00160016000000` is an ad-hoc network allowing only SSH, while `ff0000ffff000000` is an ad-hoc network allowing any UDP or TCP port.
Keep in mind that these networks are public and anyone in the entire world can join them. Care must be taken to avoid exposing vulnerable services or sharing unwanted files or other resources.
The network controller has been largely rewritten to use a simple in-filesystem JSON data store in place of SQLite, and it is now included by default in all Windows, Mac, Linux, and BSD builds. This means any desktop or server node running ZeroTier One can now be a controller with no recompilation needed.
If you have data in an old SQLite3 controller we've included a NodeJS script in `controller/migrate-sqlite` to migrate data to the new format. If you don't migrate, members will start getting `NOT_FOUND` when they attempt to query for updates.
* **The Windows HyperV 100% CPU bug is FINALLY DEAD**: This long-running problem turns out to have been an issue with Windows itself, but one we were triggering by placing invalid data into the Windows registry. Microsoft is aware of the issue but we've also fixed the triggering problem on our side. ZeroTier should now co-exist quite well with HyperV and should now be able to be bridged with a HyperV virtual switch.
* **Segmenation faults on musl-libc based Linux systems**: Alpine Linux and some embedded Linux systems that use musl libc (a minimal libc) experienced segmentation faults. These were due to a smaller default stack size. A work-around that sets the stack size for new threads has been added.
* **Windows firewall blocks local JSON API**: On some Windows systems the firewall likes to block 127.0.0.1:9993 for mysterious reasons. This is now fixed in the installer via the addition of another firewall exemption rule.
* **UI crash on embedded Windows due to missing fonts**: The MSI installer now ships fonts and will install them if they are not present, so this should be fixed.
* **Improved dead path detection**: ZeroTier is now more aggressive about expiring paths that do not seem to be active. If a path seems marginal it is re-confirmed before re-use.
* **Minor performance improvements**: We've reduced unnecessary memcpy's and made a few other performance improvements in the core.
* **Linux static binaries**: For our official packages (the ones in the download.zerotier.com apt and yum repositories) we now build Linux binaries with static linking. Hopefully this will stop all the bug reports relating to library inconsistencies, as well as allowing our deb packages to run on a wider variety of Debian-based distributions. (There are far too many of these to support officially!) The overhead for this is very small, especially since we built our static versions against musl-libc. Distribution maintainers are of course free to build dynamically linked versions for inclusion into distributions; this only affects our official binaries.
Move setting _initialized = true until after WindowsEthernetTap::threadMain() has actually created and brought up the adapter. Also in OneService::nodeVirtualNetworkConfigFunction(), wait up to 5 seconds for WindowsEthernatTap::isInitialized() to return true before attempting to configure the interface and managed routes. Without this, the adapter doesnt actually exist yet when trying to add routes
Also, dont add the prefix length to the IP addresses in there.
Appears to fix Github Issue #357. Still waiting on some info from Microsoft for confirmation of a few things.
WindowsEthernetTap was storing the MTU for the ZeroTier Adapter as a
DWORD. When changing the value from the network adapter settings UI, it
was stored as a string. Updated the call when creating the adapter to
store it as a string as windows expects.
- ZT will only drop root privileges if zerotier-one user exists. It is created by
Debian postinst script - in other cases the user has to be created by administrator.
- Linux >=4.3 with ambient capabilities is required, otherwise ZT will silently
- "-U" option now also disables privileges dropping
Version 1.1.10 is a minor bug fix release. It fixes the following two issues:
* Routes are no longer added 'via' IP addresses of the current node's ZT interface.
* IPv6 default route is now fixed on Mac.
* IP addresses are now deleted before new ones are added to allow changes to netmask
on Mac and some other platforms.
This version also quietly introduces a feature called trusted paths. We'll be
documenting this in our community section soon. This is an experimental feature
for in-LAN SDN use.
1.1.8 is a very minor update. It contains a small fix to IP address management
on Linux but other than that there are no changes to the regular ZeroTier One
code.
The 1.1.8 controller contains several fixes and improvements:
* Network authorizedMemberCount works again
* Managed IP addresses are pushed as long as they fall within a LAN-local managed
route. The v4AssignMode and v6AssignMode settings only control automatic IP
assignment from pools. In previous versions v4AssignMode had to be on for IPs
to be pushed at all, making it impossible to just manually assign IPs.
* IPv6 now has an auto-assign mode ('zt' to match V4) that can work alongside the
special RFC4193 and 6PLANE modes (though it is not NDP emulated). If this is
enabled IPv6 addresses will be auto-assigned from IPv6 ranges in the assignment
pool list.
* Controller documentation is MUCH better! See controller/README.md for actual
useful information!
Finally 1.1.8 has several improvements to linux-build-farm including Debian wheezy
support.
(Retagged due to silly one-line bug and left over debug line... doh!)
Version 1.1.6 contains several significant improvements for use in complex network
environments along with some minor bug fixes and improvements to path stability and
dead path detection.
ROUTE MANAGEMENT AND FULL TUNNEL SUPPORT
1.1.6 is the first version of ZeroTier One to permit "full tunnel" (default route
override) operation on Linux, Mac, and Windows. This allows all Internet traffic
to be tunneled through ZeroTier while allowing ZeroTier peer-to-peer traffic to
continue to use the physical interface. 1.1.6 also brings route management support
and permissions settings for local networks to control whether networks are allowed
to modify the routing table or override default routing.
This is currently considered a beta/experimental feature and must be enabled via
the command line interface.
Route management and default route override requires support at the network controller.
When my.zerotier.com is updated and ready, we will post more information and testing
instructions at: https://www.zerotier.com/community
HIGHLY SCALABLE CONTAINER NETWORKING
1.1.6 also brings a new multicast-free (NDP emulated) IPv6 private addressing scheme
called "6plane." 6plane provides each host with a private IPv6 /80 and routes *all*
IPv6 traffic for this subnet to the host via transparent NDP emulation. This /80 can
then be assigned to Docker or other container/VM managers to assign a network-wide
IPv6 /128 to every container. Since NDP is emulated and multicast isn't needed, this
system can scale to millions of containers or more on a single backplane network with
a high degree of efficiency and reliability.
6plane also requires controller support. Look for it at my.zerotier.com once we have
upgraded our core infrastructure and web UIs.
(All hosts must be running 1.1.6 for 6plane to work properly. Other IPv6 addresses
or addressing modes are not affected and normal IPv6 NDP will continue to work
alongside 6plane in the same network.)
OTHER CHANGES
* Upgraded bundled miniupnpc, libnatpmp, and http-parser.
* New Debian and RPM packaging that is closer to compliance with distribution
guidelines, and a new Dockerized Linux package build system in linux-build-farm/
that can build every package on actual images of the correct distribution.
* Improvements to dead path detection.
* IPv6 now uses keepalive because a significant number of stateful IPv6 edge
routers have very short timeouts (30 seconds or less!).
* Significant performance improvements to network controllers under high load.
* Enable -fstack-protector-strong for better stack canary (security) support
in binaries. Note that this may require newer gcc/g++ or clang.
COMING SOON
The next version of ZeroTier One should have a new Mac UI. It's a system tray app
that looks and behaves a lot like the Mac WiFi pulldown menu. We'll also be adding
GUI support for default route and route management options and other new features.
Shortly after that we plan on adding full OpenFlow-like SDN rules engine support
to the ZeroTier core, making our planetary Ethernet switch a fully manageable smart
switch and enabling sophisticated security and flow rule management.
Version 1.1.6 contains several significant improvements for use in complex network
environments along with some minor bug fixes and improvements to path stability and
dead path detection.
ROUTE MANAGEMENT AND FULL TUNNEL SUPPORT
1.1.6 is the first version of ZeroTier One to permit "full tunnel" (default route
override) operation on Linux, Mac, and Windows. This allows all Internet traffic
to be tunneled through ZeroTier while allowing ZeroTier peer-to-peer traffic to
continue to use the physical interface. 1.1.6 also brings route management support
and permissions settings for local networks to control whether networks are allowed
to modify the routing table or override default routing.
This is currently considered a beta/experimental feature and must be enabled via
the command line interface.
Route management and default route override requires support at the network controller.
When my.zerotier.com is updated and ready, we will post more information and testing
instructions at: https://www.zerotier.com/community
HIGHLY SCALABLE CONTAINER NETWORKING
1.1.6 also brings a new multicast-free (NDP emulated) IPv6 private addressing scheme
called "6plane." 6plane provides each host with a private IPv6 /80 and routes *all*
IPv6 traffic for this subnet to the host via transparent NDP emulation. This /80 can
then be assigned to Docker or other container/VM managers to assign a network-wide
IPv6 /128 to every container. Since NDP is emulated and multicast isn't needed, this
system can scale to millions of containers or more on a single backplane network with
a high degree of efficiency and reliability.
6plane also requires controller support. Look for it at my.zerotier.com once we have
upgraded our core infrastructure and web UIs.
(All hosts must be running 1.1.6 for 6plane to work properly. Other IPv6 addresses
or addressing modes are not affected and normal IPv6 NDP will continue to work
alongside 6plane in the same network.)
OTHER CHANGES
* Upgraded bundled miniupnpc, libnatpmp, and http-parser.
* New Debian and RPM packaging that is closer to compliance with distribution
guidelines, and a new Dockerized Linux package build system in linux-build-farm/
that can build every package on actual images of the correct distribution.
* Improvements to dead path detection.
* IPv6 now uses keepalive because a significant number of stateful IPv6 edge
routers have very short timeouts (30 seconds or less!).
* Significant performance improvements to network controllers under high load.
* Enable -fstack-protector-strong for better stack canary (security) support
in binaries. Note that this may require newer gcc/g++ or clang.
COMING SOON
The next version of ZeroTier One should have a new Mac UI. It's a system tray app
that looks and behaves a lot like the Mac WiFi pulldown menu. We'll also be adding
GUI support for default route and route management options and other new features.
Shortly after that we plan on adding full OpenFlow-like SDN rules engine support
to the ZeroTier core, making our planetary Ethernet switch a fully manageable smart
switch and enabling sophisticated security and flow rule management.
(and other host friendly) IPv6 /80 magic subnetting to allow massive
multicast-free NDP emulated IPv6 networks where each host can have a
/48 worth of IPv6 IPs for internal containers, VMs, etc.
Alan Kay, thou art avenged.
https://ivanovivan.wordpress.com/2010/09/13/alan-kay-quotes/
implementing NAT, do implement stateful firewalling with absurdly short
timeouts (<60s). Keepalives are still required in IPv6.
Network engineers continue to mindlessly carry forward cruft and baggage
from IPv4 to IPv6.
Many NATs revert to symmetric behavior from friendlier modes if they cannot
preserve ports. This can occur if there is, for example, more than one ZT
device behind the NAT using port 9993.
Others (Airport Extreme?) seem to have bugs in which they completely freak
out if more than one device tries to do a lot of mappings using the same
internal local port.
Mostly to fix the latter case and somewhat to fix the former, we introduce
a secondary port. ZeroTier now binds 9993 (or whatever port you specify)
plus another port computed deterministically from your ZeroTier address.
This port is used for new links 1/4 of the time.
This mostly addresses the second problem above and partly helps to address
the first.
If uPnP/NAT-PMP is enabled we also still open a tertiary port because some
routers freak out if NAT-t is attempted using the same port as uPnP.
All of this is IPv4 only of course. IPv6 is sane.
ZeroTier One version 1.1.4 has several improvements to path stability and
path learning logic. It also contains the BETA version of Network Containers!
See netcon/README.md and https://www.zerotier.com/product-netcon.shtml for more
information.
Improvements in 1.1.4:
- Paths are now confirmed more aggressively on inactivity. This version can
usually detect a dead direct path in under 10 seconds vs 1-2 minutes in
previous versions.
- Logic to prevent recursion has been overhauled. The old versions had code
that explicitly checked packet contents (AntiRecursion.hpp). This is now
gone in favor of a function that checks IP addresses of ZeroTier devices
when paths are learned or probed. This is both more efficient and less
prone to edge case failures. Testing in several complex environments has
shown that this eliminates a few edge cases where inappropriate paths would
be used.
ZeroTier One version 1.1.2 is mostly a bug fix release, but also includes support
for NAT-PMP as a method of mapping external ports from behind supported routers.
Bugs fixed include:
- Refactored path selection code to eliminate a change that might have introduced
a reliability regression from earlier versions.
- Reworked UPnP code for improved reliability and also added NAT-PMP support.
- Fixed a crash on network preferred relays. (Not exploitable or security critical
since only network controllers can send that config option and there was no
buffer overflow or other exploitable issue.)
- Fix GUI problem on OSX where window could be larger than screen on some Macbooks.
- Fix a potential rate limit problem on PUSH_DIRECT_PATHS that could negatively
impact same-network connectivity.
- Update zerotier-cli to show preferred vs. active paths and hide inactive ones.
- Fix an issue on shutdown with thread delete order and thread synchronization.
ZeroTier 1.1.0 introduces a number of fixes and improvements in several areas.
We incremented the secondary version to indicate the significance of this release.
Version numbering has been a bit ad-hoc in the past. In future versions we will
adopt the following scheme: odd-numbered revision numbers like 1.1.1 will indicate
development versions, while even numbered ones like 1.1.2 will indicate tagged
releases. The public git repo branching has also been revised: master will always
be the latest tagged release, dev will be usually-working development, and edge
will host maybe-broken "bleeding edge" development. Pull requests on GitHub should
generally be made against dev, not master or edge. Other branches that may appear
from time to time may be feature or experimental branches. Only master is confirmed
good, with dev usually being okay but not guaranteed to be such. (To the extent
that any software is ever guaranteed to be anything.)
Change summary:
User-facing changes and improvements:
- Windows now has a new .NET-based native UI, which replaces the old WebControl
wrapper around the React UI. This just didn't work well on older Windows systems,
and we did not want to bundle 40+ megabytes of web browser with our app just for
its very simple UI.
- The web UI (still used for Mac and usable in Linux as well) is updated with
improved look and simplifications.
- Both UIs no longer have the "Peers" tab, since several users reported that non-
technical users found this confusing and even alarming (does this mean people
can access my system?). This information is visibile with "listpeers" from the
command line (zerotier-cli).
New features:
- Virtual networks that use our RFC4193-based IPv6 numbering scheme now emulate
IPv6 NDP for queries that target these addresses within the same network. This
allows for faster multicast-free connection init and improved security since
the address is now hard-wired to the device ID (which is a crypto token). This
does not affect IPv6 NDP for other IPv6 addresses or link-local, which will
continue to work normally. This also opens the potential for a reduced footprint
multicast-free build for embedded applications.
- This version includes beta support for a feature called CIRCUIT_TEST. Network
controllers for networks you have joined can now send a special message called
CIRCUIT_TEST which allows for ZeroTier-layer link testing and remote diagnosis
of link issues. Any operator of a network controller can do this; more
documentation will be forthcoming. The only information that may be gathered
in this way is IP addressing info and very basic system info (OS, 32/64 bit,
ZeroTier version). No personal information, hard drive data, location, or other
private info is available. This can only be ordered by a controller of a network
you have joined and is secured using cryptographic signatures.
- This version includes an alpha version of clustering a.k.a. multi-homing! This
powerful feature allows for a single ZeroTier device to be run from multiple
endpoints, with connecting peers being handed off to endpoints that are closer
via GeoIP lookup and/or are more lightly loaded. Currently this is only suitable
for use in our soon-to-be-upgraded root server infrastructure (details will be
blogged soon), but in the future it will be capable of hosting multi-homed
devices on user networks. This will allow things like (for example) a geo-
clustered Cassandra server that appears behind a single IP on a virtual LAN.
This feature must be enabled with the ZT_ENABLE_CLUSTER=1 build option.
Bug fixes and other improvements (including performance!):
- A faster version of the Poly1305 cryptographic MAC function was substituted
for sometimes greatly improved performance.
- C++ STL std::map was replaced throughout the entire core with a hand-rolled
Hashtable implementation for improved performance and in some cases a reduced
memory footprint. Some maps are still used in peripheral code that is not
performance critical or where ordered keys are needed.
- The zerotier-cli and zerotier-idtool symbolic links are now created in
/usr/local/bin on OSX to comply with El Capitan file security restrictions.
- The OSX tap device driver has been updated. This update may fix issues that
some users have reported with bridging on OSX. This new tap device driver
drops 32-bit support, but if you have a 32-bit system you can manually install
the old driver from ext/bin/tap-mac.
- Mac users could experience a problem with the UI if they installed ZeroTier,
then uninstalled it, then installed again. This is now fixed.
- UPnP port mappings should work better on some routers, and a different local
port is now used for UPnP mapped traffic vs. NAT-t'd traffic to get around
a bug in several popular mid-tier routers where using UPnP mapping alongside
traditional NAT traversal made a port unreachable.
- Debian package now builds with the right arch label on armv7l systems (Pi 2)
- The old "root topology" has been replaced with a similar but better thought
out concept called a World. The World defines the root servers and possibly
in the future other things, and can be updated in-band from trusted peers
allowing for software-upgrade-free network upgrades to keep up with growing
demand. See node/World.hpp for details.
- A fix was made to "self-awareness," which keeps track of your external IP
info and adapts to changes, to eliminate a problem that could cause "link
thrashing" behind some symmetric NATs.
- Escalating UDP TTLs was re-introduced to better transit some port-restricted
cone NATs such as Linux IP MASQ (used for Docker).
- An otherwise harmless crash-on-exit bug in the network controller was fixed.
- All new direct links are now confirmed in both directions. This adds a very
small amount of initial HELLO/OK traffic but fixes some edge cases where an
incomplete or unidirectional path might be used.
- [SECURITY] Better rate limiting was put in place for VERB_PUSH_DIRECT_PATHS
to prevent potential abuse for amplification attacks.
- [SECURITY] Build flags were tweaked on OSX to ensure that all code including
dependency libraries are built with full stack canary protection and ASLR
support.
Visit https://www.zerotier.com/blog or follow @ZeroTier on Twitter for updates
and announcements!
* No joining/leaving networks yet, but they do display.
* Nothing is updated yet after first load of the app. Need to set up a background task to run updates.
also update for changed function calls that now accept a local address
# Conflicts:
# include/ZeroTierOne.h
# java/CMakeLists.txt
# java/jni/Android.mk
# java/jni/ZT1_jnicache.cpp
# java/jni/ZT1_jnilookup.h
# java/jni/ZT1_jniutils.cpp
# java/jni/com_zerotierone_sdk_Node.cpp
Version 1.0.5 is a very minor release. It includes a new build of the Windows
device driver that supports Windows Vista and 2008 Server, and a fix to prevent
an issue that could occur when updating Linux installations from old pre-1.0.3
versions to 1.0.3 or 1.0.4.
It also includes a few very minor fixes and improvements to the controller code,
which doesn't affect most users.
This second commit just bumps version.h. :)
Version 1.0.5 is a very minor release. It includes a new build of the Windows
device driver that supports Windows Vista and 2008 Server, and a fix to prevent
an issue that could occur when updating Linux installations from old pre-1.0.3
versions to 1.0.3 or 1.0.4.
It also includes a few very minor fixes and improvements to the controller code,
which doesn't affect most users.
ZeroTier One version 1.0.4 brings several improvements to stability,
connectivity between hosts on the same LAN, and NAT traversal.
Direct connectivity improvements:
- ZeroTier One now opens port mappings using uPnP and/or NAT-PMP
if they are available on your network. These are then made
available to other (1.0.4 or newer) nodes. This should greatly
improve direct connectivity success rates for users on networks
that support port mapping. To build with this option, you must
include ZT_USE_MINIUPNPC=1 on the make path. Pre-build binaries
are included for many common architectures to make this easier.
- A new message has been introduced whereby nodes can "push" IP
address suggestions to other nodes. This is only done to nodes
with whom you have a trust relationship, which right now means
they are members of a network you've joined. The IP addresses
sent include local interface addresses and possibly uPnP mappings
if any are available. When nodes receive pushed IPs, they can
attempt connectivity at these addresses. This greatly improves
connectivity on local LANs, since the old broadcast mechanism
proved too unreliable under many real world scenarios.
- IPv6 addresses are also "pushed" via the aforementioned message,
allowing direct connectivity over IPv6 if both hosts have an
IPv6 address.
- Some of the aggressive port-scanning NAT-t behavior has been
removed, since this occasionally triggered intrusion alarms on
some networks and proved ineffective in the field. uPnP will be
a much bigger win, and is less "hacky."
- The rate of (tiny) UDP keepalive packet generation was slightly
increased. We were as surprised as you to learn that there are
many NAT routers in the wild with timeouts as short as 20 seconds
even though the RFC stipulates that they should be no shorter
than two minutes (120 seconds).
All of these connectivity improvements rely upon a new message
introduced with 1.0.4, so they'll only work between 1.0.4 nodes.
Older methods of connectivity establishment will continue to work
with earlier versions.
Platform-specific improvements:
- Many improvements have been made to Windows support and stability.
The NDIS6 driver is now used exclusively. If you have ports that
use NDIS5, these will automatically be re-created using the NDIS6
driver. You may see a "select this network's type" notification
after 1.0.4 upgrade for this reason.
- The dependency on the external "devcon.exe" binary on Windows has
been completely removed in favor of internal direct calls to the
Windows setup API to add and remove network ports. These are done
via dynamically loaded instances of the system setup DLLs to use
the most recent setup API code on your system for improved
compatibility.
- This version is tested with Windows 10 release, and was confirmed
to work on a clean install.
- The ARM32/Raspbian build is now back to using Debian Wheezy for
library backward compatibility (binary build only).
- The Mac icon is now a bit smaller to look better in the dock.
- The ui/ subfolder is now distributed with the Linux binary installer
and packages. This means Linux users can navigate to the UI at
http://127.0.0.1:9993/ and enter their authtoken.secret to use
the GUI locally. (This port could also be accessed via SSH port
forwarding or other mechanisms to administrate graphically from a
remote system.)
Other improvements:
- The new beta SQLite-backed controller microservice found in
controller/ and built with the ZT_ENABLE_NETWORK_CONTROLLER=1 make
option is now in a much more "working" state. Feel free to give
it a try! If you tried it before, delete controller.db before
starting the new version.
- A few tweaks were made to the path selection logic in the hope of
eliminating some flaky network behavior reported by users.
The next version of ZeroTier One will focus on performance and memory
footprint reduction, and may also include perfect forward security/secrecy
(a.k.a. PFS) once our design is finalized and reviewed.
(2) Go back to timestamp as certificate revision number. This is simpler
and more robust than using the network revision number for this and
forcing network revision fast-forward, which could cause some peers
to fall off the horizon when you don't want them to.
and go ahead and bump version to 1.0.4.
For a while in 1.0.3 -dev I was trying to optimize out repeated network controller
requests by using a ratcheting mechanism. If the client received a network config
that was indeed different from the one it had, it would respond by instantlly
requesting it again.
Not sure what I was thinking. It's fundamentally unsafe to respond to a message
with another message of the same type -- it risks a race condition. In this case
that's exactly what could happen.
It just isn't worth the added complexity to avoid a tiny, tiny amount of network
overhead, so I've taken this whole path out.
A few extra bytes every two minutes isn't worth fretting about, but as I recall
the reason for this optimization was to save CPU on the controller. This can be
achieved by just caching responses in memory *there* and serving those same
responses back out if they haven't changed.
I think I developed that 'ratcheting' stuff before I went full time on this. It's
hard to develop stuff like this without hours of sustained focus.
_networks is usually quite small, using binary search on a vector might be faster comapred to std::map.
This is especially true when using uClibc++, which uses a list.
Multiple statements in a sqlite3_prepare_v2 is not usable. Only
the first statement will be executed.
Since the schema now uses 'ON DELETE CASCADE', there's only
one statement needed.
If multiple statements are needed, there should be either multiple
sqlite3_prepare_v2 calls be used or the sqlite3_exec function.
The foreign keys have 'ON DELETE CASCADE' to simplify the removal
of networks etc. (controller code)
Some unique constraints are replaced with a multi column primary
key.
To update an existing database:
* install updated binaries
* stop service
* sqlite3 controller.db .dump | \
egrep '((^PRAGMA)|(^BEGIN)|(^INSERT)|(^COMMIT))' | \
grep -v 'schemaVersion' > data.sql
* mv controller.db controller.db.backup
* start service
* stop service
* sqlite3 controller.db < data.sql
* start service
This puts code accessing the data in a critical section so that the GC cannot run while JNI has access to the array. This helps with stability somewhat, but I'm still getting some crashes in the GC
After many months of intense development, ZeroTier is proud to release version
1.0.3 of ZeroTier One.
This version focuses on under-the-hood and network level improvements to
prepare the way for more user-facing improvements in the months to come, as
well as new products built around the ZeroTier core.
1.0.3 contains a large number of changes, so we're not going to push it out
via our auto-update mechanism yet. We're going to update the download links
on the web site and let users try it out for a while first. If problems are
found, we'll do a 1.0.4 before we push it out to existing installations.
--
Important note for Linux users: we've decided to stop pushing auto-updates for
Linux, and this version's official Linux binaries are built without update
checking enabled. Linux is used mostly on servers, and based on user feedback
we've found that most users don't like anything auto-updating outside of the
normal package management channels. Instead, we'll be working over the next
few months to get ZeroTier One included in a number of upstream Linux
distributions. That way you'll be able to 'apt-get' or 'yum' install it and
stay up to date that way. Auto-updates will remain for Macintosh and Windows
users until/unless we can move to 'app store' deployment on those platforms.
--
Now for the change list. It's a big one!
NEW FEATURES and IMPROVEMENTS
* Client support is in place for preferred relays on a per-network basis.
This allows you to define statically assigned nodes that act as relays for
indirect communication and connection setup for communication between peers
on a given virtual network. If defined, they'll be used in preference to
supernodes for this purpose. If they're offline, ZeroTier will fall back to
global supernodes. This will require support on the web control panel for
most users to use.
* This version sends NAT "keepalive" packets every 20 seconds, which is
similar to the behavior of most SIP phones. This should improve reliability
behind NATs with short timeouts and certain cheap consumer NAT devices.
* Improvements have been made to NAT traversal to traverse more symmetric
NAT configurations, and to rate limit traversal attempts to avoid looking
like a port scan.
* New direct paths are now confirmed prior to use. This should improve
reliability in cases where a NAT traversal attempt "half succeeds" by
preventing the use of direct links that aren't actually usable.
* A new geo-located TCP tunneling fallback implementation should improve
performance for heavily firewalled users who can't use UDP.
* ZeroTier now uses remotely reported accounts of your external IP address
to detect changes in your network connectivity instead of relying on
"fingerprinting" of the OS-level local network environment. This should
improve reliability in cases where external routers have dynamic IPs or
when changing between networks with different external IPs but similar
internal addressing schemes. This should also improve reliability for
use within virtual machines, since the external link address might change
but the VM's link addresses will not.
* We've eliminated the old Unix domain socket (or named pipe on Windows)
control bus in favor of a local HTTP JSON API. It runs on 127.0.0.1
port 9993 and can be accessed via standard HTTP. This improves
interoperability with scripts and other tools and allows us to use
HTML5 for the desktop UI component. See README.md in the service/
subfolder of the source tree for JSON API documentation.
* The old Qt GUI has been dropped in favor of a React-based HTML5 UI. The
code for this is found in ui/, and if ui/ is present in the ZeroTier
home folder the JSON API HTTP server will serve it on 127.0.0.1/9993.
The Windows and Mac UIs are now web control wrappers which access this
UI locally and automate the process of token lookup and login.
PACKAGING / INSTALLATION IMPROVEMENTS
* The Macintosh version is now packaged as a .pkg file instead of the old
.app that would download its components and bootstrap itself. Several
Mac users had problems with this, so we made it a package instead.
* The Windows installer now includes a cleaner driver installation module
that installs the driver as part of the main MSI file instead of spawning
a subprocess.
BUG FIXES
* Windows now comes with an updated NDIS6 Ethernet tap device driver.
If you still have the old NDIS5 driver installed you'll keep using it,
so if you want to switch to the NDIS6 driver uninstall your old version
and do a fresh install of the new one. NDIS5 is deprecated, so we are
hoping an NDIS6 driver will fix a number of difficult to reproduce
issues that some Windows users have reported. It will also likely
improve performance.
* Fixed a "pseudo" memory leak related to the old auto-update code. The
amount of "committed" (but not used) memory would increase over time.
Since this was not actual used memory it didn't cause real problems,
but the issue is gone now.
* A very rare threading deadlock was fixed.
* Fix for Linux installer that would falsely recognize systemd on some
Ubuntu systems.
DEEP UNDER THE HOOD
* This version has been heavily refactored at the source code level! The
node/ subfolder now contains the core ZeroTier network virtualization
engine without any OS-specific or transport-specific code. This is in
preparation for future plans around embedded devices, etc. There is now
a public C-level API in include/ZeroTierOne.h that defines an interface
to the node core.
* A new network controller implementation is in 1.0.3 based on SQLite and
the local JSON API control bus. IT HAS NOT BEEN HEAVILY TESTED YET, so we
do not recommend using it in production until the next version. You are
welcome to experiment with it.
Fixed VirtualNetworkConfigFunction (creating array to send to Java)
Fixed the creating InetAddress instead of InetSocketAddress in virtual network config callback
Just delete the local reference rather than do an array delete. Array delete was causing a crash on loading network config files. Deleting the local ref works fine, though, as it still lets the JVM garbage collecter know that the native code is done with the array
(1) Yes, you *can* create a mixed 32/64 bit Windows MSI installer that installs drivers. All you have to do is... umm... create individual sub-MSI files for each driver (one for 32, one for 64) and then package those in the main MSI files as "chained" MSI installers. Each of these must only be considered a prerequisite on 32 or 64 bit machines, respectively.
(2) Upgrade Advanced Installer version, add rules to uninstall NDIS6 tap device on uninstall.
(3) Fix IE issue in UI code.
Version 1.0.2 brings experimental FreeBSD support. It has ONLY been tested
on FreeBSD 10 on an x64 system, and should be considered alpha for this
platform for now.
This version is not going to be pushed out to the entire world via software
update, and the binary version distributed for other platforms via the
zerotier.com web site will remain 1.0.1 as there are no other meaningful
user-facing changes. This is just an interim release to let FreeBSD users
try it out. If you find bugs, please enter them on GitHub or do a pull
request and fix them yourself.
Version 1.0.2 brings experimental FreeBSD support. It has ONLY been tested
on FreeBSD 10 on an x64 system, and should be considered alpha for this
platform for now.
This version is not going to be pushed out to the entire world via software
update, and the binary version distributed for other platforms via the
zerotier.com web site will remain 1.0.1 as there are no other meaningful
user-facing changes. This is just an interim release to let FreeBSD users
try it out. If you find bugs, please enter them on GitHub or do a pull
request and fix them yourself.
This version is mostly a bug fix release. It fixes a bug that could cause
the service to crash on Windows while running the GUI application. It also
contains a number of fixes to the Linux installer and Linux support for
systemd-based init systems.
It also includes a minor tweak to the multicast algorithm. Version 1.0.0
sent multicasts in a deterministic order, while this version randomizes
the order. The vast majority of users will notice nothing, but this may result
in superior coverage for service announcements on very large networks. It's
a hard variation to test, so we're releasing like this to gather information
from users about the effect. Nothing will change on small networks, and
ordinary multicast functions like ARP and NDP should be unaffected.
The next version will likely focus on additional improvements to Microsoft
Windows support, since there are several known Windows issues in need of
attention. We're working on an NDIS6-based Tap driver that should address
the driver issues experienced by a small number of Windows 7 users.
This should not affect most users, but on large networks it should cause service
announcements to work a lot better. This is the result of a prolonged discussion
with a user about the visibility of game servers on a large network. The old
multicast algorithm was de-facto randomized due to its distributed nature, while
the new algorithm is more deterministic. This will restore some randomization
beyond limit-overflow conditions.
It won't affect small networks at all.
ZeroTier 1.0.0 brings a number of under the hood improvements and bug
fixes. These include but are not limited to:
* A simpler, faster multicast algorithm that places full burden
for multicast propagation on the sender. This results in better
fairness without CPU-intensive signature/verify on every packet,
lower latency for intra-data-center SDN operations, and avoids
distributed burdens that are intrinsically unfriendly to mobile
clients subject to wake frequency limits and battery life
constraints. In the future this may be augmented to allow
optimized delivery to multiple recipients at the same site via
federation.
* Significant code reorg to move toward mobile support (iOS, Android)
and future SDK packaging.
* A number of efficiency and stability improvements.
* Bug fixes for Linux and ARM users.
* Improved NAT traversal, including limited support for port
numbering scheme guessing to traverse some symmetric NATs.
Upgrade from 0.9.x is not required but is strongly recommended. Older
clients using the old multicast algorithm are presently supported via
legacy hooks, and these may go away at some point in the future. We'll
monitor the status of the network and try to keep legacy support around
as long as people need it, but it'll probably be pulled once 95%+ of
clients are 1.0.0 or newer.
The next versions will bring further bug fixes, improved user experience,
more enterprise-grade features for intra-data-center use, and more!
the current content of detached-network-test. This is ready for another
test in 'headless' mode.
Merge branch 'detached-network-test' into adamierymenko-dev
This version is being tagged and bagged, despite the fact that it's not
going to be released and won't be merged into master until 1.0.0 is ready.
It contains several Linux build fixes, a fix for a unix domain socket resource
leak, and build fixes for the Raspberry Pi.
The primary focus of this version is better integration with the
Microsoft Windows operating system.
Virtual networks should now be detected as "real" networks. For
each network, a message box should pop up the first time the network
is detected and classified allowing the user to choose its services
and security designation. On Windows 7 this is "work," "home," or
"public." On Windows 8 it's a simple choice of whether or not to
enable file and printer sharing and other services.
Several bugs have been fixed. Among these are a Windows threading
issue, several minor threading deadlock issues that could manifest
if rapidly adding and removing networks, and a command line interface
issue. The network list now shows the network MAC address as well,
a UI oversight in previous versions. A vectorized SSE implementation
of Salsa20 is now included for improved encrypt/decrypt performance.
The sending of low-TTL "firewall opener" packets has been disabled
in this version, since they may not be necessary and may harm NAT
traversal in some configurations. We will measure the effectiveness
of NAT traversal and see if this change improves performance in the
field.
Finally, this version obsoletes both the Tokyo and Sydney supernodes
in favor of a single larger supernode in Singapore. This decision was
made on the basis of bandwidth costs-- both Tokyo and Sydney are
significantly more expensive. We'd like to keep the basic service free,
so keeping bandwidth costs for relaying low is important. Since NAT
traversal works well and is constantly being improved, most users will
not see a speed decrease from this. Some Chinese users may see
improved performance since Singapore may be closer than Tokyo to many
Chinese cities.
The next major releases will focus on better Macintosh platform integration,
further improvements to NAT traversal, and UI improvements.
now show up as *real* networks and prompt the user to set their
location and firewall status (public, private, home/work, etc.).
The hack used to achieve #2 should not be examined by children or those
suffering from epilepsy or heart conditions.
This version fixes several bugs including an issue with networks that have
EtherType filtering disabled, a file permission issue that affected non-English
versions of Windows, a multicast propagation bug that caused multicasts to
be dropped more often than they should be, and an issue with IP auto-configuration.
It also introduces experimental support for bridging between physical and virtual
networks, a much-requested and powerful ability that's been planned from the start.
ZeroTier One can now replace the functionality of ordinary VPNs, link multiple
offices into a single LAN, and connect virtual machine backplanes in the cloud to
physical networks at home, among other things.
Bridging support isn't "officially" out yet, since the web UI part is still
in development. But when that is done, an official announcement will be
made on the blog and users can try it out. So far bridging has only
been tested under Linux with the Linux kernel's native bridging driver.
YMMV on other platforms. Try it out and let us know by filing bugs at GitHub
or e-mailing them to "contact@zerotier.com".
Version 0.9.0 adds a network-wide toggle for blanket broadcast (ff:ff:ff:ff:ff:ff), contains changes for compatibility with the new web site and netconf server code, and most importantly introduces unique non-conflicting MAC address schemes on a per-virtual-network basis.
The MAC address change is necessary to support bridging, which is the next major feature to be added. It's not absolutely required, but it makes sure that things work properly in the (probably very rare) case that two virtual networks happen to be directly or indirectly bridged together.
The MAC change means that 0.9.0 is a required update. Clients not updating will find themselves unable to communicate with older versions. The underlying protocol is the same, but MAC address resolution and routing will not work properly. Those running binary releases will be updated automatically, while those running from source must download and rebuild.
This version also fixes two minor security issues, including one involving file permissions on non-English Windows versions.
This version fixes a few more issues with TCP tunneling including GitHub issue #63.
It also adds automatic announcement and location of peers on physical LANs (GitHub
issue #56) which should greatly improve performance if you happen to be on the same
LAN or WiFi network as another peer. It can take 60 seconds or so for this to occur,
but it should.
This, quick on the heels of 0.8.0, fixes the fact that TCP tunneling was
broken. :)
There was a bug that only manifested in some cases, and not on my testnet.
I took the opportunity to clean up some of that logic generally. I need a
better testnet, but that will have to wait until we exit beta and hopefully
I can earn a little bit of money off this. A better testnet will require
a big beefy virtualization box or two to run hundreds to thousands of KVMs.
Also fixed a tiny cosmetic issue on Windows. Other than that no changes.
This version introduces a major new feature requested by several users,
both via the user survey and otherwise: TCP tunneling.
If you are not able to communicate over UDP/9993, ZeroTier One will switch to
TCP connections to ZeroTier's supernodes. This is always slower than UDP, but
will allow you to communicate behind all but the most extremely restrictive
firewalls. This TCP traffic travels over port 443 and looks like HTTPS (SSL)
traffic (though it isn't), since that port is almost always open.
This also fixes several minor bugs and attempts to improve the robustness of
Windows tap driver management. Several users have reported spurious issues
with the Windows tap device, though I was unable to reproduce any of these with
clean VMs. (Tried Windows 7 and 8.1, both x86 and x64. No luck.) But I tried
to beef up the tap code anyway in the hopes of catching it. It now tries a lot
harder to make sure the tap is up and running.
There was some significant under the hood refactoring in support of TCP, so
this was a non-trivial change.
I bumped the version to 0.8 to indicate that more and more features are being
crossed off the list as we approach 1.0 and exit from beta. After this, the next
major feature will be LAN announcement to find direct paths to peers on the
same physical LAN. But assuming that 0.8.0 goes smoothly, I am going to divert
attention to the web site. A new design is coming that is much cleaner, sharper,
and easier to use.
Thank you all for all your excellent feedback! We're well on the way to a killer
product that makes conventional VPNs and other kludges obsolete.
Version 0.7.2 marks the first release of ZeroTier One for Windows. Binaries will be
released to a few select testers at first, then to the whole world. Installation from
MSI and auto-update appear to be working. So far Windows 7 and 8 and Windows Server
2008 or newer are supported. Vista has issues so it's not supported at the moment,
and may not be since nobody seems to use it (according to Google Analytics).
This version contains fixes to Linux installation and deployment and adds a
new supernode in Tokyo, Japan. It also has a working Windows installer, though
a bit more testing is going to take place before Windows binaries are
actually released.
Version 0.7.0 commemorates public beta binaries now being in the wild for Mac
and Linux platforms, though this actually happened a few days ago with 0.6.14.
This version fixes two bugs. First, the Linux installer/updater now supports
both systemd and regular SysV init. It will detect which your distro uses at
install/update time and install the zerotier-one service accordingly.
Secondly, this fixes an issue that caused the service to always show ONLINE
in the GUI or 'zerotier-cli info' even if there was no net connection. The
online status should be more reliably reported now.
This version adds persistence of *nix device names (where possible), and fixes
a possible crash in Topology.cpp that was introduced in a previous revision.
It also adds a new supernode located in Singapore!
This version ties up some stuff that remains in the core before binary release.
It adds support for direct interface IP enumeration on *nix systems, as well
as a fix for IPv6 link-local addresses on OSX. This also contains some cleanup
in Peer and some improvements to help detect and route around dead or unreachable
supernodes.
Getting close!
This version fixes a minor NAT traversal issue. In the past, NAT-t links had a timeout
but otherwise were preserved. This version makes them more ephemeral and invalidates
them on sleep/wake or changes in network configuration or environment.
This is because many NAT setups are very fragile with regard to hole punches, so the
past stickiness of links caused dead links to persist too long and break connectivity
between peers.
This is about 75% of what needs to be done to greatly improve robustness. The other 25%
involves detecting failed links or failed relays.
This version is also almost done for the Windows platform, moving us even closer to
binary release.
Not a significant release for OSS users, but this version marks a significant
increase in workitude on the Windows platform. A properly and officially
signed x64 driver is also included. x86 drivers and more Windows work including
Qt UI are coming soon.
(1) Add a bunch of tedious type casts to eliminate unnecessary compiler warnings on Windows X64 builds.
(2) Some EthernetTap work to integrate Windows custom IOCTL for multicast group lookup (not done quite yet).
(3) Dump some more info in selftest to make sure our Windows path lookup functions are returning sane results.
Another release leading up to official binary releases... not much to the core,
but quite a bit of work on the UI, installation, and such.
This version will build and run on OSX 10.6 while previous versions would fail
due to a missing getifmaddrs() function.
This version contains no significant changes to the engine itself, just to
the installer, the GUI, and packaging and such. It's all stuff for prep for
the big release, which is fast approaching!
New versions will be coming fast and furious for a bit as bugs get fixed and
testing is done in prep to the first binary release.
This version fixes a problem with WAN path discovery and a possible security
issue in PacketDecoder. (see previous comments)
(1) The changes to path learning in the two previous releases were poorly thought out,
and this version should remedy that by introducing PROBE. This is basically a kind of
ECHO request and is used to authenticate endpoints that are not learned via a valid
request/response pair. Thus we will still passively learn endpoints, but securely.
(2) Turns out there was a security oversight in _doHELLO() that could have permitted...
well... I'm not sure it was exploitable to do anything particularly interesting since
a bad identity would be discarded anyway, but fix it just the same.
This version contains few changes that are visible to users building from source.
It contains an almost-complete version of the Qt-based GUI in ZeroTierUI, though
this is still a work in progress. It also contains the software update infrastructure,
which is not yet enabled by default but does basically work. Some cleanup and
dead code removal has also occurred.
The next release will probably be the first binary release with auto-update and a
full UI experience for Linux and Mac. Windows will follow later, as more work has
to be done on the Windows port.
ZeroTier includes the following third party code, either in ext/ or incorporated into the ZeroTier core.
* LZ4 compression algorithm by Yann Collet
* Files: node/Packet.cpp (bundled within anonymous namespace)
* Home page: http://code.google.com/p/lz4/
* License grant: BSD 2-clause
* http-parser by Joyent, Inc. (many authors)
* Files: ext/http-parser/*
* Home page: https://github.com/joyent/http-parser/
* License grant: MIT/Expat
* C++11 json (nlohmann/json) by Niels Lohmann
* Files: ext/json/*
* Home page: https://github.com/nlohmann/json
* License grant: MIT
* TunTapOSX by Mattias Nissler
* Files: ext/tap-mac/tuntap/*
* Home page: http://tuntaposx.sourceforge.net/
* License grant: BSD attribution no-endorsement
* ZeroTier Modifications: change interface name to zt#, increase max MTU, increase max devices
* tap-windows6 by the OpenVPN project
* Files: windows/TapDriver6/*
* Home page: https://github.com/OpenVPN/tap-windows6/
* License grant: GNU GPL v2
* ZeroTier Modifications: change name of driver to ZeroTier, add ioctl() to get L2 multicast memberships (source is in ext/ and modifications inherit GPL)
* Salsa20 stream cipher, Curve25519 elliptic curve cipher, Ed25519 digital signature algorithm, and Poly1305 MAC algorithm, all by Daniel J. Bernstein
The final .AIP file can only be edited on Windows with [Advanced Installer Enterprise](http://www.advancedinstaller.com/). In addition to incrementing the version be sure that a new product code is generated. (The "upgrade code" GUID on the other hand must never change.)
# Building for Supported Platforms
## Macintosh
Mac's easy. Just type:
make official
You will need [Packages](http://s.sudre.free.fr/Software/Packages/about.html) and our release signing key in the keychain.
## Linux
Mount the GPG key for *contact@zerotier.com* and then on an x86_64 box with a recent version of Docker and an Internet connection run:
make distclean
cd linux-build-farm
./build.sh
This will build i386 and x86_64 packages. Now ssh into our build Raspberry Pi and type `make debian` there to build the Raspbian armhf package. Copy it to `debian-jessie/` inside `linux-build-farm` so that it will be included in the repositories we generate. Now generate the YUM and APT repos:
rm -rf ~/.aptly*
rm -rf /tmp/zt-rpm-repo
./make-apt-repos.sh
./make-rpm-repos.sh
This will require the passphrase for *contact@zerotier.com*.
The contents of ~/.aptly/public must be published as `debian/` on `download.zerotier.com`. The contents of /tmp/zt-rpm-repo are published as `redhat/` on same.
## Windows
First load the Visual Studio solution and rebuild the UI and ZeroTier One in both x64 and i386 `Release` mode. Then load [Advanced Installer Enterprise](http://www.advancedinstaller.com/), check that the version is correct, and build. The build will fail if any build artifacts are missing, and Windows must have our product singing key (from DigiCert) available to sign the resulting MSI file. The MSI must then be tested on at least a few different CLEAN Windows VMs to ensure that the installer is valid and properly signed.
*After the MSI is published to download.zerotier.com in the proper RELEASE/#.#.#/dist subfolder for its version* the Chocolatey package must be rebuilt and published. Open a command prompt, change to `ext/installfiles/windows/chocolatey`, and type `choco pack`. Then use `choco push` to push it to Chocolatey (API key required).
Note that this does not cover rebuilding the drivers or their containing MSI projects, as this is typically not necessary and they are shipped in binary form in the repository for convenience.
## iOS, Android
... no docs here yet since this is done entirely out of band with regular installs.
ZeroTier One creates flat virtual Ethernet networks of almost unlimited size. [Visit ZeroTier Networks on the web](https://www.zerotier.com/) for more information.
ZeroTier is an enterprise Ethernet switch for planet Earth.
This project is now in beta, which means that major incompatible protocol shifts should now become considerably more rare. You should still update very often if you plan to use it. Binary packages with auto-update are coming very soon.
It erases the LAN/WAN distinction and makes VPNs, tunnels, proxies, and other kludges arising from the inflexible nature of physical networks obsolete. Everything is encrypted end-to-end and traffic takes the most direct (peer to peer) path available.
Follow the [ZeroTier blog](http://blog.zerotier.com/) and the [GitHub project](https://github.com/zerotier/ZeroTierOne) to stay up to date.
Visit [ZeroTier's site](https://www.zerotier.com/) for more information and [pre-built binary packages](https://www.zerotier.com/download.shtml). Apps for Android and iOS are available for free in the Google Play and Apple app stores.
See BUILDING.txt and RUNNING.txt for instructions. It currently runs on Mac and Linux. A Windows port is well along the way to completion. If you're brave you can load the Visual Studio 2012 solution and play around.
### Getting Started
Once you have it running you can join the Earth network -- a LAN for the planet -- by using the 'zerotier-cli' tool: 'sudo zerotier-cli join 8056c2e21c000001'. Right now Earth is the only network. Ability to create private networks, as well as some additional special-purpose public ones, will be coming soon. (Note: previous alpha versions auto-joined Earth, but since 0.5.0 it no longer does this.)
ZeroTier's basic operation is easy to understand. Devices have 10-digit *ZeroTier addresses* like `89e92ceee5` and networks have 16-digit network IDs like `8056c2e21c000001`. All it takes for a device to join a network is its 16-digit ID, and all it takes for a network to authorize a device is its 10-digit address. Everything else is automatic.
Note that this won't work if your firewall does not allow outbound UDP. It must allow two way UDP conversations on port 9993 at a minimum.
A "device" in our terminology is any "unit of compute" capable of talking to a network: desktops, laptops, phones, servers, VMs/VPSes, containers, and even user-space applications via our [SDK](https://github.com/zerotier/ZeroTierSDK).
ZeroTier One is licensed under the GNU General Public License version 3. You are free to use, modify, or redistribute it under the terms of that license. If you would like to embed ZeroTier One in a closed source product or create a closed source derivative product, contact ZeroTier Networks LLC.
For testing purposes we provide a public virtual network called *Earth* with network ID `8056c2e21c000001`. You can join it with:
Now wait about 30 seconds and check your system with `ip addr list` or `ifconfig`. You'll see a new interface whose name starts with *zt* and it should quickly get an IPv4 and an IPv6 address. Once you see it get an IP, try pinging `earth.zerotier.net` at `29.209.112.93`. If you've joined Earth from more than one system, try pinging your other machine. If you don't want to belong to a giant Ethernet party line anymore, just type:
sudo zerotier-cli leave 8056c2e21c000001
The *zt* interface will disappear. You're no longer on the network.
To create networks of your own, you'll need a network controller. ZeroTier One (for desktops and servers) includes controller functionality in its default build that can be configured via its JSON API (see [README.md in controller/](controller/)). ZeroTier provides a hosted solution with a nice web UI and SaaS add-ons at [my.zerotier.com](https://my.zerotier.com/). Basic controller functionality is free for up to 100 devices.
### Project Layout
-`artwork/`: icons, logos, etc.
-`attic/`: old stuff and experimental code that we want to keep around for reference.
-`controller/`: the reference network controller implementation, which is built and included by default on desktop and server build targets.
-`debian/`: files for building Debian packages on Linux.
-`doc/`: manual pages and other documentation.
-`ext/`: third party libraries, binaries that we ship for convenience on some platforms (Mac and Windows), and installation support files.
-`include/`: include files for the ZeroTier core.
-`java/`: a JNI wrapper used with our Android mobile app. (The whole Android app is not open source but may be made so in the future.)
-`macui/`: a Macintosh menu-bar app for controlling ZeroTier One, written in Objective C.
-`node/`: the ZeroTier virtual Ethernet switch core, which is designed to be entirely separate from the rest of the code and able to be built as a stand-alone OS-independent library. Note to developers: do not use C++11 features in here, since we want this to build on old embedded platforms that lack C++11 support. C++11 can be used elsewhere.
-`osdep/`: code to support and integrate with OSes, including platform-specific stuff only built for certain targets.
-`service/`: the ZeroTier One service, which wraps the ZeroTier core and provides VPN-like connectivity to virtual networks for desktops, laptops, servers, VMs, and containers.
-`tcp-proxy/`: TCP proxy code run by ZeroTier, Inc. to provide TCP fallback (this will die soon!).
-`windows/`: Visual Studio solution files, Windows service code for ZeroTier One, and the Windows task bar app UI.
The base path contains the ZeroTier One service main entry point (`one.cpp`), self test code, makefiles, etc.
### Build and Platform Notes
To build on Mac and Linux just type `make`. On FreeBSD and OpenBSD `gmake` (GNU make) is required and can be installed from packages or ports. For Windows there is a Visual Studio solution in `windows/'.
- **Mac**
- Xcode command line tools for OSX 10.7 or newer are required.
- Tap device driver kext source is in `ext/tap-mac` and a signed pre-built binary can be found in `ext/bin/tap-mac`. You should not need to build it yourself. It's a fork of [tuntaposx](http://tuntaposx.sourceforge.net) with device names changed to `zt#`, support for a larger MTU, and tun functionality removed.
- **Linux**
- The minimum compiler versions required are GCC/G++ 4.9.3 or CLANG/CLANG++ 3.4.2.
- Linux makefiles automatically detect and prefer clang/clang++ if present as it produces smaller and slightly faster binaries in most cases. You can override by supplying CC and CXX variables on the make command line.
- CentOS 7 ships with a version of GCC/G++ that is too old, but a new enough version of CLANG can be found in the *epel* repositories. Type `yum install epel-release` and then `yum install clang` to build there.
- **Windows**
- Windows 7 or newer (and equivalent server versions) are supported. This *may* work on Vista but you're on your own there. Windows XP is not supported since it lacks many important network API functions.
- We build with Visual Studio 2015. Older versions may not work with the solution file and project files we ship and may not have new enough C++11 support.
- Pre-built signed Windows drivers are included in `ext/bin/tap-windows-ndis6`. The MSI files found there will install them on 32-bit and 64-bit systems. (These are included in our multi-architecture installer as chained MSIs.)
- Windows builds are more painful in general than other platforms and are for the adventurous.
- **FreeBSD**
- Tested most recently on FreeBSD-11. Older versions may work but we're not sure.
- GCC/G++ 4.9 and gmake are required. These can be installed from packages or ports. Type `gmake` to build.
- **OpenBSD**
- There is a limit of four network memberships on OpenBSD as there are only four tap devices (`/dev/tap0` through `/dev/tap3`). We're not sure if this can be increased.
- OpenBSD lacks `getifmaddrs` (or any equivalent method) to get interface multicast memberships. As a result multicast will only work on OpenBSD for ARP and NDP (IP/MAC lookup) and not for other purposes.
- Only tested on OpenBSD 6.0. Older versions may not work.
- GCC/G++ 4.9 and gmake are required and can be installed using `pkg_add` or from ports. They get installed in `/usr/local/bin` as `egcc` and `eg++` and our makefile is pre-configured to use them on OpenBSD.
Typing `make selftest` will build a *zerotier-selftest* binary which unit tests various internals and reports on a few aspects of the build environment. It's a good idea to try this on novel platforms or architectures.
### Running
Running *zerotier-one* with -h will show help.
On Linux and BSD you can start the service with:
sudo ./zerotier-one -d
A home folder for your system will automatically be created.
The service is controlled via the JSON API, which by default is available at 127.0.0.1 port 9993. We include a *zerotier-cli* command line utility to make API calls for standard things like joining and leaving networks. The *authtoken.secret* file in the home folder contains the secret token for accessing this API. See README.md in [service/](service/) for API documentation.
Here's where home folders live (by default) on each OS:
* **Windows**: `\ProgramData\ZeroTier\One` (That's for Windows 7. The base 'shared app data' folder might be different on different Windows versions.)
Running ZeroTier One on a Mac is the same, but OSX requires a kernel extension. We ship a signed binary build of the ZeroTier tap device driver, which can be installed on Mac with:
sudo make install-mac-tap
This will create the home folder for Mac, place *tap.kext* there, and set its modes correctly to enable ZeroTier One to manage it with *kextload* and *kextunload*.
### Troubleshooting
For most users, it just works.
If you are running a local system firewall, we recommend adding a rule permitting UDP port 9993 inbound and outbound. If you installed binaries for Windows this should be done automatically. Other platforms might require manual editing of local firewall rules depending on your configuration.
The Mac firewall can be found under "Security" in System Preferences. Linux has a variety of firewall configuration systems and tools. If you're using Ubuntu's *ufw*, you can do this:
sudo ufw allow 9993/udp
On CentOS check `/etc/sysconfig/iptables` for IPTables rules. For other distributions consult your distribution's documentation. You'll also have to check the UIs or documentation for commercial third party firewall applications like Little Snitch (Mac), McAfee Firewall Enterprise (Windows), etc. if you are running any of those. Some corporate environments might have centrally managed firewall software, so you might also have to contact IT.
ZeroTier One peers will automatically locate each other and communicate directly over a local wired LAN *if UDP port 9993 inbound is open*. If that port is filtered, they won't be able to see each others' LAN announcement packets. If you're experiencing poor performance between devices on the same physical network, check their firewall settings. Without LAN auto-location peers must attempt "loopback" NAT traversal, which sometimes fails and in any case requires that every packet traverse your external router twice.
Users behind certain types of firewalls and "symmetric" NAT devices may not able able to connect to external peers directly at all. ZeroTier has limited support for port prediction and will *attempt* to traverse symmetric NATs, but this doesn't always work. If P2P connectivity fails you'll be bouncing UDP packets off our relay servers resulting in slower performance. Some NAT router(s) have a configurable NAT mode, and setting this to "full cone" will eliminate this problem. If you do this you may also see a magical improvement for things like VoIP phones, Skype, BitTorrent, WebRTC, certain games, etc., since all of these use NAT traversal techniques similar to ours.
If you're interested, there's a [technical deep dive about NAT traversal on our blog](https://www.zerotier.com/blog/?p=226). A troubleshooting tool to help you diagnose NAT issues is planned for the future as are uPnP/IGD/NAT-PMP and IPv6 transport.
If a firewall between you and the Internet blocks ZeroTier's UDP traffic, you will fall back to last-resort TCP tunneling to rootservers over port 443 (https impersonation). This will work almost anywhere but is *very slow* compared to UDP or direct peer to peer connectivity.
### Contributing
Please make pull requests against the `dev` branch. The `master` branch is release, and `edge` is for unstable and work in progress changes and is not likely to work.
### License
The ZeroTier source code is open source and is licensed under the GNU GPL v3 (not LGPL). If you'd like to embed it in a closed-source commercial product or appliance, please e-mail [contact@zerotier.com](mailto:contact@zerotier.com) to discuss commercial licensing. Otherwise it can be used for free.
Version 1.2.0 is a major milestone release representing almost nine months of work. It includes our rules engine for distributed network packet filtering and security monitoring, federated roots, and many other architectural and UI improvements and bug fixes.
## New Features in 1.2.0
### The ZeroTier Rules Engine
The largest new feature in 1.2.0, and the product of many months of work, is our advanced network rules engine. With this release we achieve traffic control, security monitoring, and micro-segmentation capability on par with many enterprise SDN solutions designed for use in advanced data centers and corporate networks.
Rules allow you to filter packets on your network and vector traffic to security observers. Security observation can be performed in-band using REDIRECT or out of band using TEE.
Tags and capabilites provide advanced methods for implementing fine grained permission structures and micro-segmentation schemes without bloating the size and complexity of your rules table.
See the [rules engine announcement blog post](https://www.zerotier.com/blog/?p=927) for an in-depth discussion of theory and implementation. The [manual](https://www.zerotier.com/manual.shtml) contains detailed information on rule, tag, and capability use, and the `rule-compiler/` subfolder of the ZeroTier source tree contains a JavaScript function to compile rules in our human-readable rule definition language into rules suitable for import into a network controller. (ZeroTier Central uses this same script to compile rules on [my.zerotier.com](https://my.zerotier.com/).)
### Root Server Federation
It's now possible to create your own root servers and add them to the root server pool on your nodes. This is done by creating what's called a "moon," which is a signed enumeration of root servers and their stable points on the network. Refer to the [manual](https://www.zerotier.com/manual.shtml) for instructions.
Federated roots achieve a number of things:
* You can deploy your own infrastructure to reduce dependency on ours.
* You can deploy roots *inside your LAN* to ensure that network connectivity inside your facility still works if the Internet goes down. This is the first step toward making ZeroTier viable as an in-house SDN solution.
* Roots can be deployed inside national boundaries for countries with data residency laws or "great firewalls." (As of 1.2.0 there is still no way to force all traffic to use these roots, but that will be easy to do in a later version.)
* Last but not least this makes ZeroTier somewhat less centralized by eliminating any hard dependency on ZeroTier, Inc.'s infrastructure.
Our roots will of course remain and continue to provide zero-configuration instant-on deployment, a secure global authority for identities, and free traffic relaying for those who can't establish peer to peer connections.
### Local Configuration
An element of our design philosophy is "features are bugs." This isn't an absolute dogma but more of a guiding principle. We try as hard as we can to avoid adding features, especially "knobs" that must be tweaked by a user.
As of 1.2.0 we've decided that certain knobs are unavoidable, and so there is now a `local.conf` file that can be used to configure them. See the ZeroTier One documentation for these. They include:
* Blacklisting interfaces you want to make sure ZeroTier doesn't use for network traffic, such as VPNs, slow links, or backplanes designated for only certain kinds of traffic.
* Turning uPnP/NAT-PMP on or off.
* Configuring software updates on Windows and Mac platforms.
* Defining trusted paths (the old trusted paths file is now deprecated)
* Setting the ZeroTier main port so it doesn't have to be changed on the command line, which is very inconvenient in many cases.
### Improved In-Band Software Updates
A good software update system for Windows and Mac clients has been a missing feature in previous versions. It does exist but we've been shy about using it so far due to its fragility in some environments.
We've greatly improved this mechanism in 1.2.0. Not only does it now do a better job of actually invoking the update, but it also transfers updates in-band using the ZeroTier protocol. This means it can work in environments that do not allows http/https traffic or that force it through proxies. There's also now an update channel setting: `beta` or `release` (the default).
Software updates are authenticated three ways:
1. ZeroTier's own signing key is used to sign all updates and this signature is checked prior to installation. ZeroTier, Inc.'s signatures are performed on an air-gapped machine.
2. Updates for Mac and Windows are signed using Apple and Microsoft (DigiCert EV) keys and will not install unless these signatures are also valid.
3. The new in-band update mechanism also authenticates the source of the update via ZeroTier's built-in security features. This provides transport security, while 1 and 2 provide security of the update at rest.
Updates are now configurable via `local.conf`. There are three options: `disable`, `download`, and `apply`. The third (apply) is the default for official builds on Windows and Mac, making updates happen silently and automatically as they do for popular browsers like Chrome and Firefox. Updates are disabled by default on Linux and other Unix-type systems as these are typically updated through package managers.
### Path Link Quality Awareness
Version 1.2.0 is now aware of the link quality of direct paths with other 1.2.0 nodes. This information isn't used yet but is visible through the JSON API. (Quality always shows as 100% with pre-1.2.0 nodes.) Quality is measured passively with no additional overhead using a counter based packet loss detection algorithm.
This information is visible from the command line via `listpeers`:
The first peer's path is at 100% (1.00), while the second peer's path is suffering quite a bit of packet loss (0.44).
Link quality awareness is a precursor to intelligent multi-path and QoS support, which will in future versions bring us to feature parity with SD-WAN products like Cisco iWAN.
### Security Improvements
Version 1.2.0 adds anti-DOS (denial of service) rate limits and other hardening for improved resiliency against a number of denial of service attack scenarios.
It also adds a mechanism for instantaneous credential revocation. This can be used to revoke certificates of membership instantly to kick a node off a network (for private networks) and also to revoke capabilities and tags. The new controller sends revocations by default when a peer is de-authorized.
Revocations propagate using a "rumor mill" peer to peer algorithm. This means that a controller need only successfully send a revocation to at least one member of a network with connections to other active members. At this point the revocation will flood through the network peer to peer very quickly. This helps make revocations more robust in the face of poor connectivity with the controller or attempts to incapacitate the controller with denial of service attacks, as well as making revocations faster on huge networks.
### Windows and Macintosh UI Improvements (ZeroTier One)
The Mac has a whole new UI built natively in Objective-C. It provides a pulldown similar in appearance and operation to the Mac WiFi task bar menu.
The Windows UI has also been improved and now provides a task bar icon that can be right-clicked to manage networks. Both now expose managed route and IP permissions, allowing nodes to easily opt in to full tunnel operation if you have a router configured on your network.
### Ad-Hoc Networks
A special kind of public network called an ad-hoc network may be accessed by joining a network ID with the format:
ffSSSSEEEE000000
| | | |
| | | Reserved for future use, must be 0
| | End of port range (hex)
| Start of port range (hex)
Reserved ZeroTier address prefix indicating a controller-less network
Ad-hoc networks are public (no access control) networks that have no network controller. Instead their configuration and other credentials are generated locally. Ad-hoc networks permit only IPv6 UDP and TCP unicast traffic (no multicast or broadcast) using 6plane format NDP-emulated IPv6 addresses. In addition an ad-hoc network ID encodes an IP port range. UDP packets and TCP SYN (connection open) packets are only allowed to desintation ports within the encoded range.
For example `ff00160016000000` is an ad-hoc network allowing only SSH, while `ff0000ffff000000` is an ad-hoc network allowing any UDP or TCP port.
Keep in mind that these networks are public and anyone in the entire world can join them. Care must be taken to avoid exposing vulnerable services or sharing unwanted files or other resources.
### Network Controller (Partial) Rewrite
The network controller has been largely rewritten to use a simple in-filesystem JSON data store in place of SQLite, and it is now included by default in all Windows, Mac, Linux, and BSD builds. This means any desktop or server node running ZeroTier One can now be a controller with no recompilation needed.
If you have data in an old SQLite3 controller we've included a NodeJS script in `controller/migrate-sqlite` to migrate data to the new format. If you don't migrate, members will start getting `NOT_FOUND` when they attempt to query for updates.
## Major Bug Fixes in 1.2.0
* **The Windows HyperV 100% CPU bug is FINALLY DEAD**: This long-running problem turns out to have been an issue with Windows itself, but one we were triggering by placing invalid data into the Windows registry. Microsoft is aware of the issue but we've also fixed the triggering problem on our side. ZeroTier should now co-exist quite well with HyperV and should now be able to be bridged with a HyperV virtual switch.
* **Segmenation faults on musl-libc based Linux systems**: Alpine Linux and some embedded Linux systems that use musl libc (a minimal libc) experienced segmentation faults. These were due to a smaller default stack size. A work-around that sets the stack size for new threads has been added.
* **Windows firewall blocks local JSON API**: On some Windows systems the firewall likes to block 127.0.0.1:9993 for mysterious reasons. This is now fixed in the installer via the addition of another firewall exemption rule.
* **UI crash on embedded Windows due to missing fonts**: The MSI installer now ships fonts and will install them if they are not present, so this should be fixed.
## Other Improvements in 1.2.0
* **Improved dead path detection**: ZeroTier is now more aggressive about expiring paths that do not seem to be active. If a path seems marginal it is re-confirmed before re-use.
* **Minor performance improvements**: We've reduced unnecessary memcpy's and made a few other performance improvements in the core.
* **Linux static binaries**: For our official packages (the ones in the download.zerotier.com apt and yum repositories) we now build Linux binaries with static linking. Hopefully this will stop all the bug reports relating to library inconsistencies, as well as allowing our deb packages to run on a wider variety of Debian-based distributions. (There are far too many of these to support officially!) The overhead for this is very small, especially since we built our static versions against musl-libc. Distribution maintainers are of course free to build dynamically linked versions for inclusion into distributions; this only affects our official binaries.
// Start at utun8 to leave lower utuns unused since other stuff might
// want them -- OpenVPN, cjdns, etc. I'm not sure if those are smart
// enough to scan upward like this.
for(utunNo=8;utunNo<=256;++utunNo) {
if ((_fd = _make_utun(utunNo)) > 0)
break;
}
}
if (_fd <= 0)
throw std::runtime_error("unable to find/load ZeroTier tap driver OR use built-in utun driver in OSX; permission or system problem or too many open devices?");
*This is really internal use code. You're free to test it out but expect to do some editing/tweaking to make it work. We used this to run some massive scale tests of our new geo-cluster-based root server infrastructure prior to taking it live.*
Before using this code you will want to edit agent.js to change SERVER_HOST to the IP address of where you will run server.js. This should typically be an open Internet IP, since this makes reporting not dependent upon the thing being tested. Also note that this thing does no security of any kind. It's designed for one-off tests run over a short period of time, not to be anything that runs permanently. You will also want to edit the Dockerfile if you want to build containers and change the network ID to the network you want to run tests over.
This code can be deployed across a large number of VMs or containers to test and benchmark HTTP traffic within a virtual network at scale. The agent acts as a server and can query other agents, while the server collects agent data and tells agents about each other. It's designed to use RFC4193-based ZeroTier IPv6 addresses within the cluster, which allows the easy provisioning of a large cluster without IP conflicts.
The Dockerfile builds an image that launches the agent. The image must be "docker run" with "--device=/dev/net/tun --privileged" to permit it to open a tun/tap device within the container. (Unfortunately CAP_NET_ADMIN may not work due to a bug in Docker and/or Linux.) You can run a bunch with a command like:
for ((n=0;n<10;n++));dodockerrun--device=/dev/net/tun--privileged-dzerotier/http-test;done
With this update we've expanded upon the previous CLI's functionality, so things should seem pretty familiar. Here are some of the new features we've introduced:
- Create and administer networks on ZeroTier Central directly from the console.
- Service configurations, allows you to control local/remote instances of ZeroTier One
- Identity generation and management is now part of the same CLI tool
***
## Configurations
Configurations are a way for you to nickname and logically organize the control of ZeroTier services running locally or remotely (this includes ZeroTier Central!). They're merely groupings of service API url's and auth tokens. The CLI's settings data is contained within `.zerotierCliSettings`.
For instance, you can control your local instance of ZeroTier One via the `@local` config. By default it is represented as follows:
```
"local": {
"auth": "7tyqRoFytajf21j2l2t9QPm5",
"type": "one",
"url": "http://127.0.0.1:9993/"
}
```
As an example, if you issue the command `zerotier ls` is it implicitly stating `zerotier @local ls`.
With the same line of thinking, you could create a `@my.zerotier.com` which would allow for something like `zerotier @my.zerotier.com net-create` which talks to our hosted ZeroTier Central to create a new network.
## Command families
- `cli-` is for configuring the settings data for the CLI itself, such as adding/removing `@thing` configurations, variables, etc.
- `net-` is for operating on a *ZeroTier Central* service such as `https://my.zerotier.com`
- `id-` is for handling ZeroTier identities.
And those commands with no prefix are there to allow you to operate ZeroTier One instances either local or remote.
std::cout<<WARN_STR<<"No @thing specified, assuming default for ZeroTier One command: "<<state.settings["defaultOne"].get<std::string>().c_str()<<std::endl;
configName=state.settings["defaultOne"].get<std::string>().erase(0,1);// get default
}
else{
std::cout<<WARN_STR<<"No @thing specified, and no default is known."<<std::endl;
std::cout<<"HELP: To set a default: zerotier cli-set defaultOne @my_default_thing"<<std::endl;
std::cout<<WARN_STR<<"No @thing specified, assuming default for ZeroTier Central command: "<<state.settings["defaultCentral"].get<std::string>().c_str()<<std::endl;
configName=state.settings["defaultCentral"].get<std::string>().erase(0,1);// get default
}
else{
std::cout<<WARN_STR<<"No @thing specified, and no default is known."<<std::endl;
std::cout<<"HELP: To set a default: zerotier cli-set defaultCentral @my_default_thing"<<std::endl;
std::cerr<<"FATAL: unable to write "<<getSettingsFilePath()<<std::endl;
exit(-1);
}
if(initSuccess){
std::cerr<<"INFO: initialized new config at "<<getSettingsFilePath()<<std::endl;
}else{
std::cerr<<"INFO: initialized new config at "<<getSettingsFilePath()<<" but could not auto-init local ZeroTier One service config from "<<oneHome<<" -- you will need to set local service URL and port manually if you want to control a local instance of ZeroTier One. (This happens if you are not root/administrator.)"<<std::endl;
fprintf(stderr,"vanity address: found %.10llx !\n",(unsignedlonglong)id.address().toInt());
}
break;
}else{
fprintf(stderr,"vanity address: tried %.10llx looking for first %d bits of %.10llx\n",(unsignedlonglong)id.address().toInt(),vanityBits,(unsignedlonglong)(vanity<<(40-vanityBits)));
}
}
std::stringidser=id.toString(true);
if(argc>=3){
if(!OSUtils::writeFile(argv[2],idser)){
std::cerr<<"Error writing to "<<argv[2]<<std::endl;
return1;
}elsestd::cout<<argv[2]<<" written"<<std::endl;
if(argc>=4){
idser=id.toString(false);
if(!OSUtils::writeFile(argv[3],idser)){
std::cerr<<"Error writing to "<<argv[3]<<std::endl;
This subfolder contains Dockerfiles and a script to build Linux packages for a variety of Linux distributions. It's also an excellent way to test your CPU fans and stress test your disk.
Running `build.sh` with no arguments builds everything. You can run `build.sh` with the name of a distro (e.g. centos-7) to only build that. Both 32 and 64 bit packages are built except where no 32-bit version of the distribution exists.
The `make-apt-repos.sh` and `make-rpm-repos.sh` scripts build repositories. They may require some editing for outside-of-ZeroTier use, and be careful with the apt one if you have an existing *aptly* configuration.
cat 'ext/installfiles/linux/DEBIAN/control.in'| sed "s/__VERSION__/${vmajor}.${vminor}.${revision}/"| sed "s/__ARCH__/${debian_arch}/" >$debfolder/control
echo Found rpmbuild, trying to build RedHat/CentOS package.
rm -f /tmp/zerotier-one.spec
curr_dir=`pwd`
cat ext/installfiles/linux/RPM/zerotier-one.spec.in | sed "s/__VERSION__/${vmajor}.${vminor}.${revision}/g"| sed "s/__INSTALLER__/${targ}/g" >/tmp/zerotier-one.spec
fprintf(stdout,"FATAL ERROR: no token specified on command line and could not read '%s' or '%s'"ZT_EOL_S,dotZeroTierAuthToken.c_str(),systemAuthTokenPath);
return-2;
}
#else // __WINDOWS__
fprintf(stdout,"FATAL ERROR: no token specified on command line and could not read '%s'"ZT_EOL_S,dotZeroTierAuthToken.c_str());
return-2;
#endif // __WINDOWS__
}
}
}
if(!authToken.length()){
fprintf(stdout,"FATAL ERROR: could not find auth token"ZT_EOL_S);
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.