* ARM32 platform build and flag fixes
* Add a clarification line to LICENSE.txt
* Fix license message in CLI
* Windows service now looks for service command line arguments
* Fixed a bug that could cause excessive queued multicasts
Major Changes
* Mac version no longer requires a kernel extension, instead making use of the [feth interfaces](https://apple.stackexc
* Added support for concurrent multipath (multiple paths at once) with traffic weighting by link quality and faster rec
* Added under-the-hood support for QoS (not yet exposed) that will eventually be configurable via our rules engine.
Minor Changes and Bug Fixes
* Experimental DB driver for [LF](https://github.com/zerotier/lf) to store network controller data (LFDB.cpp / LFDB.hpp
* Modified credential push and direct path push timings and algorithms to somewhat reduce "chattiness" of the protocol
* Removed our beta/half-baked integration of Central with the Windows UI. We're going to do a whole new UI of some kind
* Fixed stack overflow issues on Linux versions using musl libc.
* Fixed some alignment problems reported on ARM and ARM64, but some reports we could not reproduce so please report any
* Fixed numerous other small issues and bugs such as ARM alignment issues causing crashes on some devices.
* Windows now sets the adapter name such that it is consistent in both the Windows UI and command line utilities.
Because zerotier-one depends on "one", which is a phony target,
making it relinks zerotier-one every time, which is probably
not the desired behavior.
In any case, zerotier-one, zerotier-cli, and zerotier-idtool are
real target, so they should have real dependencies and actions.
The "one" target now depends on these three files.
Behavior is mostly unchanged otherwise.
Made the smallest change I could figure out.
Had to remove -Werror because something in the postgres adapter woudn't
compile.
brew install postgresql rabbitmq-c
and seems to put the libs in the right place
This commit changes the Dockerfile ot use the multi-stage build and help
to get an automated build on Docker.
The idea of the multi-stage build is to use the already stable Debian
distribution channel to provide up-to-date versions of ZeroTier.
The benefit is that it would be possible to automate the image build,
either on Docker Hub, Travis or taking advantage of the [docker-library/official-images] infrastructure.
This changes follows the best-practices suggested by
[docker-library/official-images], such as using a High Availability GPG
keyserver, providing a default CMD, allowing "bash" on `docker run` and
others.
Given that both the builder `debian:stretch` and `alpine:latest` are
official messages and have [manifests], this means that this Dockerfile
is also multi-platform. This means that this same Dockerfile will
pick-up the correct Debian package according to the architecture of the
running system during build.
With this changes we could try to promote the image to be parte of
[docker-library/official-images], and take advantage of automated
publishing of multi-architecture images. Others would be able to use
`docker run zerotier` and download the latest version appropriate to
their system.
Related to #682
[docker-library/official-images]: https://github.com/docker-library/official-images
[manifests]: https://blog.docker.com/2017/09/docker-official-images-now-multi-platform/
1) Use existing standard libraries to convert to/from IPv4/IPv6 strings and binary representation.
2) Move null terminator assignment InetAddress::toIpString to top of function, this way if ANY errors occurs that don't write content to the buffer, we're not passing a potentially dangerous buffer around.
* Fixed a bug that caused exits to take a long time on Mac due to huge numbers of redundant attempts to delete managed routes.
* Fixed a socket limit problem on Windows that caused the ZeroTier service to run out of sockets, causing the UI and CLI to be unable to access the API.
* Fixed a threading bug in the ZeroTier Core, albeit one that never manifested on the regular ZeroTier One service/client.
* Fixed a bug that could cause the service to crash if an authorized local client accessed an invalid URL via the control API. (Not exploitable since you needed admin access anyway.)
Added LinuxNetLink to talk to the rtnetlink socket for adding interfaces, addresses routes. Not yet complete. Can currently monitor changes on the system.
when building with `ZT_DEBUG=1` this hint produces a warning:
> node/Packet.cpp:335:43: error: 'register' storage class specifier is deprecated and incompatible with C++17 [-Werror,-Wdeprecated-register]
See http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2015/n4340
The Hyperledger implementation (https://github.com/hyperledger/iroha-ed25519)
contains changes to the assembly code to allow PIC. This in turn fixes
compilation/linking of ZeroTier One when "full hardening" flags are used.
* Features and Core Improvements
* Path selection has been overhauled to improve path stability, simplify code, and prepare for multi-path and trunking in the next major release.
* This version introduces remote tracing for remote diagnostics. Network controllers can set a node (usually the controller itself) to receive remote tracing events from all members of the network or from select members. Events are only sent if they pertain to a given network for security reasons.
* Multicast replication can now be done by designated multicast replicators on a network (flagged as such at the controller) rather than by the sender. Most users won't want this, but it's useful for specialized use cases on hub-and-spoke networks and for low-power devices.
* Cryptographic performance improvements on several platforms.
* Multithreaded performance improvements throughout the code base, including the use of an inline lightweight spinlock for low-contention resources.
* Bugs fixed
* Disappearing routes on Mac (GitHub issue #600)
* Route flapping and path instability in some dual-stack V4/V6 networks
* Blacklist (in local.conf) doesn't work reliably (GitHub issue #656)
* Connection instabilities due to unsigned integer overflows in timing comparisons (use int64_t instead of uint64_t)
* Binaries don't run on some older or lower-end 32-bit ARM chips (build problem)
* ARM NEON crypto code crashes (build problem)
* Fixed some lock ordering issues revealed by "valgrind" tool
* The "zerotier-idtool" command could not be accessed from "zerotier-one" via command line switch
* Leaking sockets on some platforms when uPnP/NAT-PMP is enabled
* Fixed two very rare multithreading issues that were only observed on certain systems
* Platform-Specific Changes
* MacOS
* Installer now loads the kernel extension right away so that High Sierra users will see the prompt to authorize it. This is done in the "Security & Privacy" preference pane and must be done driectly on the console (not via remote desktop). On High Sierra and newer kexts must be authorized at the console via security settings system preferences pane.
* Windows
* The Windows installer should now install the driver without requiring a special prompt in most cases. This should make it easier for our packages to be accepted into and updated in the Chocolatey repository and should make it easier to perform remote installs across groups of machines using IT management and provisioning tools.
* The Windows official packages are now signed with an EV certificate (with hardware key).
* The Windows UI can now log into ZeroTier Central and join networks via the Central API.
* The `zerotier-idtool` command should now work on Windows without ugly hacks.
* Upgraded the installer version.
* Made a few changes to hopefully fix sporadic "will not uninstall" problems, though we cannot duplicate these issues ourselves.
* Linux
* Device names are now generated deterministically based on network IDs for all newly joined networks.
* Android
* Multicast now works on Android in most cases! Android apps can send and receive multicast and subscribe to multicast group IPs. Note that in some cases the app must bind to the specific correct interface for this to work.
* IPv6 can be disabled in UI for cases where it causes problems.
* Path selection has been overhauled to improve path stability, simplify code, and prepare for multi-path and trunking in the next major release.
* This version introduces remote tracing for remote diagnostics. Network controllers can set a node (usually the controller itself) to receive remote tracing events from all members of the network or from select members. Events are only sent if they pertain to a given network for security reasons.
* Multicast replication can now be done by designated multicast replicators on a network (flagged as such at the controller) rather than by the sender. Most users won't want this, but it's useful for specialized use cases on hub-and-spoke networks and for low-power devices.
* Cryptographic performance improvements on several platforms.
* Multithreaded performance improvements throughout the code base, including the use of an inline lightweight spinlock for low-contention resources.
* Bugs fixed
* Disappearing routes on Mac (GitHub issue #600)
* Route flapping and path instability in some dual-stack V4/V6 networks
* Blacklist (in local.conf) doesn't work reliably (GitHub issue #656)
* Connection instabilities due to unsigned integer overflows in timing comparisons (use int64_t instead of uint64_t)
* Binaries don't run on some older or lower-end 32-bit ARM chips (build problem)
* ARM NEON crypto code crashes (build problem)
* Fixed some lock ordering issues revealed by "valgrind" tool
* The "zerotier-idtool" command could not be accessed from "zerotier-one" via command line switch
* Leaking sockets on some platforms when uPnP/NAT-PMP is enabled
* Fixed two very rare multithreading issues that were only observed on certain systems
* Platform-Specific Changes
* MacOS
* Installer now loads the kernel extension right away so that High Sierra users will see the prompt to authorize it. This is done in the "Security & Privacy" preference pane and must be done driectly on the console (not via remote desktop). On High Sierra and newer kexts must be authorized at the console via security settings system preferences pane.
* Windows
* The Windows installer should now install the driver without requiring a special prompt in most cases. This should make it easier for our packages to be accepted into and updated in the Chocolatey repository and should make it easier to perform remote installs across groups of machines using IT management and provisioning tools.
* The Windows official packages are now signed with an EV certificate (with hardware key).
* The Windows UI can now log into ZeroTier Central and join networks via the Central API.
* The `zerotier-idtool` command should now work on Windows without ugly hacks.
* Upgraded the installer version.
* Made a few changes to hopefully fix sporadic "will not uninstall" problems, though we cannot duplicate these issues ourselves.
* Linux
* Device names are now generated deterministically based on network IDs for all newly joined networks.
* Android
* Multicast now works on Android in most cases! Android apps can send and receive multicast and subscribe to multicast group IPs. Note that in some cases the app must bind to the specific correct interface for this to work.
* IPv6 can be disabled in UI for cases where it causes problems.
Adds a "vault" section to local.conf. Example local.conf:
{
"config": {
"vault": {
"vaultURL": "https://some.vault.host:8200",
"vaultToken": "my-super-secret-vault-token",
"vaultPath": "secure/place/to/put/identity"
}
}
Additionally, the following environment variables can be set. Environment variables override local.conf:
VAULT_ADDR
VAULT_TOKEN
VAULT_PATH
Identities will be placed in the keys "public" and "secret" under the user specified path. If no path is specified, they will be placed in the token specific cubbyhole.
If identity.public and identity.secret exist on disk and vault is configured, they will be automatically added to Vault and removed from disk.
TODO:
* Decide behavior for if Vault cannot be reached.
* Add libcurl as a dependency in Linux & Mac builds
* Add libcurl as a requirement for linux packages
When running ArchlinuxARM with 64bit support on Raspberry Pi3, the default build fails due to the architecture look-up at the beginning of the build process. A simple addition of the Armv7l section above allows the build to continue and successfully run.
There were cases in the code where time calculations and comparisons were overflowing and causing connection instability. This will keep time calculations within expected ranges.
I've only tested locally, on a power8 box running Ubuntu 16.10, but everything (and earth) checks out.
Let me know if more testing infrastructure is needed.
* Managed routes are now only bifurcated for the default route. This is a change in behavior, though few people will probably notice. Bifurcating all managed routes was causing more trouble than it was worth for most users.
* Up to 2X crypto speedup on x86-64 (except Windows, which will take some porting) and 32-bit ARM platforms due to integration of fast assembly language implementations of Salsa20/12 from the [supercop](http://bench.cr.yp.to/supercop.html) code base. These were written by Daniel J. Bernstein and are in the public domain. My Macbook Pro (Core i5 2.8ghz) now does almost 1.5GiB/sec Salsa20/12 per core and a Raspberry Pi got a 2X boost. 64-bit ARM support and Windows support will take some work but should not be too hard.
* Refactored code that manages credentials to greatly reduce memory use in most cases. This may also result in a small performance improvement.
* Reworked and simplified path selection and priority logic to fix path instability and dead path persistence edge cases. There have been some sporadic reports of persistent path instabilities and dead paths hanging around that take minutes to resolve. These have proven difficult to reproduce in house, but hopefully this will fix them. In any case it seems to speed up path establishment in our tests and it makes the code simpler and more readable.
* Eliminated some unused cruft from the code around path management and in the peer class.
* Fixed an issue causing build problems on some MIPS architecture systems.
* Fixed Windows forgetting routes on sleep/wake or in some other circumstances. (GitHub issue #465)
Version 1.2.2 fixes a few bugs discovered after the 1.2.0 release. These are:
* A bug causing unreliable multicast propagation (GitHub issue #461).
* A crash in ARM binaries due to a build chain and flags problem.
* A bug in the network controller preventing members from being listed (GitHub issue #460).
The largest new feature in 1.2.0, and the product of many months of work, is our advanced network rules engine. With this release we achieve traffic control, security monitoring, and micro-segmentation capability on par with many enterprise SDN solutions designed for use in advanced data centers and corporate networks.
Rules allow you to filter packets on your network and vector traffic to security observers. Security observation can be performed in-band using REDIRECT or out of band using TEE.
Tags and capabilites provide advanced methods for implementing fine grained permission structures and micro-segmentation schemes without bloating the size and complexity of your rules table.
See the [rules engine announcement blog post](https://www.zerotier.com/blog/?p=927) for an in-depth discussion of theory and implementation. The [manual](https://www.zerotier.com/manual.shtml) contains detailed information on rule, tag, and capability use, and the `rule-compiler/` subfolder of the ZeroTier source tree contains a JavaScript function to compile rules in our human-readable rule definition language into rules suitable for import into a network controller. (ZeroTier Central uses this same script to compile rules on [my.zerotier.com](https://my.zerotier.com/).)
It's now possible to create your own root servers and add them to the root server pool on your nodes. This is done by creating what's called a "moon," which is a signed enumeration of root servers and their stable points on the network. Refer to the [manual](https://www.zerotier.com/manual.shtml) for instructions.
Federated roots achieve a number of things:
* You can deploy your own infrastructure to reduce dependency on ours.
* You can deploy roots *inside your LAN* to ensure that network connectivity inside your facility still works if the Internet goes down. This is the first step toward making ZeroTier viable as an in-house SDN solution.
* Roots can be deployed inside national boundaries for countries with data residency laws or "great firewalls." (As of 1.2.0 there is still no way to force all traffic to use these roots, but that will be easy to do in a later version.)
* Last but not least this makes ZeroTier somewhat less centralized by eliminating any hard dependency on ZeroTier, Inc.'s infrastructure.
Our roots will of course remain and continue to provide zero-configuration instant-on deployment, a secure global authority for identities, and free traffic relaying for those who can't establish peer to peer connections.
An element of our design philosophy is "features are bugs." This isn't an absolute dogma but more of a guiding principle. We try as hard as we can to avoid adding features, especially "knobs" that must be tweaked by a user.
As of 1.2.0 we've decided that certain knobs are unavoidable, and so there is now a `local.conf` file that can be used to configure them. See the ZeroTier One documentation for these. They include:
* Blacklisting interfaces you want to make sure ZeroTier doesn't use for network traffic, such as VPNs, slow links, or backplanes designated for only certain kinds of traffic.
* Turning uPnP/NAT-PMP on or off.
* Configuring software updates on Windows and Mac platforms.
* Defining trusted paths (the old trusted paths file is now deprecated)
* Setting the ZeroTier main port so it doesn't have to be changed on the command line, which is very inconvenient in many cases.
A good software update system for Windows and Mac clients has been a missing feature in previous versions. It does exist but we've been shy about using it so far due to its fragility in some environments.
We've greatly improved this mechanism in 1.2.0. Not only does it now do a better job of actually invoking the update, but it also transfers updates in-band using the ZeroTier protocol. This means it can work in environments that do not allows http/https traffic or that force it through proxies. There's also now an update channel setting: `beta` or `release` (the default).
Software updates are authenticated three ways:
1. ZeroTier's own signing key is used to sign all updates and this signature is checked prior to installation. ZeroTier, Inc.'s signatures are performed on an air-gapped machine.
2. Updates for Mac and Windows are signed using Apple and Microsoft (DigiCert EV) keys and will not install unless these signatures are also valid.
3. The new in-band update mechanism also authenticates the source of the update via ZeroTier's built-in security features. This provides transport security, while 1 and 2 provide security of the update at rest.
Updates are now configurable via `local.conf`. There are three options: `disable`, `download`, and `apply`. The third (apply) is the default for official builds on Windows and Mac, making updates happen silently and automatically as they do for popular browsers like Chrome and Firefox. Updates are disabled by default on Linux and other Unix-type systems as these are typically updated through package managers.
Version 1.2.0 is now aware of the link quality of direct paths with other 1.2.0 nodes. This information isn't used yet but is visible through the JSON API. (Quality always shows as 100% with pre-1.2.0 nodes.) Quality is measured passively with no additional overhead using a counter based packet loss detection algorithm.
This information is visible from the command line via `listpeers`:
200 listpeers XXXXXXXXXX 199.XXX.XXX.XXX/9993;10574;15250;1.00 48 1.2.0 LEAF
200 listpeers XXXXXXXXXX 195.XXX.XXX.XXX/45584;467;7608;0.44 290 1.2.0 LEAF
The first peer's path is at 100% (1.00), while the second peer's path is suffering quite a bit of packet loss (0.44).
Link quality awareness is a precursor to intelligent multi-path and QoS support, which will in future versions bring us to feature parity with SD-WAN products like Cisco iWAN.
Version 1.2.0 adds anti-DOS (denial of service) rate limits and other hardening for improved resiliency against a number of denial of service attack scenarios.
It also adds a mechanism for instantaneous credential revocation. This can be used to revoke certificates of membership instantly to kick a node off a network (for private networks) and also to revoke capabilities and tags. The new controller sends revocations by default when a peer is de-authorized.
Revocations propagate using a "rumor mill" peer to peer algorithm. This means that a controller need only successfully send a revocation to at least one member of a network with connections to other active members. At this point the revocation will flood through the network peer to peer very quickly. This helps make revocations more robust in the face of poor connectivity with the controller or attempts to incapacitate the controller with denial of service attacks, as well as making revocations faster on huge networks.
The Mac has a whole new UI built natively in Objective-C. It provides a pulldown similar in appearance and operation to the Mac WiFi task bar menu.
The Windows UI has also been improved and now provides a task bar icon that can be right-clicked to manage networks. Both now expose managed route and IP permissions, allowing nodes to easily opt in to full tunnel operation if you have a router configured on your network.
A special kind of public network called an ad-hoc network may be accessed by joining a network ID with the format:
ffSSSSEEEE000000
| | | |
| | | Reserved for future use, must be 0
| | End of port range (hex)
| Start of port range (hex)
Reserved ZeroTier address prefix indicating a controller-less network
Ad-hoc networks are public (no access control) networks that have no network controller. Instead their configuration and other credentials are generated locally. Ad-hoc networks permit only IPv6 UDP and TCP unicast traffic (no multicast or broadcast) using 6plane format NDP-emulated IPv6 addresses. In addition an ad-hoc network ID encodes an IP port range. UDP packets and TCP SYN (connection open) packets are only allowed to desintation ports within the encoded range.
For example `ff00160016000000` is an ad-hoc network allowing only SSH, while `ff0000ffff000000` is an ad-hoc network allowing any UDP or TCP port.
Keep in mind that these networks are public and anyone in the entire world can join them. Care must be taken to avoid exposing vulnerable services or sharing unwanted files or other resources.
The network controller has been largely rewritten to use a simple in-filesystem JSON data store in place of SQLite, and it is now included by default in all Windows, Mac, Linux, and BSD builds. This means any desktop or server node running ZeroTier One can now be a controller with no recompilation needed.
If you have data in an old SQLite3 controller we've included a NodeJS script in `controller/migrate-sqlite` to migrate data to the new format. If you don't migrate, members will start getting `NOT_FOUND` when they attempt to query for updates.
* **The Windows HyperV 100% CPU bug is FINALLY DEAD**: This long-running problem turns out to have been an issue with Windows itself, but one we were triggering by placing invalid data into the Windows registry. Microsoft is aware of the issue but we've also fixed the triggering problem on our side. ZeroTier should now co-exist quite well with HyperV and should now be able to be bridged with a HyperV virtual switch.
* **Segmenation faults on musl-libc based Linux systems**: Alpine Linux and some embedded Linux systems that use musl libc (a minimal libc) experienced segmentation faults. These were due to a smaller default stack size. A work-around that sets the stack size for new threads has been added.
* **Windows firewall blocks local JSON API**: On some Windows systems the firewall likes to block 127.0.0.1:9993 for mysterious reasons. This is now fixed in the installer via the addition of another firewall exemption rule.
* **UI crash on embedded Windows due to missing fonts**: The MSI installer now ships fonts and will install them if they are not present, so this should be fixed.
* **Improved dead path detection**: ZeroTier is now more aggressive about expiring paths that do not seem to be active. If a path seems marginal it is re-confirmed before re-use.
* **Minor performance improvements**: We've reduced unnecessary memcpy's and made a few other performance improvements in the core.
* **Linux static binaries**: For our official packages (the ones in the download.zerotier.com apt and yum repositories) we now build Linux binaries with static linking. Hopefully this will stop all the bug reports relating to library inconsistencies, as well as allowing our deb packages to run on a wider variety of Debian-based distributions. (There are far too many of these to support officially!) The overhead for this is very small, especially since we built our static versions against musl-libc. Distribution maintainers are of course free to build dynamically linked versions for inclusion into distributions; this only affects our official binaries.
Merge branch 'master' of github.com:zerotier/ZeroTierOne
The largest new feature in 1.2.0, and the product of many months of work, is our advanced network rules engine. With this release we achieve traffic control, security monitoring, and micro-segmentation capability on par with many enterprise SDN solutions designed for use in advanced data centers and corporate networks.
Rules allow you to filter packets on your network and vector traffic to security observers. Security observation can be performed in-band using REDIRECT or out of band using TEE.
Tags and capabilites provide advanced methods for implementing fine grained permission structures and micro-segmentation schemes without bloating the size and complexity of your rules table.
See the [rules engine announcement blog post](https://www.zerotier.com/blog/?p=927) for an in-depth discussion of theory and implementation. The [manual](https://www.zerotier.com/manual.shtml) contains detailed information on rule, tag, and capability use, and the `rule-compiler/` subfolder of the ZeroTier source tree contains a JavaScript function to compile rules in our human-readable rule definition language into rules suitable for import into a network controller. (ZeroTier Central uses this same script to compile rules on [my.zerotier.com](https://my.zerotier.com/).)
It's now possible to create your own root servers and add them to the root server pool on your nodes. This is done by creating what's called a "moon," which is a signed enumeration of root servers and their stable points on the network. Refer to the [manual](https://www.zerotier.com/manual.shtml) for instructions.
Federated roots achieve a number of things:
* You can deploy your own infrastructure to reduce dependency on ours.
* You can deploy roots *inside your LAN* to ensure that network connectivity inside your facility still works if the Internet goes down. This is the first step toward making ZeroTier viable as an in-house SDN solution.
* Roots can be deployed inside national boundaries for countries with data residency laws or "great firewalls." (As of 1.2.0 there is still no way to force all traffic to use these roots, but that will be easy to do in a later version.)
* Last but not least this makes ZeroTier somewhat less centralized by eliminating any hard dependency on ZeroTier, Inc.'s infrastructure.
Our roots will of course remain and continue to provide zero-configuration instant-on deployment, a secure global authority for identities, and free traffic relaying for those who can't establish peer to peer connections.
An element of our design philosophy is "features are bugs." This isn't an absolute dogma but more of a guiding principle. We try as hard as we can to avoid adding features, especially "knobs" that must be tweaked by a user.
As of 1.2.0 we've decided that certain knobs are unavoidable, and so there is now a `local.conf` file that can be used to configure them. See the ZeroTier One documentation for these. They include:
* Blacklisting interfaces you want to make sure ZeroTier doesn't use for network traffic, such as VPNs, slow links, or backplanes designated for only certain kinds of traffic.
* Turning uPnP/NAT-PMP on or off.
* Configuring software updates on Windows and Mac platforms.
* Defining trusted paths (the old trusted paths file is now deprecated)
* Setting the ZeroTier main port so it doesn't have to be changed on the command line, which is very inconvenient in many cases.
A good software update system for Windows and Mac clients has been a missing feature in previous versions. It does exist but we've been shy about using it so far due to its fragility in some environments.
We've greatly improved this mechanism in 1.2.0. Not only does it now do a better job of actually invoking the update, but it also transfers updates in-band using the ZeroTier protocol. This means it can work in environments that do not allows http/https traffic or that force it through proxies. There's also now an update channel setting: `beta` or `release` (the default).
Software updates are authenticated three ways:
1. ZeroTier's own signing key is used to sign all updates and this signature is checked prior to installation. ZeroTier, Inc.'s signatures are performed on an air-gapped machine.
2. Updates for Mac and Windows are signed using Apple and Microsoft (DigiCert EV) keys and will not install unless these signatures are also valid.
3. The new in-band update mechanism also authenticates the source of the update via ZeroTier's built-in security features. This provides transport security, while 1 and 2 provide security of the update at rest.
Updates are now configurable via `local.conf`. There are three options: `disable`, `download`, and `apply`. The third (apply) is the default for official builds on Windows and Mac, making updates happen silently and automatically as they do for popular browsers like Chrome and Firefox. Updates are disabled by default on Linux and other Unix-type systems as these are typically updated through package managers.
Version 1.2.0 is now aware of the link quality of direct paths with other 1.2.0 nodes. This information isn't used yet but is visible through the JSON API. (Quality always shows as 100% with pre-1.2.0 nodes.) Quality is measured passively with no additional overhead using a counter based packet loss detection algorithm.
This information is visible from the command line via `listpeers`:
200 listpeers XXXXXXXXXX 199.XXX.XXX.XXX/9993;10574;15250;1.00 48 1.2.0 LEAF
200 listpeers XXXXXXXXXX 195.XXX.XXX.XXX/45584;467;7608;0.44 290 1.2.0 LEAF
The first peer's path is at 100% (1.00), while the second peer's path is suffering quite a bit of packet loss (0.44).
Link quality awareness is a precursor to intelligent multi-path and QoS support, which will in future versions bring us to feature parity with SD-WAN products like Cisco iWAN.
Version 1.2.0 adds anti-DOS (denial of service) rate limits and other hardening for improved resiliency against a number of denial of service attack scenarios.
It also adds a mechanism for instantaneous credential revocation. This can be used to revoke certificates of membership instantly to kick a node off a network (for private networks) and also to revoke capabilities and tags. The new controller sends revocations by default when a peer is de-authorized.
Revocations propagate using a "rumor mill" peer to peer algorithm. This means that a controller need only successfully send a revocation to at least one member of a network with connections to other active members. At this point the revocation will flood through the network peer to peer very quickly. This helps make revocations more robust in the face of poor connectivity with the controller or attempts to incapacitate the controller with denial of service attacks, as well as making revocations faster on huge networks.
The Mac has a whole new UI built natively in Objective-C. It provides a pulldown similar in appearance and operation to the Mac WiFi task bar menu.
The Windows UI has also been improved and now provides a task bar icon that can be right-clicked to manage networks. Both now expose managed route and IP permissions, allowing nodes to easily opt in to full tunnel operation if you have a router configured on your network.
A special kind of public network called an ad-hoc network may be accessed by joining a network ID with the format:
ffSSSSEEEE000000
| | | |
| | | Reserved for future use, must be 0
| | End of port range (hex)
| Start of port range (hex)
Reserved ZeroTier address prefix indicating a controller-less network
Ad-hoc networks are public (no access control) networks that have no network controller. Instead their configuration and other credentials are generated locally. Ad-hoc networks permit only IPv6 UDP and TCP unicast traffic (no multicast or broadcast) using 6plane format NDP-emulated IPv6 addresses. In addition an ad-hoc network ID encodes an IP port range. UDP packets and TCP SYN (connection open) packets are only allowed to desintation ports within the encoded range.
For example `ff00160016000000` is an ad-hoc network allowing only SSH, while `ff0000ffff000000` is an ad-hoc network allowing any UDP or TCP port.
Keep in mind that these networks are public and anyone in the entire world can join them. Care must be taken to avoid exposing vulnerable services or sharing unwanted files or other resources.
The network controller has been largely rewritten to use a simple in-filesystem JSON data store in place of SQLite, and it is now included by default in all Windows, Mac, Linux, and BSD builds. This means any desktop or server node running ZeroTier One can now be a controller with no recompilation needed.
If you have data in an old SQLite3 controller we've included a NodeJS script in `controller/migrate-sqlite` to migrate data to the new format. If you don't migrate, members will start getting `NOT_FOUND` when they attempt to query for updates.
* **The Windows HyperV 100% CPU bug is FINALLY DEAD**: This long-running problem turns out to have been an issue with Windows itself, but one we were triggering by placing invalid data into the Windows registry. Microsoft is aware of the issue but we've also fixed the triggering problem on our side. ZeroTier should now co-exist quite well with HyperV and should now be able to be bridged with a HyperV virtual switch.
* **Segmenation faults on musl-libc based Linux systems**: Alpine Linux and some embedded Linux systems that use musl libc (a minimal libc) experienced segmentation faults. These were due to a smaller default stack size. A work-around that sets the stack size for new threads has been added.
* **Windows firewall blocks local JSON API**: On some Windows systems the firewall likes to block 127.0.0.1:9993 for mysterious reasons. This is now fixed in the installer via the addition of another firewall exemption rule.
* **UI crash on embedded Windows due to missing fonts**: The MSI installer now ships fonts and will install them if they are not present, so this should be fixed.
* **Improved dead path detection**: ZeroTier is now more aggressive about expiring paths that do not seem to be active. If a path seems marginal it is re-confirmed before re-use.
* **Minor performance improvements**: We've reduced unnecessary memcpy's and made a few other performance improvements in the core.
* **Linux static binaries**: For our official packages (the ones in the download.zerotier.com apt and yum repositories) we now build Linux binaries with static linking. Hopefully this will stop all the bug reports relating to library inconsistencies, as well as allowing our deb packages to run on a wider variety of Debian-based distributions. (There are far too many of these to support officially!) The overhead for this is very small, especially since we built our static versions against musl-libc. Distribution maintainers are of course free to build dynamically linked versions for inclusion into distributions; this only affects our official binaries.
Move setting _initialized = true until after WindowsEthernetTap::threadMain() has actually created and brought up the adapter. Also in OneService::nodeVirtualNetworkConfigFunction(), wait up to 5 seconds for WindowsEthernatTap::isInitialized() to return true before attempting to configure the interface and managed routes. Without this, the adapter doesnt actually exist yet when trying to add routes
Also, dont add the prefix length to the IP addresses in there.
Appears to fix Github Issue #357. Still waiting on some info from Microsoft for confirmation of a few things.
WindowsEthernetTap was storing the MTU for the ZeroTier Adapter as a
DWORD. When changing the value from the network adapter settings UI, it
was stored as a string. Updated the call when creating the adapter to
store it as a string as windows expects.
- ZT will only drop root privileges if zerotier-one user exists. It is created by
Debian postinst script - in other cases the user has to be created by administrator.
- Linux >=4.3 with ambient capabilities is required, otherwise ZT will silently
- "-U" option now also disables privileges dropping
Version 1.1.10 is a minor bug fix release. It fixes the following two issues:
* Routes are no longer added 'via' IP addresses of the current node's ZT interface.
* IPv6 default route is now fixed on Mac.
* IP addresses are now deleted before new ones are added to allow changes to netmask
on Mac and some other platforms.
This version also quietly introduces a feature called trusted paths. We'll be
documenting this in our community section soon. This is an experimental feature
for in-LAN SDN use.
1.1.8 is a very minor update. It contains a small fix to IP address management
on Linux but other than that there are no changes to the regular ZeroTier One
code.
The 1.1.8 controller contains several fixes and improvements:
* Network authorizedMemberCount works again
* Managed IP addresses are pushed as long as they fall within a LAN-local managed
route. The v4AssignMode and v6AssignMode settings only control automatic IP
assignment from pools. In previous versions v4AssignMode had to be on for IPs
to be pushed at all, making it impossible to just manually assign IPs.
* IPv6 now has an auto-assign mode ('zt' to match V4) that can work alongside the
special RFC4193 and 6PLANE modes (though it is not NDP emulated). If this is
enabled IPv6 addresses will be auto-assigned from IPv6 ranges in the assignment
pool list.
* Controller documentation is MUCH better! See controller/README.md for actual
useful information!
Finally 1.1.8 has several improvements to linux-build-farm including Debian wheezy
support.
(Retagged due to silly one-line bug and left over debug line... doh!)
Version 1.1.6 contains several significant improvements for use in complex network
environments along with some minor bug fixes and improvements to path stability and
dead path detection.
ROUTE MANAGEMENT AND FULL TUNNEL SUPPORT
1.1.6 is the first version of ZeroTier One to permit "full tunnel" (default route
override) operation on Linux, Mac, and Windows. This allows all Internet traffic
to be tunneled through ZeroTier while allowing ZeroTier peer-to-peer traffic to
continue to use the physical interface. 1.1.6 also brings route management support
and permissions settings for local networks to control whether networks are allowed
to modify the routing table or override default routing.
This is currently considered a beta/experimental feature and must be enabled via
the command line interface.
Route management and default route override requires support at the network controller.
When my.zerotier.com is updated and ready, we will post more information and testing
instructions at: https://www.zerotier.com/community
HIGHLY SCALABLE CONTAINER NETWORKING
1.1.6 also brings a new multicast-free (NDP emulated) IPv6 private addressing scheme
called "6plane." 6plane provides each host with a private IPv6 /80 and routes *all*
IPv6 traffic for this subnet to the host via transparent NDP emulation. This /80 can
then be assigned to Docker or other container/VM managers to assign a network-wide
IPv6 /128 to every container. Since NDP is emulated and multicast isn't needed, this
system can scale to millions of containers or more on a single backplane network with
a high degree of efficiency and reliability.
6plane also requires controller support. Look for it at my.zerotier.com once we have
upgraded our core infrastructure and web UIs.
(All hosts must be running 1.1.6 for 6plane to work properly. Other IPv6 addresses
or addressing modes are not affected and normal IPv6 NDP will continue to work
alongside 6plane in the same network.)
OTHER CHANGES
* Upgraded bundled miniupnpc, libnatpmp, and http-parser.
* New Debian and RPM packaging that is closer to compliance with distribution
guidelines, and a new Dockerized Linux package build system in linux-build-farm/
that can build every package on actual images of the correct distribution.
* Improvements to dead path detection.
* IPv6 now uses keepalive because a significant number of stateful IPv6 edge
routers have very short timeouts (30 seconds or less!).
* Significant performance improvements to network controllers under high load.
* Enable -fstack-protector-strong for better stack canary (security) support
in binaries. Note that this may require newer gcc/g++ or clang.
COMING SOON
The next version of ZeroTier One should have a new Mac UI. It's a system tray app
that looks and behaves a lot like the Mac WiFi pulldown menu. We'll also be adding
GUI support for default route and route management options and other new features.
Shortly after that we plan on adding full OpenFlow-like SDN rules engine support
to the ZeroTier core, making our planetary Ethernet switch a fully manageable smart
switch and enabling sophisticated security and flow rule management.
Version 1.1.6 contains several significant improvements for use in complex network
environments along with some minor bug fixes and improvements to path stability and
dead path detection.
ROUTE MANAGEMENT AND FULL TUNNEL SUPPORT
1.1.6 is the first version of ZeroTier One to permit "full tunnel" (default route
override) operation on Linux, Mac, and Windows. This allows all Internet traffic
to be tunneled through ZeroTier while allowing ZeroTier peer-to-peer traffic to
continue to use the physical interface. 1.1.6 also brings route management support
and permissions settings for local networks to control whether networks are allowed
to modify the routing table or override default routing.
This is currently considered a beta/experimental feature and must be enabled via
the command line interface.
Route management and default route override requires support at the network controller.
When my.zerotier.com is updated and ready, we will post more information and testing
instructions at: https://www.zerotier.com/community
HIGHLY SCALABLE CONTAINER NETWORKING
1.1.6 also brings a new multicast-free (NDP emulated) IPv6 private addressing scheme
called "6plane." 6plane provides each host with a private IPv6 /80 and routes *all*
IPv6 traffic for this subnet to the host via transparent NDP emulation. This /80 can
then be assigned to Docker or other container/VM managers to assign a network-wide
IPv6 /128 to every container. Since NDP is emulated and multicast isn't needed, this
system can scale to millions of containers or more on a single backplane network with
a high degree of efficiency and reliability.
6plane also requires controller support. Look for it at my.zerotier.com once we have
upgraded our core infrastructure and web UIs.
(All hosts must be running 1.1.6 for 6plane to work properly. Other IPv6 addresses
or addressing modes are not affected and normal IPv6 NDP will continue to work
alongside 6plane in the same network.)
OTHER CHANGES
* Upgraded bundled miniupnpc, libnatpmp, and http-parser.
* New Debian and RPM packaging that is closer to compliance with distribution
guidelines, and a new Dockerized Linux package build system in linux-build-farm/
that can build every package on actual images of the correct distribution.
* Improvements to dead path detection.
* IPv6 now uses keepalive because a significant number of stateful IPv6 edge
routers have very short timeouts (30 seconds or less!).
* Significant performance improvements to network controllers under high load.
* Enable -fstack-protector-strong for better stack canary (security) support
in binaries. Note that this may require newer gcc/g++ or clang.
COMING SOON
The next version of ZeroTier One should have a new Mac UI. It's a system tray app
that looks and behaves a lot like the Mac WiFi pulldown menu. We'll also be adding
GUI support for default route and route management options and other new features.
Shortly after that we plan on adding full OpenFlow-like SDN rules engine support
to the ZeroTier core, making our planetary Ethernet switch a fully manageable smart
switch and enabling sophisticated security and flow rule management.
(and other host friendly) IPv6 /80 magic subnetting to allow massive
multicast-free NDP emulated IPv6 networks where each host can have a
/48 worth of IPv6 IPs for internal containers, VMs, etc.
Alan Kay, thou art avenged.
https://ivanovivan.wordpress.com/2010/09/13/alan-kay-quotes/
implementing NAT, do implement stateful firewalling with absurdly short
timeouts (<60s). Keepalives are still required in IPv6.
Network engineers continue to mindlessly carry forward cruft and baggage
from IPv4 to IPv6.
Many NATs revert to symmetric behavior from friendlier modes if they cannot
preserve ports. This can occur if there is, for example, more than one ZT
device behind the NAT using port 9993.
Others (Airport Extreme?) seem to have bugs in which they completely freak
out if more than one device tries to do a lot of mappings using the same
internal local port.
Mostly to fix the latter case and somewhat to fix the former, we introduce
a secondary port. ZeroTier now binds 9993 (or whatever port you specify)
plus another port computed deterministically from your ZeroTier address.
This port is used for new links 1/4 of the time.
This mostly addresses the second problem above and partly helps to address
the first.
If uPnP/NAT-PMP is enabled we also still open a tertiary port because some
routers freak out if NAT-t is attempted using the same port as uPnP.
All of this is IPv4 only of course. IPv6 is sane.
ZeroTier One version 1.1.4 has several improvements to path stability and
path learning logic. It also contains the BETA version of Network Containers!
See netcon/README.md and https://www.zerotier.com/product-netcon.shtml for more
information.
Improvements in 1.1.4:
- Paths are now confirmed more aggressively on inactivity. This version can
usually detect a dead direct path in under 10 seconds vs 1-2 minutes in
previous versions.
- Logic to prevent recursion has been overhauled. The old versions had code
that explicitly checked packet contents (AntiRecursion.hpp). This is now
gone in favor of a function that checks IP addresses of ZeroTier devices
when paths are learned or probed. This is both more efficient and less
prone to edge case failures. Testing in several complex environments has
shown that this eliminates a few edge cases where inappropriate paths would
be used.
ZeroTier One version 1.1.2 is mostly a bug fix release, but also includes support
for NAT-PMP as a method of mapping external ports from behind supported routers.
Bugs fixed include:
- Refactored path selection code to eliminate a change that might have introduced
a reliability regression from earlier versions.
- Reworked UPnP code for improved reliability and also added NAT-PMP support.
- Fixed a crash on network preferred relays. (Not exploitable or security critical
since only network controllers can send that config option and there was no
buffer overflow or other exploitable issue.)
- Fix GUI problem on OSX where window could be larger than screen on some Macbooks.
- Fix a potential rate limit problem on PUSH_DIRECT_PATHS that could negatively
impact same-network connectivity.
- Update zerotier-cli to show preferred vs. active paths and hide inactive ones.
- Fix an issue on shutdown with thread delete order and thread synchronization.
ZeroTier 1.1.0 introduces a number of fixes and improvements in several areas.
We incremented the secondary version to indicate the significance of this release.
Version numbering has been a bit ad-hoc in the past. In future versions we will
adopt the following scheme: odd-numbered revision numbers like 1.1.1 will indicate
development versions, while even numbered ones like 1.1.2 will indicate tagged
releases. The public git repo branching has also been revised: master will always
be the latest tagged release, dev will be usually-working development, and edge
will host maybe-broken "bleeding edge" development. Pull requests on GitHub should
generally be made against dev, not master or edge. Other branches that may appear
from time to time may be feature or experimental branches. Only master is confirmed
good, with dev usually being okay but not guaranteed to be such. (To the extent
that any software is ever guaranteed to be anything.)
Change summary:
User-facing changes and improvements:
- Windows now has a new .NET-based native UI, which replaces the old WebControl
wrapper around the React UI. This just didn't work well on older Windows systems,
and we did not want to bundle 40+ megabytes of web browser with our app just for
its very simple UI.
- The web UI (still used for Mac and usable in Linux as well) is updated with
improved look and simplifications.
- Both UIs no longer have the "Peers" tab, since several users reported that non-
technical users found this confusing and even alarming (does this mean people
can access my system?). This information is visibile with "listpeers" from the
command line (zerotier-cli).
New features:
- Virtual networks that use our RFC4193-based IPv6 numbering scheme now emulate
IPv6 NDP for queries that target these addresses within the same network. This
allows for faster multicast-free connection init and improved security since
the address is now hard-wired to the device ID (which is a crypto token). This
does not affect IPv6 NDP for other IPv6 addresses or link-local, which will
continue to work normally. This also opens the potential for a reduced footprint
multicast-free build for embedded applications.
- This version includes beta support for a feature called CIRCUIT_TEST. Network
controllers for networks you have joined can now send a special message called
CIRCUIT_TEST which allows for ZeroTier-layer link testing and remote diagnosis
of link issues. Any operator of a network controller can do this; more
documentation will be forthcoming. The only information that may be gathered
in this way is IP addressing info and very basic system info (OS, 32/64 bit,
ZeroTier version). No personal information, hard drive data, location, or other
private info is available. This can only be ordered by a controller of a network
you have joined and is secured using cryptographic signatures.
- This version includes an alpha version of clustering a.k.a. multi-homing! This
powerful feature allows for a single ZeroTier device to be run from multiple
endpoints, with connecting peers being handed off to endpoints that are closer
via GeoIP lookup and/or are more lightly loaded. Currently this is only suitable
for use in our soon-to-be-upgraded root server infrastructure (details will be
blogged soon), but in the future it will be capable of hosting multi-homed
devices on user networks. This will allow things like (for example) a geo-
clustered Cassandra server that appears behind a single IP on a virtual LAN.
This feature must be enabled with the ZT_ENABLE_CLUSTER=1 build option.
Bug fixes and other improvements (including performance!):
- A faster version of the Poly1305 cryptographic MAC function was substituted
for sometimes greatly improved performance.
- C++ STL std::map was replaced throughout the entire core with a hand-rolled
Hashtable implementation for improved performance and in some cases a reduced
memory footprint. Some maps are still used in peripheral code that is not
performance critical or where ordered keys are needed.
- The zerotier-cli and zerotier-idtool symbolic links are now created in
/usr/local/bin on OSX to comply with El Capitan file security restrictions.
- The OSX tap device driver has been updated. This update may fix issues that
some users have reported with bridging on OSX. This new tap device driver
drops 32-bit support, but if you have a 32-bit system you can manually install
the old driver from ext/bin/tap-mac.
- Mac users could experience a problem with the UI if they installed ZeroTier,
then uninstalled it, then installed again. This is now fixed.
- UPnP port mappings should work better on some routers, and a different local
port is now used for UPnP mapped traffic vs. NAT-t'd traffic to get around
a bug in several popular mid-tier routers where using UPnP mapping alongside
traditional NAT traversal made a port unreachable.
- Debian package now builds with the right arch label on armv7l systems (Pi 2)
- The old "root topology" has been replaced with a similar but better thought
out concept called a World. The World defines the root servers and possibly
in the future other things, and can be updated in-band from trusted peers
allowing for software-upgrade-free network upgrades to keep up with growing
demand. See node/World.hpp for details.
- A fix was made to "self-awareness," which keeps track of your external IP
info and adapts to changes, to eliminate a problem that could cause "link
thrashing" behind some symmetric NATs.
- Escalating UDP TTLs was re-introduced to better transit some port-restricted
cone NATs such as Linux IP MASQ (used for Docker).
- An otherwise harmless crash-on-exit bug in the network controller was fixed.
- All new direct links are now confirmed in both directions. This adds a very
small amount of initial HELLO/OK traffic but fixes some edge cases where an
incomplete or unidirectional path might be used.
- [SECURITY] Better rate limiting was put in place for VERB_PUSH_DIRECT_PATHS
to prevent potential abuse for amplification attacks.
- [SECURITY] Build flags were tweaked on OSX to ensure that all code including
dependency libraries are built with full stack canary protection and ASLR
support.
Visit https://www.zerotier.com/blog or follow @ZeroTier on Twitter for updates
and announcements!
* No joining/leaving networks yet, but they do display.
* Nothing is updated yet after first load of the app. Need to set up a background task to run updates.
also update for changed function calls that now accept a local address
# Conflicts:
# include/ZeroTierOne.h
# java/CMakeLists.txt
# java/jni/Android.mk
# java/jni/ZT1_jnicache.cpp
# java/jni/ZT1_jnilookup.h
# java/jni/ZT1_jniutils.cpp
# java/jni/com_zerotierone_sdk_Node.cpp
This puts code accessing the data in a critical section so that the GC cannot run while JNI has access to the array. This helps with stability somewhat, but I'm still getting some crashes in the GC
# Authors and Third Party Code Licensing Information
* ZeroTier protocol design and core network virtualization engine, ZeroTier One service, React web UI, packaging for most platforms, kitchen sink...<br>
## Primary Authors
* ZeroTier Core and ZeroTier One virtual networking service<br>
Adam Ierymenko / adam.ierymenko@zerotier.com
Joseph Henry / joseph.henry@zerotier.com (QoS and multipath)
* Java JNI Interface to enable Android application development, and Android app itself (code for that is elsewhere)<br>
Grant Limberg / glimberg@gmail.com
## Contributors
* ZeroTier SDK / libzt<br>
Joseph Henry / joseph.henry@zerotier.com
## Third Party Contributors
* A number of fixes and improvements to the new controller, other stuff.<br>
Kees Bos / https://github.com/keesbos
Kees Bos / https://github.com/keesbos/
* Debugging and testing, OpenWRT support fixes.<br>
Moritz Warning / moritzwarning@web.de
* Debian GNU/Linux packaging, manual pages, and license compliance edits.<br>
Ben Finney <ben+zerotier@benfinney.id.au>
* Several others made smaller contributions, which GitHub tracks here:<br>
* LZ4 compression algorithm by Yann Collet (BSD license)<br>
http://code.google.com/p/lz4/
ZeroTier includes the following third party code, either in ext/ or incorporated into the ZeroTier core. This third party code remains licensed under its original license and is not subject to ZeroTier's BSL license.
*http-parser by many authors (MIT license)<br>
https://github.com/joyent/http-parser
*LZ4 compression algorithm by Yann Collet
*json-parser by James McLaughlin (BSD license)<br>
https://github.com/udp/json-parser
*Files: node/LZ4.cpp
* Home page: http://code.google.com/p/lz4/
* License: BSD 2-clause
*TunTapOSX by Mattias Nissler (BSD license)<br>
http://tuntaposx.sourceforge.net
*C++11 json (nlohmann/json) by Niels Lohmann
*tap-windows and tap-windows6 by the OpenVPN project (GPL)<br>
ZeroTier is a software defined networking layer for Earth.
ZeroTier is a smart programmable Ethernet switch for planet Earth. It allows all networked devices, VMs, containers, and applications to communicate as if they all reside in the same physical data center or cloud region.
It can be used for on-premise network virtualization, as a peer to peer VPN for mobile teams, for hybrid or multi-data-center cloud deployments, or just about anywhere else secure software defined virtual networking is useful.
This is accomplished by combining a cryptographically addressed and secure peer to peer network (termed VL1) with an Ethernet emulation layer somewhat similar to VXLAN (termed VL2). Our VL2 Ethernet virtualization layer includes advanced enterprise SDN features like fine grained access control rules for network micro-segmentation and security monitoring.
ZeroTier One is our OS-level client service. It allows Mac, Linux, Windows, FreeBSD, and soon other types of clients to join ZeroTier virtual networks like conventional VPNs or VLANs. It can run on native systems, VMs, or containers (Docker, OpenVZ, etc.).
All ZeroTier traffic is encrypted end-to-end using secret keys that only you control. Most traffic flows peer to peer, though we offer free (but slow) relaying for users who cannot establish peer to peer connetions.
Visit [ZeroTier's site](https://www.zerotier.com/) for more information. You can also download professionally packaged binary installers/packages for a variety of supported OSes there if you don't want to build ZeroTier One from source.
The goals and design principles of ZeroTier are inspired by among other things the original [Google BeyondCorp](https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/43231.pdf) paper and the [Jericho Forum](https://en.wikipedia.org/wiki/Jericho_Forum) with its notion of "deperimeterization."
### Building from Source
Visit [ZeroTier's site](https://www.zerotier.com/) for more information and [pre-built binary packages](https://www.zerotier.com/download/). Apps for Android and iOS are available for free in the Google Play and Apple app stores.
For Mac, Linux, and BSD, just type "make" (or "gmake" on BSD). You won't need much installed; here are the requirements for various platforms:
ZeroTier is licensed under the [BSL version 1.1](https://mariadb.com/bsl11/). See [LICENSE.txt](LICENSE.txt) and the [ZeroTier pricing page](https://www.zerotier.com/pricing) for details. ZeroTier is free to use internally in businesses and academic institutions and for non-commercial purposes. Certain types of commercial use such as building closed-source apps and devices based on ZeroTier or offering ZeroTier network controllers and network management as a SaaS service require a commercial license.
* Mac: Xcode command line tools, and [Packages](http://s.sudre.free.fr/Software/Packages/about.html) if you want to build an OSX .pkg installer ("make mac-dist-pkg"). It should build on OSX 10.7 or newer.
* Linux: gcc/g++ or clang/clang++ (Makefile will use clang by default if available.)
* FreeBSD (and other BSD): C++ compiler (G++ usually) and GNU make (gmake).
A small amount of third party code is also included in ZeroTier and is not subject to our BSL license. See [AUTHORS.md] for a list of third party code, where it is included, and the licenses that apply to it. All of the third party code in ZeroTier is liberally licensed (MIT, BSD, Apache, public domain, etc.).
Each supported platform has its own *make-XXX.mk* file that contains the actual make rules for the platform. The right .mk file is included by the main Makefile based on the GNU make *OSTYPE* variable. Take a look at the .mk file for your platform for other targets, debug build rules, etc.
### Getting Started
Windows, of course, is special. We build for Windows with Microsoft Visual Studio 2012 on Windows 7. A solution file is located in the *windows* subfolder. Newer versions of Visual Studio (and Windows) may work but haven't been tested. Older versions almost certainly will not, since they lack things like *stdint.h* and certain STL features. MinGW or other ports of gcc/clang to Windows should also work but haven't been tested. Build steps for Windows are a bit more complicated. For the moment you are on your own there.
Everything in the ZeroTier world is controlled by two types of identifier: 40-bit/10-digit *ZeroTier addresses* and 64-bit/16-digit *network IDs*. These identifiers are easily distinguished by their length. A ZeroTier address identifies a node or "device" (laptop, phone, server, VM, app, etc.) while a network ID identifies a virtual Ethernet network that can be joined by devices.
Mobile versions are in progress. They don't work yet, and in any case only the glue code will be included in this repository. The full mobile apps are in private repositories on our own git server.
ZeroTier addresses can be thought of as port numbers on an enormous planet-wide enterprise Ethernet smart switch supporting VLANs. Network IDs are VLAN IDs to which these ports may be assigned. A single port can be assigned to more than one VLAN.
### Supported Platforms
A ZeroTier address looks like `8056c2e21c` and a network ID looks like `8056c2e21c000001`. Network IDs are composed of the ZeroTier address of that network's primary controller and an arbitrary 24-bit ID that identifies the network on this controller. Network controllers are roughly analogous to SDN controllers in SDN protocols like [OpenFlow](https://en.wikipedia.org/wiki/OpenFlow), though as with the analogy between VXLAN and VL2 this should not be read to imply that the protocols or design are the same. You can use our convenient and inexpensive SaaS hosted controllers at [my.zerotier.com](https://my.zerotier.com/) or [run your own controller](controller/) if you don't mind messing around with JSON configuration files or writing scripts to do so.
CPU architecture shouldn't matter unless it's smaller than 32-bit or something really bizarre like a "middle-endian" processor. We have reports of ZeroTier One running on arm32, arm64, and MIPS. It builds and runs out of the box on Raspberry Pi, BeagleBone, BananaPi, and other ARM-based developer/hobbyist boards.
### Project Layout
ZeroTier is written in C and C++ (C++03 / ISO/IEC 14882:2003) and uses data structures and algorithms from the C++03 STL. We do not use any C++11 features (yet), since we want to support a few old and embedded platforms that don't have C++11 compilers. You *will* require a compiler and headers new enough to support 64-bit integers (long long) and the *stdint.h* header. The latter could also be faked by adding defines for things like *uint32\_t*, *int64\_t*, etc.
The base path contains the ZeroTier One service main entry point (`one.cpp`), self test code, makefiles, etc.
Typing "make selftest" will build a *zerotier-selftest* binary which unit tests various internals and reports on a few aspects of the build environment. It's a good idea to try this on novel platforms or architectures.
-`artwork/`: icons, logos, etc.
-`attic/`: old stuff and experimental code that we want to keep around for reference.
-`controller/`: the reference network controller implementation, which is built and included by default on desktop and server build targets.
-`debian/`: files for building Debian packages on Linux.
-`doc/`: manual pages and other documentation.
-`ext/`: third party libraries, binaries that we ship for convenience on some platforms (Mac and Windows), and installation support files.
-`include/`: include files for the ZeroTier core.
-`java/`: a JNI wrapper used with our Android mobile app. (The whole Android app is not open source but may be made so in the future.)
-`macui/`: a Macintosh menu-bar app for controlling ZeroTier One, written in Objective C.
-`node/`: the ZeroTier virtual Ethernet switch core, which is designed to be entirely separate from the rest of the code and able to be built as a stand-alone OS-independent library. Note to developers: do not use C++11 features in here, since we want this to build on old embedded platforms that lack C++11 support. C++11 can be used elsewhere.
-`osdep/`: code to support and integrate with OSes, including platform-specific stuff only built for certain targets.
-`rule-compiler/`: JavaScript rules language compiler for defining network-level rules.
-`service/`: the ZeroTier One service, which wraps the ZeroTier core and provides VPN-like connectivity to virtual networks for desktops, laptops, servers, VMs, and containers.
-`windows/`: Visual Studio solution files, Windows service code, and the Windows task bar app UI.
### Build and Platform Notes
To build on Mac and Linux just type `make`. On FreeBSD and OpenBSD `gmake` (GNU make) is required and can be installed from packages or ports. For Windows there is a Visual Studio solution in `windows/'.
- **Mac**
- Xcode command line tools for OSX 10.8 or newer are required.
- **Linux**
- The minimum compiler versions required are GCC/G++ 4.9.3 or CLANG/CLANG++ 3.4.2. (Install `clang` on CentOS 7 as G++ is too old.)
- Linux makefiles automatically detect and prefer clang/clang++ if present as it produces smaller and slightly faster binaries in most cases. You can override by supplying CC and CXX variables on the make command line.
- **Windows**
- Windows 7 or newer is supported. This *may* work on Vista but isn't officially supported there. It will not work on Windows XP.
- We build with Visual Studio 2017. Older versions may not work. Clang or MinGW will also probably work but may require some makefile hacking.
- **FreeBSD**
- GNU make is required. Type `gmake` to build.
- **OpenBSD**
- There is a limit of four network memberships on OpenBSD as there are only four tap devices (`/dev/tap0` through `/dev/tap3`).
- GNU make is required. Type `gmake` to build.
Typing `make selftest` will build a *zerotier-selftest* binary which unit tests various internals and reports on a few aspects of the build environment. It's a good idea to try this on novel platforms or architectures.
### Running
@ -45,10 +76,10 @@ The service is controlled via the JSON API, which by default is available at 127
Here's where home folders live (by default) on each OS:
* Linux: /var/lib/zerotier-one
* BSD: /var/db/zerotier-one
* Mac: /Library/Application Support/ZeroTier/One
*Windows: \\ProgramData\\ZeroTier\\One (That's for Windows 7. The base 'shared app data' folder might be different on different Windows versions.)
* **Windows**: `\ProgramData\ZeroTier\One` (That's for Windows 7. The base 'shared app data' folder might be different on different Windows versions.)
Running ZeroTier One on a Mac is the same, but OSX requires a kernel extension. We ship a signed binary build of the ZeroTier tap device driver, which can be installed on Mac with:
@ -56,116 +87,22 @@ Running ZeroTier One on a Mac is the same, but OSX requires a kernel extension.
This will create the home folder for Mac, place *tap.kext* there, and set its modes correctly to enable ZeroTier One to manage it with *kextload* and *kextunload*.
We recommend using our binary packages on Windows, since there are several prerequisites such as a tap driver that must be installed on the system *and* in the home folder.
### Joining A Network
ZeroTier virtual networks are identified by 16-digit hexadecimal network IDs, while devices are identified by 10-digit addresses. To get your address run:
sudo zerotier-cli status
(Use ./zerotier-cli if you're running it right from your build folder.)
You should see something like:
200 info ########## ONLINE #.#.#
That 10-digit hex code is you. It's derived via a one-way proof of work function from your cryptographic public key. Your public key can be found in *identity.public* in ZeroTier's home folder, while *identity.secret* contains your full identity including the secret portion of the key pair.
(The identity files define your device's *identity*. Moving them to another system will move that identity. Be careful when cloning virtual machines that have identities stored on them. If two devices have the same identity, they'll "fight" over it and you won't know which device will receive network packets.)
If you want to do a quick test, you can join [Earth](https://www.zerotier.com/earth.html). It's a global public network that anyone can join. Type:
sudo zerotier-cli join 8056c2e21c000001
Then:
sudo zerotier-cli listnetworks
At first it'll be in *REQUESTING\_CONFIGURATION* state. In a few seconds to a minute you should see something like:
200 listnetworks 8056c2e21c000001 earth.zerotier.net ##:##:##:##:##:## OK PUBLIC zt0 ##.##.##.##/##
Earth will assign you an IP address in the "unofficially available" globally unrouted 28.0.0.0/7 IP block so as to avoid conflicts with local networks. (Your networks can use any IP scheme, or can even leave IP addresses unmanaged.) Once you get an IP, you should be able to ping something:
ping earth.zerotier.net
Go to [http://earth.zerotier.net/](http://earth.zerotier.net/) to see a short little welcome page that will tell you your IP and Ethernet MAC address.
Earth is a public place. If you don't want to stick around run:
sudo zerotier-cli leave 8056c2e21c000001
The network (and associated interface) should be gone.
Networks are created and administrated by network controllers. Most users will want to use our hosted controllers. Visit [our web site](https://www.zerotier.com/) for more information. Later in this README there are brief instructions about building ZeroTier One with network controller support for those who want to try running their own.
Macintosh and Windows installers also install a GUI application.
### Installing
We don't have a "make install" rule quite yet. On Linux you can type:
make installer
This will build a binary that, when run, will install ZeroTier One on most current Linux distributions. We also have RPM and DEB build files in *ext/installfiles/linux* that wrap this installer in packages for RedHat/CentOS and Debian/Ubuntu derived distributions. If *rpmbuild* is present on the system, the RPM will be built. If *dpkg-deb* is present, the DEB package will be built.
On Mac the best way is to install [Packages](http://s.sudre.free.fr/Software/Packages/about.html) and use:
make mac-dist-pkg
This builds a .pkg file that can be installed.
BSD has no installer yet. We're working on it.
Linux/BSD and Mac installations have an *uninstall.sh* file in their ZeroTier home folder that cleanly removes ZeroTier One from the system. Run this with:
sudo /path/to/ZeroTier/home/folder/uninstall.sh
Windows installers are insane. We build our .MSI installers with [Advanced Installer Enterprise](http://www.advancedinstaller.com). The Advanced Installer project file is in *ext/installfiles/windows*. To avoid lasting psychological trauma we recommend leaving Windows installers alone and using the pre-built Windows binaries on our web site.
### Using ZeroTier One in Docker Containers
To run the ZeroTier One service in a Docker container, run it with "--device=/dev/net/tun --cap-add=NET_ADMIN". This will allow ZeroTier One to open a "tap" virtual network port inside the container.
Alternately, you can use Ethernet bridging to bridge the *docker0* device on your system to a ZeroTier virtual network. This allows you to run ZeroTier One on the host and bridge the entire Docker network backplane to a virtual network or other hosts.
We're working on better "official" Docker support. In the meantime there is a [user-contributed project here](https://github.com/davide/docker-zerotier).
### Building with Network Controller Support
**Warning: as of beta version 1.0.3 the new network controller is not heavily tested. We recommend waiting for 1.0.4 to deploy this in production.**
Network controllers are nodes responsible for issuing configurations and certificates to members of ZeroTier virtual networks. Most users won't need to run their own, so this code is by default not included in the ZeroTier One binary.
You can build a network controller on Linux or Mac with:
make ZT_ENABLE_NETWORK_CONTROLLER=1
This will build a version that contains the Sqlite-backed network controller and associated extensions to the JSON local service control API. You will need the development headers for sqlite3 installed. On Mac these ship as part of Xcode, while on Linux they'll be found in packages for the various distributions.
See the JSON API documentation in [service/](service/) for more information about how to control controllers.
### Troubleshooting
### Basic Troubleshooting
For most users, it just works.
If you are running a local system firewall, we recommend adding a rule permitting UDP port 9993 inbound and outbound. If you installed binaries for Windows this should be done automatically. Other platforms might require manual editing of local firewall rules depending on your configuration.
The Mac firewall can be founder under "Security" in System Preferences. Linux has a variety of firewall configuration systems and tools. If you're using Ubuntu's *ufw*, you can do this:
The Mac firewall can be found under "Security" in System Preferences. Linux has a variety of firewall configuration systems and tools. If you're using Ubuntu's *ufw*, you can do this:
sudo ufw allow 9993/udp
On CentOS check */etc/sysconfig/iptables* for IPTables rules. For other distributions consult your distribution's documentation. You'll also have to check the UIs or documentation for commercial third party firewall applications like Little Snitch (Mac), McAfee Firewall Enterprise (Windows), etc. if you are running any of those. Some corporate environments might have centrally managed firewall software, so you might also have to contact IT.
On CentOS check `/etc/sysconfig/iptables` for IPTables rules. For other distributions consult your distribution's documentation. You'll also have to check the UIs or documentation for commercial third party firewall applications like Little Snitch (Mac), McAfee Firewall Enterprise (Windows), etc. if you are running any of those. Some corporate environments might have centrally managed firewall software, so you might also have to contact IT.
ZeroTier One peers will automatically locate each other and communicate directly over a local wired LAN *if UDP port 9993 inbound is open*. If that port is filtered, they won't be able to see each others' LAN announcement packets. If you're experiencing poor performance between devices on the same physical network, check their firewall settings. Without LAN auto-location peers must attempt "loopback" NAT traversal, which sometimes fails and in any case requires that every packet traverse your external router twice.
Users behind certain types of firewalls and "symmetric" NAT devices may not able able to connect to external peers directly at all. ZeroTier has limited support for port prediction and will *attempt* to traverse symmetric NATs, but this doesn't always work. If P2P connectivity fails you'll be bouncing UDP packets off our relay servers resulting in slower performance. Some NAT router(s) have a configurable NAT mode, and setting this to "full cone" will eliminate this problem. If you do this you may also see a magical improvement for things like VoIP phones, Skype, BitTorrent, WebRTC, certain games, etc., since all of these use NAT traversal techniques similar to ours.
If you're interested, there's a [technical deep dive about NAT traversal on our blog](https://www.zerotier.com/blog/?p=226). A troubleshooting tool to help you diagnose NAT issues is planned for the future as are uPnP/IGD/NAT-PMP and IPv6 transport.
If a firewall between you and the Internet blocks ZeroTier's UDP traffic, you will fall back to last-resort TCP tunneling to rootservers over port 443 (https impersonation). This will work almost anywhere but is *very slow* compared to UDP or direct peer to peer connectivity.
### License
The ZeroTier source code is open source and is licensed under the GNU GPL v3 (not LGPL). If you'd like to embed it in a closed-source commercial product or appliance, please e-mail [contact@zerotier.com](mailto:contact@zerotier.com) to discuss commercial licensing. Otherwise it can be used for free.
Additional help [can be found in our knowledge base](https://zerotier.atlassian.net/wiki/spaces/SD/overview).
Version 2.0 is a very significant release with many changes. It remains backward compatibility to version 1.4.0 (and possibly earlier versions but this is not guaranteed) but makes numerous local and behavioral changes that should be reviewed before upgrading production systems.
After this release we're going to be working to get to a more frequent, less extreme, more "agile" release cadence.
Protocol changes:
* Trusted paths have been completely removed. The new AES mode is so fast on CPUs with AES acceleration that much of the rationale for this is gone, and this feature was never used much to begin with due to inconvenience and obvious security concerns. Environments using trusted paths will need to upgrade all nodes at once.
* The symmetric encryption algorithm and mode is now AES-GMAC-SIV, a variation of AES-GCM using the same primitives but offering superior security bounds and behavior under non-ideal conditions. It's also a lot faster than Salsa20/12 and Poly1305 on CPUs with AES acceleration (almost all desktops, laptops, and newer routers and phones). Salsa20/12 with Poly1305 is still supported for communication with older versions and small devices that lack AES acceleration.
* A new identity type (1) has been introduced that contains both Curve25519 and NIST P-384 public key types, but classic type 0 remains the default for new identities for now. ECDH key agreement between V1 identities uses both keys and hashes the resulting secrets to yield security equal to the best of the two, but V1 identities can also agree with V0 identities using only their Curve25519 component.
* Roots can now be joined and left like networks in a much more convenient way, and the old "moon" and "planet" terminology is deprecated.
* A new peer to peer multicast algorithm has been introduced that offers much better scalability and better performance, especially when the physical network itself is hub-and-spoke with many low latency peers connected by higher latency WAN links.
* Forward secrecy is finally supported via periodic re-keying using ephemeral asymmetric keys. Both Curve25519 and NIST P-384 keys are used with secrets being hashed to provide security equal to the stronger of the two curves.
* As part of forward secrecy implementation peers now always exchange HELLO messages even if they don't have a direct path.
* Compression is only enabled for control packets as almost all data packets are largely un-compressable.
* New NAT traversal tricks have been added, such as (ab)use of port 500.
Code changes:
* Migrated from GNU make to cmake for easier cross platform builds and simplified build files.
* The core network hypervisor has been significantly refactored, almost amounting to a partial rewrite.
* Critical packet handling paths have been streamlined with unnecessary memcpy() steps removed.
* Host service code has been completely rewritten in Go. Packet handling code remains in C++, but Go offers superior developer productivity when it comes to implementing more complex local service and local API features. Go imposes a little bit more memory overhead but not much and has been tuned to minimize memory use.
User interface:
* Command line interface has been redesigned and rewritten. Old commands names are supported but their output will be different.
Other things:
* The V2 design, protocol, and cryptographic primitives (AES-GMAC-SIV) have been security audited by [Trail of Bits](https://www.trailofbits.com), and the code is being audited as well prior to full 2.0 release.
---
# Older version release notes
# 2019-08-30 -- Version 1.4.6
* Update default root list to latest
* ARM32 platform build and flag fixes
* Add a clarification line to LICENSE.txt
* Fix license message in CLI
* Windows service now looks for service command line arguments
* Fixed a bug that could cause excessive queued multicasts
# 2019-08-23 -- Version 1.4.4
* Change license from GPL3 to BSL 1.1, see LICENSE.txt
* Fix an issue with the "ipauth" rule and auto-generated unforgeable IPv6 addresses
* Fix socket/bind errors setting IPs and routes on Linux
# 2019-08-12 -- Version 1.4.2
* Fix high CPU use bug on some platforms
* Fix issues with PostgreSQL controller DB (only affects Central)
* Restore backward compatibility with MacOS versions prior to 10.13
# 2019-07-29 -- Version 1.4.0
### Major Changes
* Mac version no longer requires a kernel extension, instead making use of the [feth interfaces](https://apple.stackexchange.com/questions/337715/fake-ethernet-interfaces-feth-if-fake-anyone-ever-seen-this).
* Added support for concurrent multipath (multiple paths at once) with traffic weighting by link quality and faster recovery from lost links.
* Added under-the-hood support for QoS (not yet exposed) that will eventually be configurable via our rules engine.
### Minor Changes and Bug Fixes
* Experimental controller DB driver for [LF](https://github.com/zerotier/lf) to store network controller data (LFDB.cpp / LFDB.hpp).
* Modified credential push and direct path push timings and algorithms to somewhat reduce "chattiness" of the protocol when idle. More radical background overhead reductions will have to wait for the 2.x line.
* Removed our beta/half-baked integration of Central with the Windows UI. We're going to do a whole new UI of some kind in the future at least for Windows and Mac.
* Fixed stack overflow issues on Linux versions using musl libc.
* Fixed some alignment problems reported on ARM and ARM64, but some reports we could not reproduce so please report any issues with exact chip, OS/distro, and ZeroTier version in use.
* Fixed numerous other small issues and bugs such as ARM alignment issues causing crashes on some devices.
* Windows now sets the adapter name such that it is consistent in both the Windows UI and command line utilities.
# 2018-07-27 -- Version 1.2.12
* Fixed a bug that caused exits to take a long time on Mac due to huge numbers of redundant attempts to delete managed routes.
* Fixed a socket limit problem on Windows that caused the ZeroTier service to run out of sockets, causing the UI and CLI to be unable to access the API.
* Fixed a threading bug in the ZeroTier Core, albeit one that never manifested on the regular ZeroTier One service/client.
* Fixed a bug that could cause the service to crash if an authorized local client accessed an invalid URL via the control API. (Not exploitable since you needed admin access anyway.)
# 2018-05-08 -- Version 1.2.10
* Fix bug loading `moons.d/` files for federated root operation.
* Fix compile problem with ZT_DEBUG on some versions of `clang`
* Fix slow network startup bug related to loading of `networks.d/` cache files
# 2018-04-27 -- Version 1.2.8
* Linux version once again builds with PIE (position independent executable) flags
* Fixed bug in zerotier-idtool file sign and verify
* Fixed minor OSX app typo
* Merged alpha NetBSD support (mostly untested, so YMMV)
* Merged several minor typo and one-liner bug fixes
# 2018-04-17 -- Version 1.2.6
* Features and Core Improvements
* Path selection has been overhauled to improve path stability, simplify code, and prepare for multi-path and trunking in the next major release.
* This version introduces remote tracing for remote diagnostics. Network controllers can set a node (usually the controller itself) to receive remote tracing events from all members of the network or from select members. Events are only sent if they pertain to a given network for security reasons.
* Multicast replication can now be done by designated multicast replicators on a network (flagged as such at the controller) rather than by the sender. Most users won't want this, but it's useful for specialized use cases on hub-and-spoke networks and for low-power devices.
* Cryptographic performance improvements on several platforms.
* Multithreaded performance improvements throughout the code base, including the use of an inline lightweight spinlock for low-contention resources.
* Bugs fixed
* Disappearing routes on Mac (GitHub issue #600)
* Route flapping and path instability in some dual-stack V4/V6 networks
* Blacklist (in local.conf) doesn't work reliably (GitHub issue #656)
* Connection instabilities due to unsigned integer overflows in timing comparisons (use int64_t instead of uint64_t)
* Binaries don't run on some older or lower-end 32-bit ARM chips (build problem)
* ARM NEON crypto code crashes (build problem)
* Fixed some lock ordering issues revealed by "valgrind" tool
* The "zerotier-idtool" command could not be accessed from "zerotier-one" via command line switch
* Leaking sockets on some platforms when uPnP/NAT-PMP is enabled
* Fixed two very rare multithreading issues that were only observed on certain systems
* Platform-Specific Changes
* MacOS
* Installer now loads the kernel extension right away so that High Sierra users will see the prompt to authorize it. This is done in the "Security & Privacy" preference pane and must be done directly on the console (not via remote desktop). On High Sierra and newer kexts must be authorized at the console via security settings system preferences pane.
* Windows
* The Windows installer should now install the driver without requiring a special prompt in most cases. This should make it easier for our packages to be accepted into and updated in the Chocolatey repository and should make it easier to perform remote installs across groups of machines using IT management and provisioning tools.
* The Windows official packages are now signed with an EV certificate (with hardware key).
* The Windows UI can now log into ZeroTier Central and join networks via the Central API.
* The `zerotier-idtool` command should now work on Windows without ugly hacks.
* Upgraded the installer version.
* Made a few changes to hopefully fix sporadic "will not uninstall" problems, though we cannot duplicate these issues ourselves.
* Linux
* Device names are now generated deterministically based on network IDs for all newly joined networks.
* Android
* Multicast now works on Android in most cases! Android apps can send and receive multicast and subscribe to multicast group IPs. Note that in some cases the app must bind to the specific correct interface for this to work.
* IPv6 can be disabled in UI for cases where it causes problems.
# 2017-04-20 -- Version 1.2.4
* Managed routes are now only bifurcated for the default route. This is a change in behavior, though few people will probably notice. Bifurcating all managed routes was causing more trouble than it was worth for most users.
* Up to 2X crypto speedup on x86-64 (except Windows, which will take some porting) and 32-bit ARM platforms due to integration of fast assembly language implementations of Salsa20/12 from the [supercop](http://bench.cr.yp.to/supercop.html) code base. These were written by Daniel J. Bernstein and are in the public domain. My Macbook Pro (Core i5 2.8ghz) now does almost 1.5GiB/sec Salsa20/12 per core and a Raspberry Pi got a 2X boost. 64-bit ARM support and Windows support will take some work but should not be too hard.
* Refactored code that manages credentials to greatly reduce memory use in most cases. This may also result in a small performance improvement.
* Reworked and simplified path selection and priority logic to fix path instability and dead path persistence edge cases. There have been some sporadic reports of persistent path instabilities and dead paths hanging around that take minutes to resolve. These have proven difficult to reproduce in house, but hopefully this will fix them. In any case it seems to speed up path establishment in our tests and it makes the code simpler and more readable.
* Eliminated some unused cruft from the code around path management and in the peer class.
* Fixed an issue causing build problems on some MIPS architecture systems.
* Fixed Windows forgetting routes on sleep/wake or in some other circumstances. (GitHub issue #465)
# 2017-03-17 -- Version 1.2.2
* A bug causing unreliable multicast propagation (GitHub issue #461).
* A crash in ARM binaries due to a build chain and flags problem.
* A bug in the network controller preventing members from being listed (GitHub issue #460).
# 2017-03-14 -- Version 1.2.0
Version 1.2.0 is a major milestone release representing almost nine months of work. It includes our rules engine for distributed network packet filtering and security monitoring, federated roots, and many other architectural and UI improvements and bug fixes.
## New Features in 1.2.0
### The ZeroTier Rules Engine
The largest new feature in 1.2.0, and the product of many months of work, is our advanced network rules engine. With this release we achieve traffic control, security monitoring, and micro-segmentation capability on par with many enterprise SDN solutions designed for use in advanced data centers and corporate networks.
Rules allow you to filter packets on your network and vector traffic to security observers. Security observation can be performed in-band using REDIRECT or out of band using TEE.
Tags and capabilities provide advanced methods for implementing fine grained permission structures and micro-segmentation schemes without bloating the size and complexity of your rules table.
See the [rules engine announcement blog post](https://www.zerotier.com/blog/?p=927) for an in-depth discussion of theory and implementation. The [manual](https://www.zerotier.com/manual.shtml) contains detailed information on rule, tag, and capability use, and the `rule-compiler/` subfolder of the ZeroTier source tree contains a JavaScript function to compile rules in our human-readable rule definition language into rules suitable for import into a network controller. (ZeroTier Central uses this same script to compile rules on [my.zerotier.com](https://my.zerotier.com/).)
### Root Server Federation
It's now possible to create your own root servers and add them to the root server pool on your nodes. This is done by creating what's called a "moon," which is a signed enumeration of root servers and their stable points on the network. Refer to the [manual](https://www.zerotier.com/manual.shtml) for instructions.
Federated roots achieve a number of things:
* You can deploy your own infrastructure to reduce dependency on ours.
* You can deploy roots *inside your LAN* to ensure that network connectivity inside your facility still works if the Internet goes down. This is the first step toward making ZeroTier viable as an in-house SDN solution.
* Roots can be deployed inside national boundaries for countries with data residency laws or "great firewalls." (As of 1.2.0 there is still no way to force all traffic to use these roots, but that will be easy to do in a later version.)
* Last but not least this makes ZeroTier somewhat less centralized by eliminating any hard dependency on ZeroTier, Inc.'s infrastructure.
Our roots will of course remain and continue to provide zero-configuration instant-on deployment, a secure global authority for identities, and free traffic relaying for those who can't establish peer to peer connections.
### Local Configuration
An element of our design philosophy is "features are bugs." This isn't an absolute dogma but more of a guiding principle. We try as hard as we can to avoid adding features, especially "knobs" that must be tweaked by a user.
As of 1.2.0 we've decided that certain knobs are unavoidable, and so there is now a `local.conf` file that can be used to configure them. See the ZeroTier One documentation for these. They include:
* Blacklisting interfaces you want to make sure ZeroTier doesn't use for network traffic, such as VPNs, slow links, or backplanes designated for only certain kinds of traffic.
* Turning uPnP/NAT-PMP on or off.
* Configuring software updates on Windows and Mac platforms.
* Defining trusted paths (the old trusted paths file is now deprecated)
* Setting the ZeroTier main port so it doesn't have to be changed on the command line, which is very inconvenient in many cases.
### Improved In-Band Software Updates
A good software update system for Windows and Mac clients has been a missing feature in previous versions. It does exist but we've been shy about using it so far due to its fragility in some environments.
We've greatly improved this mechanism in 1.2.0. Not only does it now do a better job of actually invoking the update, but it also transfers updates in-band using the ZeroTier protocol. This means it can work in environments that do not allows http/https traffic or that force it through proxies. There's also now an update channel setting: `beta` or `release` (the default).
Software updates are authenticated three ways:
1. ZeroTier's own signing key is used to sign all updates and this signature is checked prior to installation. ZeroTier, Inc.'s signatures are performed on an air-gapped machine.
2. Updates for Mac and Windows are signed using Apple and Microsoft (DigiCert EV) keys and will not install unless these signatures are also valid.
3. The new in-band update mechanism also authenticates the source of the update via ZeroTier's built-in security features. This provides transport security, while 1 and 2 provide security of the update at rest.
Updates are now configurable via `local.conf`. There are three options: `disable`, `download`, and `apply`. The third (apply) is the default for official builds on Windows and Mac, making updates happen silently and automatically as they do for popular browsers like Chrome and Firefox. Updates are disabled by default on Linux and other Unix-type systems as these are typically updated through package managers.
### Path Link Quality Awareness
Version 1.2.0 is now aware of the link quality of direct paths with other 1.2.0 nodes. This information isn't used yet but is visible through the JSON API. (Quality always shows as 100% with pre-1.2.0 nodes.) Quality is measured passively with no additional overhead using a counter based packet loss detection algorithm.
This information is visible from the command line via `listpeers`:
The first peer's path is at 100% (1.00), while the second peer's path is suffering quite a bit of packet loss (0.44).
Link quality awareness is a precursor to intelligent multi-path and QoS support, which will in future versions bring us to feature parity with SD-WAN products like Cisco iWAN.
### Security Improvements
Version 1.2.0 adds anti-DOS (denial of service) rate limits and other hardening for improved resiliency against a number of denial of service attack scenarios.
It also adds a mechanism for instantaneous credential revocation. This can be used to revoke certificates of membership instantly to kick a node off a network (for private networks) and also to revoke capabilities and tags. The new controller sends revocations by default when a peer is de-authorized.
Revocations propagate using a "rumor mill" peer to peer algorithm. This means that a controller need only successfully send a revocation to at least one member of a network with connections to other active members. At this point the revocation will flood through the network peer to peer very quickly. This helps make revocations more robust in the face of poor connectivity with the controller or attempts to incapacitate the controller with denial of service attacks, as well as making revocations faster on huge networks.
### Windows and Macintosh UI Improvements (ZeroTier One)
The Mac has a whole new UI built natively in Objective-C. It provides a pulldown similar in appearance and operation to the Mac WiFi task bar menu.
The Windows UI has also been improved and now provides a task bar icon that can be right-clicked to manage networks. Both now expose managed route and IP permissions, allowing nodes to easily opt in to full tunnel operation if you have a router configured on your network.
### Ad-Hoc Networks
A special kind of public network called an ad-hoc network may be accessed by joining a network ID with the format:
ffSSSSEEEE000000
| | | |
| | | Reserved for future use, must be 0
| | End of port range (hex)
| Start of port range (hex)
Reserved ZeroTier address prefix indicating a controller-less network
Ad-hoc networks are public (no access control) networks that have no network controller. Instead their configuration and other credentials are generated locally. Ad-hoc networks permit only IPv6 UDP and TCP unicast traffic (no multicast or broadcast) using 6plane format NDP-emulated IPv6 addresses. In addition an ad-hoc network ID encodes an IP port range. UDP packets and TCP SYN (connection open) packets are only allowed to destination ports within the encoded range.
For example `ff00160016000000` is an ad-hoc network allowing only SSH, while `ff0000ffff000000` is an ad-hoc network allowing any UDP or TCP port.
Keep in mind that these networks are public and anyone in the entire world can join them. Care must be taken to avoid exposing vulnerable services or sharing unwanted files or other resources.
### Network Controller (Partial) Rewrite
The network controller has been largely rewritten to use a simple in-filesystem JSON data store in place of SQLite, and it is now included by default in all Windows, Mac, Linux, and BSD builds. This means any desktop or server node running ZeroTier One can now be a controller with no recompilation needed.
If you have data in an old SQLite3 controller we've included a NodeJS script in `controller/migrate-sqlite` to migrate data to the new format. If you don't migrate, members will start getting `NOT_FOUND` when they attempt to query for updates.
## Major Bug Fixes in 1.2.0
* **The Windows HyperV 100% CPU bug is FINALLY DEAD**: This long-running problem turns out to have been an issue with Windows itself, but one we were triggering by placing invalid data into the Windows registry. Microsoft is aware of the issue but we've also fixed the triggering problem on our side. ZeroTier should now co-exist quite well with HyperV and should now be able to be bridged with a HyperV virtual switch.
* **Segmentation faults on musl-libc based Linux systems**: Alpine Linux and some embedded Linux systems that use musl libc (a minimal libc) experienced segmentation faults. These were due to a smaller default stack size. A work-around that sets the stack size for new threads has been added.
* **Windows firewall blocks local JSON API**: On some Windows systems the firewall likes to block 127.0.0.1:9993 for mysterious reasons. This is now fixed in the installer via the addition of another firewall exemption rule.
* **UI crash on embedded Windows due to missing fonts**: The MSI installer now ships fonts and will install them if they are not present, so this should be fixed.
## Other Improvements in 1.2.0
* **Improved dead path detection**: ZeroTier is now more aggressive about expiring paths that do not seem to be active. If a path seems marginal it is re-confirmed before re-use.
* **Minor performance improvements**: We've reduced unnecessary memcpy's and made a few other performance improvements in the core.
* **Linux static binaries**: For our official packages (the ones in the download.zerotier.com apt and yum repositories) we now build Linux binaries with static linking. Hopefully this will stop all the bug reports relating to library inconsistencies, as well as allowing our deb packages to run on a wider variety of Debian-based distributions. (There are far too many of these to support officially!) The overhead for this is very small, especially since we built our static versions against musl-libc. Distribution maintainers are of course free to build dynamically linked versions for inclusion into distributions; this only affects our official binaries.
This is mostly for ZeroTier internal use, but others who want to do builds might find it helpful.
Note: Many of these steps will require GPG and other signing keys that are kept in cold storage and must be mounted.
# Bumping the Version and Preparing Installers
The version must be incremented in all of the following files:
/version.h
/zerotier-one.spec
/debian/changelog
/ext/installfiles/mac/ZeroTier One.pkgproj
/ext/installfiles/windows/ZeroTier One.aip
/windows/WinUI/AboutView.xaml
The final .AIP file can only be edited on Windows with [Advanced Installer Enterprise](http://www.advancedinstaller.com/). In addition to incrementing the version be sure that a new product code is generated. (The "upgrade code" GUID on the other hand must never change.)
# Building for Supported Platforms
## Macintosh
Mac's easy. Just type:
make official
You will need [Packages](http://s.sudre.free.fr/Software/Packages/about.html) and our release signing key in the keychain.
## Windows
First load the Visual Studio solution and rebuild the UI and ZeroTier One in both x64 and i386 `Release` mode. Then load [Advanced Installer Enterprise](http://www.advancedinstaller.com/), check that the version is correct, and build. The build will fail if any build artifacts are missing, and Windows must have our product singing key (from DigiCert) available to sign the resulting MSI file. The MSI must then be tested on at least a few different CLEAN Windows VMs to ensure that the installer is valid and properly signed.
You'll need either NodeJS/npm installed (script will then automatically install the npm *marked-man* package) or */usr/bin/ronn*. The latter is a Ruby program packaged on some distributions as *rubygem-ronn* or *ruby-ronn* or installable as *gem install ronn*. The Node *marked-man* package and *ronn* from rubygems are two roughly equivalent alternatives for compiling MarkDown into roff/man format.
This document describes the core components of ZeroTier's cryptographic and security architecture. It focuses primarily on version 2.0 and only briefly touches on v1.x constructions that are being phased out.
The intended audience for this document is developers, auditors, and security professionals wishing to understand ZeroTier's design from a security posture point of view. It's also written to serve as the basis for professional security audits of the ZeroTier protocol and code base.
## High-Level Protocol Design
ZeroTier's protocol is split into two conceptual layers that we term **VL1** and **VL2**.
VL1 stands for *virtual layer 1* and is a cryptographically addressed secure global peer-to-peer network responsible for moving packets between ZeroTier nodes. It's a virtual analogue of the physical wire or radio transciever in an Ethernet or WiFi network respectively. Think of it as a gigantic wire closet for planet Earth.
VL2 stands for *virtual layer 2* and is a full Ethernet emulation layer incorporating cryptographic certificate and token based access control. It is similar (but not identical) to other Ethernet virtualization protocols like VXLAN. VL2 is conceptually separate from VL1 but for the sake of simplicity and ease of use leverages VL1's cryptographic infrastructure for its own authentication needs.
## VL1 Asymmetric Cryptography: Identities, and Addressing
VL1 peers are cryptographically addressed, meaning addresses are strongly bound to public keys. Cryptographic addressing is extremely convenient in peer-to-peer networks as it leverages authenticated (AEAD) encryption to implicity authenticate endpoint addresses.
A ZeroTier identity is comprised of one or more cryptographic public keys and a short **ZeroTier address** derived from a hash of those keys. In addition to this short address there also exists a longer fingerprint in the form of a SHA-384 hash of identity public key(s).
#### Identity Types and Corresponding Algorithms
* **Type 0** (v1.x and v2.x): one Curve25519 key for elliptic curve Diffie-Hellman and one Ed25519 key for Ed25519 signatures, with the address and fingerprint computed from a hash of both.
* **Type 1** (v2.x only): Curve25519, Ed25519, and NIST P-384 public keys, with the latter being used for signatures (the Ed25519 key is still there but is presently unused) and with *both* Curve25519 and NIST P-384 being used for elliptic curve Diffie-Hellman key agreement. In key agreement the resulting raw secret keys are hashed together using SHA-384 to combine them and yield a single session key.
Session keys resulting from identity key exchange and agreement are *long-lived keys* that remain static for the lifetime of a particular pair of identities. A different mechanism is used for ephemeral key negotiation.
#### ZeroTier Addresses and Identity Fingerprints
In the simplest form of cryptographic addressing, keys are used directly as addresses throughout the system. Unfortunately even public key cryptosystems with short keys like Curve25519 still result in string representations that are prohibitively long for human beings to type. ZeroTier mitigates this usability problem by using a short hash of the public key termed a **ZeroTier address** to refer to a peer's full identity. This short address is also used at the wire level to reduce the size of the packet header. Peers may request full identities based on addresses from from root servers.
ZeroTier addresses are very short: only 40 bits or 10 hexadecimal digits, e.g. `89e92ceee5.` This makes them convenient to type, but such a short hash would in a naive implementation introduce a significant risk that an attacker could create a duplicate identity with a different key pair but the same address. With 40 bits an intentional collision would require only an average of about 549,755,813,888 attempts for a 50% chance of colliding. If an attempt requires 0.5ms of CPU time on a typical contemporary desktop or server CPU, this would require about 3,000 CPU-days. Since this type of search is easy to parallelize, it would take only a few days for someone with access to a few thousand CPU cores.
To provide this short hash with a larger security margin, an intentionally slow one-way "hashcash" or "proof of work" function is required during identity generation. This work function is slow to compute but fast to verify, and an address is not valid unless its work checks out. This gives identity address derivation the following costs:
* Type 1 identities: an average of about 500ms per key pair per typical 2.4ghz CPU core, requiring around 3 million CPU-days to reach a 50% collision probability.
* Type 2 identities: an average of about one second per key pair per typical 2.4ghz CPU core, requiring around 6.3 million CPU-days to reach a 50% collision probability.
While too costly for the vast majority of attackers, this cost may not be prohibitive to a nation-state level attacker or to a criminal with significant funds and/or access to a very large "botnet." It's also possible that FPGA, GPU, or ASIC acceleration could be leveraged to decrease this time in a manner similar to what's been accomplished in the area of cryptocurrency mining.
Fingerprints are full SHA-384 hashes of identity public keys. In base32-encoding they look like this:
These are too large to type but not to copy/paste, store in databases, or use in scripts and APIs.
Once a device has joined a network, network controllers will remember and check its full identity or identity fingerprint (depending on implementation) rather than just the device's ZeroTier address.
## VL1 Wire Protocol
ZeroTier's wire protocol is packet based with packets having the following format:
```
[0:8] 64-bit packet ID and cryptographic nonce
[8:13] 40-bit destination ZeroTier address
[13:18] 40-bit source ZeroTier address
[18:19] 8-bit cleartext flags, cipher, and hop count (bits: FFCCCHHH)
[19:27] 64-bit message authentication code (MAC)
-- BEGIN ENCRYPTED SECTION --
[27:28] 8-bit inner flags and 5-bit protocol verb (bits: FFFVVVVV)
[28:...] Verb-specific packet payload
```
All fields (both those that remain cleartext and those that are encrypted) in a packet are authenticated except for the last three "hops" bits of the combined flags/cipher/hops field. These are masked to zero during MAC computation and verification. This is because the hops field is the only field that can be modified by third party peers in transit. It's incremented whenever a packet is forwarded by a root server or connectivity-assisting peer and is checked against a limit to prevent infinite forwarding loops.
Packets can be up to 16,384 bytes in size. Since the most common transport is UDP and this transport does not reliably support fragmentation, ZeroTier implements its own packet fragmentation and re-assembly scheme using fragments with the following wire format:
```
[0:8] 64-bit packet ID of packet of which this is a fragment
[8:13] 40-bit destination ZeroTier address
[13:14] 0xff here indicates a fragment since addresses cannot start with this byte
[14:15] 4-bit total fragments and 4-bit fragment number (bits: TTTTNNNN)
[15:16] 5 reserved bits, 3-bit hop count (bits: rrrrrHHH)
[16:...] Fragment data
```
A fragmented packet is indicated by the presence of the flag 0x40 in its cleartext flags field. If this flag is present the receiver must expect the receipt of one or more fragments in addition to the packet's header and first fragment. The total number of fragments expected is not contained in the header but will be contained within each subsequent fragment. If a fragment is received prior to its head, it's held in the event that its head arrives as the protocol does support out of order receipt of fragments.
Fragmentation can be effectively ignored from a security point of view (with the exception of denial of service concerns, which are mitigated by way of limits and heuristics in the code) since packet message authentication codes are checked at the packet level. Any improperly fragmented packet will fail cryptographic MAC check and be discarded.
*Legacy: In v1.x the packet ID and nonce field was assigned from a counter maintained to avoid duplicate nonce assignment and the MAC field was the first 64 bits of a Poly1305 MAC of the packet. The overall construction was identical in form to the NaCl Salsa20/Poly1305 "secret box" construction in which the first 32 bytes of Salsa20 output are used as a one-time Poly1305 key for each packet.*
In v2.x the packet ID and MAC field are in reality a single split 128-bit encrypted nonce and MAC field. See AES-GMAC-SIV below.
## Symmetric Encryption: AES-GMAC-SIV
**This is a draft and may change based on peer review and feedback.**
In v1.x there is a risk of nonce re-use due in part to the small size of the MAC and in part to the way ZeroTier is used. More specifically the risk arises when ZeroTier VMs are cloned or ZeroTier is used on small devices that have the potential to lack both accurate timekeeping and native strong random sources.
Salsa20 was used in v1.x since at the time the protocol was initially designed AES acceleration was not available on most mobile phones, embedded chips, and small ARM processors such as those use on Raspberry Pi and similar devices. This is no longer the case.
For v2.x our design has three objectives:
- Make use of FIPS140-compliant cryptographic primitives that would be available in a FIPS140-ceritified library (e.g. a FIPS build of crypto++ or openssl libcrypto), and use them in a way that could pass FIPS/NIST/NSA review.
- Use AES with hardware acceleration for extremely high performance processors with AES hardware acceleration, which is most non-trivial CPUs today.
- Incorporate some form of nonce-reuse-resistance to reduce the risk of duplicate nonces when virtual machines are cloned or on small devices, and to mitigate the short MAC.
The proposed AES-GMAC-SIV construction attempts to achieve all these objectives by using GMAC combined with AES-CTR (both FIPS140 primitives) in a way that achieves the security bounds and characteristics of AES-GCM-SIV but could be certified as FIPS compliant. The design is almost identical to another proposed mode called AES-GCM-SIV except that GMAC is used "as-is" for FIPS-certifiability reasons.
#### AES-GMAC-SIV Session Setup
For each new session key, derive two sub-keys **K0** and **K1** using a key derivation function such as KBKDF-HMAC-SHA384.
#### AES-GMAC-SIV Encryption
<imgsrc="AES-GMAC-SIV.png">
As with all other SIV (synthetic IV) modes of operation, encryption requires two passes. Since messages are small in our system it's very likely that the second pass would be operating on data already in CPU L0 cache, reducing the additional overhead of this two-pass requirement.
1. Generate a new unique 64-bit packet ID in the same manner as v1.x.
2. Expand this 64-bit ID to a 96-bit AES-GMAC nonce by padding the remainder with the size of the packet in bytes and the direction of communication (sender > recipient or recipient > sender). This adds a small amount of additional entropy taken from characteristics of the packet.
3. Using session sub-key **K0** compute AES-GMAC(K0, plaintext) to yield a 128-bit GMAC tag.
4. Take the first 64 bits of this 128-bit GMAC tag and append it to the 64-bit packet ID generated in step 1 to yield a 128-bit combined nonce+MAC field.
5. Obtain a 128-bit AES-CTR nonce by encrypting this 128-bit combined nonce+MAC field as a single AES block using **K1**. This is done because GMAC alone is not a cryptographic PRF (pseudo-random function) and we want to ensure that we destroy any algebraic structure before using it with AES-CTR.
6. Using the encrypted nonce+MAC field as a 128-bit nonce, encrypt the encrypted section of the packet with AES-CTR. This is also done using **K1**, meaning that the first block of CTR padding data is actually AES(K1,AES(K1,nonce+MAC)).
7. Encrypt the 128-bit AES-CTR nonce again as a single AES block using **K0** to yield a final encrypted 128-bit combined nonce and MAC. *(Question for peer review: does this step have any attack-mitigating value? AES-CTR does not require that its nonce/IV be a secret.)*
8. Split this final encrypted nonce+IV into two 64-bit chunks, replacing the packet ID with one and placing the other in the packet MAC field.
#### AES-GMAC-SIV Decryption
Unlike encryption, SIV decryption can be performed in a single pass if there is a performance benefit to doing so.
1. Re-combine the packet ID and MAC fields into a single 128-bit block.
2. Decrypt this block with AES using **K0** to yield the AES-CTR nonce.
3. Decrypt the packet using AES-CTR with **K1**.
4. Decrypt the 128-bit AES-CTR nonce field *again* as a single AES block using **K1** to obtain the original packet nonce and 64-bit truncated GMAC tag.
5. Expand the 64-bit packet ID / nonce into a 96-bit GMAC nonce as in encryption step 2.
6. Compute AES-GMAC(K0,plaintext) as in encryption step 3.
7. Verify that the first 64 bytes of the resulting GMAC tag equals the tag (last 64 bits) obtained in decryption step 4 and discard the packet if they do not match.
#### Discussion
Most standard stream cipher modes such as AES-GCM or Salsa20/Poly1305 require that message nonce/IV values are never duplicated for the same session key. Since these stream modes generate key streams that are simply XORed with message plaintext, nonce duplication reveals the plaintext of both messages for which the nonce is duplicated due to the commutativity of the XOR operation. It may also allow the MAC (GMAC or Poly1305) itself to be attacked in such a way as to enable message forgery.
SIV modes mitigate these attacks by making the actual cryptographic nonce used for stream encryption dependent on the content of the message. If a nonce is repeated when two messages differ, ciphertext will still be unique unless a MAC collision also occurs. The chance of this is quite small, only 1/2^64 in our system for any given pair of repeated nonce values. If a repeated nonce occurs and both messages are the same, the protocol will leak only the fact that a message was repeated. The actual plaintext and MAC are not compromised.
Our AES-GMAC-SIV mode is almost identical to a proposed mode called [AES-GCM-SIV](https://cyber.biu.ac.il/aes-gcm-siv/). The proposed AES-GCM-SIV mode uses a variant of GMAC called POLYVAL with very minor performance improvements while ours retains standard GMAC for compatibility with existing standards and libraries. We call our mode AES-GMAC-SIV to distinguish it.
*Question for peer review: both GMAC and AES-CTR are FIPS140 approved primitives, and the use of AES-CTR with an approved MAC is permitted. Is it actually feasible that this could be FIPS certified if it were documented in a correct and "strategic" way? It would be described as GMAC authenticated AES-CTR with the CTR IV being constructed via keyed hash (AES) from an initial plaintext IV and a "salt" taken from the MAC, or some similar description.*
\fBzerotier\-cli\fR provides a simple command line interface to the local JSON API of the ZeroTier virtual network endpoint service zerotier\-one(8)\.
.P
By default \fBzerotier\-cli\fR must be run as root or with \fBsudo\fP\|\. If you want to allow an unprivileged user to use \fBzerotier\-cli\fR to control the system ZeroTier service, you can create a local copy of the ZeroTier service authorization token in the user's home directory:
(The location of ZeroTier's service home may differ by platform\. See zerotier\-one(8)\.)
.P
Note that this gives the user the power to connect or disconnect the system to or from any virtual network, which is a significant permission\.
.P
\fBzerotier\-cli\fR has several command line arguments that are visible in \fBhelp\fP output\. The two most commonly used are \fB\-j\fP for raw JSON output and \fB\-D<path>\fP to specify an alternative ZeroTier service working directory\. Raw JSON output is easier to parse in scripts and also contains verbose details not present in the tabular output\. The \fB\-D<path>\fP option specifies where the service's zerotier\-one\.port and authtoken\.secret files are located if the service is not running at the default location for your system\.
.SHCOMMANDS
.RS0
.IP\(bu2
\fBhelp\fP:
Displays \fBzerotier\-cli\fR help\.
.IP\(bu2
\fBinfo\fP:
Shows information about this device including its 10\-digit ZeroTier address and apparent connection status\. Use \fB\-j\fP for more verbose output\.
.IP\(bu2
\fBlistpeers\fP:
This command lists the ZeroTier VL1 (virtual layer 1, the peer to peer network) peers this service knows about and has recently (within the past 30 minutes or so) communicated with\. These are not necessarily all the devices on your virtual network(s), and may also include a few devices not on any virtual network you've joined\. These are typically either root servers or network controllers\.
.IP\(bu2
\fBlistnetworks\fP:
This lists the networks your system belongs to and some information about them, such as any ZeroTier\-managed IP addresses you have been assigned\. (IP addresses assigned manually to ZeroTier interfaces will not be listed here\. Use the standard network interface commands to see these\.)
.IP\(bu2
\fBjoin\fP:
To join a network just use \fBjoin\fP and its 16\-digit hex network ID\. That's it\. Then use \fBlistnetworks\fP to see the status\. You'll either get a reply from the network controller with a certificate and other info such as IP assignments, or you'll get "access denied\." In this case you'll need the administrator of this network to authorize your device by its 10\-digit device ID (visible with \fBinfo\fP) on the network's controller\.
.IP\(bu2
\fBleave\fP:
Leaving a network is as easy as joining it\. This disconnects from the network and deletes its interface from the system\. Note that peers on the network may hang around in \fBlistpeers\fP for up to 30 minutes until they time out due to lack of traffic\. But if they no longer share a network with you, they can't actually communicate with you in any meaningful way\.
.RE
.SHEXAMPLES
.P
Join "Earth," ZeroTier's big public party line network:
.P
.RS2
.nf
$ sudo zerotier\-cli join 8056c2e21c000001
$ sudo zerotier\-cli listnetworks
( wait until you get an Earth IP )
$ ping earth\.zerotier\.net
( you should now be able to ping our Earth test IP )
**zerotier-cli** provides a simple command line interface to the local JSON API of the ZeroTier virtual network endpoint service zerotier-one(8).
By default **zerotier-cli** must be run as root or with `sudo`. If you want to allow an unprivileged user to use **zerotier-cli** to control the system ZeroTier service, you can create a local copy of the ZeroTier service authorization token in the user's home directory:
(The location of ZeroTier's service home may differ by platform. See zerotier-one(8).)
Note that this gives the user the power to connect or disconnect the system to or from any virtual network, which is a significant permission.
**zerotier-cli** has several command line arguments that are visible in `help` output. The two most commonly used are `-j` for raw JSON output and `-D<path>` to specify an alternative ZeroTier service working directory. Raw JSON output is easier to parse in scripts and also contains verbose details not present in the tabular output. The `-D<path>` option specifies where the service's zerotier-one.port and authtoken.secret files are located if the service is not running at the default location for your system.
## COMMANDS
*`help`:
Displays **zerotier-cli** help.
*`info`:
Shows information about this device including its 10-digit ZeroTier address and apparent connection status. Use `-j` for more verbose output.
*`listpeers`:
This command lists the ZeroTier VL1 (virtual layer 1, the peer to peer network) peers this service knows about and has recently (within the past 30 minutes or so) communicated with. These are not necessarily all the devices on your virtual network(s), and may also include a few devices not on any virtual network you've joined. These are typically either root servers or network controllers.
*`listnetworks`:
This lists the networks your system belongs to and some information about them, such as any ZeroTier-managed IP addresses you have been assigned. (IP addresses assigned manually to ZeroTier interfaces will not be listed here. Use the standard network interface commands to see these.)
*`join`:
To join a network just use `join` and its 16-digit hex network ID. That's it. Then use `listnetworks` to see the status. You'll either get a reply from the network controller with a certificate and other info such as IP assignments, or you'll get "access denied." In this case you'll need the administrator of this network to authorize your device by its 10-digit device ID (visible with `info`) on the network's controller.
*`leave`:
Leaving a network is as easy as joining it. This disconnects from the network and deletes its interface from the system. Note that peers on the network may hang around in `listpeers` for up to 30 minutes until they time out due to lack of traffic. But if they no longer share a network with you, they can't actually communicate with you in any meaningful way.
## EXAMPLES
Join "Earth," ZeroTier's big public party line network:
$ sudo zerotier-cli join 8056c2e21c000001
$ sudo zerotier-cli listnetworks
( wait until you get an Earth IP )
$ ping earth.zerotier.net
( you should now be able to ping our Earth test IP )
Leave "Earth":
$ sudo zerotier-cli leave 8056c2e21c000001
List VL1 peers:
$ sudo zerotier-cli listpeers
## COPYRIGHT
(c)2011-2016 ZeroTier, Inc. -- https://www.zerotier.com/ -- https://github.com/zerotier
\fBzerotier-idtool\fR\- tool for creating and manipulating ZeroTier identities
.SHSYNOPSIS
.P
\fBzerotier\-idtool\fP <command> [args]
.SHDESCRIPTION
.P
\fBzerotier\-idtool\fR is a command line utility for doing things with ZeroTier identities\. A ZeroTier identity consists of a public/private key pair (or just the public if it's only an identity\.public) and a 10\-digit hexadecimal ZeroTier address derived from the public key by way of a proof of work based hash function\.
.SHCOMMANDS
.P
When command arguments call for a public or secret (full) identity, the identity can be specified as a path to a file or directly on the command line\.
.RS0
.IP\(bu2
\fBhelp\fP:
Display help\. (Also running with no command does this\.)
Generate a new ZeroTier identity\. If a secret file is specified, the full identity including the private key will be written to this file\. If the public file is specified, the public portion will be written there\. If no file paths are specified the full secret identity is output to STDOUT\. The vanity prefix is a series of hexadecimal digits that the generated identity's address should start with\. Typically this isn't used, and if it's specified generation can take a very long time due to the intrinsic cost of generating identities with their proof of work function\. Generating an identity with a known 16\-bit (4 digit) prefix on a 2\.8ghz Core i5 (using one core) takes an average of two hours\.
.IP\(bu2
\fBvalidate\fP <identity, only public part required>:
Locally validate an identity's key and proof of work function correspondence\.
.IP\(bu2
\fBgetpublic\fP <full identity with secret>:
Extract the public portion of an identity\.secret and print to STDOUT\.
.IP\(bu2
\fBsign\fP <full identity with secret> <file to sign>:
Sign a file's contents with SHA512+ECC\-256 (ed25519)\. The signature is output in hex to STDOUT\.
.IP\(bu2
\fBverify\fP <identity, only public part required> <file to check> <signature in hex>:
Verify a signature created with \fBsign\fP\|\.
.IP\(bu2
\fBmkcom\fP <full identity with secret> [id,value,maxdelta] [\|\.\.\.]:
Create and sign a network membership certificate\. This is not generally useful since network controllers do this automatically and is included mostly for testing purposes\.
.RE
.SHEXAMPLES
.P
Generate and dump a new identity:
.P
.RS2
.nf
$ zerotier\-idtool generate
.fi
.RE
.P
Generate and write a new identity, both secret and public parts:
**zerotier-idtool** is a command line utility for doing things with ZeroTier identities. A ZeroTier identity consists of a public/private key pair (or just the public if it's only an identity.public) and a 10-digit hexadecimal ZeroTier address derived from the public key by way of a proof of work based hash function.
## COMMANDS
When command arguments call for a public or secret (full) identity, the identity can be specified as a path to a file or directly on the command line.
*`help`:
Display help. (Also running with no command does this.)
*`generate` [secret file] [public file] [vanity]:
Generate a new ZeroTier identity. If a secret file is specified, the full identity including the private key will be written to this file. If the public file is specified, the public portion will be written there. If no file paths are specified the full secret identity is output to STDOUT. The vanity prefix is a series of hexadecimal digits that the generated identity's address should start with. Typically this isn't used, and if it's specified generation can take a very long time due to the intrinsic cost of generating identities with their proof of work function. Generating an identity with a known 16-bit (4 digit) prefix on a 2.8ghz Core i5 (using one core) takes an average of two hours.
*`validate`<identity,onlypublicpartrequired>:
Locally validate an identity's key and proof of work function correspondence.
*`getpublic`<fullidentitywithsecret>:
Extract the public portion of an identity.secret and print to STDOUT.
*`sign`<fullidentitywithsecret><filetosign>:
Sign a file's contents with SHA512+ECC-256 (ed25519). The signature is output in hex to STDOUT.
Create and sign a network membership certificate. This is not generally useful since network controllers do this automatically and is included mostly for testing purposes.
## EXAMPLES
Generate and dump a new identity:
$ zerotier-idtool generate
Generate and write a new identity, both secret and public parts:
\fBzerotier\-one\fR is the service/daemon responsible for connecting a Unix (Linux/BSD/OSX) system to one or more ZeroTier virtual networks and presenting those networks to the system as virtual network ports\. You can think of it as a peer to peer VPN client\.
.P
It's typically run by init systems like systemd (Linux) or launchd (Mac) rather than directly by the user, and it must be run as root unless you give it the \fB\-U\fP switch and don't plan on actually joining networks (e\.g\. to run a network controller microservice only)\.
.P
The \fBzerotier\-one\fR service keeps its state and other files in a working directory\. If this directory is not specified at launch it defaults to "/var/lib/zerotier\-one" on Linux, "/Library/Application Support/ZeroTier/One" on Mac, and "/var/db/zerotier\-one" on FreeBSD and other similar BSDs\. The working directory should persist\. It shouldn't be automatically cleaned by system cleanup daemons or stored in a volatile location\. Loss of its identity\.secret file results in loss of this system's unique 10\-digit ZeroTier address and key\.
.P
Multiple instances of \fBzerotier\-one\fR can be run on the same system as long as they are run with different primary ports (see switches) and a different working directory\. But since a single service can join any number of networks, typically there's no point in doing this\.
.P
The \fBzerotier\-one\fR service is controlled via a JSON API available at 127\.0\.0\.1:<primary port> with the default primary port being 9993\. Access to this API requires an authorization token normally found in the authtoken\.secret file in the service's working directory\. On some platforms access may be guarded by other measures such as socket peer UID/GID lookup if additional security options are enabled (this is not the default)\.
.P
The first time the service is started in a fresh working directory, it generates a ZeroTier identity\. On slow systems this process can take ten seconds or more due to an anti\-DDOS/anti\-counterfeit proof of work function used by ZeroTier in address generation\. This only happens once, and once generated the result is saved in identity\.secret in the working directory\. This file represents and defines/claims your ZeroTier address and associated ECC\-256 key pair\.
.SHSWITCHES
.RS0
.IP\(bu2
\fB\-h\fP:
Display help\.
.IP\(bu2
\fB\-v\fP:
Display ZeroTier One version\.
.IP\(bu2
\fB\-U\fP:
Skip privilege check and allow to be run by non\-privileged user\. This is typically used when \fBzerotier\-one\fR is built with the network controller option included\. In this case the ZeroTier service might only be acting as a network controller and might never actually join networks, in which case it does not require elevated system permissions\.
.IP\(bu2
\fB\-p<port>\fP:
Specify a different primary port\. If this is not given the default is 9993\. If zero is given a random port is chosen each time\.
.IP\(bu2
\fB\-d\fP:
Fork and run as a daemon\.
.IP\(bu2
\fB\-i\fP:
Invoke the \fBzerotier\-idtool\fR personality, in which case the binary behaves like zerotier\-idtool(1)\. This happens automatically if the name of the binary (or a symlink to it) is zerotier\-idtool\.
.IP\(bu2
\fB\-q\fP:
Invoke the \fBzerotier\-cli\fR personality, in which case the binary behaves like zerotier\-cli(1)\. This happens automatically if the name of the binary (or a symlink to it) is zerotier\-cli\.
.RE
.SHEXAMPLES
.P
Run as daemon with OS default working directory and default port:
.P
.RS2
.nf
$ sudo zerotier\-one \-d
.fi
.RE
.P
Run as daemon with a different working directory and port:
These are found in the service's working directory\.
.RS0
.IP\(bu2
\fBidentity\.public\fP:
The public portion of your ZeroTier identity, which is your 10\-digit hex address and the associated public key\.
.IP\(bu2
\fBidentity\.secret\fP:
Your full ZeroTier identity including its private key\. This file identifies the system on the network, which means you can move a ZeroTier address around by copying this file and you should back up this file if you want to save your system's static ZeroTier address\. This file must be protected, since theft of its secret key will allow anyone to impersonate your device on any network and decrypt traffic\. For network controllers this file is particularly sensitive since it constitutes the private key for a certificate authority for the controller's networks\.
.IP\(bu2
\fBauthtoken\.secret\fP:
The secret token used to authenticate requests to the service's local JSON API\. If it does not exist it is generated from a secure random source on service start\. To use, send it in the "X\-ZT1\-Auth" header with HTTP requests to 127\.0\.0\.1:<primary port>\|\.
.IP\(bu2
\fBdevicemap\fP:
Remembers mappings of zt# interface numbers to ZeroTier networks so they'll persist across restarts\. On some systems that support longer interface names that can encode the network ID (such as FreeBSD) this file may not be present\.
.IP\(bu2
\fBzerotier\-one\.pid\fP:
ZeroTier's PID\. This file is deleted on normal shutdown\.
.IP\(bu2
\fBzerotier\-one\.port\fP:
ZeroTier's primary port, which is also where its JSON API is found at 127\.0\.0\.1:<this port>\|\. This file is created on startup and is read by zerotier\-cli(1) to determine where it should find the control API\.
.IP\(bu2
\fBcontroller\.db\fP:
If the ZeroTier One service is built with the network controller enabled, this file contains the controller's SQLite3 database\.
.IP\(bu2
\fBcontroller\.db\.backup\fP:
If the ZeroTier One service is built with the network controller enabled, it periodically backs up its controller\.db database in this file (currently every 5 minutes if there have been changes)\. Since this file is not a currently in use SQLite3 database it's safer to back up without corruption\. On new backups the file is rotated out rather than being rewritten in place\.
.IP\(bu2
\fBiddb\.d/\fP (directory):
Caches the public identity of every peer ZeroTier has spoken with in the last 60 days\. This directory and its contents can be deleted, but this may result in slower connection initations since it will require that we go out and re\-fetch full identities for peers we're speaking to\.
.IP\(bu2
\fBnetworks\.d\fP (directory):
This caches network configurations and certificate information for networks you belong to\. ZeroTier scans this directory for <network ID>\|\.conf files on startup to recall its networks, so "touch"ing an empty <network ID>\|\.conf file in this directory is a way of pre\-configuring ZeroTier to join a specific network on startup without using the API\. If the config file is empty ZeroTIer will just fetch it from the network's controller\.
**zerotier-one** is the service/daemon responsible for connecting a Unix (Linux/BSD/OSX) system to one or more ZeroTier virtual networks and presenting those networks to the system as virtual network ports. You can think of it as a peer to peer VPN client.
It's typically run by init systems like systemd (Linux) or launchd (Mac) rather than directly by the user, and it must be run as root unless you give it the `-U` switch and don't plan on actually joining networks (e.g. to run a network controller microservice only).
The **zerotier-one** service keeps its state and other files in a working directory. If this directory is not specified at launch it defaults to "/var/lib/zerotier-one" on Linux, "/Library/Application Support/ZeroTier/One" on Mac, and "/var/db/zerotier-one" on FreeBSD and other similar BSDs. The working directory should persist. It shouldn't be automatically cleaned by system cleanup daemons or stored in a volatile location. Loss of its identity.secret file results in loss of this system's unique 10-digit ZeroTier address and key.
Multiple instances of **zerotier-one** can be run on the same system as long as they are run with different primary ports (see switches) and a different working directory. But since a single service can join any number of networks, typically there's no point in doing this.
The **zerotier-one** service is controlled via a JSON API available at 127.0.0.1:<primaryport> with the default primary port being 9993. Access to this API requires an authorization token normally found in the authtoken.secret file in the service's working directory. On some platforms access may be guarded by other measures such as socket peer UID/GID lookup if additional security options are enabled (this is not the default).
The first time the service is started in a fresh working directory, it generates a ZeroTier identity. On slow systems this process can take ten seconds or more due to an anti-DDOS/anti-counterfeit proof of work function used by ZeroTier in address generation. This only happens once, and once generated the result is saved in identity.secret in the working directory. This file represents and defines/claims your ZeroTier address and associated ECC-256 key pair.
## SWITCHES
*`-h`:
Display help.
*`-v`:
Display ZeroTier One version.
*`-U`:
Skip privilege check and allow to be run by non-privileged user. This is typically used when **zerotier-one** is built with the network controller option included. In this case the ZeroTier service might only be acting as a network controller and might never actually join networks, in which case it does not require elevated system permissions.
*`-p<port>`:
Specify a different primary port. If this is not given the default is 9993. If zero is given a random port is chosen each time.
*`-d`:
Fork and run as a daemon.
*`-i`:
Invoke the **zerotier-idtool** personality, in which case the binary behaves like zerotier-idtool(1). This happens automatically if the name of the binary (or a symlink to it) is zerotier-idtool.
*`-q`:
Invoke the **zerotier-cli** personality, in which case the binary behaves like zerotier-cli(1). This happens automatically if the name of the binary (or a symlink to it) is zerotier-cli.
## EXAMPLES
Run as daemon with OS default working directory and default port:
$ sudo zerotier-one -d
Run as daemon with a different working directory and port:
These are found in the service's working directory.
*`identity.public`:
The public portion of your ZeroTier identity, which is your 10-digit hex address and the associated public key.
*`identity.secret`:
Your full ZeroTier identity including its private key. This file identifies the system on the network, which means you can move a ZeroTier address around by copying this file and you should back up this file if you want to save your system's static ZeroTier address. This file must be protected, since theft of its secret key will allow anyone to impersonate your device on any network and decrypt traffic. For network controllers this file is particularly sensitive since it constitutes the private key for a certificate authority for the controller's networks.
*`authtoken.secret`:
The secret token used to authenticate requests to the service's local JSON API. If it does not exist it is generated from a secure random source on service start. To use, send it in the "X-ZT1-Auth" header with HTTP requests to 127.0.0.1:<primaryport>.
*`devicemap`:
Remembers mappings of zt# interface numbers to ZeroTier networks so they'll persist across restarts. On some systems that support longer interface names that can encode the network ID (such as FreeBSD) this file may not be present.
*`zerotier-one.pid`:
ZeroTier's PID. This file is deleted on normal shutdown.
*`zerotier-one.port`:
ZeroTier's primary port, which is also where its JSON API is found at 127.0.0.1:<thisport>. This file is created on startup and is read by zerotier-cli(1) to determine where it should find the control API.
*`controller.db`:
If the ZeroTier One service is built with the network controller enabled, this file contains the controller's SQLite3 database.
*`controller.db.backup`:
If the ZeroTier One service is built with the network controller enabled, it periodically backs up its controller.db database in this file (currently every 5 minutes if there have been changes). Since this file is not a currently in use SQLite3 database it's safer to back up without corruption. On new backups the file is rotated out rather than being rewritten in place.
*`iddb.d/` (directory):
Caches the public identity of every peer ZeroTier has spoken with in the last 60 days. This directory and its contents can be deleted, but this may result in slower connection initations since it will require that we go out and re-fetch full identities for peers we're speaking to.
*`networks.d` (directory):
This caches network configurations and certificate information for networks you belong to. ZeroTier scans this directory for <networkID>.conf files on startup to recall its networks, so "touch"ing an empty <networkID>.conf file in this directory is a way of pre-configuring ZeroTier to join a specific network on startup without using the API. If the config file is empty ZeroTIer will just fetch it from the network's controller.
## COPYRIGHT
(c)2011-2016 ZeroTier, Inc. -- https://www.zerotier.com/ -- https://github.com/zerotier
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.