ZeroTierOne/node/Packet.cpp
Adam Ierymenko 0e5651f353
1.12.0 merge to main (#2104)
* add note about forceTcpRelay

* Create a sample systemd unit for tcp proxy

* set gitattributes for rust & cargo so hashes dont conflict on Windows

* Revert "set gitattributes for rust & cargo so hashes dont conflict on Windows"

This reverts commit 032dc5c108.

* Turn off autocrlf for rust source

Doesn't appear to play nice well when it comes to git and vendored cargo package hashes

* Fix #1883 (#1886)

Still unknown as to why, but the call to `nc->GetProperties()` can fail
when setting a friendly name on the Windows virtual ethernet adapter.
Ensure that `ncp` is not null before continuing and accessing the device
GUID.

* Don't vendor packages for zeroidc (#1885)

* Added docker environment way to join networks (#1871)

* add StringUtils

* fix headers
use recommended headers and remove unused headers

* move extern "C"
only JNI functions need to be exported

* cleanup

* fix ANDROID-50: RESULT_ERROR_BAD_PARAMETER typo

* fix typo in log message

* fix typos in JNI method signatures

* fix typo

* fix ANDROID-51: fieldName is uninitialized

* fix ANDROID-35: memory leak

* fix missing DeleteLocalRef in loops

* update to use unique error codes

* add GETENV macro

* add LOG_TAG defines

* ANDROID-48: add ZT_jnicache.cpp

* ANDROID-48: use ZT_jnicache.cpp and remove ZT_jnilookup.cpp and ZT_jniarray.cpp

* add Event.fromInt

* add PeerRole.fromInt

* add ResultCode.fromInt

* fix ANDROID-36: issues with ResultCode

* add VirtualNetworkConfigOperation.fromInt

* fix ANDROID-40: VirtualNetworkConfigOperation out-of-sync with ZT_VirtualNetworkConfigOperation enum

* add VirtualNetworkStatus.fromInt

* fix ANDROID-37: VirtualNetworkStatus out-of-sync with ZT_VirtualNetworkStatus enum

* add VirtualNetworkType.fromInt

* make NodeStatus a plain data class

* fix ANDROID-52: synchronization bug with nodeMap

* Node init work: separate Node construction and init

* add Node.toString

* make PeerPhysicalPath a plain data class

* remove unused PeerPhysicalPath.fixed

* add array functions

* make Peer a plain data class

* make Version a plain data class

* fix ANDROID-42: copy/paste error

* fix ANDROID-49: VirtualNetworkConfig.equals is wrong

* reimplement VirtualNetworkConfig.equals

* reimplement VirtualNetworkConfig.compareTo

* add VirtualNetworkConfig.hashCode

* make VirtualNetworkConfig a plain data class

* remove unused VirtualNetworkConfig.enabled

* reimplement VirtualNetworkDNS.equals

* add VirtualNetworkDNS.hashCode

* make VirtualNetworkDNS a plain data class

* reimplement VirtualNetworkRoute.equals

* reimplement VirtualNetworkRoute.compareTo

* reimplement VirtualNetworkRoute.toString

* add VirtualNetworkRoute.hashCode

* make VirtualNetworkRoute a plain data class

* add isSocketAddressEmpty

* add addressPort

* add fromSocketAddressObject

* invert logic in a couple of places and return early

* newInetAddress and newInetSocketAddress work
allow newInetSocketAddress to return NULL if given empty address

* fix ANDROID-38: stack corruption in onSendPacketRequested

* use GETENV macro

* JniRef work
JniRef does not use callbacks struct, so remove
fix NewGlobalRef / DeleteGlobalRef mismatch

* use PRId64 macros

* switch statement work

* comments and logging

* Modifier 'public' is redundant for interface members

* NodeException can be made a checked Exception

* 'NodeException' does not define a 'serialVersionUID' field

* 'finalize()' should not be overridden
this is fine to do because ZeroTierOneService calls close() when it is done

* error handling, error reporting, asserts, logging

* simplify loadLibrary

* rename Node.networks -> Node.networkConfigs

* Windows file permissions fix (#1887)

* Allow macOS interfaces to use multiple IP addresses (#1879)

Co-authored-by: Sean OMeara <someara@users.noreply.github.com>
Co-authored-by: Grant Limberg <glimberg@users.noreply.github.com>

* Fix condition where full HELLOs might not be sent when necessary (#1877)

Co-authored-by: Grant Limberg <glimberg@users.noreply.github.com>

* 1.10.4 version bumps

* Add security policy to repo (#1889)

* [+] add e2k64 arch (#1890)

* temp fix for ANDROID-56: crash inside newNetworkConfig from too many args

* 1.10.4 release notes

* Windows 1.10.4 Advanced Installer bump

* Revert "temp fix for ANDROID-56: crash inside newNetworkConfig from too many args"

This reverts commit dd627cd7f4.

* actual fix for ANDROID-56: crash inside newNetworkConfig
cast all arguments to varargs functions as good style

* Fix addIp being called with applied ips (#1897)

This was getting called outside of the check for existing ips
Because of the added ifdef and a brace getting moved to the
wrong place.

```
if (! n.tap()->addIp(*ip)) {
	fprintf(stderr, "ERROR: unable to add ip address %s" ZT_EOL_S, ip->toString(ipbuf));
}
WinFWHelper::newICMPRule(*ip, n.config().nwid);

```

* 1.10.5 (#1905)

* 1.10.5 bump

* 1.10.5 for Windows

* 1.10.5

* Prevent path-learning loops (#1914)

* Prevent path-learning loops

* Only allow new overwrite if not bonded

* fix binding temporary ipv6 addresses on macos (#1910)

The check code wasn't running.

I don't know why !defined(TARGET_OS_IOS) would exclude code on
desktop macOS. I did a quick search and changed it to defined(TARGET_OS_MAC).
Not 100% sure what the most correct solution there is.

You can verify the old and new versions with

`ifconfig | grep temporary`

plus

`zerotier-cli info -j` -> listeningOn

* 1.10.6 (#1929)

* 1.10.5 bump

* 1.10.6

* 1.10.6 AIP for Windows.

* Release notes for 1.10.6 (#1931)

* Minor tweak to Synology Docker image script (#1936)

* Change if_def again so ios can build (#1937)

All apple's variables are "defined"
but sometimes they are defined as "0"

* move begin/commit into try/catch block (#1932)

Thread was exiting in some cases

* Bump openssl from 0.10.45 to 0.10.48 in /zeroidc (#1938)

Bumps [openssl](https://github.com/sfackler/rust-openssl) from 0.10.45 to 0.10.48.
- [Release notes](https://github.com/sfackler/rust-openssl/releases)
- [Commits](https://github.com/sfackler/rust-openssl/compare/openssl-v0.10.45...openssl-v0.10.48)

---
updated-dependencies:
- dependency-name: openssl
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* new drone bits

* Fix multiple network join from environment entrypoint.sh.release (#1961)

* _bond_m guards _bond, not _paths_m (#1965)

* Fix: warning: mutex '_aqm_m' is not held on every path through here [-Wthread-safety-analysis] (#1964)

* Bump h2 from 0.3.16 to 0.3.17 in /zeroidc (#1963)

Bumps [h2](https://github.com/hyperium/h2) from 0.3.16 to 0.3.17.
- [Release notes](https://github.com/hyperium/h2/releases)
- [Changelog](https://github.com/hyperium/h2/blob/master/CHANGELOG.md)
- [Commits](https://github.com/hyperium/h2/compare/v0.3.16...v0.3.17)

---
updated-dependencies:
- dependency-name: h2
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Grant Limberg <glimberg@users.noreply.github.com>

* Add note that binutils is required on FreeBSD (#1968)

* Add prometheus metrics for Central controllers (#1969)

* add header-only prometheus lib to ext

* rename folder

* Undo rename directory

* prometheus simpleapi included on mac & linux

* wip

* wire up some controller stats

* Get windows building with prometheus

* bsd build flags for prometheus

* Fix multiple network join from environment entrypoint.sh.release (#1961)

* _bond_m guards _bond, not _paths_m (#1965)

* Fix: warning: mutex '_aqm_m' is not held on every path through here [-Wthread-safety-analysis] (#1964)

* Serve prom metrics from /metrics endpoint

* Add prom metrics for Central controller specific things

* reorganize metric initialization

* testing out a labled gauge on Networks

* increment error counter on throw

* Consolidate metrics definitions

Put all metric definitions into node/Metrics.hpp.  Accessed as needed
from there.

* Revert "testing out a labled gauge on Networks"

This reverts commit 499ed6d95e.

* still blows up but adding to the record for completeness right now

* Fix runtime issues with metrics

* Add metrics files to visual studio project

* Missed an "extern"

* add copyright headers to new files

* Add metrics for sent/received bytes (total)

* put /metrics endpoint behind auth

* sendto returns int on Win32

---------

Co-authored-by: Leonardo Amaral <leleobhz@users.noreply.github.com>
Co-authored-by: Brenton Bostick <bostick@gmail.com>

* Central startup update (#1973)

* allow specifying authtoken in central startup

* set allowManagedFrom

* move redis_mem_notification to the correct place

* add node checkins metric

* wire up min/max connection pool size metrics

* x86_64-unknown-linux-gnu on ubuntu runner (#1975)

* adding incoming zt packet type metrics (#1976)

* use cpp-httplib for HTTP control plane (#1979)

refactored the old control plane code to use [cpp-httplib](https://github.com/yhirose/cpp-httplib) instead of a hand rolled HTTP server.  Makes the control plane code much more legible.  Also no longer randomly stops responding.

* Outgoing Packet Metrics (#1980)

add tx/rx labels to packet counters and add metrics for outgoing packets

* Add short-term validation test workflow (#1974)

Add short-term validation test workflow

* Brenton/curly braces (#1971)

* fix formatting

* properly adjust various lines
breakup multiple statements onto multiple lines

* insert {} around if, for, etc.

* Fix rust dependency caching (#1983)

* fun with rust caching

* kick

* comment out invalid yaml keys for now

* Caching should now work

* re-add/rename key directives

* bump

* bump

* bump

* Don't force rebuild on Windows build GH Action (#1985)

Switching `/t:ZeroTierOne:Rebuild` to just `/t:ZeroTierOne` allows the Windows build to use the rust cache.  `/t:ZeroTierOne:Rebuild` cleared the cache before building.

* More packet metrics (#1982)

* found path negotation sends that weren't accounted for

* Fix histogram so it will actually compile

* Found more places for packet metrics

* separate the bind & listen calls on the http backplane (#1988)

* fix memory leak (#1992)

* fix a couple of metrics (#1989)

* More aggressive CLI spamming (#1993)

* fix type signatures (#1991)

* Network-metrics (#1994)

* Add a couple quick functions for converting a uint64_t network ID/node ID into std::string

* Network metrics

* Peer metrics (#1995)

* Adding peer metrics

still need to be wired up for use

* per peer packet metrics

* Fix crash from bad instantiation of histogram

* separate alive & dead path counts

* Add peer metric update block

* add peer latency values in doPingAndKeepalive

* prevent deadlock

* peer latency histogram actually works now

* cleanup

* capture counts of packets to specific peers

---------

Co-authored-by: Joseph Henry <joseph.henry@zerotier.com>

* Metrics consolidation (#1997)

* Rename zt_packet_incoming -> zt_packet

Also consolidate zt_peer_packets into a single metric with tx and rx labels.  Same for ztc_tcp_data and ztc_udp_data

* Further collapse tcp & udp into metric labels for zt_data

* Fix zt_data metric description

* zt_peer_packets description fix

* Consolidate incoming/outgoing network packets to a single metric

* zt_incoming_packet_error -> zt_packet_error

* Disable peer metrics for central controllers

Can change in the future if needed, but given the traffic our controllers serve, that's going to be a *lot* of data

* Disable peer metrics for controllers pt 2

* Update readme files for metrics (#2000)

* Controller Metrics & Network Config Request Fix (#2003)

* add new metrics for network config request queue size and sso expirations
* move sso expiration to its own thread in the controller
* fix potential undefined behavior when modifying a set

* Enable RTTI in Windows build

The new prometheus histogram stuff needs it.

Access violation - no RTTI data!INVALID packet 636ebd9ee8cac6c0 from cafe9efeb9(2605:9880:200:1200:30:571:e34:51/9993) (unexpected exception in tryDecode())

* Don't re-apply routes on BSD

See issue #1986

* Capture setContent by-value instead of by-reference (#2006)

Co-authored-by: Grant Limberg <glimberg@users.noreply.github.com>

* fix typos (#2010)

* central controller metrics & request path updates (#2012)

* internal db metrics

* use shared mutexes for read/write locks

* remove this lock. only used for a metric

* more metrics

* remove exploratory metrics

place controller request benchmarks behind ifdef

* Improve validation test (#2013)

* fix init order for EmbeddedNetworkController (#2014)

* add constant for getifaddrs cache time

* cache getifaddrs - mac

* cache getifaddrs - linux

* cache getifaddrs - bsd

* cache getifaddrs - windows

* Fix oidc client lookup query

join condition referenced the wrong table.  Worked fine unless there were multiple identical client IDs

* Fix udp sent metric

was only incrementing by 1 for each packet sent

* Allow sending all surface addresses to peer in low-bandwidth mode

* allow enabling of low bandwidth mode on controllers

* don't unborrow bad connections

pool will clean them up later

* Multi-arch controller container (#2037)

create arm64 & amd64 images for central controller

* Update README.md

issue #2009

* docker tags change

* fix oidc auth url memory leak (#2031)

getAuthURL() was not calling zeroidc::free_cstr(url);

the only place authAuthURL is called, the url can be retrieved
from the network config instead.

You could alternatively copy the string and call free_cstr in getAuthURL.
If that's better we can change the PR.

Since now there are no callers of getAuthURL I deleted it.

Co-authored-by: Grant Limberg <glimberg@users.noreply.github.com>

* Bump openssl from 0.10.48 to 0.10.55 in /zeroidc (#2034)

Bumps [openssl](https://github.com/sfackler/rust-openssl) from 0.10.48 to 0.10.55.
- [Release notes](https://github.com/sfackler/rust-openssl/releases)
- [Commits](https://github.com/sfackler/rust-openssl/compare/openssl-v0.10.48...openssl-v0.10.55)

---
updated-dependencies:
- dependency-name: openssl
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Grant Limberg <glimberg@users.noreply.github.com>

* zeroidc cargo warnings (#2029)

* fix unused struct member cargo warning

* fix unused import cargo warning

* fix unused return value cargo warning

---------

Co-authored-by: Grant Limberg <glimberg@users.noreply.github.com>

* fix memory leak in macos ipv6/dns helper (#2030)

Co-authored-by: Grant Limberg <glimberg@users.noreply.github.com>

* Consider ZEROTIER_JOIN_NETWORKS in healthcheck (#1978)

* Add a 2nd auth token only for access to /metrics (#2043)

* Add a 2nd auth token for /metrics

Allows administrators to distribute a token that only has access to read
metrics and nothing else.

Also added support for using bearer auth tokens for both types of tokens

Separate endpoint for metrics #2041

* Update readme

* fix a couple of cases of writing the wrong token

* Add warning to cli for allow default on FreeBSD

It doesn't work.
Not possible to fix with deficient network
stack and APIs.

ZeroTierOne-freebsd # zerotier-cli set 9bee8941b5xxxxxx allowDefault=1
400 set Allow Default does not work properly on FreeBSD. See #580
root@freebsd13-a:~/ZeroTierOne-freebsd # zerotier-cli get 9bee8941b5xxxxxx allowDefault
1

* ARM64 Support for TapDriver6 (#1949)

* Release memory previously allocated by UPNP_GetValidIGD

* Fix ifdef that breaks libzt on iOS (#2050)

* less drone (#2060)

* Exit if loading an invalid identity from disk (#2058)

* Exit if loading an invalid identity from disk

Previously, if an invalid identity was loaded from disk, ZeroTier would
generate a new identity & chug along and generate a brand new identity
as if nothing happened.  When running in containers, this introduces the
possibility for key matter loss; especially when running in containers
where the identity files are mounted in the container read only.  In
this case, ZT will continue chugging along with a brand new identity
with no possibility of recovering the private key.

ZeroTier should exit upon loading of invalid identity.public/identity.secret #2056

* add validation test for #2056

* tcp-proxy: fix build

* Adjust tcp-proxy makefile to support metrics

There's no way to get the metrics yet. Someone will
have to add the http service.

* remove ZT_NO_METRIC ifdef

* Implement recvmmsg() for Linux to reduce syscalls. (#2046)

Between 5% and 40% speed improvement on Linux, depending on system configuration and load.

* suppress warnings: comparison of integers of different signs: 'int64_t' (aka 'long') and 'uint64_t' (aka 'unsigned long') [-Wsign-compare] (#2063)

* fix warning: 'OS_STRING' macro redefined [-Wmacro-redefined] (#2064)

Even though this is in ext, these particular chunks of code were added
by us, so are ok to modify.

* Apply default route a different way - macOS

The original way we applied default route, by forking
0.0.0.0/0 into 0/1 and 128/1 works, but if mac os has any networking
hiccups -if you change SSIDs or sleep/wake- macos erases the system default route.
And then all networking on the computer is broken.

to summarize the new way:
allowDefault=1
```
sudo route delete default 192.168.82.1
sudo route add default 10.2.0.2
sudo route add -ifscope en1 default 192.168.82.1
```

gives us this routing table
```
Destination        Gateway            RT_IFA             Flags        Refs      Use    Mtu          Netif Expire    rtt(ms) rttvar(ms)
default            10.2.0.2           10.2.0.18          UGScg          90        1   2800       feth4823
default            192.168.82.1       192.168.82.217     UGScIg
```

allowDefault=0
```
sudo route delete default
sudo route delete -ifscope en1 default
sudo route add default 192.168.82.1
```

Notice the I flag, for -ifscope, on the physical default route.

route change does not seem to work reliably.

* fix docker tag for controllers (#2066)

* Update build.sh (#2068)

fix mkwork compilation errors

* Fix network DNS on macOS

It stopped working for ipv4 only networks in Monterey.
See #1696

We add some config like so to System Configuration

```
scutil
show State:/Network/Service/9bee8941b5xxxxxx/IPv4
<dictionary> {
  Addresses : <array> {
    0 : 10.2.1.36
  }
  InterfaceName : feth4823
  Router : 10.2.1.36
  ServerAddress : 127.0.0.1
}

```

* Add search domain to macos dns configuration

Stumbled upon this while debugging something else.
If we add search domain to our system configuration for
network DNS, then search domains work:

```
ping server1                                                                                                                                                                                    ~
PING server1.my.domain (10.123.3.1): 56 data bytes
64 bytes from 10.123.3.1
```

* Fix reporting of secondaryPort and tertiaryPort See: #2039

* Fix typos (#2075)

* Disable executable stacks on assembly objects (#2071)

Add `--noexecstack` to the assembler flags so the resulting binary
will link with a non-executable stack.

Fixes zerotier/ZeroTierOne#1179

Co-authored-by: Joseph Henry <joseph.henry@zerotier.com>

* Test that starting zerotier before internet works

* Don't skip hellos when there are no paths available

working on #2082

* Update validate-1m-linux.sh

* Save zt node log files on abort

* Separate test and summary step in validator script

* Don't apply default route until zerotier is "online"

I was running into issues with restarting the zerotier service while
"full tunnel" mode is enabled.
When zerotier first boots, it gets network state from the cache
on disk. So it immediately applies all the routes it knew about
before it shutdown.
The network config may have change in this time.
If it has, then your default route is via a route
you are blocked from talking on. So you  can't get the current
network config, so your internet does not work.

Other options include
- don't use cached network state on boot
- find a better criteria than "online"

* Fix node time-to-online counter in validator script

* Export variables so that they are accessible by exit function

* Fix PortMapper issue on ZeroTier startup

See issue #2082

We use a call to libnatpmp::ininatpp to make sure the computer
has working network sockets before we go into the main
nat-pmp/upnp logic.

With basic exponenetial delay up to 30 seconds.

* testing

* Comment out PortMapper debug

this got left turned on in a confusing merge previously

* fix macos default route again

see commit fb6af1971 * Fix network DNS on macOS
adding that stuff to System Config causes this extra route to be added
which breaks ipv4 default route.
We figured out a weird System Coniguration setting
that works.

--- old
couldn't figure out how to fix it in SystemConfiguration
so here we are# Please enter the commit message for your changes. Lines starting

We also moved the dns setter to before the syncIps stuff
to help with a race condition. It didn't always work when
you re-joined a network with default route enabled.

* Catch all conditions in switch statement, remove trailing whitespaces

* Add setmtu command, fix bond lifetime issue

* Basic cleanups

* Check if null is passed to VirtualNetworkConfig.equals and name fixes

* ANDROID-96: Simplify and use return code from node_init directly

* Windows arm64 (#2099)

* ARM64 changes for 1.12

* 1.12 Windows advanced installer updates and updates for ARM64

* 1.12.0

* Linux build fixes for old distros.

* release notes

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: travis laduke <travisladuke@gmail.com>
Co-authored-by: Grant Limberg <grant.limberg@zerotier.com>
Co-authored-by: Grant Limberg <glimberg@users.noreply.github.com>
Co-authored-by: Leonardo Amaral <leleobhz@users.noreply.github.com>
Co-authored-by: Brenton Bostick <bostick@gmail.com>
Co-authored-by: Sean OMeara <someara@users.noreply.github.com>
Co-authored-by: Joseph Henry <joseph-henry@users.noreply.github.com>
Co-authored-by: Roman Peshkichev <roman.peshkichev@gmail.com>
Co-authored-by: Joseph Henry <joseph.henry@zerotier.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Stavros Kois <47820033+stavros-k@users.noreply.github.com>
Co-authored-by: Jake Vis <jakevis@outlook.com>
Co-authored-by: Jörg Thalheim <joerg@thalheim.io>
Co-authored-by: lison <imlison@foxmail.com>
Co-authored-by: Kenny MacDermid <kenny@macdermid.ca>
2023-08-23 14:24:21 -04:00

1197 lines
38 KiB
C++

/*
* Copyright (c)2013-2020 ZeroTier, Inc.
*
* Use of this software is governed by the Business Source License included
* in the LICENSE.TXT file in the project's root directory.
*
* Change Date: 2025-01-01
*
* On the date above, in accordance with the Business Source License, use
* of this software will be governed by version 2.0 of the Apache License.
*/
/****/
#include <stdint.h>
#include <stddef.h>
#include <string.h>
#include <stdlib.h>
#include <stdio.h>
#include "Packet.hpp"
#if defined(ZT_USE_X64_ASM_SALSA2012) && defined(ZT_ARCH_X64)
#include "../ext/x64-salsa2012-asm/salsa2012.h"
#endif
#ifdef ZT_USE_ARM32_NEON_ASM_SALSA2012
#include "../ext/arm32-neon-salsa2012-asm/salsa2012.h"
#endif
#ifdef _MSC_VER
#define FORCE_INLINE static __forceinline
#include <intrin.h>
#pragma warning(disable : 4127) /* disable: C4127: conditional expression is constant */
#pragma warning(disable : 4293) /* disable: C4293: too large shift (32-bits) */
#else
#define FORCE_INLINE static inline
#endif
namespace ZeroTier {
/************************************************************************** */
/* Set up macros for fast single-pass ASM Salsa20/12 crypto, if we have it */
// x64 SSE crypto
#if defined(ZT_USE_X64_ASM_SALSA2012) && defined(ZT_ARCH_X64)
#define ZT_HAS_FAST_CRYPTO() (true)
#define ZT_FAST_SINGLE_PASS_SALSA2012(b,l,n,k) zt_salsa2012_amd64_xmm6(reinterpret_cast<unsigned char *>(b),(l),reinterpret_cast<const unsigned char *>(n),reinterpret_cast<const unsigned char *>(k))
#endif
// ARM (32-bit) NEON crypto (must be detected)
#ifdef ZT_USE_ARM32_NEON_ASM_SALSA2012
class _FastCryptoChecker
{
public:
_FastCryptoChecker() : canHas(zt_arm_has_neon()) {}
bool canHas;
};
static const _FastCryptoChecker _ZT_FAST_CRYPTO_CHECK;
#define ZT_HAS_FAST_CRYPTO() (_ZT_FAST_CRYPTO_CHECK.canHas)
#define ZT_FAST_SINGLE_PASS_SALSA2012(b,l,n,k) zt_salsa2012_armneon3_xor(reinterpret_cast<unsigned char *>(b),(const unsigned char *)0,(l),reinterpret_cast<const unsigned char *>(n),reinterpret_cast<const unsigned char *>(k))
#endif
// No fast crypto available
#ifndef ZT_HAS_FAST_CRYPTO
#define ZT_HAS_FAST_CRYPTO() (false)
#define ZT_FAST_SINGLE_PASS_SALSA2012(b,l,n,k) {}
#endif
/************************************************************************** */
/* LZ4 is shipped encapsulated into Packet in an anonymous namespace.
*
* We're doing this as a deliberate workaround for various Linux distribution
* policies that forbid static linking of support libraries.
*
* The reason is that relying on distribution versions of LZ4 has been too
* big a source of bugs and compatibility issues. The LZ4 API is not stable
* enough across versions, and dependency hell ensues. So fark it. */
/* Needless to say the code in this anonymous namespace should be considered
* BSD 2-clause licensed. */
namespace {
/* lz4.h ------------------------------------------------------------------ */
/*
* LZ4 - Fast LZ compression algorithm
* Header File
* Copyright (C) 2011-2016, Yann Collet.
BSD 2-Clause License (http://www.opensource.org/licenses/bsd-license.php)
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are
met:
* Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above
copyright notice, this list of conditions and the following disclaimer
in the documentation and/or other materials provided with the
distribution.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
You can contact the author at :
- LZ4 homepage : http://www.lz4.org
- LZ4 source repository : https://github.com/lz4/lz4
*/
/**
Introduction
LZ4 is lossless compression algorithm, providing compression speed at 400 MB/s per core,
scalable with multi-cores CPU. It features an extremely fast decoder, with speed in
multiple GB/s per core, typically reaching RAM speed limits on multi-core systems.
The LZ4 compression library provides in-memory compression and decompression functions.
Compression can be done in:
- a single step (described as Simple Functions)
- a single step, reusing a context (described in Advanced Functions)
- unbounded multiple steps (described as Streaming compression)
lz4.h provides block compression functions. It gives full buffer control to user.
Decompressing an lz4-compressed block also requires metadata (such as compressed size).
Each application is free to encode such metadata in whichever way it wants.
An additional format, called LZ4 frame specification (doc/lz4_Frame_format.md),
take care of encoding standard metadata alongside LZ4-compressed blocks.
If your application requires interoperability, it's recommended to use it.
A library is provided to take care of it, see lz4frame.h.
*/
#define LZ4_VERSION_MAJOR 1 /* for breaking interface changes */
#define LZ4_VERSION_MINOR 7 /* for new (non-breaking) interface capabilities */
#define LZ4_VERSION_RELEASE 5 /* for tweaks, bug-fixes, or development */
#define LZ4_VERSION_NUMBER (LZ4_VERSION_MAJOR *100*100 + LZ4_VERSION_MINOR *100 + LZ4_VERSION_RELEASE)
#define LZ4_LIB_VERSION LZ4_VERSION_MAJOR.LZ4_VERSION_MINOR.LZ4_VERSION_RELEASE
#define LZ4_QUOTE(str) #str
#define LZ4_EXPAND_AND_QUOTE(str) LZ4_QUOTE(str)
#define LZ4_VERSION_STRING LZ4_EXPAND_AND_QUOTE(LZ4_LIB_VERSION)
#define LZ4_MEMORY_USAGE 14
#define LZ4_MAX_INPUT_SIZE 0x7E000000 /* 2 113 929 216 bytes */
#define LZ4_COMPRESSBOUND(isize) ((unsigned)(isize) > (unsigned)LZ4_MAX_INPUT_SIZE ? 0 : (isize) + ((isize)/255) + 16)
typedef union LZ4_stream_u LZ4_stream_t; /* incomplete type (defined later) */
static inline void LZ4_resetStream (LZ4_stream_t* streamPtr);
#define LZ4_HASHLOG (LZ4_MEMORY_USAGE-2)
#define LZ4_HASHTABLESIZE (1 << LZ4_MEMORY_USAGE)
#define LZ4_HASH_SIZE_U32 (1 << LZ4_HASHLOG) /* required as macro for static allocation */
typedef struct {
uint32_t hashTable[LZ4_HASH_SIZE_U32];
uint32_t currentOffset;
uint32_t initCheck;
const uint8_t* dictionary;
uint8_t* bufferStart; /* obsolete, used for slideInputBuffer */
uint32_t dictSize;
} LZ4_stream_t_internal;
typedef struct {
const uint8_t* externalDict;
size_t extDictSize;
const uint8_t* prefixEnd;
size_t prefixSize;
} LZ4_streamDecode_t_internal;
#define LZ4_STREAMSIZE_U64 ((1 << (LZ4_MEMORY_USAGE-3)) + 4)
#define LZ4_STREAMSIZE (LZ4_STREAMSIZE_U64 * sizeof(unsigned long long))
union LZ4_stream_u {
unsigned long long table[LZ4_STREAMSIZE_U64];
LZ4_stream_t_internal internal_donotuse;
} ; /* previously typedef'd to LZ4_stream_t */
#define LZ4_STREAMDECODESIZE_U64 4
#define LZ4_STREAMDECODESIZE (LZ4_STREAMDECODESIZE_U64 * sizeof(unsigned long long))
union LZ4_streamDecode_u {
unsigned long long table[LZ4_STREAMDECODESIZE_U64];
LZ4_streamDecode_t_internal internal_donotuse;
} ; /* previously typedef'd to LZ4_streamDecode_t */
#ifndef HEAPMODE
#define HEAPMODE 0
#endif
#ifdef ZT_NO_TYPE_PUNNING
#define LZ4_FORCE_MEMORY_ACCESS 0
#else
#define LZ4_FORCE_MEMORY_ACCESS 2
#endif
#if defined(_MSC_VER) && defined(_WIN32_WCE) /* Visual Studio for Windows CE does not support Hardware bit count */
#define LZ4_FORCE_SW_BITCOUNT
#endif
#ifndef FORCE_INLINE
#define FORCE_INLINE static inline
#endif
#define ALLOCATOR(n,s) calloc(n,s)
#define FREEMEM free
#define MEM_INIT memset
typedef uint8_t BYTE;
typedef uint16_t U16;
typedef uint32_t U32;
typedef int32_t S32;
typedef uint64_t U64;
typedef uintptr_t uptrval;
typedef uintptr_t reg_t;
static inline unsigned LZ4_isLittleEndian(void)
{
const union { U32 u; BYTE c[4]; } one = { 1 }; /* don't use static : performance detrimental */
return one.c[0];
}
#if defined(LZ4_FORCE_MEMORY_ACCESS) && (LZ4_FORCE_MEMORY_ACCESS==2)
static U16 LZ4_read16(const void* memPtr) { return *(const U16*) memPtr; }
static U32 LZ4_read32(const void* memPtr) { return *(const U32*) memPtr; }
static reg_t LZ4_read_ARCH(const void* memPtr) { return *(const reg_t*) memPtr; }
static void LZ4_write16(void* memPtr, U16 value) { *(U16*)memPtr = value; }
static void LZ4_write32(void* memPtr, U32 value) { *(U32*)memPtr = value; }
#elif defined(LZ4_FORCE_MEMORY_ACCESS) && (LZ4_FORCE_MEMORY_ACCESS==1)
typedef union { U16 u16; U32 u32; reg_t uArch; } __attribute__((packed)) unalign;
static U16 LZ4_read16(const void* ptr) { return ((const unalign*)ptr)->u16; }
static U32 LZ4_read32(const void* ptr) { return ((const unalign*)ptr)->u32; }
static reg_t LZ4_read_ARCH(const void* ptr) { return ((const unalign*)ptr)->uArch; }
static void LZ4_write16(void* memPtr, U16 value) { ((unalign*)memPtr)->u16 = value; }
static void LZ4_write32(void* memPtr, U32 value) { ((unalign*)memPtr)->u32 = value; }
#else /* safe and portable access through memcpy() */
static inline U16 LZ4_read16(const void* memPtr)
{
U16 val; memcpy(&val, memPtr, sizeof(val)); return val;
}
static inline U32 LZ4_read32(const void* memPtr)
{
U32 val; memcpy(&val, memPtr, sizeof(val)); return val;
}
static inline reg_t LZ4_read_ARCH(const void* memPtr)
{
reg_t val; memcpy(&val, memPtr, sizeof(val)); return val;
}
static inline void LZ4_write16(void* memPtr, U16 value)
{
memcpy(memPtr, &value, sizeof(value));
}
static inline void LZ4_write32(void* memPtr, U32 value)
{
memcpy(memPtr, &value, sizeof(value));
}
#endif /* LZ4_FORCE_MEMORY_ACCESS */
static inline U16 LZ4_readLE16(const void* memPtr)
{
if (LZ4_isLittleEndian()) {
return LZ4_read16(memPtr);
} else {
const BYTE* p = (const BYTE*)memPtr;
return (U16)((U16)p[0] + (p[1]<<8));
}
}
static inline void LZ4_writeLE16(void* memPtr, U16 value)
{
if (LZ4_isLittleEndian()) {
LZ4_write16(memPtr, value);
} else {
BYTE* p = (BYTE*)memPtr;
p[0] = (BYTE) value;
p[1] = (BYTE)(value>>8);
}
}
static inline void LZ4_copy8(void* dst, const void* src)
{
memcpy(dst,src,8);
}
static inline void LZ4_wildCopy(void* dstPtr, const void* srcPtr, void* dstEnd)
{
BYTE* d = (BYTE*)dstPtr;
const BYTE* s = (const BYTE*)srcPtr;
BYTE* const e = (BYTE*)dstEnd;
do {
LZ4_copy8(d,s);
d+=8;
s+=8;
} while (d<e);
}
#define MINMATCH 4
#define WILDCOPYLENGTH 8
#define LASTLITERALS 5
#define MFLIMIT (WILDCOPYLENGTH+MINMATCH)
static const int LZ4_minLength = (MFLIMIT+1);
#define KB *(1 <<10)
#define MB *(1 <<20)
#define GB *(1U<<30)
#define MAXD_LOG 16
#define MAX_DISTANCE ((1 << MAXD_LOG) - 1)
#define ML_BITS 4
#define ML_MASK ((1U<<ML_BITS)-1)
#define RUN_BITS (8-ML_BITS)
#define RUN_MASK ((1U<<RUN_BITS)-1)
#define LZ4_STATIC_ASSERT(c) { enum { LZ4_static_assert = 1/(int)(!!(c)) }; } /* use only *after* variable declarations */
static inline unsigned LZ4_NbCommonBytes (reg_t val)
{
if (LZ4_isLittleEndian()) {
if (sizeof(val)==8) {
# if defined(_MSC_VER) && defined(_WIN64) && !defined(LZ4_FORCE_SW_BITCOUNT)
unsigned long r = 0;
_BitScanForward64( &r, (U64)val );
return (int)(r>>3);
# elif (defined(__clang__) || (defined(__GNUC__) && (__GNUC__>=3))) && !defined(LZ4_FORCE_SW_BITCOUNT)
return (__builtin_ctzll((U64)val) >> 3);
# else
static const int DeBruijnBytePos[64] = { 0, 0, 0, 0, 0, 1, 1, 2, 0, 3, 1, 3, 1, 4, 2, 7, 0, 2, 3, 6, 1, 5, 3, 5, 1, 3, 4, 4, 2, 5, 6, 7, 7, 0, 1, 2, 3, 3, 4, 6, 2, 6, 5, 5, 3, 4, 5, 6, 7, 1, 2, 4, 6, 4, 4, 5, 7, 2, 6, 5, 7, 6, 7, 7 };
return DeBruijnBytePos[((U64)((val & -(long long)val) * 0x0218A392CDABBD3FULL)) >> 58];
# endif
} else /* 32 bits */ {
# if defined(_MSC_VER) && !defined(LZ4_FORCE_SW_BITCOUNT)
unsigned long r;
_BitScanForward( &r, (U32)val );
return (int)(r>>3);
# elif (defined(__clang__) || (defined(__GNUC__) && (__GNUC__>=3))) && !defined(LZ4_FORCE_SW_BITCOUNT)
return (__builtin_ctz((U32)val) >> 3);
# else
static const int DeBruijnBytePos[32] = { 0, 0, 3, 0, 3, 1, 3, 0, 3, 2, 2, 1, 3, 2, 0, 1, 3, 3, 1, 2, 2, 2, 2, 0, 3, 1, 2, 0, 1, 0, 1, 1 };
return DeBruijnBytePos[((U32)((val & -(S32)val) * 0x077CB531U)) >> 27];
# endif
}
} else /* Big Endian CPU */ {
if (sizeof(val)==8) {
# if defined(_MSC_VER) && defined(_WIN64) && !defined(LZ4_FORCE_SW_BITCOUNT)
unsigned long r = 0;
_BitScanReverse64( &r, val );
return (unsigned)(r>>3);
# elif (defined(__clang__) || (defined(__GNUC__) && (__GNUC__>=3))) && !defined(LZ4_FORCE_SW_BITCOUNT)
return (__builtin_clzll((U64)val) >> 3);
# else
unsigned r;
if (!(val>>32)) {
r=4;
} else {
r=0;
val>>=32;
}
if (!(val>>16)) {
r+=2;
val>>=8;
} else {
val>>=24;
}
r += (!val);
return r;
# endif
} else /* 32 bits */ {
# if defined(_MSC_VER) && !defined(LZ4_FORCE_SW_BITCOUNT)
unsigned long r = 0;
_BitScanReverse( &r, (unsigned long)val );
return (unsigned)(r>>3);
# elif (defined(__clang__) || (defined(__GNUC__) && (__GNUC__>=3))) && !defined(LZ4_FORCE_SW_BITCOUNT)
return (__builtin_clz((U32)val) >> 3);
# else
unsigned r;
if (!(val>>16)) {
r=2;
val>>=8;
} else {
r=0;
val>>=24;
}
r += (!val);
return r;
# endif
}
}
}
#define STEPSIZE sizeof(reg_t)
static inline unsigned LZ4_count(const BYTE* pIn, const BYTE* pMatch, const BYTE* pInLimit)
{
const BYTE* const pStart = pIn;
while (likely(pIn<pInLimit-(STEPSIZE-1))) {
reg_t const diff = LZ4_read_ARCH(pMatch) ^ LZ4_read_ARCH(pIn);
if (!diff) {
pIn+=STEPSIZE;
pMatch+=STEPSIZE; continue;
}
pIn += LZ4_NbCommonBytes(diff);
return (unsigned)(pIn - pStart);
}
if ((STEPSIZE==8) && (pIn<(pInLimit-3)) && (LZ4_read32(pMatch) == LZ4_read32(pIn))) {
pIn+=4; pMatch+=4;
}
if ((pIn<(pInLimit-1)) && (LZ4_read16(pMatch) == LZ4_read16(pIn))) {
pIn+=2; pMatch+=2;
}
if ((pIn<pInLimit) && (*pMatch == *pIn)) {
pIn++;
}
return (unsigned)(pIn - pStart);
}
static const int LZ4_64Klimit = ((64 KB) + (MFLIMIT-1));
static const U32 LZ4_skipTrigger = 6; /* Increase this value ==> compression run slower on incompressible data */
typedef enum { notLimited = 0, limitedOutput = 1 } limitedOutput_directive;
typedef enum { byPtr, byU32, byU16 } tableType_t;
typedef enum { noDict = 0, withPrefix64k, usingExtDict } dict_directive;
typedef enum { noDictIssue = 0, dictSmall } dictIssue_directive;
typedef enum { endOnOutputSize = 0, endOnInputSize = 1 } endCondition_directive;
typedef enum { full = 0, partial = 1 } earlyEnd_directive;
static inline int LZ4_compressBound(int isize) { return LZ4_COMPRESSBOUND(isize); }
static inline U32 LZ4_hash4(U32 sequence, tableType_t const tableType)
{
if (tableType == byU16) {
return ((sequence * 2654435761U) >> ((MINMATCH*8)-(LZ4_HASHLOG+1)));
} else {
return ((sequence * 2654435761U) >> ((MINMATCH*8)-LZ4_HASHLOG));
}
}
static inline U32 LZ4_hash5(U64 sequence, tableType_t const tableType)
{
static const U64 prime5bytes = 889523592379ULL;
static const U64 prime8bytes = 11400714785074694791ULL;
const U32 hashLog = (tableType == byU16) ? LZ4_HASHLOG+1 : LZ4_HASHLOG;
if (LZ4_isLittleEndian()) {
return (U32)(((sequence << 24) * prime5bytes) >> (64 - hashLog));
} else {
return (U32)(((sequence >> 24) * prime8bytes) >> (64 - hashLog));
}
}
FORCE_INLINE U32 LZ4_hashPosition(const void* const p, tableType_t const tableType)
{
if ((sizeof(reg_t)==8) && (tableType != byU16)) {
return LZ4_hash5(LZ4_read_ARCH(p), tableType);
}
return LZ4_hash4(LZ4_read32(p), tableType);
}
static inline void LZ4_putPositionOnHash(const BYTE* p, U32 h, void* tableBase, tableType_t const tableType, const BYTE* srcBase)
{
switch (tableType) {
case byPtr: {
const BYTE** hashTable = (const BYTE**)tableBase; hashTable[h] = p;
return;
}
case byU32: {
U32* hashTable = (U32*) tableBase; hashTable[h] = (U32)(p-srcBase);
return;
}
case byU16: {
U16* hashTable = (U16*) tableBase; hashTable[h] = (U16)(p-srcBase);
return;
}
}
}
FORCE_INLINE void LZ4_putPosition(const BYTE* p, void* tableBase, tableType_t tableType, const BYTE* srcBase)
{
U32 const h = LZ4_hashPosition(p, tableType);
LZ4_putPositionOnHash(p, h, tableBase, tableType, srcBase);
}
static inline const BYTE* LZ4_getPositionOnHash(U32 h, void* tableBase, tableType_t tableType, const BYTE* srcBase)
{
if (tableType == byPtr) {
const BYTE** hashTable = (const BYTE**) tableBase;
return hashTable[h];
}
if (tableType == byU32) {
const U32* const hashTable = (U32*) tableBase;
return hashTable[h] + srcBase;
}
{ /* default, to ensure a return */
const U16* const hashTable = (U16*) tableBase;
return hashTable[h] + srcBase;
}
}
FORCE_INLINE const BYTE* LZ4_getPosition(const BYTE* p, void* tableBase, tableType_t tableType, const BYTE* srcBase)
{
U32 const h = LZ4_hashPosition(p, tableType);
return LZ4_getPositionOnHash(h, tableBase, tableType, srcBase);
}
FORCE_INLINE int LZ4_compress_generic(
LZ4_stream_t_internal* const cctx,
const char* const source,
char* const dest,
const int inputSize,
const int maxOutputSize,
const limitedOutput_directive outputLimited,
const tableType_t tableType,
const dict_directive dict,
const dictIssue_directive dictIssue,
const U32 acceleration)
{
const BYTE* ip = (const BYTE*) source;
const BYTE* base;
const BYTE* lowLimit;
const BYTE* const lowRefLimit = ip - cctx->dictSize;
const BYTE* const dictionary = cctx->dictionary;
const BYTE* const dictEnd = dictionary + cctx->dictSize;
const ptrdiff_t dictDelta = dictEnd - (const BYTE*)source;
const BYTE* anchor = (const BYTE*) source;
const BYTE* const iend = ip + inputSize;
const BYTE* const mflimit = iend - MFLIMIT;
const BYTE* const matchlimit = iend - LASTLITERALS;
BYTE* op = (BYTE*) dest;
BYTE* const olimit = op + maxOutputSize;
U32 forwardH;
/* Init conditions */
if ((U32)inputSize > (U32)LZ4_MAX_INPUT_SIZE) {
return 0; /* Unsupported inputSize, too large (or negative) */
}
switch(dict) {
case noDict:
default:
base = (const BYTE*)source;
lowLimit = (const BYTE*)source;
break;
case withPrefix64k:
base = (const BYTE*)source - cctx->currentOffset;
lowLimit = (const BYTE*)source - cctx->dictSize;
break;
case usingExtDict:
base = (const BYTE*)source - cctx->currentOffset;
lowLimit = (const BYTE*)source;
break;
}
if ((tableType == byU16) && (inputSize>=LZ4_64Klimit)) {
return 0; /* Size too large (not within 64K limit) */
}
if (inputSize<LZ4_minLength) {
goto _last_literals; /* Input too small, no compression (all literals) */
}
/* First Byte */
LZ4_putPosition(ip, cctx->hashTable, tableType, base);
ip++;
forwardH = LZ4_hashPosition(ip, tableType);
/* Main Loop */
for ( ; ; ) {
ptrdiff_t refDelta = 0;
const BYTE* match;
BYTE* token;
/* Find a match */
{
const BYTE* forwardIp = ip;
unsigned step = 1;
unsigned searchMatchNb = acceleration << LZ4_skipTrigger;
do {
U32 const h = forwardH;
ip = forwardIp;
forwardIp += step;
step = (searchMatchNb++ >> LZ4_skipTrigger);
if (unlikely(forwardIp > mflimit)) {
goto _last_literals;
}
match = LZ4_getPositionOnHash(h, cctx->hashTable, tableType, base);
if (dict==usingExtDict) {
if (match < (const BYTE*)source) {
refDelta = dictDelta;
lowLimit = dictionary;
} else {
refDelta = 0;
lowLimit = (const BYTE*)source;
}
}
forwardH = LZ4_hashPosition(forwardIp, tableType);
LZ4_putPositionOnHash(ip, h, cctx->hashTable, tableType, base);
} while ( ((dictIssue==dictSmall) ? (match < lowRefLimit) : 0)
|| ((tableType==byU16) ? 0 : (match + MAX_DISTANCE < ip))
|| (LZ4_read32(match+refDelta) != LZ4_read32(ip)) );
}
/* Catch up */
while (((ip>anchor) & (match+refDelta > lowLimit)) && (unlikely(ip[-1]==match[refDelta-1]))) {
ip--;
match--;
}
/* Encode Literals */
{
unsigned const litLength = (unsigned)(ip - anchor);
token = op++;
if ((outputLimited) && /* Check output buffer overflow */
(unlikely(op + litLength + (2 + 1 + LASTLITERALS) + (litLength/255) > olimit))) {
return 0;
}
if (litLength >= RUN_MASK) {
int len = (int)litLength-RUN_MASK;
*token = (RUN_MASK<<ML_BITS);
for(; len >= 255 ; len-=255) {
*op++ = 255;
}
*op++ = (BYTE)len;
} else {
*token = (BYTE)(litLength<<ML_BITS);
}
/* Copy Literals */
LZ4_wildCopy(op, anchor, op+litLength);
op+=litLength;
}
_next_match:
/* Encode Offset */
LZ4_writeLE16(op, (U16)(ip-match));
op+=2;
/* Encode MatchLength */
{
unsigned matchCode;
if ((dict==usingExtDict) && (lowLimit==dictionary)) {
const BYTE* limit;
match += refDelta;
limit = ip + (dictEnd-match);
if (limit > matchlimit) {
limit = matchlimit;
}
matchCode = LZ4_count(ip+MINMATCH, match+MINMATCH, limit);
ip += MINMATCH + matchCode;
if (ip==limit) {
unsigned const more = LZ4_count(ip, (const BYTE*)source, matchlimit);
matchCode += more;
ip += more;
}
} else {
matchCode = LZ4_count(ip+MINMATCH, match+MINMATCH, matchlimit);
ip += MINMATCH + matchCode;
}
if ( outputLimited && /* Check output buffer overflow */
(unlikely(op + (1 + LASTLITERALS) + (matchCode>>8) > olimit)) ) {
return 0;
}
if (matchCode >= ML_MASK) {
*token += ML_MASK;
matchCode -= ML_MASK;
LZ4_write32(op, 0xFFFFFFFF);
while (matchCode >= 4*255) {
op+=4;
LZ4_write32(op, 0xFFFFFFFF);
matchCode -= 4*255;
}
op += matchCode / 255;
*op++ = (BYTE)(matchCode % 255);
} else {
*token += (BYTE)(matchCode);
}
}
anchor = ip;
/* Test end of chunk */
if (ip > mflimit) {
break;
}
/* Fill table */
LZ4_putPosition(ip-2, cctx->hashTable, tableType, base);
/* Test next position */
match = LZ4_getPosition(ip, cctx->hashTable, tableType, base);
if (dict==usingExtDict) {
if (match < (const BYTE*)source) {
refDelta = dictDelta;
lowLimit = dictionary;
} else {
refDelta = 0;
lowLimit = (const BYTE*)source;
}
}
LZ4_putPosition(ip, cctx->hashTable, tableType, base);
if ( ((dictIssue==dictSmall) ? (match>=lowRefLimit) : 1)
&& (match+MAX_DISTANCE>=ip)
&& (LZ4_read32(match+refDelta)==LZ4_read32(ip)) ) {
token=op++;
*token=0;
goto _next_match;
}
/* Prepare next loop */
forwardH = LZ4_hashPosition(++ip, tableType);
}
_last_literals:
/* Encode Last Literals */
{
size_t const lastRun = (size_t)(iend - anchor);
if ( (outputLimited) && /* Check output buffer overflow */
((op - (BYTE*)dest) + lastRun + 1 + ((lastRun+255-RUN_MASK)/255) > (U32)maxOutputSize) ) {
return 0;
}
if (lastRun >= RUN_MASK) {
size_t accumulator = lastRun - RUN_MASK;
*op++ = RUN_MASK << ML_BITS;
for(; accumulator >= 255 ; accumulator-=255) {
*op++ = 255;
}
*op++ = (BYTE) accumulator;
} else {
*op++ = (BYTE)(lastRun<<ML_BITS);
}
memcpy(op, anchor, lastRun);
op += lastRun;
}
/* End */
return (int) (((char*)op)-dest);
}
static inline int LZ4_compress_fast_extState(void* state, const char* source, char* dest, int inputSize, int maxOutputSize, int acceleration)
{
LZ4_stream_t_internal* ctx = &((LZ4_stream_t*)state)->internal_donotuse;
LZ4_resetStream((LZ4_stream_t*)state);
//if (acceleration < 1) acceleration = ACCELERATION_DEFAULT;
if (maxOutputSize >= LZ4_compressBound(inputSize)) {
if (inputSize < LZ4_64Klimit) {
return LZ4_compress_generic(ctx, source, dest, inputSize, 0, notLimited, byU16, noDict, noDictIssue, acceleration);
} else {
return LZ4_compress_generic(ctx, source, dest, inputSize, 0, notLimited, (sizeof(void*)==8) ? byU32 : byPtr, noDict, noDictIssue, acceleration);
}
} else {
if (inputSize < LZ4_64Klimit) {
return LZ4_compress_generic(ctx, source, dest, inputSize, maxOutputSize, limitedOutput, byU16, noDict, noDictIssue, acceleration);
} else {
return LZ4_compress_generic(ctx, source, dest, inputSize, maxOutputSize, limitedOutput, (sizeof(void*)==8) ? byU32 : byPtr, noDict, noDictIssue, acceleration);
}
}
}
static inline int LZ4_compress_fast(const char* source, char* dest, int inputSize, int maxOutputSize, int acceleration)
{
#if (HEAPMODE)
void* ctxPtr = ALLOCATOR(1, sizeof(LZ4_stream_t)); /* malloc-calloc always properly aligned */
#else
LZ4_stream_t ctx;
void* const ctxPtr = &ctx;
#endif
int const result = LZ4_compress_fast_extState(ctxPtr, source, dest, inputSize, maxOutputSize, acceleration);
#if (HEAPMODE)
FREEMEM(ctxPtr);
#endif
return result;
}
static inline void LZ4_resetStream (LZ4_stream_t* LZ4_stream)
{
MEM_INIT(LZ4_stream, 0, sizeof(LZ4_stream_t));
}
FORCE_INLINE int LZ4_decompress_generic(
const char* const source,
char* const dest,
int inputSize,
int outputSize, /* If endOnInput==endOnInputSize, this value is the max size of Output Buffer. */
int endOnInput, /* endOnOutputSize, endOnInputSize */
int partialDecoding, /* full, partial */
int targetOutputSize, /* only used if partialDecoding==partial */
int dict, /* noDict, withPrefix64k, usingExtDict */
const BYTE* const lowPrefix, /* == dest when no prefix */
const BYTE* const dictStart, /* only if dict==usingExtDict */
const size_t dictSize /* note : = 0 if noDict */
)
{
/* Local Variables */
const BYTE* ip = (const BYTE*) source;
const BYTE* const iend = ip + inputSize;
BYTE* op = (BYTE*) dest;
BYTE* const oend = op + outputSize;
BYTE* cpy;
BYTE* oexit = op + targetOutputSize;
const BYTE* const lowLimit = lowPrefix - dictSize;
const BYTE* const dictEnd = (const BYTE*)dictStart + dictSize;
const unsigned dec32table[] = {0, 1, 2, 1, 4, 4, 4, 4};
const int dec64table[] = {0, 0, 0, -1, 0, 1, 2, 3};
const int safeDecode = (endOnInput==endOnInputSize);
const int checkOffset = ((safeDecode) && (dictSize < (int)(64 KB)));
/* Special cases */
if ((partialDecoding) && (oexit > oend-MFLIMIT)) {
oexit = oend-MFLIMIT; /* targetOutputSize too high => decode everything */
}
if ((endOnInput) && (unlikely(outputSize==0))) {
return ((inputSize==1) && (*ip==0)) ? 0 : -1; /* Empty output buffer */
}
if ((!endOnInput) && (unlikely(outputSize==0))) {
return (*ip==0?1:-1);
}
/* Main Loop : decode sequences */
while (1) {
size_t length;
const BYTE* match;
size_t offset;
/* get literal length */
unsigned const token = *ip++;
if ((length=(token>>ML_BITS)) == RUN_MASK) {
unsigned s;
do {
s = *ip++;
length += s;
} while ( likely(endOnInput ? ip<iend-RUN_MASK : 1) & (s==255) );
if ((safeDecode) && unlikely((uptrval)(op)+length<(uptrval)(op))) {
goto _output_error; /* overflow detection */
}
if ((safeDecode) && unlikely((uptrval)(ip)+length<(uptrval)(ip))) {
goto _output_error; /* overflow detection */
}
}
/* copy literals */
cpy = op+length;
if ( ((endOnInput) && ((cpy>(partialDecoding?oexit:oend-MFLIMIT)) || (ip+length>iend-(2+1+LASTLITERALS))) )
|| ((!endOnInput) && (cpy>oend-WILDCOPYLENGTH)) ) {
if (partialDecoding) {
if (cpy > oend) {
goto _output_error; /* Error : write attempt beyond end of output buffer */
}
if ((endOnInput) && (ip+length > iend)) {
goto _output_error; /* Error : read attempt beyond end of input buffer */
}
} else {
if ((!endOnInput) && (cpy != oend)) {
goto _output_error; /* Error : block decoding must stop exactly there */
}
if ((endOnInput) && ((ip+length != iend) || (cpy > oend))) {
goto _output_error; /* Error : input must be consumed */
}
}
memcpy(op, ip, length);
ip += length;
op += length;
break; /* Necessarily EOF, due to parsing restrictions */
}
LZ4_wildCopy(op, ip, cpy);
ip += length;
op = cpy;
/* get offset */
offset = LZ4_readLE16(ip);
ip += 2;
match = op - offset;
if ((checkOffset) && (unlikely(match < lowLimit))) {
goto _output_error; /* Error : offset outside buffers */
}
LZ4_write32(op, (U32)offset); /* costs ~1%; silence an msan warning when offset==0 */
/* get matchlength */
length = token & ML_MASK;
if (length == ML_MASK) {
unsigned s;
do {
s = *ip++;
if ((endOnInput) && (ip > iend-LASTLITERALS)) {
goto _output_error;
}
length += s;
} while (s==255);
if ((safeDecode) && unlikely((uptrval)(op)+length<(uptrval)op)) {
goto _output_error; /* overflow detection */
}
}
length += MINMATCH;
/* check external dictionary */
if ((dict==usingExtDict) && (match < lowPrefix)) {
if (unlikely(op+length > oend-LASTLITERALS)) {
goto _output_error; /* doesn't respect parsing restriction */
}
if (length <= (size_t)(lowPrefix-match)) {
/* match can be copied as a single segment from external dictionary */
memmove(op, dictEnd - (lowPrefix-match), length);
op += length;
} else {
/* match encompass external dictionary and current block */
size_t const copySize = (size_t)(lowPrefix-match);
size_t const restSize = length - copySize;
memcpy(op, dictEnd - copySize, copySize);
op += copySize;
if (restSize > (size_t)(op-lowPrefix)) { /* overlap copy */
BYTE* const endOfMatch = op + restSize;
const BYTE* copyFrom = lowPrefix;
while (op < endOfMatch) {
*op++ = *copyFrom++;
}
} else {
memcpy(op, lowPrefix, restSize);
op += restSize;
}
}
continue;
}
/* copy match within block */
cpy = op + length;
if (unlikely(offset<8)) {
const int dec64 = dec64table[offset];
op[0] = match[0];
op[1] = match[1];
op[2] = match[2];
op[3] = match[3];
match += dec32table[offset];
memcpy(op+4, match, 4);
match -= dec64;
} else {
LZ4_copy8(op, match);
match+=8;
}
op += 8;
if (unlikely(cpy>oend-12)) {
BYTE* const oCopyLimit = oend-(WILDCOPYLENGTH-1);
if (cpy > oend-LASTLITERALS) {
goto _output_error; /* Error : last LASTLITERALS bytes must be literals (uncompressed) */
}
if (op < oCopyLimit) {
LZ4_wildCopy(op, match, oCopyLimit);
match += oCopyLimit - op;
op = oCopyLimit;
}
while (op<cpy) {
*op++ = *match++;
}
} else {
LZ4_copy8(op, match);
if (length>16) {
LZ4_wildCopy(op+8, match+8, cpy);
}
}
op=cpy; /* correction */
}
/* end of decoding */
if (endOnInput) {
return (int) (((char*)op)-dest); /* Nb of output bytes decoded */
} else {
return (int) (((const char*)ip)-source); /* Nb of input bytes read */
}
/* Overflow error detected */
_output_error:
return (int) (-(((const char*)ip)-source))-1;
}
static inline int LZ4_decompress_safe(const char* source, char* dest, int compressedSize, int maxDecompressedSize)
{
return LZ4_decompress_generic(source, dest, compressedSize, maxDecompressedSize, endOnInputSize, full, 0, noDict, (BYTE*)dest, NULL, 0);
}
} // anonymous namespace
/************************************************************************** */
/************************************************************************** */
const unsigned char Packet::ZERO_KEY[32] = { 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0 };
void Packet::armor(const void *key,bool encryptPayload,const AES aesKeys[2])
{
uint8_t *const data = reinterpret_cast<uint8_t *>(unsafeData());
if ((aesKeys) && (encryptPayload)) {
//char tmp0[16],tmp1[16];
setCipher(ZT_PROTO_CIPHER_SUITE__AES_GMAC_SIV);
uint8_t *const payload = data + ZT_PACKET_IDX_VERB;
const unsigned int payloadLen = size() - ZT_PACKET_IDX_VERB;
AES::GMACSIVEncryptor enc(aesKeys[0],aesKeys[1]);
enc.init(Utils::loadMachineEndian<uint64_t>(data + ZT_PACKET_IDX_IV),payload);
enc.aad(data + ZT_PACKET_IDX_DEST,11);
enc.update1(payload,payloadLen);
enc.finish1();
enc.update2(payload,payloadLen);
const uint64_t *const tag = enc.finish2();
#ifdef ZT_NO_UNALIGNED_ACCESS
Utils::copy<8>(data,tag);
Utils::copy<8>(data + ZT_PACKET_IDX_MAC,tag + 1);
#else
*reinterpret_cast<uint64_t *>(data + ZT_PACKET_IDX_IV) = tag[0];
*reinterpret_cast<uint64_t *>(data + ZT_PACKET_IDX_MAC) = tag[1];
#endif
} else {
setCipher(encryptPayload ? ZT_PROTO_CIPHER_SUITE__C25519_POLY1305_SALSA2012 : ZT_PROTO_CIPHER_SUITE__C25519_POLY1305_NONE);
uint8_t mangledKey[32];
_salsa20MangleKey((const unsigned char *)key,mangledKey);
if (ZT_HAS_FAST_CRYPTO()) {
const unsigned int payloadLen = (encryptPayload) ? (size() - ZT_PACKET_IDX_VERB) : 0;
uint64_t keyStream[(ZT_PROTO_MAX_PACKET_LENGTH + 64 + 8) / 8];
ZT_FAST_SINGLE_PASS_SALSA2012(keyStream,payloadLen + 64,(data + ZT_PACKET_IDX_IV),mangledKey);
Salsa20::memxor(data + ZT_PACKET_IDX_VERB,reinterpret_cast<const uint8_t *>(keyStream + 8),payloadLen);
uint64_t mac[2];
Poly1305::compute(mac,data + ZT_PACKET_IDX_VERB,size() - ZT_PACKET_IDX_VERB,keyStream);
#ifdef ZT_NO_TYPE_PUNNING
memcpy(data + ZT_PACKET_IDX_MAC,mac,8);
#else
(*reinterpret_cast<uint64_t *>(data + ZT_PACKET_IDX_MAC)) = mac[0];
#endif
} else {
Salsa20 s20(mangledKey,data + ZT_PACKET_IDX_IV);
uint64_t macKey[4];
s20.crypt12(ZERO_KEY,macKey,sizeof(macKey));
uint8_t *const payload = data + ZT_PACKET_IDX_VERB;
const unsigned int payloadLen = size() - ZT_PACKET_IDX_VERB;
if (encryptPayload) {
s20.crypt12(payload,payload,payloadLen);
}
uint64_t mac[2];
Poly1305::compute(mac,payload,payloadLen,macKey);
memcpy(data + ZT_PACKET_IDX_MAC,mac,8);
}
}
}
bool Packet::dearmor(const void *key,const AES aesKeys[2])
{
uint8_t *const data = reinterpret_cast<uint8_t *>(unsafeData());
const unsigned int payloadLen = size() - ZT_PACKET_IDX_VERB;
unsigned char *const payload = data + ZT_PACKET_IDX_VERB;
const unsigned int cs = cipher();
if (cs == ZT_PROTO_CIPHER_SUITE__AES_GMAC_SIV) {
if (aesKeys) {
uint64_t tag[2];
#ifdef ZT_NO_UNALIGNED_ACCESS
Utils::copy<8>(tag, data);
Utils::copy<8>(tag + 1, data + ZT_PACKET_IDX_MAC);
#else
tag[0] = *reinterpret_cast<uint64_t *>(data + ZT_PACKET_IDX_IV);
tag[1] = *reinterpret_cast<uint64_t *>(data + ZT_PACKET_IDX_MAC);
#endif
AES::GMACSIVDecryptor dec(aesKeys[0],aesKeys[1]);
dec.init(tag, payload);
const uint8_t oldFlags = data[ZT_PACKET_IDX_FLAGS];
data[ZT_PACKET_IDX_FLAGS] &= 0xf8;
dec.aad(data + ZT_PACKET_IDX_DEST,11);
data[ZT_PACKET_IDX_FLAGS] = oldFlags;
dec.update(payload, payloadLen);
return dec.finish();
}
} else if ((cs == ZT_PROTO_CIPHER_SUITE__C25519_POLY1305_NONE)||(cs == ZT_PROTO_CIPHER_SUITE__C25519_POLY1305_SALSA2012)) {
uint8_t mangledKey[32];
_salsa20MangleKey((const unsigned char *)key,mangledKey);
if (ZT_HAS_FAST_CRYPTO()) {
uint64_t keyStream[(ZT_PROTO_MAX_PACKET_LENGTH + 64 + 8) / 8];
ZT_FAST_SINGLE_PASS_SALSA2012(keyStream,((cs == ZT_PROTO_CIPHER_SUITE__C25519_POLY1305_SALSA2012) ? (payloadLen + 64) : 64),(data + ZT_PACKET_IDX_IV),mangledKey);
uint64_t mac[2];
Poly1305::compute(mac,payload,payloadLen,keyStream);
#ifdef ZT_NO_TYPE_PUNNING
if (!Utils::secureEq(mac,data + ZT_PACKET_IDX_MAC,8)) {
return false;
}
#else
if ((*reinterpret_cast<const uint64_t *>(data + ZT_PACKET_IDX_MAC)) != mac[0]) { // also secure, constant time
return false;
}
#endif
if (cs == ZT_PROTO_CIPHER_SUITE__C25519_POLY1305_SALSA2012) {
Salsa20::memxor(data + ZT_PACKET_IDX_VERB,reinterpret_cast<const uint8_t *>(keyStream + 8),payloadLen);
}
} else {
Salsa20 s20(mangledKey,data + ZT_PACKET_IDX_IV);
uint64_t macKey[4];
s20.crypt12(ZERO_KEY,macKey,sizeof(macKey));
uint64_t mac[2];
Poly1305::compute(mac,payload,payloadLen,macKey);
#ifdef ZT_NO_TYPE_PUNNING
if (!Utils::secureEq(mac,data + ZT_PACKET_IDX_MAC,8)) {
return false;
}
#else
if ((*reinterpret_cast<const uint64_t *>(data + ZT_PACKET_IDX_MAC)) != mac[0]) { // also secure, constant time
return false;
}
#endif
if (cs == ZT_PROTO_CIPHER_SUITE__C25519_POLY1305_SALSA2012) {
s20.crypt12(payload,payload,payloadLen);
}
}
return true;
}
return false;
}
void Packet::cryptField(const void *key,unsigned int start,unsigned int len)
{
uint8_t *const data = reinterpret_cast<uint8_t *>(unsafeData());
uint8_t iv[8];
for(int i=0;i<8;++i) {
iv[i] = data[i];
}
iv[7] &= 0xf8; // mask off least significant 3 bits of packet ID / IV since this is unset when this function gets called
Salsa20 s20(key,iv);
s20.crypt12(data + start,data + start,len);
}
bool Packet::compress()
{
char *const data = reinterpret_cast<char *>(unsafeData());
char buf[ZT_PROTO_MAX_PACKET_LENGTH * 2];
if ((!compressed())&&(size() > (ZT_PACKET_IDX_PAYLOAD + 64))) { // don't bother compressing tiny packets
int pl = (int)(size() - ZT_PACKET_IDX_PAYLOAD);
int cl = LZ4_compress_fast(data + ZT_PACKET_IDX_PAYLOAD,buf,pl,ZT_PROTO_MAX_PACKET_LENGTH * 2,1);
if ((cl > 0)&&(cl < pl)) {
data[ZT_PACKET_IDX_VERB] |= (char)ZT_PROTO_VERB_FLAG_COMPRESSED;
setSize((unsigned int)cl + ZT_PACKET_IDX_PAYLOAD);
memcpy(data + ZT_PACKET_IDX_PAYLOAD,buf,cl);
return true;
}
}
data[ZT_PACKET_IDX_VERB] &= (char)(~ZT_PROTO_VERB_FLAG_COMPRESSED);
return false;
}
bool Packet::uncompress()
{
char *const data = reinterpret_cast<char *>(unsafeData());
char buf[ZT_PROTO_MAX_PACKET_LENGTH];
if ((compressed())&&(size() >= ZT_PROTO_MIN_PACKET_LENGTH)) {
if (size() > ZT_PACKET_IDX_PAYLOAD) {
unsigned int compLen = size() - ZT_PACKET_IDX_PAYLOAD;
int ucl = LZ4_decompress_safe((const char *)data + ZT_PACKET_IDX_PAYLOAD,buf,compLen,sizeof(buf));
if ((ucl > 0)&&(ucl <= (int)(capacity() - ZT_PACKET_IDX_PAYLOAD))) {
setSize((unsigned int)ucl + ZT_PACKET_IDX_PAYLOAD);
memcpy(data + ZT_PACKET_IDX_PAYLOAD,buf,ucl);
} else {
return false;
}
}
data[ZT_PACKET_IDX_VERB] &= (char)(~ZT_PROTO_VERB_FLAG_COMPRESSED);
}
return true;
}
} // namespace ZeroTier