ExternalVendorCode/ZeroTierOne
1
0
Fork 0
mirror of https://github.com/zerotier/ZeroTierOne.git synced 2025-01-18 02:40:13 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity

Get rid of DBM, which technically is a case of YAGNI. Supernodes will need a way to save identities, but that can be a different feature. Regular clients do not really need a permanent cache (yet). When/if we do need one we can do it then. Until then it only caused problems.

Browse Source
This commit is contained in:
Adam Ierymenko 2013-10-21 10:29:44 -04:00
parent bbfd43e036
commit 6e217dfcb0
12 changed files with 50 additions and 918 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
Makefile.linux
View File

@ -7,7 +7,7 @@ DEFS=-DZT_ARCH="$(ARCH)" -DZT_OSNAME="linux" -DZT_TRACE
LIBS=
# Uncomment for a release optimized build
CFLAGS=-Wall -O3 -fno-unroll-loops -fstack-protector -pthread $(INCLUDES) -DNDEBUG $(DEFS)
CFLAGS=-Wall -O3 -fno-unroll-loops -fvisibility=hidden -fstack-protector -pthread $(INCLUDES) -DNDEBUG $(DEFS)
STRIP=strip --strip-all
# Uncomment for a debug build

70
ext/kissdb/README.md
View File

@ -1,70 +0,0 @@
kissdb
======
(Keep It) Simple Stupid Database
KISSDB is about the simplest key/value store you'll ever see, anywhere.
It's written in plain vanilla C using only the standard string and FILE
I/O functions, and should port to just about anything with a disk or
something that acts like one.
It stores keys and values of fixed length in a stupid-simple file format
based on fixed-size hash tables. If a hash collision occurrs, a new "page"
of hash table is appended to the database. The format is append-only.
There is no delete. Puts that replace an existing value, however, will not
grow the file as they will overwrite the existing entry.
Hash table size is a space/speed trade-off parameter. Larger hash tables
will reduce collisions and speed things up a bit, at the expense of memory
and disk space. A good size is usually about 1/2 the average number of
entries you expect.
Features:
* Tiny, compiles to ~4k on an x86_64 Linux system
* Small memory footprint (only caches hash tables)
* Very space-efficient (on disk) if small hash tables are used
* Makes a decent effort to be robust on power loss
* Pretty respectably fast, especially given its simplicity
* 64-bit, file size limit is 2^64 bytes
* Ports to anything with a C compiler and stdlib/stdio
* Public domain
Limitations:
* Fixed-size keys and values, must recreate and copy to change any init size parameter
* Add/update only, no delete
* Iteration is supported but key order is undefined
* No search for subsets of keys/values
* No indexes
* No transactions
* No special recovery features if a database gets corrupted
* No built-in thread-safety (guard it with a mutex in MT code)
* No built-in caching of data (only hash tables are cached for lookup speed)
* No endian-awareness (currently), so big-endian DBs won't read on little-endian machines
Alternative key/value stores and embedded databases:
* [MDB](http://symas.com/mdb/) uses mmap() and is very fast (not quite as tiny/simple/portable)
* [CDB](http://cr.yp.to/cdb.html) is also minimal and fast, probably the closest thing to this (but has a 4gb size limit)
* [Kyoto Cabinet](http://fallabs.com/kyotocabinet/) is very fast, full-featured, and modern (license required for commercial use)
* [SQLite](http://www.sqlite.org/) gives you a complete embedded SQL server (public domain, very mature, much larger)
* Others include GDBM, NDBM, Berkeley DB, etc. Use your Googles. :)
KISSDB is good if you want space-efficient relatively fast write-once/read-many storage
of keys mapped to values. It's not a good choice if you need searches, indexes, delete,
structured storage, or widely varying key/value sizes. It's also probably not a good
choice if you need a long-lived database for critical data, since it lacks recovery
features and is brittle if its internals are modified. It would be better for a cache
of data that can be restored or "re-learned," such as keys, Bitcoin transactions, nodes
on a peer-to-peer network, log analysis results, rendered web pages, session cookies,
auth tokens, etc.
KISSDB is in the public domain. One reason it was written was the
poverty of simple key/value databases with wide open licensing. Even old
ones like GDBM have GPL, not LGPL, licenses.
See comments in kissdb.h for documentation. Makefile can be used to build
a test program on systems with gcc.
Author: Adam Ierymenko / ZeroTier Networks LLC

60
ext/kissdb/SPEC.txt
View File

@ -1,60 +0,0 @@
-----
KISSDB file format (version 2)
Author: Adam Ierymenko <adam.ierymenko@zerotier.com>
-----
In keeping with the goal of minimalism the file format is very simple, the
sort of thing that would be given as an example in an introductory course in
data structures. It's a basic hash table that adds additional pages of hash
table entries on collision.
It consists of a 28 byte header followed by a series of hash tables and data.
All integer values are stored in the native word order of the target
architecture (in the future the code might be fixed to make everything
little-endian if anyone cares about that).
The header consists of the following fields:
[0-3] magic numbers: (ASCII) 'K', 'd', 'B', KISSDB_VERSION (currently 2)
[4-11] 64-bit hash table size in entries
[12-19] 64-bit key size in bytes
[20-27] 64-bit value size in bytes
Hash tables are arrays of [hash table size + 1] 64-bit integers. The extra
entry, if nonzero, is the offset in the file of the next hash table, forming
a linked list of hash tables across the file.
Immediately following the header, the first hash table will be written when
the first key/value is added. The algorithm for adding new entries is as
follows:
(1) The key is hashed using a 64-bit variant of the DJB2 hash function, and
this is taken modulo hash table size to get a bucket number.
(2) Hash tables are checked in order, starting with the first hash table,
until a zero (empty) bucket is found. If one is found, skip to step (4).
(3) If no empty buckets are found in any hash table, a new table is appended
to the file and the final pointer in the previous hash table is set to
its offset. (In the code the update of the next hash table pointer in
the previous hash table happens last, after the whole write is complete,
to avoid corruption on power loss.)
(4) The key and value are appended, in order with no additional meta-data,
to the database file. Before appending the offset in the file stream
where they will be stored is saved. After appending, this offset is
written to the empty hash table bucket we chose in steps 2/3. Hash table
updates happen last to avoid corruption if the write does not complete.
Lookup of a key/value pair occurs as follows:
(1) The key is hashed and taken modulo hash table size to get a bucket
number.
(2) If this bucket's entry in the hash table is nonzero, the key at the
offset specified by this bucket is compared to the key being looked up.
If they are equal, the value is read and returned.
(3) If the keys are not equal, the next hash table is checked and step (2)
is repeated. If an empty bucket is encountered or if we run out of hash
tables, the key was not found.
To update an existing value, its location is looked up and the value portion
of the entry is rewritten.

447
ext/kissdb/kissdb.c
View File

@ -1,447 +0,0 @@
/* (Keep It) Simple Stupid Database
*
* Written by Adam Ierymenko <adam.ierymenko@zerotier.com>
* KISSDB is in the public domain and is distributed with NO WARRANTY. */
/* Compile with KISSDB_TEST to build as a test program. */
/* Note: big-endian systems will need changes to implement byte swapping
* on hash table file I/O. Or you could just use it as-is if you don't care
* that your database files will be unreadable on little-endian systems. */
#define _FILE_OFFSET_BITS 64
#include "kissdb.h"
#include <string.h>
#include <stdlib.h>
#include <stdint.h>
#ifdef _WIN32
#define fseeko _fseeki64
#define ftello _ftelli64
#endif
#define KISSDB_HEADER_SIZE ((sizeof(uint64_t) * 3) + 4)
/* djb2 hash function */
static uint64_t KISSDB_hash(const void *b,unsigned long len)
{
unsigned long i;
uint64_t hash = 5381;
for(i=0;i<len;++i)
hash = ((hash << 5) + hash) + (uint64_t)(((const uint8_t *)b)[i]);
return hash;
}
int KISSDB_open(
KISSDB *db,
const char *path,
int mode,
unsigned long hash_table_size,
unsigned long key_size,
unsigned long value_size)
{
uint64_t tmp;
uint8_t tmp2[4];
uint64_t *httmp;
uint64_t *hash_tables_rea;
#ifdef _WIN32
db->f = (FILE *)0;
fopen_s(&db->f,path,((mode == KISSDB_OPEN_MODE_RWREPLACE) ? "w+b" : (((mode == KISSDB_OPEN_MODE_RDWR)||(mode == KISSDB_OPEN_MODE_RWCREAT)) ? "r+b" : "rb")));
#else
db->f = fopen(path,((mode == KISSDB_OPEN_MODE_RWREPLACE) ? "w+b" : (((mode == KISSDB_OPEN_MODE_RDWR)||(mode == KISSDB_OPEN_MODE_RWCREAT)) ? "r+b" : "rb")));
#endif
if (!db->f) {
if (mode == KISSDB_OPEN_MODE_RWCREAT) {
#ifdef _WIN32
db->f = (FILE *)0;
fopen_s(&db->f,path,"w+b");
#else
db->f = fopen(path,"w+b");
#endif
}
if (!db->f)
return KISSDB_ERROR_IO;
}
if (fseeko(db->f,0,SEEK_END)) {
fclose(db->f);
return KISSDB_ERROR_IO;
}
if (ftello(db->f) < KISSDB_HEADER_SIZE) {
/* write header if not already present */
if ((hash_table_size)&&(key_size)&&(value_size)) {
if (fseeko(db->f,0,SEEK_SET)) { fclose(db->f); return KISSDB_ERROR_IO; }
tmp2[0] = 'K'; tmp2[1] = 'd'; tmp2[2] = 'B'; tmp2[3] = KISSDB_VERSION;
if (fwrite(tmp2,4,1,db->f) != 1) { fclose(db->f); return KISSDB_ERROR_IO; }
tmp = hash_table_size;
if (fwrite(&tmp,sizeof(uint64_t),1,db->f) != 1) { fclose(db->f); return KISSDB_ERROR_IO; }
tmp = key_size;
if (fwrite(&tmp,sizeof(uint64_t),1,db->f) != 1) { fclose(db->f); return KISSDB_ERROR_IO; }
tmp = value_size;
if (fwrite(&tmp,sizeof(uint64_t),1,db->f) != 1) { fclose(db->f); return KISSDB_ERROR_IO; }
fflush(db->f);
} else {
fclose(db->f);
return KISSDB_ERROR_INVALID_PARAMETERS;
}
} else {
if (fseeko(db->f,0,SEEK_SET)) { fclose(db->f); return KISSDB_ERROR_IO; }
if (fread(tmp2,4,1,db->f) != 1) { fclose(db->f); return KISSDB_ERROR_IO; }
if ((tmp2[0] != 'K')||(tmp2[1] != 'd')||(tmp2[2] != 'B')||(tmp2[3] != KISSDB_VERSION)) {
fclose(db->f);
return KISSDB_ERROR_CORRUPT_DBFILE;
}
if (fread(&tmp,sizeof(uint64_t),1,db->f) != 1) { fclose(db->f); return KISSDB_ERROR_IO; }
if (!tmp) {
fclose(db->f);
return KISSDB_ERROR_CORRUPT_DBFILE;
}
hash_table_size = (unsigned long)tmp;
if (fread(&tmp,sizeof(uint64_t),1,db->f) != 1) { fclose(db->f); return KISSDB_ERROR_IO; }
if (!tmp) {
fclose(db->f);
return KISSDB_ERROR_CORRUPT_DBFILE;
}
key_size = (unsigned long)tmp;
if (fread(&tmp,sizeof(uint64_t),1,db->f) != 1) { fclose(db->f); return KISSDB_ERROR_IO; }
if (!tmp) {
fclose(db->f);
return KISSDB_ERROR_CORRUPT_DBFILE;
}
value_size = (unsigned long)tmp;
}
db->hash_table_size = hash_table_size;
db->key_size = key_size;
db->value_size = value_size;
db->hash_table_size_bytes = sizeof(uint64_t) * (hash_table_size + 1); /* [hash_table_size] == next table */
httmp = malloc(db->hash_table_size_bytes);
if (!httmp) {
fclose(db->f);
return KISSDB_ERROR_MALLOC;
}
db->num_hash_tables = 0;
db->hash_tables = (uint64_t *)0;
while (fread(httmp,db->hash_table_size_bytes,1,db->f) == 1) {
hash_tables_rea = realloc(db->hash_tables,db->hash_table_size_bytes * (db->num_hash_tables + 1));
if (!hash_tables_rea) {
KISSDB_close(db);
free(httmp);
return KISSDB_ERROR_MALLOC;
}
db->hash_tables = hash_tables_rea;
memcpy(((uint8_t *)db->hash_tables) + (db->hash_table_size_bytes * db->num_hash_tables),httmp,db->hash_table_size_bytes);
++db->num_hash_tables;
if (httmp[db->hash_table_size]) {
if (fseeko(db->f,httmp[db->hash_table_size],SEEK_SET)) {
KISSDB_close(db);
free(httmp);
return KISSDB_ERROR_IO;
}
} else break;
}
free(httmp);
return 0;
}
void KISSDB_close(KISSDB *db)
{
if (db->hash_tables)
free(db->hash_tables);
if (db->f)
fclose(db->f);
memset(db,0,sizeof(KISSDB));
}
int KISSDB_get(KISSDB *db,const void *key,void *vbuf)
{
uint8_t tmp[4096];
const uint8_t *kptr;
unsigned long klen,i;
uint64_t hash = KISSDB_hash(key,db->key_size) % (uint64_t)db->hash_table_size;
uint64_t offset;
uint64_t *cur_hash_table;
long n;
cur_hash_table = db->hash_tables;
for(i=0;i<db->num_hash_tables;++i) {
offset = cur_hash_table[hash];
if (offset) {
if (fseeko(db->f,offset,SEEK_SET))
return KISSDB_ERROR_IO;
kptr = (const uint8_t *)key;
klen = db->key_size;
while (klen) {
n = (long)fread(tmp,1,(klen > sizeof(tmp)) ? sizeof(tmp) : klen,db->f);
if (n > 0) {
if (memcmp(kptr,tmp,n))
goto get_no_match_next_hash_table;
kptr += n;
klen -= (unsigned long)n;
} else return 1; /* not found */
}
if (fread(vbuf,db->value_size,1,db->f) == 1)
return 0; /* success */
else return KISSDB_ERROR_IO;
} else return 1; /* not found */
get_no_match_next_hash_table:
cur_hash_table += db->hash_table_size + 1;
}
return 1; /* not found */
}
int KISSDB_put(KISSDB *db,const void *key,const void *value)
{
uint8_t tmp[4096];
const uint8_t *kptr;
unsigned long klen,i;
uint64_t hash = KISSDB_hash(key,db->key_size) % (uint64_t)db->hash_table_size;
uint64_t offset;
uint64_t htoffset,lasthtoffset;
uint64_t endoffset;
uint64_t *cur_hash_table;
uint64_t *hash_tables_rea;
long n;
lasthtoffset = htoffset = KISSDB_HEADER_SIZE;
cur_hash_table = db->hash_tables;
for(i=0;i<db->num_hash_tables;++i) {
offset = cur_hash_table[hash];
if (offset) {
/* rewrite if already exists */
if (fseeko(db->f,offset,SEEK_SET))
return KISSDB_ERROR_IO;
kptr = (const uint8_t *)key;
klen = db->key_size;
while (klen) {
n = (long)fread(tmp,1,(klen > sizeof(tmp)) ? sizeof(tmp) : klen,db->f);
if (n > 0) {
if (memcmp(kptr,tmp,n))
goto put_no_match_next_hash_table;
kptr += n;
klen -= (unsigned long)n;
}
}
if (fwrite(value,db->value_size,1,db->f) == 1) {
fflush(db->f);
return 0; /* success */
} else return KISSDB_ERROR_IO;
} else {
/* add if an empty hash table slot is discovered */
if (fseeko(db->f,0,SEEK_END))
return KISSDB_ERROR_IO;
endoffset = ftello(db->f);
if (fwrite(key,db->key_size,1,db->f) != 1)
return KISSDB_ERROR_IO;
if (fwrite(value,db->value_size,1,db->f) != 1)
return KISSDB_ERROR_IO;
if (fseeko(db->f,htoffset + (sizeof(uint64_t) * hash),SEEK_SET))
return KISSDB_ERROR_IO;
if (fwrite(&endoffset,sizeof(uint64_t),1,db->f) != 1)
return KISSDB_ERROR_IO;
cur_hash_table[hash] = endoffset;
fflush(db->f);
return 0; /* success */
}
put_no_match_next_hash_table:
lasthtoffset = htoffset;
htoffset = cur_hash_table[db->hash_table_size];
cur_hash_table += (db->hash_table_size + 1);
}
/* if no existing slots, add a new page of hash table entries */
if (fseeko(db->f,0,SEEK_END))
return KISSDB_ERROR_IO;
endoffset = ftello(db->f);
hash_tables_rea = realloc(db->hash_tables,db->hash_table_size_bytes * (db->num_hash_tables + 1));
if (!hash_tables_rea)
return KISSDB_ERROR_MALLOC;
db->hash_tables = hash_tables_rea;
cur_hash_table = &(db->hash_tables[(db->hash_table_size + 1) * db->num_hash_tables]);
memset(cur_hash_table,0,db->hash_table_size_bytes);
cur_hash_table[hash] = endoffset + db->hash_table_size_bytes; /* where new entry will go */
if (fwrite(cur_hash_table,db->hash_table_size_bytes,1,db->f) != 1)
return KISSDB_ERROR_IO;
if (fwrite(key,db->key_size,1,db->f) != 1)
return KISSDB_ERROR_IO;
if (fwrite(value,db->value_size,1,db->f) != 1)
return KISSDB_ERROR_IO;
if (db->num_hash_tables) {
if (fseeko(db->f,lasthtoffset + (sizeof(uint64_t) * db->hash_table_size),SEEK_SET))
return KISSDB_ERROR_IO;
if (fwrite(&endoffset,sizeof(uint64_t),1,db->f) != 1)
return KISSDB_ERROR_IO;
db->hash_tables[((db->hash_table_size + 1) * (db->num_hash_tables - 1)) + db->hash_table_size] = endoffset;
}
++db->num_hash_tables;
fflush(db->f);
return 0; /* success */
}
void KISSDB_Iterator_init(KISSDB *db,KISSDB_Iterator *dbi)
{
dbi->db = db;
dbi->h_no = 0;
dbi->h_idx = 0;
}
int KISSDB_Iterator_next(KISSDB_Iterator *dbi,void *kbuf,void *vbuf)
{
uint64_t offset;
if ((dbi->h_no < dbi->db->num_hash_tables)&&(dbi->h_idx < dbi->db->hash_table_size)) {
while (!(offset = dbi->db->hash_tables[((dbi->db->hash_table_size + 1) * dbi->h_no) + dbi->h_idx])) {
if (++dbi->h_idx >= dbi->db->hash_table_size) {
dbi->h_idx = 0;
if (++dbi->h_no >= dbi->db->num_hash_tables)
return 0;
}
}
if (fseeko(dbi->db->f,offset,SEEK_SET))
return KISSDB_ERROR_IO;
if (fread(kbuf,dbi->db->key_size,1,dbi->db->f) != 1)
return KISSDB_ERROR_IO;
if (fread(vbuf,dbi->db->value_size,1,dbi->db->f) != 1)
return KISSDB_ERROR_IO;
if (++dbi->h_idx >= dbi->db->hash_table_size) {
dbi->h_idx = 0;
++dbi->h_no;
}
return 1;
}
return 0;
}
#ifdef KISSDB_TEST
#include <inttypes.h>
int main(int argc,char **argv)
{
uint64_t i,j;
uint64_t v[8];
KISSDB db;
KISSDB_Iterator dbi;
char got_all_values[10000];
int q;
printf("Opening new empty database test.db...\n");
if (KISSDB_open(&db,"test.db",KISSDB_OPEN_MODE_RWREPLACE,1024,8,sizeof(v))) {
printf("KISSDB_open failed\n");
return 1;
}
printf("Adding and then re-getting 10000 64-byte values...\n");
for(i=0;i<10000;++i) {
for(j=0;j<8;++j)
v[j] = i;
if (KISSDB_put(&db,&i,v)) {
printf("KISSDB_put failed (%"PRIu64")\n",i);
return 1;
}
memset(v,0,sizeof(v));
if ((q = KISSDB_get(&db,&i,v))) {
printf("KISSDB_get (1) failed (%"PRIu64") (%d)\n",i,q);
return 1;
}
for(j=0;j<8;++j) {
if (v[j] != i) {
printf("KISSDB_get (1) failed, bad data (%"PRIu64")\n",i);
return 1;
}
}
}
printf("Getting 10000 64-byte values...\n");
for(i=0;i<10000;++i) {
if ((q = KISSDB_get(&db,&i,v))) {
printf("KISSDB_get (2) failed (%"PRIu64") (%d)\n",i,q);
return 1;
}
for(j=0;j<8;++j) {
if (v[j] != i) {
printf("KISSDB_get (2) failed, bad data (%"PRIu64")\n",i);
return 1;
}
}
}
printf("Closing and re-opening database in read-only mode...\n");
KISSDB_close(&db);
if (KISSDB_open(&db,"test.db",KISSDB_OPEN_MODE_RDONLY,1024,8,sizeof(v))) {
printf("KISSDB_open failed\n");
return 1;
}
printf("Getting 10000 64-byte values...\n");
for(i=0;i<10000;++i) {
if ((q = KISSDB_get(&db,&i,v))) {
printf("KISSDB_get (3) failed (%"PRIu64") (%d)\n",i,q);
return 1;
}
for(j=0;j<8;++j) {
if (v[j] != i) {
printf("KISSDB_get (3) failed, bad data (%"PRIu64")\n",i);
return 1;
}
}
}
printf("Iterator test...\n");
KISSDB_Iterator_init(&db,&dbi);
i = 0xdeadbeef;
memset(got_all_values,0,sizeof(got_all_values));
while (KISSDB_Iterator_next(&dbi,&i,&v) > 0) {
if (i < 10000)
got_all_values[i] = 1;
else {
printf("KISSDB_Iterator_next failed, bad data (%"PRIu64")\n",i);
return 1;
}
}
for(i=0;i<10000;++i) {
if (!got_all_values[i]) {
printf("KISSDB_Iterator failed, missing value index %"PRIu64"\n",i);
return 1;
}
}
KISSDB_close(&db);
printf("All tests OK!\n");
return 0;
}
#endif

171
ext/kissdb/kissdb.h
View File

@ -1,171 +0,0 @@
/* (Keep It) Simple Stupid Database
*
* Written by Adam Ierymenko <adam.ierymenko@zerotier.com>
* KISSDB is in the public domain and is distributed with NO WARRANTY. */
#ifndef ___KISSDB_H
#define ___KISSDB_H
#include <stdio.h>
#include <stdint.h>
#ifdef __cplusplus
extern "C" {
#endif
/**
* Version: 2
*
* This is the file format identifier, and changes any time the file
* format changes. The code version will be this dot something, and can
* be seen in tags in the git repository.
*/
#define KISSDB_VERSION 2
/**
* KISSDB database state
*
* These fields can be read by a user, e.g. to look up key_size and
* value_size, but should never be changed.
*/
typedef struct {
unsigned long hash_table_size;
unsigned long key_size;
unsigned long value_size;
unsigned long hash_table_size_bytes;
unsigned long num_hash_tables;
uint64_t *hash_tables;
FILE *f;
} KISSDB;
/**
* I/O error or file not found
*/
#define KISSDB_ERROR_IO -1
/**
* Out of memory
*/
#define KISSDB_ERROR_MALLOC -2
/**
* Invalid paramters (e.g. missing _size paramters on init to create database)
*/
#define KISSDB_ERROR_INVALID_PARAMETERS -3
/**
* Database file appears corrupt
*/
#define KISSDB_ERROR_CORRUPT_DBFILE -4
/**
* Open mode: read only
*/
#define KISSDB_OPEN_MODE_RDONLY 1
/**
* Open mode: read/write
*/
#define KISSDB_OPEN_MODE_RDWR 2
/**
* Open mode: read/write, create if doesn't exist
*/
#define KISSDB_OPEN_MODE_RWCREAT 3
/**
* Open mode: truncate database, open for reading and writing
*/
#define KISSDB_OPEN_MODE_RWREPLACE 4
/**
* Open database
*
* The three _size parameters must be specified if the database could
* be created or re-created. Otherwise an error will occur. If the
* database already exists, these parameters are ignored and are read
* from the database. You can check the struture afterwords to see what
* they were.
*
* @param db Database struct
* @param path Path to file
* @param mode One of the KISSDB_OPEN_MODE constants
* @param hash_table_size Size of hash table in 64-bit entries (must be >0)
* @param key_size Size of keys in bytes
* @param value_size Size of values in bytes
* @return 0 on success, nonzero on error
*/
extern int KISSDB_open(
KISSDB *db,
const char *path,
int mode,
unsigned long hash_table_size,
unsigned long key_size,
unsigned long value_size);
/**
* Close database
*
* @param db Database struct
*/
extern void KISSDB_close(KISSDB *db);
/**
* Get an entry
*
* @param db Database struct
* @param key Key (key_size bytes)
* @param vbuf Value buffer (value_size bytes capacity)
* @return -1 on I/O error, 0 on success, 1 on not found
*/
extern int KISSDB_get(KISSDB *db,const void *key,void *vbuf);
/**
* Put an entry (overwriting it if it already exists)
*
* In the already-exists case the size of the database file does not
* change.
*
* @param db Database struct
* @param key Key (key_size bytes)
* @param value Value (value_size bytes)
* @return -1 on I/O error, 0 on success
*/
extern int KISSDB_put(KISSDB *db,const void *key,const void *value);
/**
* Cursor used for iterating over all entries in database
*/
typedef struct {
KISSDB *db;
unsigned long h_no;
unsigned long h_idx;
} KISSDB_Iterator;
/**
* Initialize an iterator
*
* @param db Database struct
* @param i Iterator to initialize
*/
extern void KISSDB_Iterator_init(KISSDB *db,KISSDB_Iterator *dbi);
/**
* Get the next entry
*
* The order of entries returned by iterator is undefined. It depends on
* how keys hash.
*
* @param Database iterator
* @param kbuf Buffer to fill with next key (key_size bytes)
* @param vbuf Buffer to fill with next value (value_size bytes)
* @return 0 if there are no more entries, negative on error, positive if an kbuf/vbuf have been filled
*/
extern int KISSDB_Iterator_next(KISSDB_Iterator *dbi,void *kbuf,void *vbuf);
#ifdef __cplusplus
}
#endif
#endif

8
node/Identity.hpp
View File

@ -256,13 +256,15 @@ public:
p += ZT_ADDRESS_LENGTH;
if (b[p++] != IDENTITY_TYPE_C25519)
throw std::invalid_argument("Identity: deserialize(): unsupported identity type");
throw std::invalid_argument("unsupported identity type");
memcpy(_publicKey.data,b.field(p,_publicKey.size()),_publicKey.size());
p += _publicKey.size();
unsigned int privateKeyLength = b[p++];
if ((privateKeyLength)&&(privateKeyLength == ZT_C25519_PRIVATE_KEY_LEN)) {
unsigned int privateKeyLength = (unsigned int)b[p++];
if (privateKeyLength) {
if (privateKeyLength != ZT_C25519_PRIVATE_KEY_LEN)
throw std::invalid_argument("invalid private key");
_privateKey = new C25519::Private();
memcpy(_privateKey->data,b.field(p,ZT_C25519_PRIVATE_KEY_LEN),ZT_C25519_PRIVATE_KEY_LEN);
p += ZT_C25519_PRIVATE_KEY_LEN;

3
node/Node.cpp
View File

@ -380,6 +380,7 @@ Node::ReasonForTermination Node::run()
// Clean up some obsolete files if present -- this will be removed later
Utils::rm((_r->homePath + ZT_PATH_SEPARATOR_S + "status"));
Utils::rm((_r->homePath + ZT_PATH_SEPARATOR_S + "thisdeviceismine"));
Utils::rm((_r->homePath + ZT_PATH_SEPARATOR_S + "peer.db"));
// Make sure networks.d exists
#ifdef __WINDOWS__
@ -407,7 +408,7 @@ Node::ReasonForTermination Node::run()
_r->mc = new Multicaster();
_r->sw = new Switch(_r);
_r->demarc = new Demarc(_r);
_r->topology = new Topology(_r,(_r->homePath + ZT_PATH_SEPARATOR_S + "peer.db").c_str());
_r->topology = new Topology(_r);
_r->sysEnv = new SysEnv(_r);
try {
_r->nc = new NodeConfig(_r,configAuthToken.c_str(),impl->controlPort);

25
node/Peer.cpp
View File

@ -34,12 +34,13 @@ Peer::Peer() :
_id(),
_ipv4p(),
_ipv6p(),
_lastUsed(0),
_lastUnicastFrame(0),
_lastMulticastFrame(0),
_lastAnnouncedTo(0),
_vMajor(0),
_vMinor(0),
_vRevision(0),
_dirty(false)
_vRevision(0)
{
}
@ -48,12 +49,13 @@ Peer::Peer(const Identity &myIdentity,const Identity &peerIdentity)
_id(peerIdentity),
_ipv4p(),
_ipv6p(),
_lastUsed(0),
_lastUnicastFrame(0),
_lastMulticastFrame(0),
_lastAnnouncedTo(0),
_vMajor(0),
_vMinor(0),
_vRevision(0),
_dirty(true)
_vRevision(0)
{
if (!myIdentity.agree(peerIdentity,_key,ZT_PEER_SECRET_KEY_LENGTH))
throw std::runtime_error("new peer identity key agreement failed");
@ -72,16 +74,12 @@ void Peer::onReceive(const RuntimeEnvironment *_r,Demarc::Port localPort,const I
_lastAnnouncedTo = now;
_r->sw->announceMulticastGroups(SharedPtr<Peer>(this));
}
_dirty = true;
}
if (verb == Packet::VERB_FRAME) {
_lastUnicastFrame = now;
_dirty = true;
} else if (verb == Packet::VERB_MULTICAST_FRAME) {
_lastMulticastFrame = now;
_dirty = true;
}
}
@ -90,7 +88,6 @@ bool Peer::send(const RuntimeEnvironment *_r,const void *data,unsigned int len,u
if ((_ipv6p.isActive(now))||((!(_ipv4p.addr))&&(_ipv6p.addr))) {
if (_r->demarc->send(_ipv6p.localPort,_ipv6p.addr,data,len,-1)) {
_ipv6p.lastSend = now;
_dirty = true;
return true;
}
}
@ -98,7 +95,6 @@ bool Peer::send(const RuntimeEnvironment *_r,const void *data,unsigned int len,u
if (_ipv4p.addr) {
if (_r->demarc->send(_ipv4p.localPort,_ipv4p.addr,data,len,-1)) {
_ipv4p.lastSend = now;
_dirty = true;
return true;
}
}
@ -112,14 +108,12 @@ bool Peer::sendFirewallOpener(const RuntimeEnvironment *_r,uint64_t now)
if (_ipv4p.addr) {
if (_r->demarc->send(_ipv4p.localPort,_ipv4p.addr,"\0",1,ZT_FIREWALL_OPENER_HOPS)) {
_ipv4p.lastFirewallOpener = now;
_dirty = true;
sent = true;
}
}
if (_ipv6p.addr) {
if (_r->demarc->send(_ipv6p.localPort,_ipv6p.addr,"\0",1,ZT_FIREWALL_OPENER_HOPS)) {
_ipv6p.lastFirewallOpener = now;
_dirty = true;
sent = true;
}
}
@ -132,14 +126,12 @@ bool Peer::sendPing(const RuntimeEnvironment *_r,uint64_t now)
if (_ipv4p.addr) {
if (_r->sw->sendHELLO(SharedPtr<Peer>(this),_ipv4p.localPort,_ipv4p.addr)) {
_ipv4p.lastSend = now;
_dirty = true;
sent = true;
}
}
if (_ipv6p.addr) {
if (_r->sw->sendHELLO(SharedPtr<Peer>(this),_ipv6p.localPort,_ipv6p.addr)) {
_ipv6p.lastSend = now;
_dirty = true;
sent = true;
}
}
@ -151,11 +143,9 @@ void Peer::setPathAddress(const InetAddress &addr,bool fixed)
if (addr.isV4()) {
_ipv4p.addr = addr;
_ipv4p.fixed = fixed;
_dirty = true;
} else if (addr.isV6()) {
_ipv6p.addr = addr;
_ipv6p.fixed = fixed;
_dirty = true;
}
}
@ -165,15 +155,12 @@ void Peer::clearFixedFlag(InetAddress::AddressType t)
case InetAddress::TYPE_NULL:
_ipv4p.fixed = false;
_ipv6p.fixed = false;
_dirty = true;
break;
case InetAddress::TYPE_IPV4:
_ipv4p.fixed = false;
_dirty = true;
break;
case InetAddress::TYPE_IPV6:
_ipv6p.fixed = false;
_dirty = true;
break;
}
}

60
node/Peer.hpp
View File

@ -48,24 +48,6 @@
#include "NonCopyable.hpp"
#include "Mutex.hpp"
/**
* Max length of serialized peer record
*/
#define ZT_PEER_MAX_SERIALIZED_LENGTH ( \
ZT_PEER_SECRET_KEY_LENGTH + \
ZT_IDENTITY_MAX_BINARY_SERIALIZED_LENGTH + \
( ( \
(sizeof(uint64_t) * 4) + \
sizeof(uint16_t) + \
1 + \
sizeof(uint16_t) + \
16 + \
1 \
) * 2) + \
(sizeof(uint64_t) * 3) + \
(sizeof(uint16_t) * 3) \
)
namespace ZeroTier {
/**
@ -99,6 +81,16 @@ public:
Peer(const Identity &myIdentity,const Identity &peerIdentity)
throw(std::runtime_error);
/**
* @return Time peer record was last used in any way
*/
inline uint64_t lastUsed() const throw() { return _lastUsed; }
/**
* @param now New time of last use
*/
inline void setLastUsed(uint64_t now) throw() { _lastUsed = now; }
/**
* @return This peer's ZT address (short for identity().address())
*/
@ -254,10 +246,8 @@ public:
{
if (addr == _ipv4p.addr) {
_ipv4p.latency = latency;
_dirty = true;
} else if (addr == _ipv6p.addr) {
_ipv6p.latency = latency;
_dirty = true;
}
}
@ -357,33 +347,16 @@ public:
return std::string("?");
}
/**
* Get and reset dirty flag
*
* @return Previous value of dirty flag before reset
*/
inline bool getAndResetDirty()
throw()
{
bool d = _dirty;
_dirty = false;
return d;
}
/**
* @return Current value of dirty flag
*/
inline bool dirty() const throw() { return _dirty; }
template<unsigned int C>
inline void serialize(Buffer<C> &b)
throw(std::out_of_range)
{
b.append((unsigned char)3); // version
b.append((unsigned char)4); // version
b.append(_key,sizeof(_key));
_id.serialize(b,false);
_ipv4p.serialize(b);
_ipv6p.serialize(b);
b.append(_lastUsed);
b.append(_lastUnicastFrame);
b.append(_lastMulticastFrame);
b.append(_lastAnnouncedTo);
@ -398,13 +371,14 @@ public:
{
unsigned int p = startAt;
if (b[p++] != 3)
if (b[p++] != 4)
throw std::invalid_argument("Peer: deserialize(): version mismatch");
memcpy(_key,b.field(p,sizeof(_key)),sizeof(_key)); p += sizeof(_key);
p += _id.deserialize(b,p);
p += _ipv4p.deserialize(b,p);
p += _ipv6p.deserialize(b,p);
_lastUsed = b.template at<uint64_t>(p); p += sizeof(uint64_t);
_lastUnicastFrame = b.template at<uint64_t>(p); p += sizeof(uint64_t);
_lastMulticastFrame = b.template at<uint64_t>(p); p += sizeof(uint64_t);
_lastAnnouncedTo = b.template at<uint64_t>(p); p += sizeof(uint64_t);
@ -412,8 +386,6 @@ public:
_vMinor = b.template at<uint16_t>(p); p += sizeof(uint16_t);
_vRevision = b.template at<uint16_t>(p); p += sizeof(uint16_t);
_dirty = false;
return (p - startAt);
}
@ -538,14 +510,12 @@ private:
WanPath _ipv4p;
WanPath _ipv6p;
uint64_t _lastUsed;
uint64_t _lastUnicastFrame;
uint64_t _lastMulticastFrame;
uint64_t _lastAnnouncedTo;
unsigned int _vMajor,_vMinor,_vRevision;
// Fields below this line are not persisted with serialize() ---------------
bool _dirty;
AtomicCounter __refCount;
};

104
node/Topology.cpp
View File

@ -33,32 +33,14 @@
namespace ZeroTier {
#define ZT_KISSDB_HASH_TABLE_SIZE 32768
#define ZT_KISSDB_KEY_SIZE ZT_ADDRESS_LENGTH
#define ZT_KISSDB_VALUE_SIZE ZT_PEER_MAX_SERIALIZED_LENGTH
Topology::Topology(const RuntimeEnvironment *renv,const char *dbpath) :
Topology::Topology(const RuntimeEnvironment *renv) :
_r(renv),
_amSupernode(false)
{
if (KISSDB_open(&_dbm,dbpath,KISSDB_OPEN_MODE_RWCREAT,ZT_KISSDB_HASH_TABLE_SIZE,ZT_KISSDB_KEY_SIZE,ZT_KISSDB_VALUE_SIZE)) {
if (KISSDB_open(&_dbm,dbpath,KISSDB_OPEN_MODE_RWREPLACE,ZT_KISSDB_HASH_TABLE_SIZE,ZT_KISSDB_KEY_SIZE,ZT_KISSDB_VALUE_SIZE))
throw std::runtime_error("unable to open peer database (rw/create)");
}
if ((_dbm.key_size != ZT_KISSDB_KEY_SIZE)||(_dbm.value_size != ZT_KISSDB_VALUE_SIZE)||(_dbm.hash_table_size != ZT_KISSDB_HASH_TABLE_SIZE)) {
KISSDB_close(&_dbm);
if (KISSDB_open(&_dbm,dbpath,KISSDB_OPEN_MODE_RWREPLACE,ZT_KISSDB_HASH_TABLE_SIZE,ZT_KISSDB_KEY_SIZE,ZT_KISSDB_VALUE_SIZE))
throw std::runtime_error("unable to open peer database (recreate)");
}
Utils::lockDownFile(dbpath,false); // node.db caches secrets
}
Topology::~Topology()
{
// Flush last changes to disk
clean();
}
void Topology::setSupernodes(const std::map< Identity,std::vector<InetAddress> > &sn)
@ -68,6 +50,7 @@ void Topology::setSupernodes(const std::map< Identity,std::vector<InetAddress> >
_supernodes = sn;
_supernodeAddresses.clear();
_supernodePeers.clear();
uint64_t now = Utils::now();
for(std::map< Identity,std::vector<InetAddress> >::const_iterator i(sn.begin());i!=sn.end();++i) {
if (i->first != _r->identity) {
@ -76,6 +59,7 @@ void Topology::setSupernodes(const std::map< Identity,std::vector<InetAddress> >
p = addPeer(SharedPtr<Peer>(new Peer(_r->identity,i->first)));
for(std::vector<InetAddress>::const_iterator j(i->second.begin());j!=i->second.end();++j)
p->setPathAddress(*j,true);
p->setLastUsed(now);
_supernodePeers.push_back(p);
}
_supernodeAddresses.insert(i->first.address());
@ -90,27 +74,11 @@ SharedPtr<Peer> Topology::addPeer(const SharedPtr<Peer> &peer)
TRACE("BUG: addNewPeer() caught and ignored attempt to add peer for self");
throw std::logic_error("cannot add peer for self");
}
SharedPtr<Peer> actualPeer;
{
uint64_t now = Utils::now();
Mutex::Lock _l(_activePeers_m);
actualPeer = _activePeers.insert(std::pair< Address,SharedPtr<Peer> >(peer->address(),peer)).first->second;
}
uint64_t atmp[ZT_ADDRESS_LENGTH];
actualPeer->address().copyTo(atmp,ZT_ADDRESS_LENGTH);
Buffer<ZT_PEER_MAX_SERIALIZED_LENGTH> b;
actualPeer->serialize(b);
b.zeroUnused();
_dbm_m.lock();
if (KISSDB_put(&_dbm,atmp,b.data())) {
TRACE("error writing %s to peerdb",actualPeer->address().toString().c_str());
} else actualPeer->getAndResetDirty();
_dbm_m.unlock();
return actualPeer;
SharedPtr<Peer> p(_activePeers.insert(std::pair< Address,SharedPtr<Peer> >(peer->address(),peer)).first->second);
p->setLastUsed(now);
return p;
}
SharedPtr<Peer> Topology::getPeer(const Address &zta)
@ -119,34 +87,13 @@ SharedPtr<Peer> Topology::getPeer(const Address &zta)
TRACE("BUG: ignored attempt to getPeer() for self, returned NULL");
return SharedPtr<Peer>();
}
{
uint64_t now = Utils::now();
Mutex::Lock _l(_activePeers_m);
std::map< Address,SharedPtr<Peer> >::const_iterator ap(_activePeers.find(zta));
if ((ap != _activePeers.end())&&(ap->second))
if ((ap != _activePeers.end())&&(ap->second)) {
ap->second->setLastUsed(now);
return ap->second;
}
unsigned char ztatmp[ZT_ADDRESS_LENGTH];
zta.copyTo(ztatmp,ZT_ADDRESS_LENGTH);
Buffer<ZT_KISSDB_VALUE_SIZE> b(ZT_KISSDB_VALUE_SIZE);
_dbm_m.lock();
if (!KISSDB_get(&_dbm,ztatmp,b.data())) {
_dbm_m.unlock();
SharedPtr<Peer> p(new Peer());
try {
p->deserialize(b,0);
Mutex::Lock _l(_activePeers_m);
_activePeers[zta] = p;
return p;
} catch ( ... ) {
TRACE("unexpected exception deserializing peer %s from peerdb",zta.toString().c_str());
return SharedPtr<Peer>();
}
} else _dbm_m.unlock();
return SharedPtr<Peer>();
}
@ -183,8 +130,11 @@ skip_and_try_next_supernode:
++sn;
}
if ((bestSupernode)||(strictAvoid))
if (bestSupernode) {
bestSupernode->setLastUsed(now);
return bestSupernode;
} else if (strictAvoid)
return SharedPtr<Peer>();
for(std::vector< SharedPtr<Peer> >::const_iterator sn=_supernodePeers.begin();sn!=_supernodePeers.end();++sn) {
if ((*sn)->hasActiveDirectPath(now)) {
@ -203,36 +153,12 @@ skip_and_try_next_supernode:
}
if (bestSupernode)
bestSupernode->setLastUsed(now);
return bestSupernode;
return _supernodePeers[_r->prng->next32() % _supernodePeers.size()];
}
void Topology::clean()
{
TRACE("cleaning caches and flushing modified peers to disk...");
Mutex::Lock _l(_activePeers_m);
for(std::map< Address,SharedPtr<Peer> >::iterator p(_activePeers.begin());p!=_activePeers.end();++p) {
if (p->second->getAndResetDirty()) {
try {
uint64_t atmp[ZT_ADDRESS_LENGTH];
p->second->identity().address().copyTo(atmp,ZT_ADDRESS_LENGTH);
Buffer<ZT_PEER_MAX_SERIALIZED_LENGTH> b;
p->second->serialize(b);
b.zeroUnused();
_dbm_m.lock();
if (KISSDB_put(&_dbm,atmp,b.data())) {
TRACE("error writing %s to peer.db",p->second->identity().address().toString().c_str());
}
_dbm_m.unlock();
} catch ( ... ) {
TRACE("unexpected exception flushing %s to peer.db",p->second->identity().address().toString().c_str());
}
}
}
}
} // namespace ZeroTier

7
node/Topology.hpp
View File

@ -43,8 +43,6 @@
#include "InetAddress.hpp"
#include "Utils.hpp"
#include "../ext/kissdb/kissdb.h"
namespace ZeroTier {
class RuntimeEnvironment;
@ -55,7 +53,7 @@ class RuntimeEnvironment;
class Topology
{
public:
Topology(const RuntimeEnvironment *renv,const char *dbpath);
Topology(const RuntimeEnvironment *renv);
~Topology();
/**
@ -283,9 +281,6 @@ private:
// Set to true if my identity is in _supernodes
volatile bool _amSupernode;
KISSDB _dbm;
Mutex _dbm_m;
};
} // namespace ZeroTier

1
objects.mk
View File

@ -1,5 +1,4 @@
OBJS=\
ext/kissdb/kissdb.o \
ext/lz4/lz4hc.o \
ext/lz4/lz4.o \
node/C25519.o \