ExternalVendorCode/serval-dna
1
0
Fork 0
mirror of https://github.com/servalproject/serval-dna.git synced 2024-12-19 05:07:56 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity
3,810 Commits 22 Branches 0 Tags 33 MiB
988250a576
Commit Graph

266 Commits

Author SHA1 Message Date
Andrew Bettison
00cf61721d Rename http_header_complete() to is_http_header_complete() 2013-10-16 11:22:02 +10:30
Andrew Bettison
97cbebc91e Issue #11: improve types in file i/o functions 
Use open(2)/read(2)/write(2) instead of fopen(3)/fread(3)/fwrite(3) in
several places to avoid unnecessary buffering

Fix a bug in Rhizome HTTP add's handling of unconfigured manifest
template file

Improve some debug and error logging for file i/o
 2013-10-11 16:24:51 +10:30
Andrew Bettison
974c7a56a0 Issue #11: Use rhizome_filehash_t everywhere 2013-10-11 15:35:27 +10:30
Andrew Bettison
a95ef79139 Use sid_t everywhere 
Remove stowSid() at last

Change API of tohex() and strbuf_tohex(), to pass string length instead of
binary byte count.  This allows odd numbers of hex digits to be produced.

Remove alloca_tohex_sid(); replace with alloca_tohex_sid_t()

New alloca_tohex_sid_t_trunc() macro
 2013-10-09 18:54:21 +10:30
Jeremy Lakeman
60e3f5a3fc Fix printf formats for 64bit compilation 2013-10-09 10:31:52 +10:30
Andrew Bettison
ffafae86b1 Issue #11: replace long long with int64_t 2013-10-07 05:54:46 +10:30
Andrew Bettison
f5e7db089c Fix bugs revealed by test failures 
Introduced by the recent change to rhizome_bid_t everywhere
 2013-10-04 03:12:52 +09:30
Andrew Bettison
ab31420faf rhizome_bid_t 
Consistent type for internal binary representation of Rhizome Bundle ID
(aka Manifest ID)
 2013-10-03 23:16:45 +09:30
Andrew Bettison
7e3a552011 Issue #69: refactor all Rhizome SQL queries to use binding 
Remove all uses of sprintf(3) and its ilk to form SQL query commands.
Use the new sqlite_bind() and sqlite_prepare_bind() functions instead.
 2013-10-03 15:15:30 +09:30
Jeremy Lakeman
79bdb5f993 Add more html debugging for rhizome sync and fetch queues 2013-09-24 14:34:51 +09:30
Jeremy Lakeman
57577abca6 Merge branch 'mavlink' into development 2013-09-20 14:58:29 +09:30
Jeremy Lakeman
9f1407043a Reduce rhizome MDP debug messages 2013-09-20 14:41:49 +09:30
Andrew Bettison
2b3119b49b Issue #20: merge branch 'sockets' into 'development' 
Daniel's improvements to the local file/abstract socket code, with many
improvements to bring it up to date.
 2013-09-19 04:30:14 +09:30
Jeremy Lakeman
2a5ba97b48 Add rhizome test over fakeradio 2013-09-13 14:02:35 +09:30
Jeremy Lakeman
8accabfcac Always store payloads with a temporary id 2013-08-27 16:15:51 +09:30
Jeremy Lakeman
f2e7391672 Fix format warnings on 64bit machines 2013-08-26 11:54:46 +09:30
gardners
65cc17b7e1 add bidVersion reporting to rhizome_tx debug. 2013-08-25 22:32:13 +09:30
gardners
39785313f3 fix rhizome/mdp debugging for when slot is NULL. 2013-08-25 21:57:41 +09:30
gardners
24f4b789f3 add some more rhizome/mdp debugging. 2013-08-25 21:49:31 +09:30
gardners
ed9971e8c1 increase Rhizome over MDP debug output to track down inability of 
MEs to do rhizome over mdp on ad-hoc wifi
 2013-08-25 21:30:17 +09:30
gardners
53c9a406e1 don't give up on an HTTP rhizome transfer on an empty read if 
the error is EAGAIN.
 2013-08-25 17:35:34 +09:30
gardners
53ce40a053 fix http->mdp rhizome fetch failover regression. 2013-08-25 16:38:24 +09:30
Jeremy Lakeman
643b21d6a7 Fetch file payloads if they are missing, even if we already have the manifest 2013-08-22 15:14:21 +09:30
Jeremy Lakeman
1468d2deb0 Pre-emptively send small payloads via MDP 2013-08-21 09:21:40 +09:30
Jeremy Lakeman
771cb4151b Fetch journal range via HTTP 2013-08-15 17:00:43 +09:30
Jeremy Lakeman
0b40ef655c Log format warnings for strbuf_sprintf 2013-08-15 16:31:26 +09:30
Jeremy Lakeman
e322114850 Only transfer new journal content via MDP 2013-08-15 14:36:56 +09:30
Jeremy Lakeman
68bf04f69d Change manifest service to MeshMS2 2013-08-01 11:37:35 +09:30
Jeremy Lakeman
15ad333195 Implicitly buffer all writes to database blobs 2013-07-18 17:04:12 +09:30
Jeremy Lakeman
a4bf4edbfc Write out of order packets with MDP transport 
- add random order writing to rhizome store API
- remove rhizome flush, force all callers to provide buffer
 2013-07-17 16:57:59 +09:30
Jeremy Lakeman
ea2e55c62c Simplify state code for swapping to MDP transfer 2013-07-16 15:38:00 +09:30
Jeremy Lakeman
3855ff35c4 Clean up string formatting for 64bit compilation 2013-07-15 09:59:24 +09:30
Jeremy Lakeman
8ac359e550 Use consistent integer types for 64bit compilation 2013-07-13 14:47:06 +09:30
Jeremy Lakeman
add895fd97 Close manifest leak 2013-07-11 13:52:38 +09:30
Jeremy Lakeman
a1ac7febe2 Add rhizome fetch status html page 2013-07-11 12:32:06 +09:30
Jeremy Lakeman
8249f65f8f Don't create unused import folder 2013-07-03 10:54:05 +09:30
Jeremy Lakeman
96f6453a43 Don't attempt to read unless the descriptor is ready 2013-06-20 11:18:20 +09:30
Jeremy Lakeman
ceb3c17b70 Perform cheap in-memory tests first, before starting fetch 2013-06-20 11:10:02 +09:30
Jeremy Lakeman
3e57add9ca Fix, initialise read / write handles so we don't close fd=0 2013-06-19 12:05:56 +09:30
Jeremy Lakeman
96c0889f9a Don't ask for manifests if we don't have room in transfer queues 2013-04-11 15:24:41 +09:30
Andrew Bettison
157fe3f1d6 Fix Solaris SEGV bug: null pointer passed to sprintf() 2013-03-06 12:56:41 +10:30
Jeremy Lakeman
4cdd3da77f Add config option to disable fetching via rhizome 2013-03-05 10:56:20 +10:30
Jeremy Lakeman
73786bcb5d Refactor all rhizome reading and writing 
- The API in rhizome_store.c is used for all reading and writing
- external storage is now usable for all transport options
 2013-02-20 16:18:59 +10:30
gardners
5a3dec28d9 make rhizome over mdp idle timeout configurable at runtime. 2013-02-20 16:18:57 +10:30
gardners
a23612428b make rhizome over mdp block size configurable. 
1K seems too big to be reliable on our prototype UHF radio links,
as CRCs fail almost all the time. #48
 2013-02-20 16:18:57 +10:30
gardners
2252fdcaa7 created tool for automatically finding IN()s without matching 
OUT()s or where return() is used instead of RETURN().
Added OUT() to end of all functions using IN() that lacked it to
make it easier to statically analyse this invariant.
Fixed several return instead of RETURNs detected through use of
this tool. #49
 2013-02-20 16:18:56 +10:30
gardners
641d749ab4 Create files for large rhizome bundles 
- configurable size threashold
 2013-02-20 15:36:32 +10:30
gardners
0ac403717d improve rhizome mdp re-request timeout handling so that it doesn't 
do amazingly bad things on low-bandwidth packet radio links.
 2013-02-20 15:36:31 +10:30
gardners
34bf9127a1 make rhizome transfer timeouts consistent. 2013-02-20 15:36:31 +10:30
gardners
f465f3a9ee Add lots of profiling to assist in debugging 
- made some debug output conditional.
- make it possible to debug slip decoding without getting swamped by other output.
- Improve reporting of unnamed schedule() calls.
- always collect performance statistics, and report on any excess
  use (>1sec in every 3), even if debug.timing is not enabled.
- include file size information in Rhizome transfer messages.
- reduce output when debug.rhizome_rx is set.
 2013-02-20 15:36:30 +10:30
gardners
e551017896 push profiling down deeper into rhizome_fetch_poll()'s children. 2013-02-20 15:36:29 +10:30
gardners
17ea1b69b3 don't look at all BARs if we are too slow. 2013-02-20 15:36:29 +10:30
gardners
f114e6d19e add progress data for each rhizome transfer. 2013-02-20 15:36:28 +10:30
gardners
3bd4c49af5 add more info to /rssi http page. 2013-02-20 15:36:27 +10:30
Jeremy Lakeman
ca5eb7d8bb Reinstate simple slip code, improve debug messages 2013-02-20 15:36:27 +10:30
Jeremy Lakeman
e4d6520c46 Don't request manifests that are already in the fetch queue 2013-02-20 15:36:25 +10:30
Jeremy Lakeman
6d9bbe2e2c Refactor interface handling to separate encapsulation from stream type 2013-02-20 15:36:23 +10:30
Jeremy Lakeman
ba1800012e Only advertise BAR's on an alarm 
- we no longer hit the database for every outgoing packet, attempting to announce bundles
- we no longer advertise manifests periodically
- when an interesting bar arrives, we ask for the manifest to be announced, which uses the existing packet format
 2013-02-20 15:36:23 +10:30
Jeremy Lakeman
84ad4debfa Throttle serial radio devices when tx buffer is in use 2013-02-20 15:36:22 +10:30
Jeremy Lakeman
2967d1e00f Don't check for duplicates if an existing id has been supplied 2013-01-03 11:18:30 +10:30
Jeremy Lakeman
a492c05d64 Re-add database cleanup 2012-12-31 15:09:12 +10:30
Jeremy Lakeman
3f45623d04 New stress test, rhizome add while transferring 2012-12-27 16:51:31 +10:30
Jeremy Lakeman
0f81c1f641 Minor code cleanup 2012-12-14 17:07:28 +10:30
Jeremy Lakeman
4264dec978 Reduce logging 2012-12-12 13:18:19 +10:30
Jeremy Lakeman
1a26a3e452 Reduce log spam 2012-12-12 10:02:43 +10:30
Andrew Bettison
5985df751d Overhaul debug flags 
Replace debugflags_t and DEBUG_XXX bit masks with config schema "debug.xxx"
entries.

No more support for "debug.all".
 2012-12-11 15:59:46 +10:30
Jeremy Lakeman
4b73e42bed Merge remote-tracking branch 'origin/master' into rhizomeovermdp 
Conflicts:
	rhizome_fetch.c
 2012-12-06 13:27:04 +10:30
Jeremy Lakeman
59babd75a2 Fix SEGV if connect fails immediately 2012-12-06 10:35:20 +10:30
gardners
e8cfac1462 don't try mdp fall back when performing an http rhizome direct transfer. 2012-12-06 09:42:27 +10:30
gardners
e2cf0ff506 fixed bug with rhizome direct sync manifest fetching. 2012-12-06 09:31:41 +10:30
gardners
5416e4baca fixed bugs that were preventing writing to database blob from 
buffer.  Increased maximum buffer size to useful size.
 2012-12-04 17:26:47 +10:30
gardners
39f99da3f2 fixing various bugs with rhizome streaming to database. 2012-12-04 17:18:31 +10:30
Jeremy Lakeman
6a5efcc445 Remove buggy debug code 2012-12-04 16:50:27 +10:30
gardners
f827c7c1c8 fixed various problems following split of files into fileblobs 2012-12-04 16:47:45 +10:30
Jeremy Lakeman
33e44ed6be Don't unwatch if the file descriptor is already closed 2012-12-04 16:30:01 +10:30
gardners
6dfe654ab0 adjusted tuning of Rhizome over MDP to use 32x1KB packets for 
better throughput.  Still sub-optimal for wired ethernet type
interfaces, but should be ok for WiFi where max broadcast throughput
is 100kb/sec.
 2012-12-04 16:19:22 +10:30
Jeremy Lakeman
bd83c192bb Use consistent length constant 2012-12-04 16:18:51 +10:30
gardners
fc0c134cbf added speed indication on completion of rhizome transfer. 
split files table into files and fileblobs to avoid pathological
delays when updating datavalid flag in files table (sqlite copies
the whole table row, including possibly large blob).
 2012-12-04 16:03:56 +10:30
gardners
23cace8455 removed writing temporary file for debugging. 2012-12-04 13:49:47 +10:30
gardners
2e4e111764 fixed bug that prevented remaining rhizome tests from passing. 2012-12-04 08:50:00 +10:30
gardners
a7e09aac2c fixed scheduling bug that stopped rhizomeprotocol tests 1 and 3. 2012-12-04 08:30:58 +10:30
gardners
254b0a4fd6 fixed problem with rhizome over http (alarm without callback 
specified).
 2012-12-04 07:12:03 +10:30
gardners
fcd5142aaf fixed bug with streaming to database of rhizome fetches. 2012-12-04 06:56:10 +10:30
gardners
abcac03981 work on tracking down hash calculation bug for files > one 
window long.
 2012-12-04 06:26:44 +10:30
gardners
43316ade3a added buffer when streaming to database so that we don't waste 
most of our time opening and closing blobs, and don't have to keep
the blob open all the time.
 2012-12-03 21:47:02 +10:30
gardners
de7a4ce2e3 fixed bugs with rhizome streaming direct to database. 
that now works, but rhizome direct pull now fails.
 2012-12-03 20:51:14 +10:30
gardners
e108fe0822 fixed bug in handling of sqlite3_blob_write(). Still more to fix. 2012-12-03 17:10:39 +10:30
gardners
5d24ba7db9 first cut at progressive hashing and checking of hash when streaming 
file into database via rhizome.
 2012-12-03 17:05:12 +10:30
gardners
e08593b466 stream manifest to buffer, and files to blob in database instead 
of via files.
NOT COMPLETE - needs to calculate progressive hash and decide
if file is ok at end of import.
 2012-12-03 16:52:08 +10:30
gardners
7ec695c940 feedback from code review by Jeremy. 
Some preparation for out-of-order handling.
Consolidated redundantly redundant fields in fetch slot structure.
Probably fixed spurious alarm bug.
 2012-12-03 11:50:35 +10:30
gardners
115736f0a4 cleaned out excess debugging messages and upped block size from 
200 bytes to 1KB to greatly improve throughput.  Will eventually
want to make block size selection be based on network interface.
 2012-12-01 10:52:08 +10:30
gardners
b7ae55c143 receiving multiple bundles via rhizome over mdp works now, although 
there is a bug with alarms being called on fetch slots after they
complete, even though they apparently get unscheduled.
 2012-12-01 10:47:19 +10:30
gardners
55df66f6d6 culled some debug output. 
mostly works, but fetch slots can get muddled.
 2012-12-01 07:43:18 +10:30
gardners
ac734ff00d fixed bugs with rhizome over mdp receive for files >32*200 bytes 
long.  also we request next block of data immediately after
receiving all of the previous block.
 2012-12-01 07:37:22 +10:30
gardners
7e6d836627 remember to update timeout when storing data via rhizome mdp. 2012-11-30 22:55:10 +10:30
gardners
41261bf122 gadzooks, rhizome over mdp appears to work. 2012-11-30 22:45:27 +10:30
gardners
dbb1fe8d1e rhizome over mdp now writes content for in-order packets, and 
import gets triggered, but file hash currently doesn't match.
 2012-11-30 22:42:28 +10:30
gardners
79c90dde6e now rhizome mdp replies get parsed and the appropriate slot, if 
any, is located.
 2012-11-30 22:26:31 +10:30
gardners
1be28246e4 we now pass bid+version in Rhizome over MDP block request. 2012-11-30 21:13:29 +10:30
gardners
0c7419da5d changed bar to bid in rhizome mdp fetch elements, because that 
is probably the better way to address the content.  Should add
version to make sure.
 2012-11-30 21:08:06 +10:30