Jeremy Lakeman
9f1407043a
Reduce rhizome MDP debug messages
2013-09-20 14:41:49 +09:30
Andrew Bettison
2b3119b49b
Issue #20 : merge branch 'sockets' into 'development'
...
Daniel's improvements to the local file/abstract socket code, with many
improvements to bring it up to date.
2013-09-19 04:30:14 +09:30
Jeremy Lakeman
2a5ba97b48
Add rhizome test over fakeradio
2013-09-13 14:02:35 +09:30
Jeremy Lakeman
8accabfcac
Always store payloads with a temporary id
2013-08-27 16:15:51 +09:30
Jeremy Lakeman
f2e7391672
Fix format warnings on 64bit machines
2013-08-26 11:54:46 +09:30
gardners
65cc17b7e1
add bidVersion reporting to rhizome_tx debug.
2013-08-25 22:32:13 +09:30
gardners
39785313f3
fix rhizome/mdp debugging for when slot is NULL.
2013-08-25 21:57:41 +09:30
gardners
24f4b789f3
add some more rhizome/mdp debugging.
2013-08-25 21:49:31 +09:30
gardners
ed9971e8c1
increase Rhizome over MDP debug output to track down inability of
...
MEs to do rhizome over mdp on ad-hoc wifi
2013-08-25 21:30:17 +09:30
gardners
53c9a406e1
don't give up on an HTTP rhizome transfer on an empty read if
...
the error is EAGAIN.
2013-08-25 17:35:34 +09:30
gardners
53ce40a053
fix http->mdp rhizome fetch failover regression.
2013-08-25 16:38:24 +09:30
Jeremy Lakeman
643b21d6a7
Fetch file payloads if they are missing, even if we already have the manifest
2013-08-22 15:14:21 +09:30
Jeremy Lakeman
1468d2deb0
Pre-emptively send small payloads via MDP
2013-08-21 09:21:40 +09:30
Jeremy Lakeman
771cb4151b
Fetch journal range via HTTP
2013-08-15 17:00:43 +09:30
Jeremy Lakeman
0b40ef655c
Log format warnings for strbuf_sprintf
2013-08-15 16:31:26 +09:30
Jeremy Lakeman
e322114850
Only transfer new journal content via MDP
2013-08-15 14:36:56 +09:30
Jeremy Lakeman
68bf04f69d
Change manifest service to MeshMS2
2013-08-01 11:37:35 +09:30
Jeremy Lakeman
15ad333195
Implicitly buffer all writes to database blobs
2013-07-18 17:04:12 +09:30
Jeremy Lakeman
a4bf4edbfc
Write out of order packets with MDP transport
...
- add random order writing to rhizome store API
- remove rhizome flush, force all callers to provide buffer
2013-07-17 16:57:59 +09:30
Jeremy Lakeman
ea2e55c62c
Simplify state code for swapping to MDP transfer
2013-07-16 15:38:00 +09:30
Jeremy Lakeman
3855ff35c4
Clean up string formatting for 64bit compilation
2013-07-15 09:59:24 +09:30
Jeremy Lakeman
8ac359e550
Use consistent integer types for 64bit compilation
2013-07-13 14:47:06 +09:30
Jeremy Lakeman
add895fd97
Close manifest leak
2013-07-11 13:52:38 +09:30
Jeremy Lakeman
a1ac7febe2
Add rhizome fetch status html page
2013-07-11 12:32:06 +09:30
Jeremy Lakeman
8249f65f8f
Don't create unused import folder
2013-07-03 10:54:05 +09:30
Jeremy Lakeman
96f6453a43
Don't attempt to read unless the descriptor is ready
2013-06-20 11:18:20 +09:30
Jeremy Lakeman
ceb3c17b70
Perform cheap in-memory tests first, before starting fetch
2013-06-20 11:10:02 +09:30
Jeremy Lakeman
3e57add9ca
Fix, initialise read / write handles so we don't close fd=0
2013-06-19 12:05:56 +09:30
Jeremy Lakeman
96c0889f9a
Don't ask for manifests if we don't have room in transfer queues
2013-04-11 15:24:41 +09:30
Andrew Bettison
157fe3f1d6
Fix Solaris SEGV bug: null pointer passed to sprintf()
2013-03-06 12:56:41 +10:30
Jeremy Lakeman
4cdd3da77f
Add config option to disable fetching via rhizome
2013-03-05 10:56:20 +10:30
Jeremy Lakeman
73786bcb5d
Refactor all rhizome reading and writing
...
- The API in rhizome_store.c is used for all reading and writing
- external storage is now usable for all transport options
2013-02-20 16:18:59 +10:30
gardners
5a3dec28d9
make rhizome over mdp idle timeout configurable at runtime.
2013-02-20 16:18:57 +10:30
gardners
a23612428b
make rhizome over mdp block size configurable.
...
1K seems too big to be reliable on our prototype UHF radio links,
as CRCs fail almost all the time. #48
2013-02-20 16:18:57 +10:30
gardners
2252fdcaa7
created tool for automatically finding IN()s without matching
...
OUT()s or where return() is used instead of RETURN().
Added OUT() to end of all functions using IN() that lacked it to
make it easier to statically analyse this invariant.
Fixed several return instead of RETURNs detected through use of
this tool. #49
2013-02-20 16:18:56 +10:30
gardners
641d749ab4
Create files for large rhizome bundles
...
- configurable size threashold
2013-02-20 15:36:32 +10:30
gardners
0ac403717d
improve rhizome mdp re-request timeout handling so that it doesn't
...
do amazingly bad things on low-bandwidth packet radio links.
2013-02-20 15:36:31 +10:30
gardners
34bf9127a1
make rhizome transfer timeouts consistent.
2013-02-20 15:36:31 +10:30
gardners
f465f3a9ee
Add lots of profiling to assist in debugging
...
- made some debug output conditional.
- make it possible to debug slip decoding without getting swamped by other output.
- Improve reporting of unnamed schedule() calls.
- always collect performance statistics, and report on any excess
use (>1sec in every 3), even if debug.timing is not enabled.
- include file size information in Rhizome transfer messages.
- reduce output when debug.rhizome_rx is set.
2013-02-20 15:36:30 +10:30
gardners
e551017896
push profiling down deeper into rhizome_fetch_poll()'s children.
2013-02-20 15:36:29 +10:30
gardners
17ea1b69b3
don't look at all BARs if we are too slow.
2013-02-20 15:36:29 +10:30
gardners
f114e6d19e
add progress data for each rhizome transfer.
2013-02-20 15:36:28 +10:30
gardners
3bd4c49af5
add more info to /rssi http page.
2013-02-20 15:36:27 +10:30
Jeremy Lakeman
ca5eb7d8bb
Reinstate simple slip code, improve debug messages
2013-02-20 15:36:27 +10:30
Jeremy Lakeman
e4d6520c46
Don't request manifests that are already in the fetch queue
2013-02-20 15:36:25 +10:30
Jeremy Lakeman
6d9bbe2e2c
Refactor interface handling to separate encapsulation from stream type
2013-02-20 15:36:23 +10:30
Jeremy Lakeman
ba1800012e
Only advertise BAR's on an alarm
...
- we no longer hit the database for every outgoing packet, attempting to announce bundles
- we no longer advertise manifests periodically
- when an interesting bar arrives, we ask for the manifest to be announced, which uses the existing packet format
2013-02-20 15:36:23 +10:30
Jeremy Lakeman
84ad4debfa
Throttle serial radio devices when tx buffer is in use
2013-02-20 15:36:22 +10:30
Jeremy Lakeman
2967d1e00f
Don't check for duplicates if an existing id has been supplied
2013-01-03 11:18:30 +10:30
Jeremy Lakeman
a492c05d64
Re-add database cleanup
2012-12-31 15:09:12 +10:30
Jeremy Lakeman
3f45623d04
New stress test, rhizome add while transferring
2012-12-27 16:51:31 +10:30
Jeremy Lakeman
0f81c1f641
Minor code cleanup
2012-12-14 17:07:28 +10:30
Jeremy Lakeman
4264dec978
Reduce logging
2012-12-12 13:18:19 +10:30
Jeremy Lakeman
1a26a3e452
Reduce log spam
2012-12-12 10:02:43 +10:30
Andrew Bettison
5985df751d
Overhaul debug flags
...
Replace debugflags_t and DEBUG_XXX bit masks with config schema "debug.xxx"
entries.
No more support for "debug.all".
2012-12-11 15:59:46 +10:30
Jeremy Lakeman
4b73e42bed
Merge remote-tracking branch 'origin/master' into rhizomeovermdp
...
Conflicts:
rhizome_fetch.c
2012-12-06 13:27:04 +10:30
Jeremy Lakeman
59babd75a2
Fix SEGV if connect fails immediately
2012-12-06 10:35:20 +10:30
gardners
e8cfac1462
don't try mdp fall back when performing an http rhizome direct transfer.
2012-12-06 09:42:27 +10:30
gardners
e2cf0ff506
fixed bug with rhizome direct sync manifest fetching.
2012-12-06 09:31:41 +10:30
gardners
5416e4baca
fixed bugs that were preventing writing to database blob from
...
buffer. Increased maximum buffer size to useful size.
2012-12-04 17:26:47 +10:30
gardners
39f99da3f2
fixing various bugs with rhizome streaming to database.
2012-12-04 17:18:31 +10:30
Jeremy Lakeman
6a5efcc445
Remove buggy debug code
2012-12-04 16:50:27 +10:30
gardners
f827c7c1c8
fixed various problems following split of files into fileblobs
2012-12-04 16:47:45 +10:30
Jeremy Lakeman
33e44ed6be
Don't unwatch if the file descriptor is already closed
2012-12-04 16:30:01 +10:30
gardners
6dfe654ab0
adjusted tuning of Rhizome over MDP to use 32x1KB packets for
...
better throughput. Still sub-optimal for wired ethernet type
interfaces, but should be ok for WiFi where max broadcast throughput
is 100kb/sec.
2012-12-04 16:19:22 +10:30
Jeremy Lakeman
bd83c192bb
Use consistent length constant
2012-12-04 16:18:51 +10:30
gardners
fc0c134cbf
added speed indication on completion of rhizome transfer.
...
split files table into files and fileblobs to avoid pathological
delays when updating datavalid flag in files table (sqlite copies
the whole table row, including possibly large blob).
2012-12-04 16:03:56 +10:30
gardners
23cace8455
removed writing temporary file for debugging.
2012-12-04 13:49:47 +10:30
gardners
2e4e111764
fixed bug that prevented remaining rhizome tests from passing.
2012-12-04 08:50:00 +10:30
gardners
a7e09aac2c
fixed scheduling bug that stopped rhizomeprotocol tests 1 and 3.
2012-12-04 08:30:58 +10:30
gardners
254b0a4fd6
fixed problem with rhizome over http (alarm without callback
...
specified).
2012-12-04 07:12:03 +10:30
gardners
fcd5142aaf
fixed bug with streaming to database of rhizome fetches.
2012-12-04 06:56:10 +10:30
gardners
abcac03981
work on tracking down hash calculation bug for files > one
...
window long.
2012-12-04 06:26:44 +10:30
gardners
43316ade3a
added buffer when streaming to database so that we don't waste
...
most of our time opening and closing blobs, and don't have to keep
the blob open all the time.
2012-12-03 21:47:02 +10:30
gardners
de7a4ce2e3
fixed bugs with rhizome streaming direct to database.
...
that now works, but rhizome direct pull now fails.
2012-12-03 20:51:14 +10:30
gardners
e108fe0822
fixed bug in handling of sqlite3_blob_write(). Still more to fix.
2012-12-03 17:10:39 +10:30
gardners
5d24ba7db9
first cut at progressive hashing and checking of hash when streaming
...
file into database via rhizome.
2012-12-03 17:05:12 +10:30
gardners
e08593b466
stream manifest to buffer, and files to blob in database instead
...
of via files.
NOT COMPLETE - needs to calculate progressive hash and decide
if file is ok at end of import.
2012-12-03 16:52:08 +10:30
gardners
7ec695c940
feedback from code review by Jeremy.
...
Some preparation for out-of-order handling.
Consolidated redundantly redundant fields in fetch slot structure.
Probably fixed spurious alarm bug.
2012-12-03 11:50:35 +10:30
gardners
115736f0a4
cleaned out excess debugging messages and upped block size from
...
200 bytes to 1KB to greatly improve throughput. Will eventually
want to make block size selection be based on network interface.
2012-12-01 10:52:08 +10:30
gardners
b7ae55c143
receiving multiple bundles via rhizome over mdp works now, although
...
there is a bug with alarms being called on fetch slots after they
complete, even though they apparently get unscheduled.
2012-12-01 10:47:19 +10:30
gardners
55df66f6d6
culled some debug output.
...
mostly works, but fetch slots can get muddled.
2012-12-01 07:43:18 +10:30
gardners
ac734ff00d
fixed bugs with rhizome over mdp receive for files >32*200 bytes
...
long. also we request next block of data immediately after
receiving all of the previous block.
2012-12-01 07:37:22 +10:30
gardners
7e6d836627
remember to update timeout when storing data via rhizome mdp.
2012-11-30 22:55:10 +10:30
gardners
41261bf122
gadzooks, rhizome over mdp appears to work.
2012-11-30 22:45:27 +10:30
gardners
dbb1fe8d1e
rhizome over mdp now writes content for in-order packets, and
...
import gets triggered, but file hash currently doesn't match.
2012-11-30 22:42:28 +10:30
gardners
79c90dde6e
now rhizome mdp replies get parsed and the appropriate slot, if
...
any, is located.
2012-11-30 22:26:31 +10:30
gardners
1be28246e4
we now pass bid+version in Rhizome over MDP block request.
2012-11-30 21:13:29 +10:30
gardners
0c7419da5d
changed bar to bid in rhizome mdp fetch elements, because that
...
is probably the better way to address the content. Should add
version to make sure.
2012-11-30 21:08:06 +10:30
gardners
e5d5f3f750
fixed bugs with Rhizome over MDP requests. Now seems to output
...
a series of MDP block requests as it should.
2012-11-30 20:20:33 +10:30
gardners
11cce162dc
fixed bugs with preparing Rhizome MDP packets (source sid was not
...
being set properly).
2012-11-30 20:06:47 +10:30
gardners
05745d9b9a
same fix as previous commit, but in another place.
2012-11-30 20:03:53 +10:30
gardners
5becae0136
fixed segfault bug with obtaining own SID.
2012-11-30 20:03:18 +10:30
gardners
7a4e3d20f8
Actually send Rhizome over MDP request packets.
...
Actually call appropriate callback when receiving Rhizome over
MDP packets.
2012-11-30 14:17:05 +10:30
gardners
eb7524e068
factored out internal MDP services into a separate file, and made
...
core case statement in that brief and clear.
2012-11-30 11:48:00 +10:30
gardners
a665750f4c
prepare and dispatch mdp frames for rhizome over mdp.
...
still need to set bodies of frames, and schedule call backs,
and handle the requests when received.
2012-11-29 21:08:11 +10:30
gardners
c05ebada8c
more work on rhizome over MDP.
2012-11-29 20:39:30 +10:30
gardners
629a36f041
call switch to mdp fetch if http request setup fails.
2012-11-29 16:43:19 +10:30
gardners
a07e5761c4
added code to try switching to mdp if http fetching fails for
...
any reason.
2012-11-29 16:15:51 +10:30
gardners
faad1f26b1
Pass SID as well as ip:port through rhizome fetch request logic.
...
This is in preparation for rhizome over MDP.
2012-11-29 15:38:04 +10:30
