Commit Graph

164 Commits

Author SHA1 Message Date
Andrew Bettison
6488f7ad65 Fix basic bugs in new HTTP server code
All 'rhizomeprotocol' HTTP tests still fail
2013-10-25 17:38:51 +10:30
Andrew Bettison
291a631095 New header file "fdqueue.h"
So that "http_server.h" does not have to include "serval.h" which
creates a circular dependency.

Remove the __SERVALDNA__HTTP_SERVER_IMPLEMENTATION hack from
"http_server.h"
2013-10-25 00:20:53 +10:30
Andrew Bettison
fa21bec880 Rewrite HTTP server 2013-10-25 00:19:37 +10:30
Andrew Bettison
640a61cbe5 Add 'debug.rhizome_httpd' config option 2013-10-16 11:23:45 +10:30
Andrew Bettison
00cf61721d Rename http_header_complete() to is_http_header_complete() 2013-10-16 11:22:02 +10:30
Andrew Bettison
974c7a56a0 Issue #11: Use rhizome_filehash_t everywhere 2013-10-11 15:35:27 +10:30
Andrew Bettison
a95ef79139 Use sid_t everywhere
Remove stowSid() at last

Change API of tohex() and strbuf_tohex(), to pass string length instead of
binary byte count.  This allows odd numbers of hex digits to be produced.

Remove alloca_tohex_sid(); replace with alloca_tohex_sid_t()

New alloca_tohex_sid_t_trunc() macro
2013-10-09 18:54:21 +10:30
Andrew Bettison
7a8d750a0e Issue #11: uint16_t instead of short for TCP port number 2013-10-07 23:37:37 +10:30
Andrew Bettison
ab31420faf rhizome_bid_t
Consistent type for internal binary representation of Rhizome Bundle ID
(aka Manifest ID)
2013-10-03 23:16:45 +09:30
Jeremy Lakeman
4838a529bd Don't busy wait while testing for MDP failover 2013-09-24 15:15:32 +09:30
Jeremy Lakeman
b38b4720cc Add html stats for http connection and mdp transfer counts 2013-09-24 14:34:51 +09:30
Jeremy Lakeman
9885280334 Squelch warning on success 2013-09-24 14:34:51 +09:30
Jeremy Lakeman
931ca60554 Refactor and simplify http server
- Define a function per page that is responsible for parsing requests
- Define a "generator" callback function for filling the buffer for a response
- Remove features that have never been used
- Remove functions that are no longer used
2013-09-24 14:34:50 +09:30
Jeremy Lakeman
08b44e96ed Improve html debug statistics
- shift radio rssi onto interace structure
- track packets sent received
- create html output for interface stats
- create html output for link state routing stats
2013-09-23 11:31:38 +09:30
gardners
0d85d60566 relocate point of action for debug.rhizome_nohttptx so that it
performs correctly, and can be genuinely used to test failover from
http to mdp.
this has been confirmed by running the resulting rhizomeprotocol test
before and after application of the fix for the regression that stopped
the failover from working in certain circumstances.
2013-08-25 16:53:26 +09:30
gardners
9b09021605 add test for falling over to MDP when HTTP rhizome fetching fails. 2013-08-25 16:37:44 +09:30
Jeremy Lakeman
f9b828c3dd Remove payload if hash doesn't match when reading back 2013-08-21 15:45:18 +09:30
Jeremy Lakeman
771cb4151b Fetch journal range via HTTP 2013-08-15 17:00:43 +09:30
Jeremy Lakeman
21132466f1 Serve a range of a file payload via HTTP 2013-08-15 16:31:26 +09:30
gardners
8968d5cf05 report RFD900 packets received as well as link margin estimate.
(non-zero packets received implies radio link).
2013-07-19 20:04:15 +09:30
Jeremy Lakeman
3855ff35c4 Clean up string formatting for 64bit compilation 2013-07-15 09:59:24 +09:30
Jeremy Lakeman
8ac359e550 Use consistent integer types for 64bit compilation 2013-07-13 14:47:06 +09:30
Jeremy Lakeman
a1ac7febe2 Add rhizome fetch status html page 2013-07-11 12:32:06 +09:30
Jeremy Lakeman
3e57add9ca Fix, initialise read / write handles so we don't close fd=0 2013-06-19 12:05:56 +09:30
Jeremy Lakeman
40364be92f Fix and improve rhizome stress test
- reduce dependance on routing
- fail immediately if a servald instance crashes
2013-06-18 16:33:41 +09:30
gardners
0898981807 add time stamp to rssi.csv 2013-05-07 13:40:38 +09:30
gardners
3692a959d5 add rssi.csv http request 2013-05-07 13:40:07 +09:30
Jeremy Lakeman
73786bcb5d Refactor all rhizome reading and writing
- The API in rhizome_store.c is used for all reading and writing
- external storage is now usable for all transport options
2013-02-20 16:18:59 +10:30
gardners
3301a51883 fix behaviour when rhizome http service is disabled so that
rhizomeprotocol test 3 can detect when rhizome http is disabled,
even though the http server itself runs for providing informational
services.
2013-02-20 16:18:56 +10:30
gardners
2252fdcaa7 created tool for automatically finding IN()s without matching
OUT()s or where return() is used instead of RETURN().
Added OUT() to end of all functions using IN() that lacked it to
make it easier to statically analyse this invariant.
Fixed several return instead of RETURNs detected through use of
this tool. #49
2013-02-20 16:18:56 +10:30
gardners
f681356f52 reformat /rssi web page to better fit on phone screens. 2013-02-20 15:36:31 +10:30
gardners
e551017896 push profiling down deeper into rhizome_fetch_poll()'s children. 2013-02-20 15:36:29 +10:30
gardners
60015a1aee keep HTTP server running when rhizome over http is disabled, but
don't serve and rhizome requests.
2013-02-20 15:36:29 +10:30
gardners
f114e6d19e add progress data for each rhizome transfer. 2013-02-20 15:36:28 +10:30
gardners
d49649ed96 add stored bundle count to /rssi web page display. 2013-02-20 15:36:28 +10:30
gardners
3bd4c49af5 add more info to /rssi http page. 2013-02-20 15:36:27 +10:30
gardners
d251f46638 add SID to rssi display 2013-02-20 15:36:27 +10:30
gardners
b9c924c5fe make rssi web page text really big for small phone screens. 2013-02-20 15:36:27 +10:30
gardners
66956d84ce add /rssi page to rhizome web server to allow easy querying of
radio link budget.
2013-02-20 15:36:27 +10:30
Jeremy Lakeman
340003ed99 Don't keep sql blob open from http server 2013-01-30 11:46:10 +11:00
Andrew Bettison
5985df751d Overhaul debug flags
Replace debugflags_t and DEBUG_XXX bit masks with config schema "debug.xxx"
entries.

No more support for "debug.all".
2012-12-11 15:59:46 +10:30
gardners
f827c7c1c8 fixed various problems following split of files into fileblobs 2012-12-04 16:47:45 +10:30
gardners
b91e832ad7 added improved control of rhizome http and mdp servers via
separate config options with their own predicate functions.
2012-11-30 14:47:27 +10:30
Andrew Bettison
c84b7e5db4 Improve str.h functions
Add strn_startswith() and strncase_startswith().

Make all str*_startswith() functions take const char * arguments, to make it
possible to do safe programming with consts.
2012-11-20 18:10:08 +10:30
Jeremy Lakeman
8b9505670d Fix handling of socket closure on OSX 2012-10-08 16:53:52 +10:30
Andrew Bettison
89343c69cd Merge branch 'rhizomedirect' into 'master' 2012-10-05 17:45:30 +09:30
Andrew Bettison
2e7d19a3f7 Issue #9, fix rhizome direct push and sync tests
The "rhizome direct push" command (and also sync) was not waiting for the
server's HTTP response, so it was exiting before the server had finished
storing the bundle, which led to a race with the subsequent "assert
bundle_received_by" test.  Fixed by adding the missing code to receive the HTTP
response.

Refactored the code used for parsing HTTP responses in rhizome_fetch.c, and
used it in rhizome_direct_http.c.
2012-10-04 17:38:33 +09:30
Andrew Bettison
3040d2b665 Add Rhizome DB retries on SQL statement preparation
It turns out that if the DB is locked, sqlite_prepare_v2() call can return
SQLITE_BUSY.  The retry logic (implemented for issue #2) only provided for
sqlite_step() to return SQLITE_BUSY.  It was a fairly straightforward matter to
extend the retry logic to cover statement preparation in an equally general
fashion.

The problem was observed while diagnosing failures in the rhizomeprotocol
DirectPush test case: the "servald rhizome list" command was failing due to a
locked database.  See issue #9.
2012-10-04 14:30:20 +09:30
gardners
33107e795f Reduced debug output. #9 2012-10-03 16:26:41 +02:00
gardners
636cba8363 rhizome direct http manifest-less file import now uses the name
supplied in the HTTP POST form. #9
2012-10-03 16:13:41 +02:00
gardners
dc3137707f rhizome import bundle from file sans-manifest via http now works.
Must be enabled by using rhizome.api.addfile.*
Certainly polishing to be done, including using filename supplied
during HTTP POST.  Now to fix that, and make it all work with
final rhizomeprotocol test case.
rhizomeprotocol test cases 8 and 9 currently fail post-merge. #9
2012-10-03 16:13:06 +02:00
gardners
e9136ec7ee first cut at code to lookup manifest by prefix and deliver the
manifest in full. #9
2012-10-03 15:50:43 +02:00
gardners
c356ead09c fixed bug in rhizome direct http push code.
Rhizome direct push via HTTP now works. #9
2012-10-03 12:22:52 +02:00
Andrew Bettison
e51745e39d Issue #9, fix HTTP server "POST /rhizome/bundle"
Handle failure cases properly.  The HttpImport test now fails because the
"servald" executable is not found by the system(3) call -- need to give the
full path name, or avoid the use of system(3) altogether.
2012-09-28 17:53:20 +09:30
Andrew Bettison
14721fab19 Merge branch 'solaris' into rhizomedirect
All test cases pass on Solaris (see issue #16) except the 'jni' tests because
there is no Java compiler available on Solaris, and the new 'rhizomeprotocol'
tests from the rhizomedirect branch, which have always failed.
2012-09-27 17:38:16 +09:30
gardners
9a5dbbd14f rhizome direct http transport now asks for response to enquiry when
enquiry received by server.  Generating responses to enquiries not
yet implemented. #9
2012-09-11 13:09:40 +09:30
gardners
d3f5c6e598 Merged rhizome direct httpd back into main rhizome httpd server.
bundles can be imported via HTTP POST multi-part form. #9
2012-09-06 13:21:13 +09:30
gardners
dc15ac5a34 Make simple http response clear request type so that it always
works. #9
2012-09-06 06:09:16 +09:30
Paul Gardner-Stephen
bba6839656 Initial stab at porting to Solaris.
It compiles without warning (with CC=gcc) but doesn't link
because NaCL doesn't build yet.
2012-09-05 20:42:50 +09:30
gardners
ecbe8e4f8c improve HTTP server response code handling, including fixing
iching calculator bug - unknown response codes not more than four
no longer result in "a suffusion of yellow".
Also stopped dropping HTTP requests on empty reads when parsing
a multi-part POST.
2012-09-05 16:35:08 +09:30
gardners
4cab16598c added code to rhizome http request client poll routine to call
multi-part form data parser.
2012-09-05 15:53:16 +09:30
Jeremy Lakeman
df0e20408c Close sockets on poll errors 2012-08-31 15:04:06 +09:30
gardners
9d2aa61792 Refactoring of Rhizome http server code so that we can supply a
different request parser, but otherwise share code between
rhizome transfers and rhizome direct. #9
2012-08-31 13:01:28 +09:30
gardners
bde9d1c56a Rhizome http server now allows specification of client socket callback
when starting, so that same server code can be shared for rhizome transfers
and rhizome direct. #9
2012-08-31 13:01:28 +09:30
Andrew Bettison
41680ff663 Move functions from testdefs.sh to testdefs_rhizome.sh
... where they should have been in the first place.

Also, clean up the RHIZOME HTTP SERVER START log message.

Issue #9.
2012-08-27 17:07:44 +09:30
Andrew Bettison
3f1e0dd763 Finish sleep-retry on locked Rhizome db
Closes #2.

Rewrite all Rhizome db query code using new retry primitives defined in
"rhizome.h": sqlite_step_retry(), sqlite_retry(), sqlite_retry_done(), etc.
Replace all calls to sqlite3_prepare_v2() with sqlite_prepare() which does
proper error logging.

Fix bug: re-invoking sqlite3_blob_close() on SQLITE_BUSY return causes process
to abort.  Use an explicit BEGIN...COMMIT around the blob writing code instead.

Tested using repeated invocations of batphone/tests/meshms1.

Delete deprecated Rhizome db code in rhizome_crypto.c that has been replaced
with keyring file.
2012-08-24 15:38:39 +09:30
Andrew Bettison
385e2bdb4e Improve Rhizome INFO messages 2012-08-17 19:42:11 +09:30
Jeremy Lakeman
7705676a9e Move str functions to c file to enable reuse 2012-08-09 13:30:52 +09:30
Andrew Bettison
12d8bc0c12 Use new typedef time_ms_t everywhere 2012-08-09 12:14:32 +09:30
Andrew Bettison
240be25793 Replace overlay_gettime_ms() with gettime_ms() 2012-07-30 17:22:38 +09:30
Andrew Bettison
04b95d2590 Merge branch 'andrew' into 'master' 2012-07-25 18:04:16 +09:30
Andrew Bettison
0e5c5e0e98 More work on dnahelper, almost done
Servald starts DNA helper, receives startup ACK, sends requests, receives
responses, handles malformed helper responses, echoes dnahelper stderr lines to
log, sends MDP reply packet, waits for dead helper process, all asynchronously.
Shuts down helper process during servald shutdown.

Remaining issues:
 - Does not impose a timeout on helper responses.
 - Only the first URI is reported by the "dna lookup" command.
2012-07-20 18:17:43 +09:30
Jeremy Lakeman
1d53726b7b Merge remote-tracking branch 'origin/master'
Conflicts:
	rhizome_fetch.c
	rhizome_http.c
	testframework.sh
2012-07-17 15:39:55 +09:30
Andrew Bettison
c791ba94d0 Refactor HTTP response parsing
Remove need to nul-terminate the received buffers in HTTP fetch reply handling
and HTTP server request parsing.

Remove redundant copying of data.

More rigorous parsing code, probably less vulnerable to overrun exploits.

Better debug logging of requests and responses.
2012-07-13 18:06:55 +09:30
Andrew Bettison
dd225bbb7f Refactor rhizome HTTP response headers 2012-07-13 12:31:10 +09:30
Jeremy Lakeman
27c24f377e Add deadline time for alarm prioritisation 2012-07-13 12:18:45 +09:30
Andrew Bettison
41f3228300 Refactor rhizome HTTP request parsing
Replace DEBUG_RHIZOMESYNC with DEBUG_RHIZOME_TX
2012-07-13 11:06:10 +09:30
Andrew Bettison
e378a8dc82 Improve logging from rhizome HTTP server 2012-07-12 16:42:41 +09:30
Andrew Bettison
d111f763c7 Fix bugs revealed by 'rhizomeprotocol' test
Was not transmitting actual HTTP server port in rhizome announcements, was
always transmitting port 4110.

When trying for a free HTTP server port, sometimes bind() succeeds but listen()
fails with EADDRINUSE, so new logic to deal with that.
2012-07-12 12:10:59 +09:30
Andrew Bettison
4eb3910dc1 Fix bug in Rhizome fetch - byte order of sin_port
Also improve some debug logging
2012-07-11 14:21:49 +09:30
Andrew Bettison
42744da371 Improve log diagnostics related to rhizome HTTP 2012-07-10 19:59:46 +09:30
Andrew Bettison
8020ea3b74 Merge branch 'eventscheduler' into master
Conflicts:
	commandline.c
	monitor.c
	overlay.c
	overlay_interface.c
	overlay_packetformats.c
	rhizome_fetch.c
	rhizome_http.c
	rhizome_packetformats.c
	serval.h
	server.c
	testdefs.sh
	testframework.sh
	tests/dnaprotocol
	tests/server
2012-07-03 10:26:22 +09:30
Andrew Bettison
08a8ec13e8 Start rhizome HTTP server on first available port 2012-07-02 16:24:07 +09:30
Jeremy Lakeman
132d3a6f9b rename profiling structure 2012-07-02 16:04:00 +09:30
Jeremy Lakeman
e705696896 separate statistics gathering from the scheduler structure 2012-07-02 15:20:30 +09:30
Jeremy Lakeman
89566e4d3d Refactor how functions are scheduled or file handes are watched 2012-07-02 13:19:54 +09:30
Jeremy Lakeman
89d3923557 Reduce log spam 2012-06-28 11:13:52 +09:30
Jeremy Lakeman
29cba17891 Fix rhizome transfers 2012-06-27 16:54:42 +09:30
Andrew Bettison
981afb795e Overhaul rhizome HTTP server start code 2012-06-26 11:07:01 +09:30
gardners
b40a468276 Merge branch 'eventscheduler'
Conflicts:
	monitor.c
	serval.h
2012-06-25 16:50:23 +09:30
Daniel O'Connor
930280a17d Move signal handler setup to main so it's only done once. 2012-06-25 16:24:33 +09:30
Andrew Bettison
208b9c15fd Factor out rhizome_hex_to_bytes() and chartonybl() 2012-06-25 14:51:21 +09:30
gardners
5dc6d122a5 significant progress towards clean callback scheduler for poll()
events and timed callbacks.
2012-06-22 16:05:49 +09:30
gardners
dceeed8d35 more work on dna helper interface.
Also added missing dna_helper.c file from previous changes.
moved sigpipe/sigio detection code to own file.
2012-06-21 16:35:28 +09:30
Andrew Bettison
86c7819f97 Remove spurious ERROR messsages
Tests assert that stderr contains no ERROR: lines after a successful exit

Rewrote sqlite_exec_int64() to separate error outcomes from legitimate
result values

Changed several WHY() calls to DEBUG()

Improved test framework
2012-06-08 13:13:26 +09:30
Andrew Bettison
1af9125392 All rhizome database keys are uppercase hex
FILES.id, MANIFESTS.id, FILEMANIFESTS.fileid, FILEMANIFESTS.manifestid
Named constants for hex and binary ID sizes
2012-05-23 16:04:00 +09:30
gardners
8e9396419c adding ignored manifest list to let us ignore repeated offerings
of broken manifests without wasting effort.
2012-05-22 14:15:35 +09:30
gardners
29bb86d9e7 first cut at supressing rhizome activity during calls. 2012-05-22 13:05:29 +09:30
gardners
bf6172f7ca clean up and use WHY() macros for all output. 2012-05-22 10:01:57 +09:30
gardners
1eb4c34015 Renamed mphlr.h to serval.h
(also some temporary debugging has been added in some places)
2012-02-23 12:45:42 +10:30