Commit Graph

151 Commits

Author SHA1 Message Date
Andrew Bettison
3678522872 Issue #17, add AUTHOR column to Rhizome MANIFESTS table
Replace ".selfsigned" column with ".author" and ".fromhere" columns in
output of "rhizome list" command.  (Note that a "sender" column is
already present.)

Add 'author' field to struct rhizome_manifest.

Log all fully rendered SQL statements on DEBUG_RHIZOME.

Update 'rhizomeops' test cases and improve the assert_rhizome_list()
test function to be able to assert authorship of files.
2012-10-09 17:48:06 +10:30
Andrew Bettison
b4c92a289e Fix SEGV bugs on Solaris
Revealed by 'rhizomeprotocol' tests FileTransferDelete, DirectPush and
DirectSync.
2012-10-05 15:49:36 +09:30
gardners
d685953ff5 Fixed storage of signatures with manifests in a codepath that
was missed in commit 1c72778c50.
2012-10-05 05:53:11 +02:00
gardners
33107e795f Reduced debug output. #9 2012-10-03 16:26:41 +02:00
gardners
1c72778c50 fixed problem with rhizome direct push / manifest import where
signatures were not being recorded.  Also fixed separate issue
where rhizomeprotocols tests expected selfsigned to be 1 on
receiver end, when it will never be, because the BK doesn't match.
rhizome direct push test in rhizomeprotocols now passes. #9
2012-10-03 12:28:23 +02:00
Andrew Bettison
5fed0460e1 Issue #9, 'rhizomeprotocol' HttpImport test now passes
Any HTTP client can import a bundle into Rhizome using a PUSH request.
2012-10-02 17:15:14 +09:30
Andrew Bettison
cc9cb8d827 Issue #9, fix old 'rhizomeprotocol' test failures
Now only the five new Rhizome Direct tests fail.
2012-10-02 16:32:48 +09:30
Andrew Bettison
17ef2249e4 Issue #9, rewrite rhizome_import_bundle()
Objective is to avoid having to call system("servald rhizome import ...") to
handle a Rhizome direct POST /rhizome/bundle request.  Antiquated code in and
around rhizome_import_bundle() needs much cleaning up, as indicated by some
TODO comments.  Invocations must unnecessarily write the manifest into a file,
when they already have it in memory, ready to pass to the function.

All the 'rhizomeops' tests pass, but two 'rhizomeprotocol' tests are broken
by the changes in this commit.
2012-09-28 17:55:43 +09:30
Jeremy Lakeman
152d46ddb6 Reduce and improve logging 2012-08-09 13:30:52 +09:30
Andrew Bettison
12d8bc0c12 Use new typedef time_ms_t everywhere 2012-08-09 12:14:32 +09:30
Andrew Bettison
240be25793 Replace overlay_gettime_ms() with gettime_ms() 2012-07-30 17:22:38 +09:30
Andrew Bettison
b50943d09c Fix handling of empty payload (filesize = 0)
Do not add 'filehash' var to manifest if filesize=0
Do not accept 'filehash' var when parsing manifest with filesize=0
When responding to a new rhizome advertisement, do not try to HTTP
request a payload if filesize=0, just import the manifest directly
Various operations, eg "rhizome file add", do not report 'filehash'
fields where 'filesize' is zero
Do not delete rows from MANIFESTS table which have empty filehash
Various related bug fixes
2012-07-16 18:40:30 +09:30
Andrew Bettison
41f3228300 Refactor rhizome HTTP request parsing
Replace DEBUG_RHIZOMESYNC with DEBUG_RHIZOME_TX
2012-07-13 11:06:10 +09:30
Andrew Bettison
208a95a233 Replace validateSid() with str_is_subscriber_id() 2012-07-12 16:14:25 +09:30
Andrew Bettison
ff2c98afb3 Add "rhizome.fetch_interval_ms" config option 2012-07-11 16:51:30 +09:30
Andrew Bettison
38a5717789 Fix 'rhizomeprotocol' test to always PASS 2012-07-11 14:51:25 +09:30
Andrew Bettison
72769e162b Replace overlay_render_sid_prefix() with alloca_tohex()
To eliminate timebomb bugs caused by rotor wraparound
2012-07-04 10:30:46 +09:30
Andrew Bettison
208b9c15fd Factor out rhizome_hex_to_bytes() and chartonybl() 2012-06-25 14:51:21 +09:30
Andrew Bettison
637ef02f30 Add _selfsigned column to "rhizome list" output
Add <pin,pin...> arg1 to "rhizome list" command, so it can open the keyring

Some tests in tests/rhizome fail
2012-06-08 18:25:43 +09:30
Andrew Bettison
7a71a521f8 Pass author SID parameters in binary, not hex 2012-06-08 15:27:15 +09:30
Andrew Bettison
86c7819f97 Remove spurious ERROR messsages
Tests assert that stderr contains no ERROR: lines after a successful exit

Rewrote sqlite_exec_int64() to separate error outcomes from legitimate
result values

Changed several WHY() calls to DEBUG()

Improved test framework
2012-06-08 13:13:26 +09:30
Andrew Bettison
b14db5c28b Improve "rhizome add file" for empty payloads
Accept the empty string for a payload pathname to mean a zero-length
payload.
2012-06-05 15:45:53 +09:30
Jeremy Lakeman
0f214ed46f Adjust database schema 2012-05-28 19:24:02 +09:30
gardners
07fc646fed various fixes to rhizome re-fetch logic after a failed partial fetch
(which seems to be what is tripping things up)
2012-05-27 17:29:50 +02:00
gardners
30e53f963c fixed various manifest synchronisation bugs, including probably
the main one that was stopping meshms (large manifests would
practically never get advertised under certain (common) conditions).
2012-05-27 08:31:48 +02:00
gardners
04cc05d6e1 refactored manifest verification into a separate function
that is called only when needed, and marks a manifest as finalised
if the verifcation fails.  reading a manifest now never sets
finalised flag, as either _finalise() or _verify() must be called.
2012-05-26 11:36:18 +02:00
gardners
e213a9eae8 renamed sanity_check to check_sanity for consistency.
Put missing check for illegal service type into check_sanity.
2012-05-26 11:36:18 +02:00
gardners
8161709c4e work on fixing handling of duplicate rhizome bundle debouncing.
(seems to work from command line, but not from the test framework).
2012-05-26 11:36:18 +02:00
gardners
e473e5b6df some fixes to new refactored bundle handling code. 2012-05-26 11:36:18 +02:00
gardners
b0dca7e03f merged in latest changes from team. 2012-05-26 11:36:18 +02:00
Andrew Bettison
49aec4d331 Improve rhizome manifest debugging
Move rhizome_new_manifest() out of rhizome_read_manifest_file() so that the
out-of-manifest report shows the names of the functions where the manifests
were really allocated.
2012-05-25 15:38:13 +09:30
Andrew Bettison
c0ac693957 Remove old payload when updating a manifest with a new payload 2012-05-25 14:29:55 +09:30
Andrew Bettison
c847478140 Add some helpful debug logging 2012-05-24 18:46:33 +09:30
Andrew Bettison
3b00110c49 Fix bug in rhizome_bundle_import()
Introduced by recent rhizome import path improvements
2012-05-24 13:00:31 +09:30
Andrew Bettison
fa26b53294 Fix SEGV bug
rhizome_write_manifest_file() was not checking for NULL fopen() result
Standardise rhizome "import" directory pathname handling
2012-05-24 11:28:32 +09:30
Andrew Bettison
3f9e20f2f0 Cosmetic edits, no change to function 2012-05-23 18:13:18 +09:30
Andrew Bettison
54d456e540 Fix and reinstate FORM_RHIZOME_DATASTORE_PATH() 2012-05-23 18:11:34 +09:30
Andrew Bettison
1af9125392 All rhizome database keys are uppercase hex
FILES.id, MANIFESTS.id, FILEMANIFESTS.fileid, FILEMANIFESTS.manifestid
Named constants for hex and binary ID sizes
2012-05-23 16:04:00 +09:30
gardners
299cb92952 further output cleanups. 2012-05-22 17:08:12 +09:30
gardners
6a83f2434f cleaned up some debug output, among other things. 2012-05-22 17:04:24 +09:30
gardners
3f1f668e8b found and fixed a bug with chopping signatures from manifests when
importing (really need to clean that code up).
2012-05-22 16:55:59 +09:30
gardners
8e9396419c adding ignored manifest list to let us ignore repeated offerings
of broken manifests without wasting effort.
2012-05-22 14:15:35 +09:30
gardners
29bb86d9e7 first cut at supressing rhizome activity during calls. 2012-05-22 13:05:29 +09:30
gardners
99bbbb0b05 make sure import path exists. 2012-05-22 10:00:26 +09:30
gardners
322806a150 force correct construction of filenames for manifests and data files
(strbuf_* not working for these right now).
2012-05-21 22:31:35 +09:30
gardners
f96ba5be18 added various debugging, including revealing problem forming
file names for rhizome bundles being imported.
2012-05-21 22:19:47 +09:30
Andrew Bettison
c2de68449b Make rhizome_add_manifest() stricter for MeshMS
If author not given, use 'sender' as author
Ensure that author and 'sender' field are identical
2012-05-21 10:37:24 +09:30
Andrew Bettison
ddd17db1c4 Make "rhizome add file" work for MeshMS 2012-05-21 00:09:14 +09:30
Andrew Bettison
e6575cb369 Introduce macro constants for Rhizome service names
Use "MeshMS1" instead of "MeshMS"... version 2 won't be far away
2012-05-20 16:07:22 +09:30
Andrew Bettison
d90508b2ba Fix bugs in rhizome_add_manifest()
Tests for service types were inverted
Fail with error if BK field cannot be added
2012-05-20 13:46:38 +09:30
Andrew Bettison
4b5e179841 Start support for MeshMS manifests 2012-05-19 14:09:50 +09:30
Andrew Bettison
9a51c76dfb Insist on 'service' field in all manifests
The "rhizome file add" command assumes service=file if no manifest supplied
or the manifest lacks a service field.

The "rhizome extract manifest" command includes the service in its CLI output.
2012-05-17 17:00:14 +09:30
Andrew Bettison
2434d51bee Remove "rhizome add authored file" command
Now "rhizome add file" takes SID and PIN args, and if SID is empty, an
unauthored bundle (no BK field) is created

Updated dna_rhizome test cases
2012-05-17 12:01:25 +09:30
gardners
09e0cc5030 removed deprecated first_byte and last_byte manifest fields. 2012-05-16 15:56:39 +09:30
gardners
8980b13bbf fixed BK calculation bug. now calculates BK correctly. 2012-05-16 15:41:41 +09:30
gardners
39be8a0b75 added new command line option for adding rhizome bundle with authorship
so that it includes a BK.
All good, except the BK isn't being added. Yet to find out why.
2012-05-15 22:15:13 +09:30
gardners
0eafc34970 BK encoding and decoding complete, but untested. 2012-05-15 20:04:41 +09:30
gardners
525d3c4154 substantial work towards implementing BK field in rhizome manifests. 2012-05-15 17:24:25 +09:30
Andrew Bettison
b02e9b9ac4 Overhaul rhizome datastore path code
Create datastore directory and all parent dirs if it does not exist
More robust error checking and buffer overflow protection
New mkdirs() and mkdirsn() functions
2012-05-14 18:37:32 +09:30
gardners
0b5a9665b2 first cut code for sending monitor socket announcements when
rhizome bundles are received.
2012-05-12 07:24:52 +09:30
Andrew Bettison
e746557a08 Fix bug in overlay_gettime_ms()
The sequence start time was only getting set in overlay_interface_init(), which
was not always called, or was called after overlay_gettime_ms() had already
been used.

Added FATALF(), FATAL() and FATAL_perror() macros.

Removed a bunch of debug statements no longer needed in monitor.c and server.c.
2012-05-11 10:38:46 +09:30
Andrew Bettison
f31c326871 CLI output from "rhizome add file", with tests 2012-05-04 15:31:03 +09:30
Andrew Bettison
146eb7cdfd Add "rhizome extract manifest" command 2012-05-02 16:03:09 +09:30
Andrew Bettison
866ddea189 Get all rhizome add file test cases passing
- new test cases to cover new payload with different versions
 - exit status of 2 when duplicate is detected
 - tighten up other test cases
 - fix bug in sqlite_exec_int64()
2012-04-16 11:46:58 +09:30
Andrew Bettison
de88d3db21 Make "dna rhizome add" work for more test cases
- adding a bundle using an existing manifest with an incorrect payload should
   fail with an error
 - adding a bundle using an existing manifest to update the payload to a new
   version should succeed
 - improve format of "dna rhizome list" output to one bundle per line
2012-04-13 18:03:19 +09:30
Andrew Bettison
7b53fd6782 Rhizome add does not add duplicate files
- If a bundle has the same payload hash and manifest.name as one already in
   the rhizome store, then the "dna rhizome add" command will not add it
 - Now passes fourth test case in tests/dna_rhizome
2012-04-12 18:33:43 +09:30
Andrew Bettison
a87a78536c Refactor rhizome_add_manifest() 2012-04-11 18:40:10 +09:30
Andrew Bettison
74986a0c30 Implement "rhizome add file" and "rhizome list" 2012-04-02 17:42:40 +09:30
Andrew Bettison
738b70b513 Test and fix ACTION_CREATEHLR idempotency code:
- refactor hlrSid() to not return pointer to static buffer, take 3rd arg instead
 - introduce SID_STRLEN macro constant, use it everywhere
 - reformat some code for readability
2012-03-14 12:00:54 +10:30
gardners
1eb4c34015 Renamed mphlr.h to serval.h
(also some temporary debugging has been added in some places)
2012-02-23 12:45:42 +10:30
gardners
c0676fd530 Fixed various bugs and memory leaks in rhizome file fetching.
Now successfully fetches manifests for files we already have in
the database, using the existing database copy of the file.
Also added filter to stop same file being fetched more than once at
same time..
2012-01-27 16:21:48 +10:30
gardners
42b5c08321 Refactored rhizome.c into several smaller files.
Manifests are now gleaned from advertisements, and the ones that
are interesting are scheduled for importing into database.
(actual importing is not yet done -- requires fetching of file).
2012-01-12 14:05:05 +10:30
gardners
670f7c51c7 Reworked Rhizome advertisement frames to include whole manifests
inline where possible.
2012-01-11 14:27:40 +10:30
gardners
0b73df19c8 Fixed bug with handling of OA_CODE_SELF intepretation. 2012-01-11 07:16:22 +10:30
gardners
f8eea651b5 Changed broadcast address to include a 64bit BPI to allow filtering
of duplicate broadcast announcements.
2012-01-10 21:56:07 +10:30
gardners
44c0765981 Various debugging code cleanups.
Fixed some address resolution bugs (for when sender address is not
fully known).
There is still a bug in dequeuing packets though it seems.
2012-01-10 17:21:26 +10:30
gardners
800f8d41eb Revamped debug/verbosity control to use flags for functions of interest
instead of general verbosity ramp.
2012-01-10 15:56:40 +10:30
gardners
9156a68c7f Added 2nd implementation of serval packet decoder for aiding debugging
of packet formation and decoding.
Fixed time calculation bug (duplicate time functions based off different
epochs).
Various debugging fiddles and tweaks.
2012-01-10 14:05:26 +10:30
gardners
23ee957169 Various minor tweaks to debugging and related things.
Fixed major memory handling error in neighbour structure allocation.
2012-01-09 16:28:44 +10:30
gardners
1240c49a92 Added more debugging to no open path messages.
Added hook for reading rhizome advertisements.
2012-01-09 09:17:54 +10:30
gardners
25458d0e2b Fixed packet length fields for Rhizome advertisements. 2012-01-09 09:07:00 +10:30
gardners
e069984c71 Found and fixed evil memory corruption error. 2012-01-09 05:10:58 +10:30
gardners
ba645687f6 Progress towards overlay mode advertising rhizome bundles.
(but some evil memory corruption error if overlay mode and
rhizome are both enabled.)
2012-01-09 04:57:13 +10:30
gardners
4e6ef2e69e Various fixes to Rhizome web server and BAR generation.
Can now present a set of BARs via HTTP.
2012-01-03 16:35:02 +10:30
gardners
16fb3a3f61 Added missing GPL license statements. 2011-12-21 20:25:05 +10:30
gardners
ae22480c23 signature reading and logging of signature verification errors now
in place.
2011-12-21 07:53:45 +10:30
gardners
b7b795d4fc manifest reading now reads and can verify signatures.
Still incomplete. Doesn't yet count verify errors etc.
2011-12-21 07:46:12 +10:30
gardners
a159edf152 updating manifests now correctly removes old files and old
versions of the manifest.
group subscriptions not yet implemented, and reading of signatures
not yet implemented.  Controls on group membership and pushing files
to groups not yet implemented.  Group ciphers not yet implemented.
2011-12-20 22:09:49 +10:30
gardners
745cade469 Further work on getting signatures and signed manifests working.
Some work on updating existing manifests, although some cascading
consequences need addressing (in particular checking if the file
associated with the old version of the manifest still needs to hang
about, and if so, whether its highest priority has changed).
2011-12-20 17:27:24 +10:30
gardners
4719281f03 Added keypair storage and retrieval (still pending encryption of
stored secrets, but the framework is there).
2011-12-20 15:57:25 +10:30
gardners
cc09a0f994 Some database schema tweaks and additional checks on manifests when
importing (we now need to have either the key for the manifest, or to
be creating the manifest).
2011-12-20 15:48:26 +10:30
gardners
8fca64d630 Importing bundles into the rhizome database now works.
Signing and groups are still missing for the moment.
2011-12-20 14:16:17 +10:30
gardners
a50c56ac92 Importing of a bundle into the database now seems to work.
(group membership not currently handled, signing not currently
handled).
2011-12-20 13:24:09 +10:30
gardners
312eaa397b Significant progress on Rhizome.
Manifest reading and writing to disk (not database) seems okay now.
Working on database side of things next.
2011-12-20 11:25:52 +10:30
gardners
34075ce84d Fixed an NaCl macro name issue for Android NDK cross-compile 2011-12-19 09:03:30 +10:30
gardners
b775e48042 further NaCl macro name fix for Android cross-compilation. 2011-12-19 08:28:39 +10:30
gardners
c110e46a08 fixes to previous commit. 2011-12-19 08:10:02 +10:30
gardners
000dd2db55 Moved rhizome prototypes into separate header file.
(needed for android building where undefined functions are
an error, not a warning).
2011-12-19 08:04:31 +10:30
gardners
1628d5cbfb Added command line options to DNA to import a rhizome bundle. 2011-12-19 05:46:34 +10:30
gardners
6ec1d6032f Further work towards sqlite3 backend for rhizome.
Wrote function to store file to db iff it is not stored already,
and update the priority flag on the stored file.
Uses blobio to progressively load large blobs to make it friendly
on small memory systems.
2011-12-17 12:11:32 +10:30