Commit Graph

106 Commits

Author SHA1 Message Date
6a049db3a3 Renames application insights keys to be more clear (#587)
* renames `telemetry_key` to `microsoft_telemetry_key`
* renames `instrumentation_key` to `instance_telemetry_key`
* renames `can_share` to `can_share_with_microsoft`
* renames the `applicationinsights-rs` instances to `internal` and `microsoft` respective of the keys used during construction.

This clarifies the underlying use of Application Insights keys and uses struct tuple to ensure the keys are used correctly via rust's type checker.
2021-02-26 17:04:49 +00:00
a3fa5f6b62 Update onefuzz-agent unit tests (#592) 2021-02-24 20:54:36 -05:00
89d7f060dd make missing symbols for coverage tasks more explicit (#554)
This moves from:

```
"Error: coverage extraction from C:\users\bcaswell\projects\bugs\andrew-coverage-fail\setup\oft-setup-5c77cfe1b181520ab0b33a16286a690a\fuzz.exe failed when processing file "11f6ad8ec52a2984abaafd7c3b516503785c2072".  target appears to be missing sancov instrumentation",
```

To even more explicit:
```
Error: Target appears to be missing sancov instrumentation.  This error can happen due to missing coverage symbols.
target_exe: C:\users\bcaswell\projects\bugs\andrew-coverage-fail\setup\oft-setup-5c77cfe1b181520ab0b33a16286a690a\fuzz.exe
input: "11f6ad8ec52a2984abaafd7c3b516503785c2072"
debugger stdout:
...
[+] disabling sympath
[+] processing fuzz.exe
[+] no tables  fuzz.exe
[+] processing C:\WINDOWS\SYSTEM32\kernel.appcore.dll
[+] no tables  C:\WINDOWS\SYSTEM32\kernel.appcore.dll
[+] processing C:\WINDOWS\System32\KERNELBASE.dll
[+] no tables  C:\WINDOWS\System32\KERNELBASE.dll
[+] processing C:\WINDOWS\System32\RPCRT4.dll
[+] no tables  C:\WINDOWS\System32\RPCRT4.dll
[+] processing C:\WINDOWS\System32\msvcrt.dll
[+] no tables  C:\WINDOWS\System32\msvcrt.dll
[+] processing C:\WINDOWS\System32\KERNEL32.DLL
[+] no tables  C:\WINDOWS\System32\KERNEL32.DLL
[+] processing ntdll.dll
[+] no tables  ntdll.dll
Error: unable to find sancov counter symbols [at DumpCounters (line 114 col 9)]
...
```
2021-02-17 16:34:09 +00:00
c160088998 expose input_blob fields needed to generate crash reports (#551) 2021-02-16 13:16:54 -05:00
360693e8a4 move verbose to debug to align with log and opentelemetry (#541) 2021-02-11 16:49:27 -05:00
a3d73a240d report the total coverage after processing all inputs in local mode (#537) 2021-02-11 19:34:09 +00:00
1e536c54d3 update error message when coverage extraction fails (#539) 2021-02-11 14:18:49 -05:00
f8046934e9 add roles to agent & supervisor (#527) 2021-02-10 20:56:22 +00:00
4900b5a920 check libfuzzer -help=1 prior to starting the heartbeat (#528) 2021-02-09 21:20:36 -05:00
6d4f45679f allow for more coverage data in total (#519) 2021-02-09 16:47:38 -05:00
c9455fdfa5 add logging to generic analysis task (#522) 2021-02-08 18:39:39 -05:00
19655b50ae update rust prereqs (#517) 2021-02-08 12:17:25 -05:00
a50eb94db5 Making input_tester and expand immutable (#500) 2021-02-04 15:18:15 -05:00
cdfdc2be84 split telemetry into it's own crate (#501)
Splits out telemetry crate such that it can be reused by other components (specifically the proxy-manager) easily.
2021-02-04 14:46:35 +00:00
02721f3ed9 address clippy issues in agent (#490) 2021-02-02 14:41:27 -05:00
0a1021447b Return an error when download_input fails (#485) 2021-01-29 17:13:36 +00:00
5acb59e5b9 Add task_id & job_id to variable expansion (#481)
Fixes #479 

Note, this is built on top of #480
2021-01-29 15:35:59 +00:00
9c7eb33149 Expand tools_dir accessibility in variable expansion (#480)
Fixes #478
2021-01-29 11:33:23 +00:00
bcf42485eb sync new inputs found by supervisor tasks (#484) 2021-01-28 21:33:30 -05:00
cfcf493a23 add context to command failures (#466)
Fixes #465
2021-01-26 21:29:59 +00:00
dc31ffc92b add support for fully self-contained fuzzers (#454) 2021-01-22 18:20:22 -05:00
f3d81566e3 add expanding {tools_dir} to supervisor_options and supervisor_env (#444)
Fixes #443
2021-01-20 20:34:25 +00:00
fd956380d4 experimental "local fuzzing" support (#405)
This PR adds an experimental "local" mode for the agent, starting with `libfuzzer`.  For tasks that poll a queue, in local mode, they just monitor a directory for new files.

Supported commands: 
* libfuzzer-fuzz (models the `libfuzzer-fuzz` task)
* libfuzzer-coverage (models the `libfuzzer-coverage` task)
* libfuzzer-crash-report (models the `libfuzzer-crash-report` task)
* libfuzzer (models the `libfuzzer basic` job template, running libfuzzer-fuzz and libfuzzer-crash-report tasks concurrently, where any files that show up in `crashes_dir` are automatically turned into reports, and optionally runs the coverage task which runs the coverage data exporter for each file that shows up in `inputs_dir`).

Under the hood, there are a handful of changes required to the rest of the system to enable this feature.
1. `SyncedDir` URLs are now optional.  In local mode, these no longer make sense.   (We've discussed moving management of `SyncedDirs` to the Supervisor.  This is tangential to that effort.)
2. `InputPoller` uses a `tempdir` rather than abusing `task_id` for temporary directory naming.
3. Moved the `agent` to only use a single tokio runtime, rather than one for each of the subcommands.
4. Sets the default log level to `info`.  (RUST_LOG can still be used as is).

Note, this removes the `onefuzz-agent debug` commands for the tasks that are now exposed via `onefuzz-agent local`, as these provide a more featureful version of the debug tasks.
2021-01-20 03:33:25 +00:00
a89065f882 adding {setup_dir} to variable expansion (#417)
## Summary of the Pull Request

Adds a new placeholder {setup_dir} for the setup directory 

## PR Checklist
* [x] Applies to work item: #221
* [x] CLA signed. If not, go over [here](https://cla.opensource.microsoft.com/microsoft/onefuzz) and sign the CLI.
* [x] Requires documentation to be updated
* [x] I've discussed this with core contributors already. If not checked, I'm ready to accept this work might be rejected in favor of a different grand plan. Issue number where discussion took place: #xxx

## Info on Pull Request

_What does this include?_

## Validation Steps Performed

_How does someone test & validate?_
2021-01-13 00:39:59 +00:00
2e2ba988ee Fix condition for triggering new unique report event (#422) 2021-01-12 14:00:34 -05:00
465727680d add context to all fs calls (#423)
Adds additional context in error handling to all `std::fs` and `tokio::fs` calls.

Fixes #309
2021-01-11 20:55:22 +00:00
7e56efa6a8 Address clippy issues (#409) 2021-01-05 15:41:46 +00:00
37f06bb324 handle libfuzzer fuzzing non-zero exits better (#381)
When running libfuzzer in 'fuzzing' mode, we expect the following on exit.

If the exit code is zero, crashing input isn't required.  This happens if the user specifies '-runs=N'

If the exit code is non-zero, then crashes are expected.  In practice, there are two causes to non-zero exits.
1. If the binary can't execute for some reason, like a missing prerequisite
2. If the binary _can_ execute, sometimes the sanitizers are put in such a bad place that they are unable to record the input that caused the crash.

This PR enables handling these two non-zero exit cases.

1. Optionally verify the libfuzzer target loads appropriately using `target_exe -help=1`.  This allows failing faster in the common issues, such a missing prerequisite library.
2. Optionally allow non-zero exits without crashes to be a warning, rather than a task failure.
2021-01-05 14:40:15 +00:00
e51d7affb7 Fixes race condition of a libfuzzer coverage without inputs (#403)
This fixes an issue running a libfuzzer coverage task and don't have any initial seeds (or there are seeds found by the fuzzer by the time the task starts), it will fail.
2021-01-05 00:05:13 +00:00
1b1af1f84f log stdout & stderr lines for supervisor & generator (#400)
This fixes #371 and #372.
2021-01-04 21:53:49 +00:00
7f5673eb21 handle non-utf8 from libfuzzer stderr (#379) 2020-12-10 15:13:14 -05:00
054989f232 Add support for ASAN print_scariness (#359) 2020-12-02 11:33:22 -05:00
33b7608aaf Adding option to merge all inputs at once (#282) 2020-11-24 08:43:08 -05:00
bb6d083768 Enable unmanaged registrations and configuration by environment variables (#318) 2020-11-18 12:19:09 -05:00
c56f72b37c Make supervisor heartbeat only fire on main loop progress (#283) 2020-11-11 18:30:02 -05:00
6c598773dd add instance_id generated at install time (#245) 2020-11-02 14:27:51 -05:00
ced8200d74 enable setting ensemble sync duration timer (#229) 2020-10-29 14:48:12 -04:00
36bae9e649 disable sympath unless it's set via target_env (#222) 2020-10-28 16:43:00 -04:00
59cfc52e9b report coverage total after batch processing prior (#218) 2020-10-28 12:51:32 -04:00
66dda9397a Address race-condition when syncing input seeds (#204) 2020-10-28 11:57:46 -04:00
db8534109f Resilient connection (#153) 2020-10-28 10:51:18 -04:00
bc2a3f816f add missing telemetry relating to reports & tools (#203) 2020-10-26 09:59:02 -04:00
b675ee75df allows hardcoding the version on build (#181) 2020-10-23 09:05:26 -04:00
3dd0f136b8 unify syncdir management & add jitter (#180) 2020-10-20 17:17:45 -04:00
a563861487 fail louder when uploads fail (#166) 2020-10-16 17:04:46 -04:00
458b21f690 Enable logging HTTP response body on error (#162) 2020-10-15 15:55:52 -04:00
3189daeeb7 implementing heartbeat for the supervisor (#30) 2020-10-14 15:13:16 -04:00
9acfbeaa2e only start appinsights threads with configured app insights instances (#142) 2020-10-14 13:33:20 -04:00
6e2a7222ed parse stderr for ASAN for libfuzzer targets (#127) 2020-10-09 17:00:19 -04:00
69832f727c Filter agent app-insights logs based on log::max_level (#125) 2020-10-09 13:52:14 -04:00