Commit Graph

1063 Commits

Author SHA1 Message Date
7f5673eb21 handle non-utf8 from libfuzzer stderr (#379) 2020-12-10 15:13:14 -05:00
69fc9f508b fix clippy issue (#367) 2020-12-04 15:04:29 -05:00
203bc22756 Allow unmaintained memmap (#364) 2020-12-02 15:34:22 -05:00
b81c6fa89e fix job_templates deletion (#360) 2020-12-02 14:02:16 -05:00
054989f232 Add support for ASAN print_scariness (#359) 2020-12-02 11:33:22 -05:00
e6b55ab95a Simplify job template management workflow (#354)
1. Merge 'create' and 'update' to a single 'save' operation.
2. Allow fetching a single template.

This enables the following workflow:

```
$ onefuzz job_templates manage get libfuzzer_linux > template.json
$ <... update template as desired ...>
$ onefuzz job_templates manage save libfuzzer_linux @./template.json
$
```
2020-12-02 14:27:42 +00:00
9b3ccf37ea use the correct instrumentation key (#355) 2020-12-01 18:44:10 -05:00
0182dc597d handle asan check failures (#358) 2020-12-01 18:23:26 -05:00
fc34725428 update rust prereqs (#357) 2020-12-01 17:22:32 -05:00
aef511efe8 Fail the task if parsing asan_log files fail (#351)
This differentiates parsing ASAN log parse failures from ASAN logs not existing, fixing the first part of #343.
2020-12-01 21:10:59 +00:00
7f97c142ed add the instrumentation key to Info (#353) 2020-12-01 11:13:06 -05:00
3f3193beeb Use disable_check_debugger on asan integration tests (#352) 2020-12-01 10:36:53 -05:00
a1af90cb83 Update deployment prerequisites to remove pyopenssl errors (#348)
Over the weekend, pyOpenssl 20.0 was released.  This causes an incompatible library issue during deployment.

Prior to this change, deployment would generate the following error
```
ERROR: pyopenssl 20.0.0 has requirement cryptography>=3.2, but you'll have cryptography 2.9.2 which is incompatible.
```
2020-12-01 14:43:53 +00:00
5092f96af4 Fix deployment of backdated versions of OneFuzz (#347)
When running automated deployments, 'tools' were not being properly replaced with the updated versions if the deployment was created _prior_ to the original instance deployment.
2020-12-01 10:59:43 +00:00
37e3251966 render the event model as json to not include error (#350) 2020-11-30 23:19:27 -05:00
30cc5d4778 ignore nodes already scheduled for re-imaging in outdated check (#341)
If a node is already scheduled to be reimaged/deleted, we should not bother checking if it's outdated.
2020-11-30 17:36:15 +00:00
079f387b88 clarify prefix-expansion errors (#342) 2020-11-24 11:51:03 -05:00
33b7608aaf Adding option to merge all inputs at once (#282) 2020-11-24 08:43:08 -05:00
79cc82098a Move integration test artifacts into primary source tree (#336) 2020-11-24 08:03:01 -05:00
905dc7c0d6 Re-enable the retry logic for App Password creation (#338) 2020-11-24 08:00:31 -05:00
d47124fe8c Fix state management in the scheduler (#337) 2020-11-24 12:43:51 +00:00
32ba86be9d Update current_thread_id when setting current thread (#340) 2020-11-23 13:39:03 -08:00
3ddb756504 Add linting to deployment tools (#332) 2020-11-20 13:00:19 -05:00
9e2a61fe66 Add user_info to Jobs & Repro (#327)
This adds information about the user that created a job or repro VM to the respective resources.

This expands on the addition made to tasks in #303.
2020-11-20 15:46:52 +00:00
d96209c659 Include the body when receiving a registration error (#321)
* Include the body when receiving a registration error to help debug issued like #215 
* increase the agent registration timeout to 20 min
2020-11-20 14:43:40 +00:00
7e3b807479 Support pre-release instance specific setup script paths (#331)
Support `instance-specific-setup/<OS>/setup` and `instance-specific-setup/setup` scripts.

Fixes #328
2020-11-20 12:42:58 +00:00
3974d680ef Support retry during function deploy (#330)
Starting earlier today, I saw roughly 1 in 3 deployments fail with the error `Azure.Functions.Cli.Common.CliException: Timed out waiting for SCM to update the Environment Settings`.  Redeploying the application resolves the issue.  New builds and past releases alike hit this exception.

According to https://github.com/Azure/azure-functions-core-tools/issues/1863, function app deployments may fail due to timeouts related to cold-start.

This PR executes the deploy in a loop with a delay in the case of failure.
2020-11-19 20:04:18 +00:00
31a661f071 Expose coverage/exec_sec for libfuzzer targets via CLI (#325)
Adds debug subcommands to the SDK/CLI that simplify querying Application Insights for libfuzzer telemetry.  

Querying for the latest execs_sec for a job, by job_id fragment.
```
$ onefuzz debug job libfuzzer_execs_sec 88 --limit 1
[
    {
        "execs_sec": "191035",
        "machine_id": "b2dbe720-4fd8-4342-957a-6cb0979d2187",
        "timestamp": "2020-11-18T00:08:53.98Z",
        "worker_id": "0"
    }
]
```

Querying for the latest coverage for a job, by job_id fragment.
```
$ onefuzz debug job libfuzzer_coverage 88 --limit 1
[
    {
        "covered": "10",
        "features": "21",
        "rate": "0.47619047619047616",
        "timestamp": "2020-11-18T00:09:40.793Z"
    }
]
```
2020-11-19 15:14:37 +00:00
bb2b18a2b9 Fix MSVC Libfuzzer coverage reporting (#324)
This PR fixes two issues:
- First, in MSVC compiled binaries both the LLVM _and_ MSVC symbols are
present, but only the MSVC symbols have correct values. For example:

```
0:000> cdb: Reading initial command '.scriptload DumpCountersOld.js ; !dumpcounters "cov" ; q'
JavaScript script successfully loaded from 'DumpCountersOld.js'
[+] not disabling sympath
INFO: Seed: 58715679
INFO: Loaded 1 modules   (3968 inline 8-bit counters): 3968 [00007FF70DB4B000, 00007FF70DB4BF80), # XXX Note
xxx.exe: Running 1 inputs 1 time(s) each.
Running: inp
[+] processing xxx.exe
[+] using LLVM 10 symbols - 0x7ff70db72b00:0x7ff70db72b08 # XXX These are wrong
```

This means the order we search for the coverage symbols is important.

- Secondly, this enables support for MSVC 8bit counter coverage.

## Validation Steps Performed

Running any recent MSVC compiled libfuzzer target should fail to actually collect coverage, instead just returning the 8 null bytes described in the linked issue.
2020-11-19 02:47:33 +00:00
b2b4a06afa Address typing issues hidden by memoization.caching (#322) 2020-11-18 15:08:40 -05:00
bb6d083768 Enable unmanaged registrations and configuration by environment variables (#318) 2020-11-18 12:19:09 -05:00
e47e89609a Use Storage Account types, rather than account_id (#320)
We need to move to supporting data sharding.

One of the steps towards that is stop passing around `account_id`, rather we need to specify the type of storage we need.
2020-11-18 14:06:14 +00:00
52eca33237 Move more run-time actions to setup-time (#317)
This script moves more of the run-time actions to setup-time.  This is important for running fuzzing within docker containers, such that installing llvm & gdb is done as part of the container, rather than on each launch.
2020-11-18 10:13:01 +00:00
64bd389eb7 Declarative templates (#266) 2020-11-17 16:00:09 -05:00
ce3356d597 Add SDK Feature Flags (#313)
## Summary of the Pull Request

This enables feature flags for the SDK, which enables gating access to preview features to those that have specifically asked for them.  This is intended to be used within #266.

Note, this change also moves to using a `pydantic` model for the config, rather than hand-crafted JSON dicts.
2020-11-17 15:40:16 +00:00
02600320b7 Don't reset webhooks by default (#307) 2020-11-16 16:51:32 -05:00
beea318968 Add User Info to created tasks (#303)
This PR makes user information from JWT tokens available as part of a Task.

Included changes:
* Renamed `verify_token` to `call_if_agent`, since this function is specific to agent token verification
* Renames `is_authorized` to `is_agent`, since this function checks if the token is an agent
* Adds support for unmanaged nodes in `is_agent` (see #133 for information) 
* Saves the user information from the JWT token on task create as part of `TaskConfig`

Note, `TaskConfig` is what is provided to notification templates.  This enables Github issues and ADO work items to tie back to the user that created the task.

Note, while `upn` _usually_ means email for AAD user tokens.  If we were going to make use of the email address, we should perform a graph lookup based on the `oid`, but we're not.
2020-11-13 11:50:52 +00:00
31f099d3d4 Event based webhooks (#296) 2020-11-12 17:44:42 -05:00
693c988854 add symbols for windows agents (#306) 2020-11-12 15:46:32 -05:00
a0b5d10c81 Add target_workers to TaskUnitConfig (#305) 2020-11-12 13:22:53 -05:00
87085bc48a Change prepare_test_result to take reference (#297)
Originally used move to avoid copying, but the copy is cheap in the
normal case (no exceptions) and is more usable this way.
2020-11-11 21:48:08 -08:00
61b3f87dd4 prevent users from enabling public access to containers (#300) 2020-11-11 20:36:49 -05:00
c56f72b37c Make supervisor heartbeat only fire on main loop progress (#283) 2020-11-11 18:30:02 -05:00
ca209eb543 refactor agent_events handler (#261) 2020-11-11 18:28:16 -05:00
4e138b0bef cli debug command to ssh/rdp to a specific node (#298) 2020-11-11 17:14:02 -05:00
9e24754150 Windows debugger updates (#294)
* Set execution context while handling breakpoint

If we don't do this, the caller will see incorrect state such
as the wrong rip.

* Improve debugger logging messages
* Tiny tweak to breakpoint data type
* Suspend threads while single stepping over breakpoint
* Fix get_current_thread_id to return actual id
2020-11-11 12:09:18 -08:00
5c45d85359 Don't use HTTP 404 on agent refresh error (#287) 2020-11-11 12:33:27 -05:00
c70007d5df include deleting unique_inputs on containers.reset (#290) 2020-11-11 12:33:03 -05:00
dec1a2d7b0 removing nodes whose ground truth is not avail (#275) 2020-11-11 12:20:05 -05:00
8678755b71 Render envelope.event as json (#288) 2020-11-11 12:18:32 -05:00