Commit Graph

2095 Commits

Author SHA1 Message Date
9b3ccf37ea use the correct instrumentation key (#355) 2020-12-01 18:44:10 -05:00
0182dc597d handle asan check failures (#358) 2020-12-01 18:23:26 -05:00
fc34725428 update rust prereqs (#357) 2020-12-01 17:22:32 -05:00
aef511efe8 Fail the task if parsing asan_log files fail (#351)
This differentiates parsing ASAN log parse failures from ASAN logs not existing, fixing the first part of #343.
2020-12-01 21:10:59 +00:00
7f97c142ed add the instrumentation key to Info (#353) 2020-12-01 11:13:06 -05:00
3f3193beeb Use disable_check_debugger on asan integration tests (#352) 2020-12-01 10:36:53 -05:00
a1af90cb83 Update deployment prerequisites to remove pyopenssl errors (#348)
Over the weekend, pyOpenssl 20.0 was released.  This causes an incompatible library issue during deployment.

Prior to this change, deployment would generate the following error
```
ERROR: pyopenssl 20.0.0 has requirement cryptography>=3.2, but you'll have cryptography 2.9.2 which is incompatible.
```
2020-12-01 14:43:53 +00:00
5092f96af4 Fix deployment of backdated versions of OneFuzz (#347)
When running automated deployments, 'tools' were not being properly replaced with the updated versions if the deployment was created _prior_ to the original instance deployment.
2020-12-01 10:59:43 +00:00
37e3251966 render the event model as json to not include error (#350) 2020-11-30 23:19:27 -05:00
30cc5d4778 ignore nodes already scheduled for re-imaging in outdated check (#341)
If a node is already scheduled to be reimaged/deleted, we should not bother checking if it's outdated.
2020-11-30 17:36:15 +00:00
2391d927f7 Updating yml file to run config endpoint command with tenant/authority ID. (#339)
## Summary of the Pull Request

Originally, the yml file printed out a semi-generalized _onefuzz config --endpoint_ comman. This command did have a specified _--authority_ and so it used the Microsoft id by default. To enable users to work with OneFuzz on tenants other than the standard Microsoft tenant, we have added a _--authority_ parameter that is printed out at the end of the deployment. 

## PR Checklist
* [ ] I've discussed this with core contributors already. If not checked, I'm ready to accept this work might be rejected in favor of a different grand plan. Issue number where discussion took place: #xxx

## Info on Pull Request

Changes to the yml file. 

## Validation Steps Performed

We have made this change to our local automation repository and tested an automated deployment pipeline with this change.
2020-11-30 14:54:42 +00:00
079f387b88 clarify prefix-expansion errors (#342) 2020-11-24 11:51:03 -05:00
33b7608aaf Adding option to merge all inputs at once (#282) 2020-11-24 08:43:08 -05:00
79cc82098a Move integration test artifacts into primary source tree (#336) 2020-11-24 08:03:01 -05:00
905dc7c0d6 Re-enable the retry logic for App Password creation (#338) 2020-11-24 08:00:31 -05:00
d47124fe8c Fix state management in the scheduler (#337) 2020-11-24 12:43:51 +00:00
32ba86be9d Update current_thread_id when setting current thread (#340) 2020-11-23 13:39:03 -08:00
2e276de0f5 Release 1.9.0 (#335) 1.9.0 2020-11-20 16:01:28 -05:00
3ddb756504 Add linting to deployment tools (#332) 2020-11-20 13:00:19 -05:00
9e2a61fe66 Add user_info to Jobs & Repro (#327)
This adds information about the user that created a job or repro VM to the respective resources.

This expands on the addition made to tasks in #303.
2020-11-20 15:46:52 +00:00
d96209c659 Include the body when receiving a registration error (#321)
* Include the body when receiving a registration error to help debug issued like #215 
* increase the agent registration timeout to 20 min
2020-11-20 14:43:40 +00:00
7e3b807479 Support pre-release instance specific setup script paths (#331)
Support `instance-specific-setup/<OS>/setup` and `instance-specific-setup/setup` scripts.

Fixes #328
2020-11-20 12:42:58 +00:00
3974d680ef Support retry during function deploy (#330)
Starting earlier today, I saw roughly 1 in 3 deployments fail with the error `Azure.Functions.Cli.Common.CliException: Timed out waiting for SCM to update the Environment Settings`.  Redeploying the application resolves the issue.  New builds and past releases alike hit this exception.

According to https://github.com/Azure/azure-functions-core-tools/issues/1863, function app deployments may fail due to timeouts related to cold-start.

This PR executes the deploy in a loop with a delay in the case of failure.
2020-11-19 20:04:18 +00:00
31a661f071 Expose coverage/exec_sec for libfuzzer targets via CLI (#325)
Adds debug subcommands to the SDK/CLI that simplify querying Application Insights for libfuzzer telemetry.  

Querying for the latest execs_sec for a job, by job_id fragment.
```
$ onefuzz debug job libfuzzer_execs_sec 88 --limit 1
[
    {
        "execs_sec": "191035",
        "machine_id": "b2dbe720-4fd8-4342-957a-6cb0979d2187",
        "timestamp": "2020-11-18T00:08:53.98Z",
        "worker_id": "0"
    }
]
```

Querying for the latest coverage for a job, by job_id fragment.
```
$ onefuzz debug job libfuzzer_coverage 88 --limit 1
[
    {
        "covered": "10",
        "features": "21",
        "rate": "0.47619047619047616",
        "timestamp": "2020-11-18T00:09:40.793Z"
    }
]
```
2020-11-19 15:14:37 +00:00
bb2b18a2b9 Fix MSVC Libfuzzer coverage reporting (#324)
This PR fixes two issues:
- First, in MSVC compiled binaries both the LLVM _and_ MSVC symbols are
present, but only the MSVC symbols have correct values. For example:

```
0:000> cdb: Reading initial command '.scriptload DumpCountersOld.js ; !dumpcounters "cov" ; q'
JavaScript script successfully loaded from 'DumpCountersOld.js'
[+] not disabling sympath
INFO: Seed: 58715679
INFO: Loaded 1 modules   (3968 inline 8-bit counters): 3968 [00007FF70DB4B000, 00007FF70DB4BF80), # XXX Note
xxx.exe: Running 1 inputs 1 time(s) each.
Running: inp
[+] processing xxx.exe
[+] using LLVM 10 symbols - 0x7ff70db72b00:0x7ff70db72b08 # XXX These are wrong
```

This means the order we search for the coverage symbols is important.

- Secondly, this enables support for MSVC 8bit counter coverage.

## Validation Steps Performed

Running any recent MSVC compiled libfuzzer target should fail to actually collect coverage, instead just returning the 8 null bytes described in the linked issue.
2020-11-19 02:47:33 +00:00
b2b4a06afa Address typing issues hidden by memoization.caching (#322) 2020-11-18 15:08:40 -05:00
bb6d083768 Enable unmanaged registrations and configuration by environment variables (#318) 2020-11-18 12:19:09 -05:00
e47e89609a Use Storage Account types, rather than account_id (#320)
We need to move to supporting data sharding.

One of the steps towards that is stop passing around `account_id`, rather we need to specify the type of storage we need.
2020-11-18 14:06:14 +00:00
52eca33237 Move more run-time actions to setup-time (#317)
This script moves more of the run-time actions to setup-time.  This is important for running fuzzing within docker containers, such that installing llvm & gdb is done as part of the container, rather than on each launch.
2020-11-18 10:13:01 +00:00
64bd389eb7 Declarative templates (#266) 2020-11-17 16:00:09 -05:00
ce3356d597 Add SDK Feature Flags (#313)
## Summary of the Pull Request

This enables feature flags for the SDK, which enables gating access to preview features to those that have specifically asked for them.  This is intended to be used within #266.

Note, this change also moves to using a `pydantic` model for the config, rather than hand-crafted JSON dicts.
2020-11-17 15:40:16 +00:00
c4f266ee00 fix webhook events doc link (#316) 2020-11-16 18:45:54 -05:00
41271c62e0 Release 1.8.0 (#315) 1.8.0 2020-11-16 17:56:50 -05:00
02600320b7 Don't reset webhooks by default (#307) 2020-11-16 16:51:32 -05:00
beea318968 Add User Info to created tasks (#303)
This PR makes user information from JWT tokens available as part of a Task.

Included changes:
* Renamed `verify_token` to `call_if_agent`, since this function is specific to agent token verification
* Renames `is_authorized` to `is_agent`, since this function checks if the token is an agent
* Adds support for unmanaged nodes in `is_agent` (see #133 for information) 
* Saves the user information from the JWT token on task create as part of `TaskConfig`

Note, `TaskConfig` is what is provided to notification templates.  This enables Github issues and ADO work items to tie back to the user that created the task.

Note, while `upn` _usually_ means email for AAD user tokens.  If we were going to make use of the email address, we should perform a graph lookup based on the `oid`, but we're not.
2020-11-13 11:50:52 +00:00
31f099d3d4 Event based webhooks (#296) 2020-11-12 17:44:42 -05:00
693c988854 add symbols for windows agents (#306) 2020-11-12 15:46:32 -05:00
484b5318d3 fix the labels used in issue templates (#302) 2020-11-12 14:00:06 -05:00
f700a06e49 Don't store the sccache directory as part of the Rust Prereqs (#301) 2020-11-12 13:58:51 -05:00
a0b5d10c81 Add target_workers to TaskUnitConfig (#305) 2020-11-12 13:22:53 -05:00
30ba30d609 Add --upgrade to the Azure Devops deployment pipeline (#304) 2020-11-12 11:34:31 -05:00
87085bc48a Change prepare_test_result to take reference (#297)
Originally used move to avoid copying, but the copy is cheap in the
normal case (no exceptions) and is more usable this way.
2020-11-11 21:48:08 -08:00
b6be6e30cb Release 1.7.0 (#299) 1.7.0 2020-11-11 21:40:32 -05:00
61b3f87dd4 prevent users from enabling public access to containers (#300) 2020-11-11 20:36:49 -05:00
c56f72b37c Make supervisor heartbeat only fire on main loop progress (#283) 2020-11-11 18:30:02 -05:00
ca209eb543 refactor agent_events handler (#261) 2020-11-11 18:28:16 -05:00
382003e4d0 Comment update to register app as owner (#262) 2020-11-11 17:58:30 -05:00
4e138b0bef cli debug command to ssh/rdp to a specific node (#298) 2020-11-11 17:14:02 -05:00
9e24754150 Windows debugger updates (#294)
* Set execution context while handling breakpoint

If we don't do this, the caller will see incorrect state such
as the wrong rip.

* Improve debugger logging messages
* Tiny tweak to breakpoint data type
* Suspend threads while single stepping over breakpoint
* Fix get_current_thread_id to return actual id
2020-11-11 12:09:18 -08:00
5c45d85359 Don't use HTTP 404 on agent refresh error (#287) 2020-11-11 12:33:27 -05:00