Commit Graph

1082 Commits

Author SHA1 Message Date
79cc82098a Move integration test artifacts into primary source tree (#336) 2020-11-24 08:03:01 -05:00
905dc7c0d6 Re-enable the retry logic for App Password creation (#338) 2020-11-24 08:00:31 -05:00
d47124fe8c Fix state management in the scheduler (#337) 2020-11-24 12:43:51 +00:00
32ba86be9d Update current_thread_id when setting current thread (#340) 2020-11-23 13:39:03 -08:00
2e276de0f5 Release 1.9.0 (#335) 1.9.0 2020-11-20 16:01:28 -05:00
3ddb756504 Add linting to deployment tools (#332) 2020-11-20 13:00:19 -05:00
9e2a61fe66 Add user_info to Jobs & Repro (#327)
This adds information about the user that created a job or repro VM to the respective resources.

This expands on the addition made to tasks in #303.
2020-11-20 15:46:52 +00:00
d96209c659 Include the body when receiving a registration error (#321)
* Include the body when receiving a registration error to help debug issued like #215 
* increase the agent registration timeout to 20 min
2020-11-20 14:43:40 +00:00
7e3b807479 Support pre-release instance specific setup script paths (#331)
Support `instance-specific-setup/<OS>/setup` and `instance-specific-setup/setup` scripts.

Fixes #328
2020-11-20 12:42:58 +00:00
3974d680ef Support retry during function deploy (#330)
Starting earlier today, I saw roughly 1 in 3 deployments fail with the error `Azure.Functions.Cli.Common.CliException: Timed out waiting for SCM to update the Environment Settings`.  Redeploying the application resolves the issue.  New builds and past releases alike hit this exception.

According to https://github.com/Azure/azure-functions-core-tools/issues/1863, function app deployments may fail due to timeouts related to cold-start.

This PR executes the deploy in a loop with a delay in the case of failure.
2020-11-19 20:04:18 +00:00
31a661f071 Expose coverage/exec_sec for libfuzzer targets via CLI (#325)
Adds debug subcommands to the SDK/CLI that simplify querying Application Insights for libfuzzer telemetry.  

Querying for the latest execs_sec for a job, by job_id fragment.
```
$ onefuzz debug job libfuzzer_execs_sec 88 --limit 1
[
    {
        "execs_sec": "191035",
        "machine_id": "b2dbe720-4fd8-4342-957a-6cb0979d2187",
        "timestamp": "2020-11-18T00:08:53.98Z",
        "worker_id": "0"
    }
]
```

Querying for the latest coverage for a job, by job_id fragment.
```
$ onefuzz debug job libfuzzer_coverage 88 --limit 1
[
    {
        "covered": "10",
        "features": "21",
        "rate": "0.47619047619047616",
        "timestamp": "2020-11-18T00:09:40.793Z"
    }
]
```
2020-11-19 15:14:37 +00:00
bb2b18a2b9 Fix MSVC Libfuzzer coverage reporting (#324)
This PR fixes two issues:
- First, in MSVC compiled binaries both the LLVM _and_ MSVC symbols are
present, but only the MSVC symbols have correct values. For example:

```
0:000> cdb: Reading initial command '.scriptload DumpCountersOld.js ; !dumpcounters "cov" ; q'
JavaScript script successfully loaded from 'DumpCountersOld.js'
[+] not disabling sympath
INFO: Seed: 58715679
INFO: Loaded 1 modules   (3968 inline 8-bit counters): 3968 [00007FF70DB4B000, 00007FF70DB4BF80), # XXX Note
xxx.exe: Running 1 inputs 1 time(s) each.
Running: inp
[+] processing xxx.exe
[+] using LLVM 10 symbols - 0x7ff70db72b00:0x7ff70db72b08 # XXX These are wrong
```

This means the order we search for the coverage symbols is important.

- Secondly, this enables support for MSVC 8bit counter coverage.

## Validation Steps Performed

Running any recent MSVC compiled libfuzzer target should fail to actually collect coverage, instead just returning the 8 null bytes described in the linked issue.
2020-11-19 02:47:33 +00:00
b2b4a06afa Address typing issues hidden by memoization.caching (#322) 2020-11-18 15:08:40 -05:00
bb6d083768 Enable unmanaged registrations and configuration by environment variables (#318) 2020-11-18 12:19:09 -05:00
e47e89609a Use Storage Account types, rather than account_id (#320)
We need to move to supporting data sharding.

One of the steps towards that is stop passing around `account_id`, rather we need to specify the type of storage we need.
2020-11-18 14:06:14 +00:00
52eca33237 Move more run-time actions to setup-time (#317)
This script moves more of the run-time actions to setup-time.  This is important for running fuzzing within docker containers, such that installing llvm & gdb is done as part of the container, rather than on each launch.
2020-11-18 10:13:01 +00:00
64bd389eb7 Declarative templates (#266) 2020-11-17 16:00:09 -05:00
ce3356d597 Add SDK Feature Flags (#313)
## Summary of the Pull Request

This enables feature flags for the SDK, which enables gating access to preview features to those that have specifically asked for them.  This is intended to be used within #266.

Note, this change also moves to using a `pydantic` model for the config, rather than hand-crafted JSON dicts.
2020-11-17 15:40:16 +00:00
c4f266ee00 fix webhook events doc link (#316) 2020-11-16 18:45:54 -05:00
41271c62e0 Release 1.8.0 (#315) 1.8.0 2020-11-16 17:56:50 -05:00
02600320b7 Don't reset webhooks by default (#307) 2020-11-16 16:51:32 -05:00
beea318968 Add User Info to created tasks (#303)
This PR makes user information from JWT tokens available as part of a Task.

Included changes:
* Renamed `verify_token` to `call_if_agent`, since this function is specific to agent token verification
* Renames `is_authorized` to `is_agent`, since this function checks if the token is an agent
* Adds support for unmanaged nodes in `is_agent` (see #133 for information) 
* Saves the user information from the JWT token on task create as part of `TaskConfig`

Note, `TaskConfig` is what is provided to notification templates.  This enables Github issues and ADO work items to tie back to the user that created the task.

Note, while `upn` _usually_ means email for AAD user tokens.  If we were going to make use of the email address, we should perform a graph lookup based on the `oid`, but we're not.
2020-11-13 11:50:52 +00:00
31f099d3d4 Event based webhooks (#296) 2020-11-12 17:44:42 -05:00
693c988854 add symbols for windows agents (#306) 2020-11-12 15:46:32 -05:00
484b5318d3 fix the labels used in issue templates (#302) 2020-11-12 14:00:06 -05:00
f700a06e49 Don't store the sccache directory as part of the Rust Prereqs (#301) 2020-11-12 13:58:51 -05:00
a0b5d10c81 Add target_workers to TaskUnitConfig (#305) 2020-11-12 13:22:53 -05:00
30ba30d609 Add --upgrade to the Azure Devops deployment pipeline (#304) 2020-11-12 11:34:31 -05:00
87085bc48a Change prepare_test_result to take reference (#297)
Originally used move to avoid copying, but the copy is cheap in the
normal case (no exceptions) and is more usable this way.
2020-11-11 21:48:08 -08:00
b6be6e30cb Release 1.7.0 (#299) 1.7.0 2020-11-11 21:40:32 -05:00
61b3f87dd4 prevent users from enabling public access to containers (#300) 2020-11-11 20:36:49 -05:00
c56f72b37c Make supervisor heartbeat only fire on main loop progress (#283) 2020-11-11 18:30:02 -05:00
ca209eb543 refactor agent_events handler (#261) 2020-11-11 18:28:16 -05:00
382003e4d0 Comment update to register app as owner (#262) 2020-11-11 17:58:30 -05:00
4e138b0bef cli debug command to ssh/rdp to a specific node (#298) 2020-11-11 17:14:02 -05:00
9e24754150 Windows debugger updates (#294)
* Set execution context while handling breakpoint

If we don't do this, the caller will see incorrect state such
as the wrong rip.

* Improve debugger logging messages
* Tiny tweak to breakpoint data type
* Suspend threads while single stepping over breakpoint
* Fix get_current_thread_id to return actual id
2020-11-11 12:09:18 -08:00
5c45d85359 Don't use HTTP 404 on agent refresh error (#287) 2020-11-11 12:33:27 -05:00
c70007d5df include deleting unique_inputs on containers.reset (#290) 2020-11-11 12:33:03 -05:00
dec1a2d7b0 removing nodes whose ground truth is not avail (#275) 2020-11-11 12:20:05 -05:00
8678755b71 Render envelope.event as json (#288) 2020-11-11 12:18:32 -05:00
7b34cf23f2 reduce calls to containers.list() (#289) 2020-11-11 12:18:14 -05:00
3e4b26951e pin urllib3 for deployment prereqs (#292) 2020-11-11 12:17:58 -05:00
ba59230187 fix create_vmss log message (#293) 2020-11-11 12:17:42 -05:00
3cade0800d Autoscale test: I (#278) 2020-11-11 10:53:47 -05:00
439a58f8a2 Add uninstrumented block coverage recording (#280) 2020-11-11 07:20:38 -05:00
e638908aac Add application-insights debug cli (#281) 2020-11-11 06:17:43 -05:00
daba3232bc Use a shortened machine_id, similar to other UUIDs (#286) 2020-11-11 05:41:24 -05:00
82806b1cf2 Keeps task/node association until the nodes are reimaged (#273) 2020-11-10 17:41:51 -05:00
5797fa1f3a adding ypy on tests (#279) 2020-11-10 11:09:18 -05:00
81a04ed81d Make the role assignment step optional in the deployment (#271) 2020-11-10 09:11:34 -05:00