228 Commits

Author SHA1 Message Date
bmc-msft
2e276de0f5
Release 1.9.0 (#335) 1.9.0 2020-11-20 16:01:28 -05:00
bmc-msft
3ddb756504
Add linting to deployment tools (#332) 2020-11-20 13:00:19 -05:00
bmc-msft
9e2a61fe66
Add user_info to Jobs & Repro (#327)
This adds information about the user that created a job or repro VM to the respective resources.

This expands on the addition made to tasks in #303.
2020-11-20 15:46:52 +00:00
Cheick Keita
d96209c659
Include the body when receiving a registration error (#321)
* Include the body when receiving a registration error to help debug issued like #215 
* increase the agent registration timeout to 20 min
2020-11-20 14:43:40 +00:00
bmc-msft
7e3b807479
Support pre-release instance specific setup script paths (#331)
Support `instance-specific-setup/<OS>/setup` and `instance-specific-setup/setup` scripts.

Fixes #328
2020-11-20 12:42:58 +00:00
bmc-msft
3974d680ef
Support retry during function deploy (#330)
Starting earlier today, I saw roughly 1 in 3 deployments fail with the error `Azure.Functions.Cli.Common.CliException: Timed out waiting for SCM to update the Environment Settings`.  Redeploying the application resolves the issue.  New builds and past releases alike hit this exception.

According to https://github.com/Azure/azure-functions-core-tools/issues/1863, function app deployments may fail due to timeouts related to cold-start.

This PR executes the deploy in a loop with a delay in the case of failure.
2020-11-19 20:04:18 +00:00
bmc-msft
31a661f071
Expose coverage/exec_sec for libfuzzer targets via CLI (#325)
Adds debug subcommands to the SDK/CLI that simplify querying Application Insights for libfuzzer telemetry.  

Querying for the latest execs_sec for a job, by job_id fragment.
```
$ onefuzz debug job libfuzzer_execs_sec 88 --limit 1
[
    {
        "execs_sec": "191035",
        "machine_id": "b2dbe720-4fd8-4342-957a-6cb0979d2187",
        "timestamp": "2020-11-18T00:08:53.98Z",
        "worker_id": "0"
    }
]
```

Querying for the latest coverage for a job, by job_id fragment.
```
$ onefuzz debug job libfuzzer_coverage 88 --limit 1
[
    {
        "covered": "10",
        "features": "21",
        "rate": "0.47619047619047616",
        "timestamp": "2020-11-18T00:09:40.793Z"
    }
]
```
2020-11-19 15:14:37 +00:00
jopletchMSFT
bb2b18a2b9
Fix MSVC Libfuzzer coverage reporting (#324)
This PR fixes two issues:
- First, in MSVC compiled binaries both the LLVM _and_ MSVC symbols are
present, but only the MSVC symbols have correct values. For example:

```
0:000> cdb: Reading initial command '.scriptload DumpCountersOld.js ; !dumpcounters "cov" ; q'
JavaScript script successfully loaded from 'DumpCountersOld.js'
[+] not disabling sympath
INFO: Seed: 58715679
INFO: Loaded 1 modules   (3968 inline 8-bit counters): 3968 [00007FF70DB4B000, 00007FF70DB4BF80), # XXX Note
xxx.exe: Running 1 inputs 1 time(s) each.
Running: inp
[+] processing xxx.exe
[+] using LLVM 10 symbols - 0x7ff70db72b00:0x7ff70db72b08 # XXX These are wrong
```

This means the order we search for the coverage symbols is important.

- Secondly, this enables support for MSVC 8bit counter coverage.

## Validation Steps Performed

Running any recent MSVC compiled libfuzzer target should fail to actually collect coverage, instead just returning the 8 null bytes described in the linked issue.
2020-11-19 02:47:33 +00:00
bmc-msft
b2b4a06afa
Address typing issues hidden by memoization.caching (#322) 2020-11-18 15:08:40 -05:00
bmc-msft
bb6d083768
Enable unmanaged registrations and configuration by environment variables (#318) 2020-11-18 12:19:09 -05:00
bmc-msft
e47e89609a
Use Storage Account types, rather than account_id (#320)
We need to move to supporting data sharding.

One of the steps towards that is stop passing around `account_id`, rather we need to specify the type of storage we need.
2020-11-18 14:06:14 +00:00
bmc-msft
52eca33237
Move more run-time actions to setup-time (#317)
This script moves more of the run-time actions to setup-time.  This is important for running fuzzing within docker containers, such that installing llvm & gdb is done as part of the container, rather than on each launch.
2020-11-18 10:13:01 +00:00
bmc-msft
64bd389eb7
Declarative templates (#266) 2020-11-17 16:00:09 -05:00
bmc-msft
ce3356d597
Add SDK Feature Flags (#313)
## Summary of the Pull Request

This enables feature flags for the SDK, which enables gating access to preview features to those that have specifically asked for them.  This is intended to be used within #266.

Note, this change also moves to using a `pydantic` model for the config, rather than hand-crafted JSON dicts.
2020-11-17 15:40:16 +00:00
bmc-msft
c4f266ee00
fix webhook events doc link (#316) 2020-11-16 18:45:54 -05:00
bmc-msft
41271c62e0
Release 1.8.0 (#315) 1.8.0 2020-11-16 17:56:50 -05:00
bmc-msft
02600320b7
Don't reset webhooks by default (#307) 2020-11-16 16:51:32 -05:00
bmc-msft
beea318968
Add User Info to created tasks (#303)
This PR makes user information from JWT tokens available as part of a Task.

Included changes:
* Renamed `verify_token` to `call_if_agent`, since this function is specific to agent token verification
* Renames `is_authorized` to `is_agent`, since this function checks if the token is an agent
* Adds support for unmanaged nodes in `is_agent` (see #133 for information) 
* Saves the user information from the JWT token on task create as part of `TaskConfig`

Note, `TaskConfig` is what is provided to notification templates.  This enables Github issues and ADO work items to tie back to the user that created the task.

Note, while `upn` _usually_ means email for AAD user tokens.  If we were going to make use of the email address, we should perform a graph lookup based on the `oid`, but we're not.
2020-11-13 11:50:52 +00:00
bmc-msft
31f099d3d4
Event based webhooks (#296) 2020-11-12 17:44:42 -05:00
bmc-msft
693c988854
add symbols for windows agents (#306) 2020-11-12 15:46:32 -05:00
bmc-msft
484b5318d3
fix the labels used in issue templates (#302) 2020-11-12 14:00:06 -05:00
bmc-msft
f700a06e49
Don't store the sccache directory as part of the Rust Prereqs (#301) 2020-11-12 13:58:51 -05:00
Choongwoo Han
a0b5d10c81
Add target_workers to TaskUnitConfig (#305) 2020-11-12 13:22:53 -05:00
Anshuman Goel
30ba30d609
Add --upgrade to the Azure Devops deployment pipeline (#304) 2020-11-12 11:34:31 -05:00
Jason Shirk
87085bc48a
Change prepare_test_result to take reference (#297)
Originally used move to avoid copying, but the copy is cheap in the
normal case (no exceptions) and is more usable this way.
2020-11-11 21:48:08 -08:00
bmc-msft
b6be6e30cb
Release 1.7.0 (#299) 1.7.0 2020-11-11 21:40:32 -05:00
Anshuman Goel
61b3f87dd4
prevent users from enabling public access to containers (#300) 2020-11-11 20:36:49 -05:00
Cheick Keita
c56f72b37c
Make supervisor heartbeat only fire on main loop progress (#283) 2020-11-11 18:30:02 -05:00
bmc-msft
ca209eb543
refactor agent_events handler (#261) 2020-11-11 18:28:16 -05:00
Anshuman Goel
382003e4d0
Comment update to register app as owner (#262) 2020-11-11 17:58:30 -05:00
bmc-msft
4e138b0bef
cli debug command to ssh/rdp to a specific node (#298) 2020-11-11 17:14:02 -05:00
Jason Shirk
9e24754150
Windows debugger updates (#294)
* Set execution context while handling breakpoint

If we don't do this, the caller will see incorrect state such
as the wrong rip.

* Improve debugger logging messages
* Tiny tweak to breakpoint data type
* Suspend threads while single stepping over breakpoint
* Fix get_current_thread_id to return actual id
2020-11-11 12:09:18 -08:00
bmc-msft
5c45d85359
Don't use HTTP 404 on agent refresh error (#287) 2020-11-11 12:33:27 -05:00
bmc-msft
c70007d5df
include deleting unique_inputs on containers.reset (#290) 2020-11-11 12:33:03 -05:00
Anshuman Goel
dec1a2d7b0
removing nodes whose ground truth is not avail (#275) 2020-11-11 12:20:05 -05:00
bmc-msft
8678755b71
Render envelope.event as json (#288) 2020-11-11 12:18:32 -05:00
bmc-msft
7b34cf23f2
reduce calls to containers.list() (#289) 2020-11-11 12:18:14 -05:00
bmc-msft
3e4b26951e
pin urllib3 for deployment prereqs (#292) 2020-11-11 12:17:58 -05:00
bmc-msft
ba59230187
fix create_vmss log message (#293) 2020-11-11 12:17:42 -05:00
Anshuman Goel
3cade0800d
Autoscale test: I (#278) 2020-11-11 10:53:47 -05:00
Joe Ranweiler
439a58f8a2
Add uninstrumented block coverage recording (#280) 2020-11-11 07:20:38 -05:00
bmc-msft
e638908aac
Add application-insights debug cli (#281) 2020-11-11 06:17:43 -05:00
bmc-msft
daba3232bc
Use a shortened machine_id, similar to other UUIDs (#286) 2020-11-11 05:41:24 -05:00
bmc-msft
82806b1cf2
Keeps task/node association until the nodes are reimaged (#273) 2020-11-10 17:41:51 -05:00
Anshuman Goel
5797fa1f3a
adding ypy on tests (#279) 2020-11-10 11:09:18 -05:00
Cheick Keita
81a04ed81d
Make the role assignment step optional in the deployment (#271) 2020-11-10 09:11:34 -05:00
Anshuman Goel
4cafee9bbf
Sorting params for Task creation (#274) 2020-11-09 16:52:09 -05:00
Cheick Keita
bbee84ab1f
Storing the user assigned managed identity in the scaleset table (#255) 2020-11-05 18:36:59 -05:00
bmc-msft
b5578381ce
default TTL for queued messages to infinite (#259) 2020-11-04 15:41:05 -05:00
anslutsk
600b3efdf7
Address formatting in logging in registration.py (#258) 2020-11-03 16:16:18 -05:00