Commit Graph

2050 Commits

Author SHA1 Message Date
9eaf92ea35 tests and bug fixes (#2379)
Co-authored-by: stas <statis@microsoft.com>
2022-09-12 14:45:45 -07:00
9f03a174ec Use self-hosted runner for Rust agent builds (#2366)
Use a 1ES hosted pool with more powerful machines to do the build for Rust agents.
2022-09-13 09:11:24 +12:00
64bfc0e9e9 Validate changes to devcontainer files (#2373) 2022-09-12 20:56:23 +00:00
615dff6a64 Update teams.md link (#2370) 2022-09-12 12:20:10 +00:00
2a2c07ed35 Extend use of validated string types (#2357)
In the Python code there were more validated string types that haven't been properly ported for use in the C# code (such as Region). Add that type back in and improve some others:

- Use `Region` type to represent regions (implicitly convertible to/from the `AzureLocation` SDK type)
- Improve validation of `Container` type to match Azure specs and use it in more places
- Restore/fix validation of `PoolName` type which was previously removed (#2080) due to being too strict: now allows 1-64 ASCII alphanumeric/hyphen/dash
  - We want to restrict pool names so that we can use them as disambiguating prefixes for scaleset names (see #2189). Note that underscore is not actually permitted in scaleset names so we will probably end up mapping it to hyphen.

Note that once C#7 lands we will be able to simplify the usage of `ValidatedString` a lot (using static abstract methods).

----

Open questions:

For deserializing from "known-good" places such as table storage or from Azure SDK APIs, should we have an `T.UnsafeAssumeValid(string input)` method which does no validation, to protect us from breakage?
2022-09-12 10:06:51 +12:00
140638f684 Release-5.13.0 (#2371)
* Release-5.13.0
5.13.0
2022-09-09 15:32:54 -07:00
16916c8653 Devcontainer: ensure venv is installed (#2372) 2022-09-09 22:07:26 +00:00
fa667e72e2 Bump url from 2.2.2 to 2.3.0 in /src/proxy-manager (#2360)
Bumps [url](https://github.com/servo/rust-url) from 2.2.2 to 2.3.0.
- [Release notes](https://github.com/servo/rust-url/releases)
- [Commits](https://github.com/servo/rust-url/compare/v2.2.2...v2.3.0)

---
updated-dependencies:
- dependency-name: url
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-09-09 01:37:21 +00:00
343dda3073 Bump url from 2.2.2 to 2.3.0 in /src/agent (#2361)
Bumps [url](https://github.com/servo/rust-url) from 2.2.2 to 2.3.0.
- [Release notes](https://github.com/servo/rust-url/releases)
- [Commits](https://github.com/servo/rust-url/compare/v2.2.2...v2.3.0)

---
updated-dependencies:
- dependency-name: url
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-09-09 01:04:00 +00:00
ef7b557599 Add debug command to download a task’s containers (#2359) 2022-09-09 00:45:15 +00:00
511eafca73 fix bug in ado notification info (#2368)
Co-authored-by: stas <statis@microsoft.com>
2022-09-08 23:43:06 +00:00
6dfa46e62f remove required field to match python behavior (#2367)
Co-authored-by: stas <statis@microsoft.com>
2022-09-08 23:10:12 +00:00
e1420e1a1d Move global tool installs into another script so they can be cached (#2365) 2022-09-08 22:31:52 +00:00
02298f2bf2 Updating all azure-cli and azure-cli-core versions to 2.40 (#2364)
* Updating all azure-cli and azure-cli-core versions to 2.40

* Updating azure-identity to 1.10.0 required for azure-cli msal-extensions dependency
2022-09-08 13:42:48 -07:00
c562bbb0ad VMSS creation fixes (#2358) 2022-09-09 07:53:02 +12:00
d61f66a667 Fixing memory leaks (#2356)
* fix a memory leak

* fix another memory leak

* remove extra queries

* formatting

* do not throw when scale set cannot be updated

Co-authored-by: stas <statis@microsoft.com>
2022-09-07 17:03:38 -07:00
df4edbea98 Coverage task should have access to readonly_inputs (#2352)
Fixes #2348.

Code duplicated & modified from line 112.

Both the generic and dotnet coverage tasks [iterate over all `readonly_inputs` that they are supplied](bd4dfdc592/src/agent/onefuzz-task/src/tasks/coverage/generic.rs (L98-L115)), so this should be sufficient.
2022-09-07 22:15:30 +00:00
44089d32f0 fix broken deploy (#2355)
Co-authored-by: stas <statis@microsoft.com>
2022-09-07 14:49:07 -07:00
02869eba9e Fix #1285 (#2353) 2022-09-08 08:36:38 +12:00
fe51e011d1 Use Graph SDK instead of manually constructing queries (#2324) 2022-09-07 01:12:36 +00:00
bd4dfdc592 Update chrono, remove time to fix audit failures (#2285)
Closes #1366 and incidentally closes #1266.

Requires using a not-yet-merged modification to the latest version of `appinsights-rs` to remove the `time` feature from the `chrono` dependency (https://github.com/dmolokanov/appinsights-rs/pull/280). 

There are changes to use the new (tokio-based) version of `appinsights-rs`, e.g., made `set_appinsights_clients` async to ensure it is always called from an async context, since the constructor for appinsights now invokes `Tokio::spawn`.
2022-09-07 00:34:31 +00:00
0a84b1466d Implement not implemented bits (#2341)
`ScalesetOperations` was missing:

* `ReimageNodes`
* `DeleteNodes`
2022-09-06 23:43:14 +00:00
c9d484778d Use 64-bit worker for function (#2349) 2022-09-06 23:30:35 +00:00
b28008b519 Check if state transitions exist at test time (#2241)
Don't ignore missing state transitions in `StatefulOrm` static constructor test, instead error out. 

`Scaleset` was missing the `Resize` transition.
2022-09-06 22:52:31 +00:00
84b2cc9992 Rework use of storage clients (#2302)
- Reuse Storage clients by caching them against account name.
  - This follows recommendations of Storage documentation: https://devblogs.microsoft.com/azure-sdk/lifetime-management-and-thread-safety-guarantees-of-azure-sdk-net-clients/
- Use strongly-typed `ResourceIdentifier` when passing around Account IDs, to prevent mixing them up with Account Names. 
- Simplify & centralize SAS generation in one place.
2022-09-06 22:11:10 +00:00
318e568cab add flag to enable memory and cpu profilers (#2345)
Co-authored-by: stas <statis@microsoft.com>
2022-09-06 10:32:30 -07:00
c175b4676a enable running dot-net function on Windows to allow attaching remote debugger (#2344)
* enable running dot-net function on Windows to allow attaching remote debugger

* rename from 'use_windows' to 'host_dotnet_on_windows'

* instructions

* reformat deploy.py

Co-authored-by: stas <statis@microsoft.com>
2022-09-06 09:34:22 -07:00
dd3aa1ce0a Bug fixes to enable timer_workers running properly (#2343)
Co-authored-by: stas <statis@microsoft.com>
2022-09-06 09:13:33 -07:00
e7abd2d8fa Bump pretty_assertions from 1.0.0 to 1.3.0 in /src/agent (#2322) 2022-09-06 14:34:43 +12:00
ee1ad0abd8 Functional tests (#2333)
* more tests

* revert compute, since "released" version has bug when creating scaleset extensions

* format

* found bug, i think...

* now should be fixed

Co-authored-by: stas <statis@microsoft.com>
2022-09-02 08:16:48 -07:00
c54db04083 Enforce that there no extra properties in request JSON, and that non-null properties are [Required] (#2328)
Closes #2314 via two fixes, one for additional properties and one for missing properties:

- Make all request types inherit from `BaseRequest` which has an `ExtensionData` property, and ensure that it is empty in `ParseRequest`.
- Add `[Required]` attribute to non-nullable properties that do not have defaults, and add a test that ensures we have this attribute where necessary.
2022-09-02 01:59:24 +00:00
52ba57bf0d Remove some response-only properties from the Task model (#2335) 2022-09-01 23:33:04 +00:00
440f19baeb Remove dead libfuzzer_coverage scripts (#2288) 2022-09-01 14:49:50 -07:00
ecc099203b Add Cobertura XML output to src-cov example binary (#2334) 2022-09-01 12:10:30 -07:00
b737dcd98a Remove the max_elapsed_time limit from az_copy (#2332)
* Remove the max_elapsed_time  limit from az_copy
The code still relies on the number of retry to limit the requests

* add MAX_FAILURE_COUNT and MAX_RETRY_COUNT
2022-09-01 00:51:59 +00:00
f7e37355d0 Remove IDisposable from Creds (#2327) 2022-08-31 22:51:06 +00:00
366b671942 Cache InstanceConfig (#2329) 2022-08-31 22:30:38 +00:00
d711172ed7 Add scriban dependency and some template tests (#2330)
* Add scriban dependency and some template tests

* Resolve

* Missed force-evaluate

* fmt
2022-08-31 13:21:22 -04:00
0c88116a0d Added deploy.py flags onefuzz_app_id and auto_create_cli_app to allow custom app registrations. (#2305)
* Update Deployment Params to use custom CLI

* Adding failure mode if app id not found.

* Formatting.

* Fixing path.

* Cleaning up condition path.

* Update param name.

* Adding identiy assignment.

* Adding correct identity.

* FIxing app role assignment.
2022-08-30 22:05:44 -07:00
8b31dfe4ef Update deploy to set all settings at once (#2325) 2022-08-30 23:34:35 +00:00
8d23df328d bump nuget packages (#2321)
Co-authored-by: stas <statis@microsoft.com>
2022-08-30 13:58:07 -07:00
ea4d589abf Functional tests (#2317)
* more functional tests

* more tests

* bug fixes

* OneFuzz results do not map one to one with datastructures. So add support for multiple datastructures to be returned per API

* more tests

* format fixes

Co-authored-by: stas <statis@microsoft.com>
2022-08-30 12:26:06 -07:00
9ecc383981 trying to fix python break (#2319)
Co-authored-by: stas <statis@microsoft.com>
2022-08-30 11:47:45 -07:00
63c54193a9 Enable AppInsights dependency tracking (#2315) 2022-08-30 16:35:35 +12:00
ef434db201 Fix ServiceConfiguration.OneFuzzVersion (#2316) 2022-08-30 00:08:56 +00:00
5a8aa33af1 functional tests and bug fixes (#2301)
Co-authored-by: stas <statis@microsoft.com>
2022-08-29 09:33:15 -07:00
ae6df1e22f Create tables on startup (#2309) 2022-08-29 15:12:19 +12:00
4e2d2d2879 Release 5.12.0 (#2304)
* Release 5.12.0

* Removing dockercontainer changes.

* Update CHANGELOG.md

Co-authored-by: Marc Greisen <mgreisen@microsoft.com>

* Update CHANGELOG.md

Co-authored-by: Marc Greisen <mgreisen@microsoft.com>

* Adding az_copy logging.

* Adding ado logging.

Co-authored-by: Marc Greisen <mgreisen@microsoft.com>
5.12.0
2022-08-26 17:44:35 -07:00
163e324227 Specifying initial size for scaleset create in integration-tests. (#2307)
* Specifying initial size for scaleset create in integration-tests.

* Spell check.
2022-08-26 16:09:36 -07:00
c94ae0bbb6 log the work item id when notifying ado (#2291)
* log the work item id when notifying ado

* format

* refactoring

* fix build

* fix

* format
2022-08-26 19:08:31 +00:00