Commit Graph

1518 Commits

Author SHA1 Message Date
3f86fc8689 fix some bugs (#2406)
* - fix queries in timer retention

- do not discard proxy record after proxy state is processed, since that record needs to persist

* addressing comments

Co-authored-by: stas <statis@microsoft.com>
2022-09-15 15:45:39 -07:00
4f1ac523da Adding error message to catch Model HttpResponseError. (#2384)
* Adding error message to catch Model HttpResponseError.

* Changing error message.

* Formatting.
2022-09-15 14:44:23 -07:00
f22dee18df CodeQL needs explicit permissions to run (#2404) 2022-09-15 09:05:18 -04:00
4cc4de9c9e Codecov setup for C# & Rust code (#2400)
Use Codecov to show coverage reports, so we get highlighted versions of the files where it is easy to see missing coverage.

- Setup Rust coverage using [`cargo-llvm-cov`](https://github.com/taiki-e/cargo-llvm-cov).
- Add the `ci/agent.sh` build script to the agent artifact cache key, since it wasn't there before.
- Don't run Rust tests in `--release` mode (have been meaning to change this so doing it at the same time).

There is some subtlety about putting the coverage result into the cached agent artifact, so that when we reuse the agent artifact we can still upload the coverage information for it to Codecov. Without this it would look like the coverage had dropped.
2022-09-15 02:29:22 +00:00
61a797e224 Restore self-hosted configuration (#2394) 2022-09-14 23:59:41 +00:00
ac9d072e1d fix linux proxy extensions provisioning failures (#2401)
* fix linux proxy extensions provisioning failures

* format

Co-authored-by: stas <statis@microsoft.com>
2022-09-14 23:52:35 +00:00
2fe73ab79c bug fix (#2392)
* bug fix

* rename anyNotStoppedJobs  to anyNotStoppedTasks

Co-authored-by: stas <statis@microsoft.com>
2022-09-14 10:07:15 -07:00
ca7b6be43b Refactor notification support (#2363)
* Add teams notifications

* .

* Fix compilation isues

* Checkpoint

* Added Ado

* Fix some TODOs

* Teams messages work! 🎉

* fmt

* Bug fix container url generator

* Some small ado changes

* 🧹

* PR comments

* Fix packages

* Get more detailed restore information to debug errors

* Maybe fixes this issue?

* Undo CI change
2022-09-14 15:07:52 +00:00
f375ee719e Two fixes to C# scheduling (#2390)
Two fixes to scheduling code:

- `GetPool` was not correct for the VM case (this code is possibly legacy and not used any more)
- `BuildWorkUnit` could fetch the same pool multiple times and then fail due to `BucketConfig` mismatch (on `TimeStamp`)
  - add a cache to the loop so that we only fetch each pool once
2022-09-14 02:06:01 +00:00
3b8cbc3f1e Cancel any previous PR builds when new one starts (#2393) 2022-09-14 01:34:42 +00:00
bb81f2ec51 Fix MarkDependantsFailed (#2389) 2022-09-14 01:21:13 +00:00
2ff758464e Use Github-hosted Ubuntu until issue with self-hosted pool is resolved (#2391) 2022-09-14 12:39:18 +12:00
f7f91df622 CSharp Refactor - Instance Config Endpoint (#2347)
* CSharp Refactor - Instance Config Endpoint

* Finshing config update.

* Formatting.

* Formatting.

* formatting.

* Fixing encoding.

* Fixing config references.

* Fixing refs.

* Trying location.

* Trying ref to location.

* Passing nsg.

* Passing nsg.

* Setting nsg to not null.

* Fixing ok reference.

* Adding Instance Config Response.

* Setting required attribute.

* Adding route specifier.

* Formatting.

* Fixing route.

* Fixing optionals.

* Trying to set default

* Trying again.

* Setting require admins

* Removing optioanl.

* Testing with instancename.

* Updating instanceconfig model.

* Updating instance config response.

* Formatting.

* Removing AllowPoolManagement.

* Readding.

* Removing arg.

* Replacing with RequireAdminPrivs.

* Fix orm test.

* Setting requireadminprivs to true.

* Requiring admin privs.

* Fix formatting.

* fix test.

* Fixing.

* Changing error message.

* Changing.

* Reordering test args.

* Flipping.

* Fixing args.

* Fixing again.

* Removing false.

* Removing from constructor.

* Setting.

* Setting string to optional.

* Formatting.

* Adding default value.

* PUshing changes to OrmModelsTest

* Updating test to not pass null.

* George's suggestions.

* Removing entityconverter changes.

* Fixing import.
2022-09-13 08:55:40 -07:00
67e55910ac Fix TaskOperations.SearchStates (#2383) 2022-09-13 00:31:35 +00:00
bc33ae1d7a Fix Scaleset response Auth inclusion (#2382)
The `Auth` property is not meant to be returned upon Create/`POST`. Fix this, and make it easier to specify when `Auth` should be included or not.
2022-09-12 17:19:10 -07:00
ecf858bbdd Updating error and fixing default value for uto_create_cli_app (#2378) 2022-09-12 16:28:41 -07:00
ce1fc773a9 implement not-implemented: GetInputContainerQueues (#2380)
* implement not-implemented: GetInputContainerQueues

* named tuple

Co-authored-by: stas <statis@microsoft.com>
2022-09-12 15:17:21 -07:00
44f74f622a DoNotRunExtensionsOnOverprovisionedVms must be false if Overprovision is false (#2375) 2022-09-12 22:00:47 +00:00
9eaf92ea35 tests and bug fixes (#2379)
Co-authored-by: stas <statis@microsoft.com>
2022-09-12 14:45:45 -07:00
9f03a174ec Use self-hosted runner for Rust agent builds (#2366)
Use a 1ES hosted pool with more powerful machines to do the build for Rust agents.
2022-09-13 09:11:24 +12:00
64bfc0e9e9 Validate changes to devcontainer files (#2373) 2022-09-12 20:56:23 +00:00
615dff6a64 Update teams.md link (#2370) 2022-09-12 12:20:10 +00:00
2a2c07ed35 Extend use of validated string types (#2357)
In the Python code there were more validated string types that haven't been properly ported for use in the C# code (such as Region). Add that type back in and improve some others:

- Use `Region` type to represent regions (implicitly convertible to/from the `AzureLocation` SDK type)
- Improve validation of `Container` type to match Azure specs and use it in more places
- Restore/fix validation of `PoolName` type which was previously removed (#2080) due to being too strict: now allows 1-64 ASCII alphanumeric/hyphen/dash
  - We want to restrict pool names so that we can use them as disambiguating prefixes for scaleset names (see #2189). Note that underscore is not actually permitted in scaleset names so we will probably end up mapping it to hyphen.

Note that once C#7 lands we will be able to simplify the usage of `ValidatedString` a lot (using static abstract methods).

----

Open questions:

For deserializing from "known-good" places such as table storage or from Azure SDK APIs, should we have an `T.UnsafeAssumeValid(string input)` method which does no validation, to protect us from breakage?
2022-09-12 10:06:51 +12:00
140638f684 Release-5.13.0 (#2371)
* Release-5.13.0
5.13.0
2022-09-09 15:32:54 -07:00
16916c8653 Devcontainer: ensure venv is installed (#2372) 2022-09-09 22:07:26 +00:00
fa667e72e2 Bump url from 2.2.2 to 2.3.0 in /src/proxy-manager (#2360)
Bumps [url](https://github.com/servo/rust-url) from 2.2.2 to 2.3.0.
- [Release notes](https://github.com/servo/rust-url/releases)
- [Commits](https://github.com/servo/rust-url/compare/v2.2.2...v2.3.0)

---
updated-dependencies:
- dependency-name: url
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-09-09 01:37:21 +00:00
343dda3073 Bump url from 2.2.2 to 2.3.0 in /src/agent (#2361)
Bumps [url](https://github.com/servo/rust-url) from 2.2.2 to 2.3.0.
- [Release notes](https://github.com/servo/rust-url/releases)
- [Commits](https://github.com/servo/rust-url/compare/v2.2.2...v2.3.0)

---
updated-dependencies:
- dependency-name: url
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-09-09 01:04:00 +00:00
ef7b557599 Add debug command to download a task’s containers (#2359) 2022-09-09 00:45:15 +00:00
511eafca73 fix bug in ado notification info (#2368)
Co-authored-by: stas <statis@microsoft.com>
2022-09-08 23:43:06 +00:00
6dfa46e62f remove required field to match python behavior (#2367)
Co-authored-by: stas <statis@microsoft.com>
2022-09-08 23:10:12 +00:00
e1420e1a1d Move global tool installs into another script so they can be cached (#2365) 2022-09-08 22:31:52 +00:00
02298f2bf2 Updating all azure-cli and azure-cli-core versions to 2.40 (#2364)
* Updating all azure-cli and azure-cli-core versions to 2.40

* Updating azure-identity to 1.10.0 required for azure-cli msal-extensions dependency
2022-09-08 13:42:48 -07:00
c562bbb0ad VMSS creation fixes (#2358) 2022-09-09 07:53:02 +12:00
d61f66a667 Fixing memory leaks (#2356)
* fix a memory leak

* fix another memory leak

* remove extra queries

* formatting

* do not throw when scale set cannot be updated

Co-authored-by: stas <statis@microsoft.com>
2022-09-07 17:03:38 -07:00
df4edbea98 Coverage task should have access to readonly_inputs (#2352)
Fixes #2348.

Code duplicated & modified from line 112.

Both the generic and dotnet coverage tasks [iterate over all `readonly_inputs` that they are supplied](bd4dfdc592/src/agent/onefuzz-task/src/tasks/coverage/generic.rs (L98-L115)), so this should be sufficient.
2022-09-07 22:15:30 +00:00
44089d32f0 fix broken deploy (#2355)
Co-authored-by: stas <statis@microsoft.com>
2022-09-07 14:49:07 -07:00
02869eba9e Fix #1285 (#2353) 2022-09-08 08:36:38 +12:00
fe51e011d1 Use Graph SDK instead of manually constructing queries (#2324) 2022-09-07 01:12:36 +00:00
bd4dfdc592 Update chrono, remove time to fix audit failures (#2285)
Closes #1366 and incidentally closes #1266.

Requires using a not-yet-merged modification to the latest version of `appinsights-rs` to remove the `time` feature from the `chrono` dependency (https://github.com/dmolokanov/appinsights-rs/pull/280). 

There are changes to use the new (tokio-based) version of `appinsights-rs`, e.g., made `set_appinsights_clients` async to ensure it is always called from an async context, since the constructor for appinsights now invokes `Tokio::spawn`.
2022-09-07 00:34:31 +00:00
0a84b1466d Implement not implemented bits (#2341)
`ScalesetOperations` was missing:

* `ReimageNodes`
* `DeleteNodes`
2022-09-06 23:43:14 +00:00
c9d484778d Use 64-bit worker for function (#2349) 2022-09-06 23:30:35 +00:00
b28008b519 Check if state transitions exist at test time (#2241)
Don't ignore missing state transitions in `StatefulOrm` static constructor test, instead error out. 

`Scaleset` was missing the `Resize` transition.
2022-09-06 22:52:31 +00:00
84b2cc9992 Rework use of storage clients (#2302)
- Reuse Storage clients by caching them against account name.
  - This follows recommendations of Storage documentation: https://devblogs.microsoft.com/azure-sdk/lifetime-management-and-thread-safety-guarantees-of-azure-sdk-net-clients/
- Use strongly-typed `ResourceIdentifier` when passing around Account IDs, to prevent mixing them up with Account Names. 
- Simplify & centralize SAS generation in one place.
2022-09-06 22:11:10 +00:00
318e568cab add flag to enable memory and cpu profilers (#2345)
Co-authored-by: stas <statis@microsoft.com>
2022-09-06 10:32:30 -07:00
c175b4676a enable running dot-net function on Windows to allow attaching remote debugger (#2344)
* enable running dot-net function on Windows to allow attaching remote debugger

* rename from 'use_windows' to 'host_dotnet_on_windows'

* instructions

* reformat deploy.py

Co-authored-by: stas <statis@microsoft.com>
2022-09-06 09:34:22 -07:00
dd3aa1ce0a Bug fixes to enable timer_workers running properly (#2343)
Co-authored-by: stas <statis@microsoft.com>
2022-09-06 09:13:33 -07:00
e7abd2d8fa Bump pretty_assertions from 1.0.0 to 1.3.0 in /src/agent (#2322) 2022-09-06 14:34:43 +12:00
ee1ad0abd8 Functional tests (#2333)
* more tests

* revert compute, since "released" version has bug when creating scaleset extensions

* format

* found bug, i think...

* now should be fixed

Co-authored-by: stas <statis@microsoft.com>
2022-09-02 08:16:48 -07:00
c54db04083 Enforce that there no extra properties in request JSON, and that non-null properties are [Required] (#2328)
Closes #2314 via two fixes, one for additional properties and one for missing properties:

- Make all request types inherit from `BaseRequest` which has an `ExtensionData` property, and ensure that it is empty in `ParseRequest`.
- Add `[Required]` attribute to non-nullable properties that do not have defaults, and add a test that ensures we have this attribute where necessary.
2022-09-02 01:59:24 +00:00
52ba57bf0d Remove some response-only properties from the Task model (#2335) 2022-09-01 23:33:04 +00:00