1882 Commits

Author SHA1 Message Date
Stas
b14bade0fc
cleanup queues for non-existent pools and non-existent tasks (#2433)
* cleanup queues for non-existent pools and non-existent tasks

* extra logs

Co-authored-by: stas <statis@microsoft.com>
2022-09-22 08:15:57 -07:00
Stas
de766dfa78
Delete pool queue when pool is deleted (#2431)
* Delete pool queue when pool is deleted

* Also delete shrink queue when pool is deleted

Co-authored-by: stas <statis@microsoft.com>
2022-09-21 14:39:30 -07:00
Noah McGregor Harper
2f42cd74d4
Adding State Transition Error Logging. (#2425)
* Adding State Transition Logging.

* Fix log call.

* Removing logging statement.

* Fix error! call.

* Adding error to error message.

* Make err var borrowed.

* Formatting.
2022-09-21 12:54:28 -07:00
Stas
a6addbf83a
Minor fixes (#2420)
* delete all temp files

* add try/catch and some logging when deleting temp files

Co-authored-by: stas <statis@microsoft.com>
2022-09-20 10:20:29 -07:00
Stas
f40a69a37f
fix linux repro extensions (#2415)
* fix OMS Linux repro extension config

* Fixing lost Node state updates

* fix bug in ReproVmss

* rewrite ssh auth

* win azure function ssh-keygen fix

* more logs

* try -P

* use empty string for password

* use argument list

* addressing comments

Co-authored-by: stas <statis@microsoft.com>
Co-authored-by: George Pollard <gpollard@microsoft.com>
2022-09-20 08:15:26 -07:00
Stas
b647e4a1fe
mark tasks as failed if a work unit cannot be created for the task (#2409)
* mark tasks as failed if a work unit cannot be created for the task

* fix up time queries

* query improvements

Co-authored-by: stas <statis@microsoft.com>
2022-09-17 12:50:41 -07:00
Stas
867cdbc06f
Port SyncAutoscaleSettings from Python to C# (#2407)
* Port SyncAutoscaleSettings from Python to C#

* address comment

Co-authored-by: stas <statis@microsoft.com>
2022-09-16 07:58:37 -07:00
Stas
3f86fc8689
fix some bugs (#2406)
* - fix queries in timer retention

- do not discard proxy record after proxy state is processed, since that record needs to persist

* addressing comments

Co-authored-by: stas <statis@microsoft.com>
2022-09-15 15:45:39 -07:00
Noah McGregor Harper
4f1ac523da
Adding error message to catch Model HttpResponseError. (#2384)
* Adding error message to catch Model HttpResponseError.

* Changing error message.

* Formatting.
2022-09-15 14:44:23 -07:00
George Pollard
4cc4de9c9e
Codecov setup for C# & Rust code (#2400)
Use Codecov to show coverage reports, so we get highlighted versions of the files where it is easy to see missing coverage.

- Setup Rust coverage using [`cargo-llvm-cov`](https://github.com/taiki-e/cargo-llvm-cov).
- Add the `ci/agent.sh` build script to the agent artifact cache key, since it wasn't there before.
- Don't run Rust tests in `--release` mode (have been meaning to change this so doing it at the same time).

There is some subtlety about putting the coverage result into the cached agent artifact, so that when we reuse the agent artifact we can still upload the coverage information for it to Codecov. Without this it would look like the coverage had dropped.
2022-09-15 02:29:22 +00:00
Stas
ac9d072e1d
fix linux proxy extensions provisioning failures (#2401)
* fix linux proxy extensions provisioning failures

* format

Co-authored-by: stas <statis@microsoft.com>
2022-09-14 23:52:35 +00:00
Stas
2fe73ab79c
bug fix (#2392)
* bug fix

* rename anyNotStoppedJobs  to anyNotStoppedTasks

Co-authored-by: stas <statis@microsoft.com>
2022-09-14 10:07:15 -07:00
Teo Voinea
ca7b6be43b
Refactor notification support (#2363)
* Add teams notifications

* .

* Fix compilation isues

* Checkpoint

* Added Ado

* Fix some TODOs

* Teams messages work! 🎉

* fmt

* Bug fix container url generator

* Some small ado changes

* 🧹

* PR comments

* Fix packages

* Get more detailed restore information to debug errors

* Maybe fixes this issue?

* Undo CI change
2022-09-14 15:07:52 +00:00
George Pollard
f375ee719e
Two fixes to C# scheduling (#2390)
Two fixes to scheduling code:

- `GetPool` was not correct for the VM case (this code is possibly legacy and not used any more)
- `BuildWorkUnit` could fetch the same pool multiple times and then fail due to `BucketConfig` mismatch (on `TimeStamp`)
  - add a cache to the loop so that we only fetch each pool once
2022-09-14 02:06:01 +00:00
George Pollard
bb81f2ec51
Fix MarkDependantsFailed (#2389) 2022-09-14 01:21:13 +00:00
Noah McGregor Harper
f7f91df622
CSharp Refactor - Instance Config Endpoint (#2347)
* CSharp Refactor - Instance Config Endpoint

* Finshing config update.

* Formatting.

* Formatting.

* formatting.

* Fixing encoding.

* Fixing config references.

* Fixing refs.

* Trying location.

* Trying ref to location.

* Passing nsg.

* Passing nsg.

* Setting nsg to not null.

* Fixing ok reference.

* Adding Instance Config Response.

* Setting required attribute.

* Adding route specifier.

* Formatting.

* Fixing route.

* Fixing optionals.

* Trying to set default

* Trying again.

* Setting require admins

* Removing optioanl.

* Testing with instancename.

* Updating instanceconfig model.

* Updating instance config response.

* Formatting.

* Removing AllowPoolManagement.

* Readding.

* Removing arg.

* Replacing with RequireAdminPrivs.

* Fix orm test.

* Setting requireadminprivs to true.

* Requiring admin privs.

* Fix formatting.

* fix test.

* Fixing.

* Changing error message.

* Changing.

* Reordering test args.

* Flipping.

* Fixing args.

* Fixing again.

* Removing false.

* Removing from constructor.

* Setting.

* Setting string to optional.

* Formatting.

* Adding default value.

* PUshing changes to OrmModelsTest

* Updating test to not pass null.

* George's suggestions.

* Removing entityconverter changes.

* Fixing import.
2022-09-13 08:55:40 -07:00
George Pollard
67e55910ac
Fix TaskOperations.SearchStates (#2383) 2022-09-13 00:31:35 +00:00
George Pollard
bc33ae1d7a
Fix Scaleset response Auth inclusion (#2382)
The `Auth` property is not meant to be returned upon Create/`POST`. Fix this, and make it easier to specify when `Auth` should be included or not.
2022-09-12 17:19:10 -07:00
Noah McGregor Harper
ecf858bbdd
Updating error and fixing default value for uto_create_cli_app (#2378) 2022-09-12 16:28:41 -07:00
Stas
ce1fc773a9
implement not-implemented: GetInputContainerQueues (#2380)
* implement not-implemented: GetInputContainerQueues

* named tuple

Co-authored-by: stas <statis@microsoft.com>
2022-09-12 15:17:21 -07:00
George Pollard
44f74f622a
DoNotRunExtensionsOnOverprovisionedVms must be false if Overprovision is false (#2375) 2022-09-12 22:00:47 +00:00
Stas
9eaf92ea35
tests and bug fixes (#2379)
Co-authored-by: stas <statis@microsoft.com>
2022-09-12 14:45:45 -07:00
George Pollard
9f03a174ec
Use self-hosted runner for Rust agent builds (#2366)
Use a 1ES hosted pool with more powerful machines to do the build for Rust agents.
2022-09-13 09:11:24 +12:00
George Pollard
2a2c07ed35
Extend use of validated string types (#2357)
In the Python code there were more validated string types that haven't been properly ported for use in the C# code (such as Region). Add that type back in and improve some others:

- Use `Region` type to represent regions (implicitly convertible to/from the `AzureLocation` SDK type)
- Improve validation of `Container` type to match Azure specs and use it in more places
- Restore/fix validation of `PoolName` type which was previously removed (#2080) due to being too strict: now allows 1-64 ASCII alphanumeric/hyphen/dash
  - We want to restrict pool names so that we can use them as disambiguating prefixes for scaleset names (see #2189). Note that underscore is not actually permitted in scaleset names so we will probably end up mapping it to hyphen.

Note that once C#7 lands we will be able to simplify the usage of `ValidatedString` a lot (using static abstract methods).

----

Open questions:

For deserializing from "known-good" places such as table storage or from Azure SDK APIs, should we have an `T.UnsafeAssumeValid(string input)` method which does no validation, to protect us from breakage?
2022-09-12 10:06:51 +12:00
dependabot[bot]
fa667e72e2
Bump url from 2.2.2 to 2.3.0 in /src/proxy-manager (#2360)
Bumps [url](https://github.com/servo/rust-url) from 2.2.2 to 2.3.0.
- [Release notes](https://github.com/servo/rust-url/releases)
- [Commits](https://github.com/servo/rust-url/compare/v2.2.2...v2.3.0)

---
updated-dependencies:
- dependency-name: url
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-09-09 01:37:21 +00:00
dependabot[bot]
343dda3073
Bump url from 2.2.2 to 2.3.0 in /src/agent (#2361)
Bumps [url](https://github.com/servo/rust-url) from 2.2.2 to 2.3.0.
- [Release notes](https://github.com/servo/rust-url/releases)
- [Commits](https://github.com/servo/rust-url/compare/v2.2.2...v2.3.0)

---
updated-dependencies:
- dependency-name: url
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-09-09 01:04:00 +00:00
George Pollard
ef7b557599
Add debug command to download a task’s containers (#2359) 2022-09-09 00:45:15 +00:00
Stas
511eafca73
fix bug in ado notification info (#2368)
Co-authored-by: stas <statis@microsoft.com>
2022-09-08 23:43:06 +00:00
Stas
6dfa46e62f
remove required field to match python behavior (#2367)
Co-authored-by: stas <statis@microsoft.com>
2022-09-08 23:10:12 +00:00
George Pollard
e1420e1a1d
Move global tool installs into another script so they can be cached (#2365) 2022-09-08 22:31:52 +00:00
Adam
02298f2bf2
Updating all azure-cli and azure-cli-core versions to 2.40 (#2364)
* Updating all azure-cli and azure-cli-core versions to 2.40

* Updating azure-identity to 1.10.0 required for azure-cli msal-extensions dependency
2022-09-08 13:42:48 -07:00
George Pollard
c562bbb0ad
VMSS creation fixes (#2358) 2022-09-09 07:53:02 +12:00
Stas
d61f66a667
Fixing memory leaks (#2356)
* fix a memory leak

* fix another memory leak

* remove extra queries

* formatting

* do not throw when scale set cannot be updated

Co-authored-by: stas <statis@microsoft.com>
2022-09-07 17:03:38 -07:00
George Pollard
df4edbea98
Coverage task should have access to readonly_inputs (#2352)
Fixes #2348.

Code duplicated & modified from line 112.

Both the generic and dotnet coverage tasks [iterate over all `readonly_inputs` that they are supplied](bd4dfdc592/src/agent/onefuzz-task/src/tasks/coverage/generic.rs (L98-L115)), so this should be sufficient.
2022-09-07 22:15:30 +00:00
Stas
44089d32f0
fix broken deploy (#2355)
Co-authored-by: stas <statis@microsoft.com>
2022-09-07 14:49:07 -07:00
George Pollard
02869eba9e
Fix #1285 (#2353) 2022-09-08 08:36:38 +12:00
George Pollard
fe51e011d1
Use Graph SDK instead of manually constructing queries (#2324) 2022-09-07 01:12:36 +00:00
George Pollard
bd4dfdc592
Update chrono, remove time to fix audit failures (#2285)
Closes #1366 and incidentally closes #1266.

Requires using a not-yet-merged modification to the latest version of `appinsights-rs` to remove the `time` feature from the `chrono` dependency (https://github.com/dmolokanov/appinsights-rs/pull/280). 

There are changes to use the new (tokio-based) version of `appinsights-rs`, e.g., made `set_appinsights_clients` async to ensure it is always called from an async context, since the constructor for appinsights now invokes `Tokio::spawn`.
2022-09-07 00:34:31 +00:00
George Pollard
0a84b1466d
Implement not implemented bits (#2341)
`ScalesetOperations` was missing:

* `ReimageNodes`
* `DeleteNodes`
2022-09-06 23:43:14 +00:00
George Pollard
c9d484778d
Use 64-bit worker for function (#2349) 2022-09-06 23:30:35 +00:00
George Pollard
b28008b519
Check if state transitions exist at test time (#2241)
Don't ignore missing state transitions in `StatefulOrm` static constructor test, instead error out. 

`Scaleset` was missing the `Resize` transition.
2022-09-06 22:52:31 +00:00
George Pollard
84b2cc9992
Rework use of storage clients (#2302)
- Reuse Storage clients by caching them against account name.
  - This follows recommendations of Storage documentation: https://devblogs.microsoft.com/azure-sdk/lifetime-management-and-thread-safety-guarantees-of-azure-sdk-net-clients/
- Use strongly-typed `ResourceIdentifier` when passing around Account IDs, to prevent mixing them up with Account Names. 
- Simplify & centralize SAS generation in one place.
2022-09-06 22:11:10 +00:00
Stas
318e568cab
add flag to enable memory and cpu profilers (#2345)
Co-authored-by: stas <statis@microsoft.com>
2022-09-06 10:32:30 -07:00
Stas
c175b4676a
enable running dot-net function on Windows to allow attaching remote debugger (#2344)
* enable running dot-net function on Windows to allow attaching remote debugger

* rename from 'use_windows' to 'host_dotnet_on_windows'

* instructions

* reformat deploy.py

Co-authored-by: stas <statis@microsoft.com>
2022-09-06 09:34:22 -07:00
Stas
dd3aa1ce0a
Bug fixes to enable timer_workers running properly (#2343)
Co-authored-by: stas <statis@microsoft.com>
2022-09-06 09:13:33 -07:00
dependabot[bot]
e7abd2d8fa
Bump pretty_assertions from 1.0.0 to 1.3.0 in /src/agent (#2322) 2022-09-06 14:34:43 +12:00
Stas
ee1ad0abd8
Functional tests (#2333)
* more tests

* revert compute, since "released" version has bug when creating scaleset extensions

* format

* found bug, i think...

* now should be fixed

Co-authored-by: stas <statis@microsoft.com>
2022-09-02 08:16:48 -07:00
George Pollard
c54db04083
Enforce that there no extra properties in request JSON, and that non-null properties are [Required] (#2328)
Closes #2314 via two fixes, one for additional properties and one for missing properties:

- Make all request types inherit from `BaseRequest` which has an `ExtensionData` property, and ensure that it is empty in `ParseRequest`.
- Add `[Required]` attribute to non-nullable properties that do not have defaults, and add a test that ensures we have this attribute where necessary.
2022-09-02 01:59:24 +00:00
George Pollard
52ba57bf0d
Remove some response-only properties from the Task model (#2335) 2022-09-01 23:33:04 +00:00
Joe Ranweiler
440f19baeb
Remove dead libfuzzer_coverage scripts (#2288) 2022-09-01 14:49:50 -07:00