Commit Graph

1629 Commits

Author SHA1 Message Date
e1851b0af4 Add more logs (#2438)
* add logs

* avoid relying on exceptions for logic flow control

* add logs to agent commands

* add more logs and fix error logging when table writes fail

* move machine ID to CustomDimensions

* log insert errors

* Log Delete failures

* more logs

* more logs

* more logs

* More logs (I think that's it there is no more...)

Co-authored-by: stas <statis@microsoft.com>
2022-09-23 11:28:37 +12:00
4f9682d3cf Do not fail task on notification failure (#2435)
* Do not fail task on notification failure

* Need to throw on the last iteration in order for it to go to poison queue

* lint
2022-09-22 21:05:07 +00:00
b14bade0fc cleanup queues for non-existent pools and non-existent tasks (#2433)
* cleanup queues for non-existent pools and non-existent tasks

* extra logs

Co-authored-by: stas <statis@microsoft.com>
2022-09-22 08:15:57 -07:00
1013e01a3d Release-5.14.0 (#2428)
* Release-5.14.0
2022-09-21 17:21:09 -07:00
de766dfa78 Delete pool queue when pool is deleted (#2431)
* Delete pool queue when pool is deleted

* Also delete shrink queue when pool is deleted

Co-authored-by: stas <statis@microsoft.com>
2022-09-21 14:39:30 -07:00
2f42cd74d4 Adding State Transition Error Logging. (#2425)
* Adding State Transition Logging.

* Fix log call.

* Removing logging statement.

* Fix error! call.

* Adding error to error message.

* Make err var borrowed.

* Formatting.
2022-09-21 12:54:28 -07:00
39c3736bef Adjust concurrency spec (#2426) 2022-09-20 22:12:53 +00:00
a6addbf83a Minor fixes (#2420)
* delete all temp files

* add try/catch and some logging when deleting temp files

Co-authored-by: stas <statis@microsoft.com>
2022-09-20 10:20:29 -07:00
f40a69a37f fix linux repro extensions (#2415)
* fix OMS Linux repro extension config

* Fixing lost Node state updates

* fix bug in ReproVmss

* rewrite ssh auth

* win azure function ssh-keygen fix

* more logs

* try -P

* use empty string for password

* use argument list

* addressing comments

Co-authored-by: stas <statis@microsoft.com>
Co-authored-by: George Pollard <gpollard@microsoft.com>
2022-09-20 08:15:26 -07:00
b647e4a1fe mark tasks as failed if a work unit cannot be created for the task (#2409)
* mark tasks as failed if a work unit cannot be created for the task

* fix up time queries

* query improvements

Co-authored-by: stas <statis@microsoft.com>
2022-09-17 12:50:41 -07:00
867cdbc06f Port SyncAutoscaleSettings from Python to C# (#2407)
* Port SyncAutoscaleSettings from Python to C#

* address comment

Co-authored-by: stas <statis@microsoft.com>
2022-09-16 07:58:37 -07:00
3f86fc8689 fix some bugs (#2406)
* - fix queries in timer retention

- do not discard proxy record after proxy state is processed, since that record needs to persist

* addressing comments

Co-authored-by: stas <statis@microsoft.com>
2022-09-15 15:45:39 -07:00
4f1ac523da Adding error message to catch Model HttpResponseError. (#2384)
* Adding error message to catch Model HttpResponseError.

* Changing error message.

* Formatting.
2022-09-15 14:44:23 -07:00
f22dee18df CodeQL needs explicit permissions to run (#2404) 2022-09-15 09:05:18 -04:00
4cc4de9c9e Codecov setup for C# & Rust code (#2400)
Use Codecov to show coverage reports, so we get highlighted versions of the files where it is easy to see missing coverage.

- Setup Rust coverage using [`cargo-llvm-cov`](https://github.com/taiki-e/cargo-llvm-cov).
- Add the `ci/agent.sh` build script to the agent artifact cache key, since it wasn't there before.
- Don't run Rust tests in `--release` mode (have been meaning to change this so doing it at the same time).

There is some subtlety about putting the coverage result into the cached agent artifact, so that when we reuse the agent artifact we can still upload the coverage information for it to Codecov. Without this it would look like the coverage had dropped.
2022-09-15 02:29:22 +00:00
61a797e224 Restore self-hosted configuration (#2394) 2022-09-14 23:59:41 +00:00
ac9d072e1d fix linux proxy extensions provisioning failures (#2401)
* fix linux proxy extensions provisioning failures

* format

Co-authored-by: stas <statis@microsoft.com>
2022-09-14 23:52:35 +00:00
2fe73ab79c bug fix (#2392)
* bug fix

* rename anyNotStoppedJobs  to anyNotStoppedTasks

Co-authored-by: stas <statis@microsoft.com>
2022-09-14 10:07:15 -07:00
ca7b6be43b Refactor notification support (#2363)
* Add teams notifications

* .

* Fix compilation isues

* Checkpoint

* Added Ado

* Fix some TODOs

* Teams messages work! 🎉

* fmt

* Bug fix container url generator

* Some small ado changes

* 🧹

* PR comments

* Fix packages

* Get more detailed restore information to debug errors

* Maybe fixes this issue?

* Undo CI change
2022-09-14 15:07:52 +00:00
f375ee719e Two fixes to C# scheduling (#2390)
Two fixes to scheduling code:

- `GetPool` was not correct for the VM case (this code is possibly legacy and not used any more)
- `BuildWorkUnit` could fetch the same pool multiple times and then fail due to `BucketConfig` mismatch (on `TimeStamp`)
  - add a cache to the loop so that we only fetch each pool once
2022-09-14 02:06:01 +00:00
3b8cbc3f1e Cancel any previous PR builds when new one starts (#2393) 2022-09-14 01:34:42 +00:00
bb81f2ec51 Fix MarkDependantsFailed (#2389) 2022-09-14 01:21:13 +00:00
2ff758464e Use Github-hosted Ubuntu until issue with self-hosted pool is resolved (#2391) 2022-09-14 12:39:18 +12:00
f7f91df622 CSharp Refactor - Instance Config Endpoint (#2347)
* CSharp Refactor - Instance Config Endpoint

* Finshing config update.

* Formatting.

* Formatting.

* formatting.

* Fixing encoding.

* Fixing config references.

* Fixing refs.

* Trying location.

* Trying ref to location.

* Passing nsg.

* Passing nsg.

* Setting nsg to not null.

* Fixing ok reference.

* Adding Instance Config Response.

* Setting required attribute.

* Adding route specifier.

* Formatting.

* Fixing route.

* Fixing optionals.

* Trying to set default

* Trying again.

* Setting require admins

* Removing optioanl.

* Testing with instancename.

* Updating instanceconfig model.

* Updating instance config response.

* Formatting.

* Removing AllowPoolManagement.

* Readding.

* Removing arg.

* Replacing with RequireAdminPrivs.

* Fix orm test.

* Setting requireadminprivs to true.

* Requiring admin privs.

* Fix formatting.

* fix test.

* Fixing.

* Changing error message.

* Changing.

* Reordering test args.

* Flipping.

* Fixing args.

* Fixing again.

* Removing false.

* Removing from constructor.

* Setting.

* Setting string to optional.

* Formatting.

* Adding default value.

* PUshing changes to OrmModelsTest

* Updating test to not pass null.

* George's suggestions.

* Removing entityconverter changes.

* Fixing import.
2022-09-13 08:55:40 -07:00
67e55910ac Fix TaskOperations.SearchStates (#2383) 2022-09-13 00:31:35 +00:00
bc33ae1d7a Fix Scaleset response Auth inclusion (#2382)
The `Auth` property is not meant to be returned upon Create/`POST`. Fix this, and make it easier to specify when `Auth` should be included or not.
2022-09-12 17:19:10 -07:00
ecf858bbdd Updating error and fixing default value for uto_create_cli_app (#2378) 2022-09-12 16:28:41 -07:00
ce1fc773a9 implement not-implemented: GetInputContainerQueues (#2380)
* implement not-implemented: GetInputContainerQueues

* named tuple

Co-authored-by: stas <statis@microsoft.com>
2022-09-12 15:17:21 -07:00
44f74f622a DoNotRunExtensionsOnOverprovisionedVms must be false if Overprovision is false (#2375) 2022-09-12 22:00:47 +00:00
9eaf92ea35 tests and bug fixes (#2379)
Co-authored-by: stas <statis@microsoft.com>
2022-09-12 14:45:45 -07:00
9f03a174ec Use self-hosted runner for Rust agent builds (#2366)
Use a 1ES hosted pool with more powerful machines to do the build for Rust agents.
2022-09-13 09:11:24 +12:00
64bfc0e9e9 Validate changes to devcontainer files (#2373) 2022-09-12 20:56:23 +00:00
615dff6a64 Update teams.md link (#2370) 2022-09-12 12:20:10 +00:00
2a2c07ed35 Extend use of validated string types (#2357)
In the Python code there were more validated string types that haven't been properly ported for use in the C# code (such as Region). Add that type back in and improve some others:

- Use `Region` type to represent regions (implicitly convertible to/from the `AzureLocation` SDK type)
- Improve validation of `Container` type to match Azure specs and use it in more places
- Restore/fix validation of `PoolName` type which was previously removed (#2080) due to being too strict: now allows 1-64 ASCII alphanumeric/hyphen/dash
  - We want to restrict pool names so that we can use them as disambiguating prefixes for scaleset names (see #2189). Note that underscore is not actually permitted in scaleset names so we will probably end up mapping it to hyphen.

Note that once C#7 lands we will be able to simplify the usage of `ValidatedString` a lot (using static abstract methods).

----

Open questions:

For deserializing from "known-good" places such as table storage or from Azure SDK APIs, should we have an `T.UnsafeAssumeValid(string input)` method which does no validation, to protect us from breakage?
2022-09-12 10:06:51 +12:00
140638f684 Release-5.13.0 (#2371)
* Release-5.13.0
5.13.0
2022-09-09 15:32:54 -07:00
16916c8653 Devcontainer: ensure venv is installed (#2372) 2022-09-09 22:07:26 +00:00
fa667e72e2 Bump url from 2.2.2 to 2.3.0 in /src/proxy-manager (#2360)
Bumps [url](https://github.com/servo/rust-url) from 2.2.2 to 2.3.0.
- [Release notes](https://github.com/servo/rust-url/releases)
- [Commits](https://github.com/servo/rust-url/compare/v2.2.2...v2.3.0)

---
updated-dependencies:
- dependency-name: url
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-09-09 01:37:21 +00:00
343dda3073 Bump url from 2.2.2 to 2.3.0 in /src/agent (#2361)
Bumps [url](https://github.com/servo/rust-url) from 2.2.2 to 2.3.0.
- [Release notes](https://github.com/servo/rust-url/releases)
- [Commits](https://github.com/servo/rust-url/compare/v2.2.2...v2.3.0)

---
updated-dependencies:
- dependency-name: url
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-09-09 01:04:00 +00:00
ef7b557599 Add debug command to download a task’s containers (#2359) 2022-09-09 00:45:15 +00:00
511eafca73 fix bug in ado notification info (#2368)
Co-authored-by: stas <statis@microsoft.com>
2022-09-08 23:43:06 +00:00
6dfa46e62f remove required field to match python behavior (#2367)
Co-authored-by: stas <statis@microsoft.com>
2022-09-08 23:10:12 +00:00
e1420e1a1d Move global tool installs into another script so they can be cached (#2365) 2022-09-08 22:31:52 +00:00
02298f2bf2 Updating all azure-cli and azure-cli-core versions to 2.40 (#2364)
* Updating all azure-cli and azure-cli-core versions to 2.40

* Updating azure-identity to 1.10.0 required for azure-cli msal-extensions dependency
2022-09-08 13:42:48 -07:00
c562bbb0ad VMSS creation fixes (#2358) 2022-09-09 07:53:02 +12:00
d61f66a667 Fixing memory leaks (#2356)
* fix a memory leak

* fix another memory leak

* remove extra queries

* formatting

* do not throw when scale set cannot be updated

Co-authored-by: stas <statis@microsoft.com>
2022-09-07 17:03:38 -07:00
df4edbea98 Coverage task should have access to readonly_inputs (#2352)
Fixes #2348.

Code duplicated & modified from line 112.

Both the generic and dotnet coverage tasks [iterate over all `readonly_inputs` that they are supplied](bd4dfdc592/src/agent/onefuzz-task/src/tasks/coverage/generic.rs (L98-L115)), so this should be sufficient.
2022-09-07 22:15:30 +00:00
44089d32f0 fix broken deploy (#2355)
Co-authored-by: stas <statis@microsoft.com>
2022-09-07 14:49:07 -07:00
02869eba9e Fix #1285 (#2353) 2022-09-08 08:36:38 +12:00
fe51e011d1 Use Graph SDK instead of manually constructing queries (#2324) 2022-09-07 01:12:36 +00:00
bd4dfdc592 Update chrono, remove time to fix audit failures (#2285)
Closes #1366 and incidentally closes #1266.

Requires using a not-yet-merged modification to the latest version of `appinsights-rs` to remove the `time` feature from the `chrono` dependency (https://github.com/dmolokanov/appinsights-rs/pull/280). 

There are changes to use the new (tokio-based) version of `appinsights-rs`, e.g., made `set_appinsights_clients` async to ensure it is always called from an async context, since the constructor for appinsights now invokes `Tokio::spawn`.
2022-09-07 00:34:31 +00:00