Commit Graph

350 Commits

Author SHA1 Message Date
6c3b465dbe Update azure cli (#2733)
* update azure-cli to v2.43 along with supporting dependencies
2023-01-10 09:19:16 -08:00
c4f1adeff6 Fixing .NET crash report no-repro (#2642)
The minidump file being generated did not have enough information for SOS to correctly locate the exception object. Switch to “full” minidumps instead.

- Fix a small bug with coverage reporting task ID as job ID.

- I had thought we needed .NET 6 installed as well to run SharpFuzz, but now I'm not so sure. In any case, installing both versions will not hurt (and may help users).
2022-11-28 20:54:41 +00:00
e0634a3365 Deploy update (#2650)
* Remove Python App Function Deployment Code and  Code.

* Updating yml and zip names.

* Fixing ci.yml.

* Typo.

* Format

* Trying to remove python service.

* Updating directories.

* Removing flag.

* Format.

* Fixng api-service-net ref.

* Re-add requirement.

* Fixing refs in bicep.

* Specifying version.

* Removing dotnet refs in integration tests.

* Updating role assignment naming convention.

* Adding ignore.

* Update src/deny.toml

Co-authored-by: George Pollard <porges@porg.es>

* Update version.

* Removing onefuzztypes dependency.

* Switch app service plan to windows.

* Update test version.

* Changing version.

* Returning version.

* Trying to add onefuzz types back.

* Force pipenv version.

* Fix.

* Syntax.

* Renaming.

* Trying different version.

* Removing build step.

* Fixing bicep parameter.

* Retrying run with older version.

* Trying pipenv with another version.

* Forcing pipenv version in tox.

* Adding pipenv fix and updating version.

Co-authored-by: George Pollard <porges@porg.es>
2022-11-23 17:48:30 -08:00
22cb201805 Fix the notification get request (#2591)
* Fix the notification get request

* remove unused import
2022-11-02 17:56:50 -07:00
ee0cbd70a0 Enable dotnet fuzzing (#2273)
Add a new CLI job template, `libfuzzer dotnet_dll`, and supporting server-side definitions.
2022-10-19 14:17:27 -07:00
80b1122a8d Undo tab change from previous PR (#2492) 2022-10-05 17:32:33 +00:00
41f973184e Prefix target_exe with setup dir at use sites (#2405) 2022-09-29 13:47:04 -07:00
e6d3b39d1a Bump protobuf from 3.20.0 to 3.20.2 in /src/api-service/__app__ (#2446)
Bumps [protobuf](https://github.com/protocolbuffers/protobuf) from 3.20.0 to 3.20.2.
- [Release notes](https://github.com/protocolbuffers/protobuf/releases)
- [Changelog](https://github.com/protocolbuffers/protobuf/blob/main/generate_changelog.py)
- [Commits](https://github.com/protocolbuffers/protobuf/compare/v3.20.0...v3.20.2)

---
updated-dependencies:
- dependency-name: protobuf
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-09-27 12:42:55 +13:00
3f35d81f4b Adding New Default Image Config Value to IC. (#2434)
* Adding New Default Image Config Value to IC.

* Removing forced image setting.

* Updating Webhook Events.

* Removing typo.

* Updating webhook_events again.

* Syncing webhook events.

* Fixing check for os type.

* Fixing import.

* PR Suggestions.

* Fix C# Model Typo.

* Removing other refs to images.

* Removing remaining refs to images outside of models.

* Removing hardcoded image values from tests.

* Update Default Proxy and Repro Images.

Co-authored-by: Marc Greisen <mgreisen@microsoft.com>
2022-09-23 10:40:44 -07:00
4f9682d3cf Do not fail task on notification failure (#2435)
* Do not fail task on notification failure

* Need to throw on the last iteration in order for it to go to poison queue

* lint
2022-09-22 21:05:07 +00:00
4f1ac523da Adding error message to catch Model HttpResponseError. (#2384)
* Adding error message to catch Model HttpResponseError.

* Changing error message.

* Formatting.
2022-09-15 14:44:23 -07:00
511eafca73 fix bug in ado notification info (#2368)
Co-authored-by: stas <statis@microsoft.com>
2022-09-08 23:43:06 +00:00
02298f2bf2 Updating all azure-cli and azure-cli-core versions to 2.40 (#2364)
* Updating all azure-cli and azure-cli-core versions to 2.40

* Updating azure-identity to 1.10.0 required for azure-cli msal-extensions dependency
2022-09-08 13:42:48 -07:00
c94ae0bbb6 log the work item id when notifying ado (#2291)
* log the work item id when notifying ado

* format

* refactoring

* fix build

* fix

* format
2022-08-26 19:08:31 +00:00
e263b245f5 Add --use_dotnet_agent_functions to deploy.py and Python service (#2292) 2022-08-25 11:50:01 +12:00
39e22b060a Only save the job once when creating it (#2289)
* Convert exception to warning

* don't save job before creating the log container

* same as the python code

* test fix

* format

* format

* don't schedule the task if the log container is missing
2022-08-24 10:19:18 -07:00
f5b647ef1b Fixing Scaleset State Check. (#2265) 2022-08-17 15:39:02 -07:00
fdd16df847 Autoscale Table Synces Current Scaleset Settings (#2255)
* Autoscale Table Syncs Current Scaleset Settings.

* Fixing formatting.

* Fixing lint.

* Remove white space.

* Adding default check.

* Formatting.

* Fix.

* Fixing import.

* Fixing ref.

* Trying regular save.

* using self.

* Trying to fix lint.

* Fixing args.

* Fixed.

* Adding function call.

* Reformat.

* Formatting.

* trying again.

* FIxing profile call.

* Fixing type.

* Fixing type.

* Fixing format.

* Fixing typing.

* Fixing.

* Fixing function call

* Getting rid of length check.

* Getting rid of length check.

* Ready for review.

* Formatting.

* Removing bad change.
2022-08-17 10:41:48 -07:00
a3f1d59f70 Set autocrlf by default, update affected files (#2261) 2022-08-17 13:07:20 +12:00
46f4aac805 Default must be greater than or equal to minimum (#2248) 2022-08-12 14:40:55 -04:00
660d943824 Adjust auto scale to scale down nodes on shutdown (#2232)
* Only scale down when scale set in shutdown state

* Bug fix + explaing the logic a bit better

* Fix some bugs

* linting and bug fixes

* lint

* Actually now

* I'm not writing sql

* last try

* It's working

* lint

* Small docs update
2022-08-11 18:26:39 +00:00
4fa6e74241 Enable .NET functions in check-pr for Agent-specific functions (#2119)
Enable the .NET functions for the agent by sending the agent the URI for the `-net` service.

Also fix some things causing failures when using the .NET functions (`CouldShrinkScaleset` was not implemented).

Improve error handling around table serialization/deserialization, fix an issue with int64/long mismatch between Python & C# code.

----

For `check-pr` testing:

1. There's a new parameter `enable_dotnet` which maps directly to the `--enable_dotnet` switch on `deploy.py`.
2. If you put `agent` there, all the `agent_*` functions will be enabled for .NET and disabled for Python.
3. If `agent_can_schedule` is disabled on the Python side, it will automatically tell the agent to use the .NET functions.

So to test the .NET agent functions, do a `check-pr` run with `enable_dotnet` set to `agent` and it should all work.
2022-07-20 20:40:30 +00:00
3347e7e67b Use .get instead of lookup (#2165) 2022-07-14 23:55:30 +00:00
0b8093b89e Reuse Agent artifacts if nothing in src/agent changes (#2115)
The agent build takes most of the CI runtime, so improve it by only rebuilding if the pre-reqs or something inside `src/agent` changes.

We will always skip the cache for builds on tags and from the `main` branch, so that version is stamped correctly there.
2022-07-12 22:09:32 +00:00
f37224e8bb Add dotnet coverage task (#2062)
* checkpoint

* some more progress

* more progress

* More progress

* Now it's time to test it

* It works locally 🎉

* Attempting clean build

* fmt

* temporarily stub out macos

* missed a few

* please be the last one

* .

* .

* .

* noop change to unstuck actions

* .

* .

* Fix setup script

* Some fixes

* It works except for a race condition -- use a directory watcher to fix it

* It works end to end!

* Execute the commands using tokio's structs and timeout mechanism

* It works.... for real this time

* Undo timer changes

* Cleanup

* 🧹

* Fix import

* .

* PR comments

* Fix clippy

* Clippy whyyy

* Only check dotnet path once

* fmt

* Fix a couple more comments
2022-07-06 16:13:45 -04:00
61fc091f88 Make the log sas url last as long as the job duration (#2116) 2022-07-02 01:40:02 +00:00
30daae2215 Potential solution for TLS errors in OneFuzz (#2087)
* proposed fix from here:

https://github.com/Azure/azure-functions-durable-python/issues/194#issuecomment-710670377

* Update src/api-service/__app__/__init__.py

Co-authored-by: George Pollard <porges@porg.es>

Co-authored-by: stas <statis@microsoft.com>
Co-authored-by: George Pollard <porges@porg.es>
Co-authored-by: George Pollard <gpollard@microsoft.com>
2022-06-29 08:24:40 -07:00
52ccf05a29 Remove deprecated libfuzzer_coverage task (#2021)
- Remove the ability to create or execute a `libfuzzer_coverage` task
- Preserve the enum variant in `onefuzztypes` to prevent errors when deserializing old data
- Remove doc references to `libfuzzer_coverage`
2022-06-13 12:38:35 -07:00
9989189e60 Adding Node State to Node Heartbeat (#2024)
* Adding Node State to Node Heartbeat.

* Updating docs.

* Fixing webhook events.

* Formatting.

* Resetting type.

* Updting param.

* Setting to nodestate.
2022-06-13 10:13:57 -07:00
60b304a220 handle messages that are too big to fit in a queue message (#2020)
* handle messages that are too big to fit in a queue message

* tests

Co-authored-by: stas <statis@microsoft.com>
2022-06-06 12:16:47 -07:00
79cc5d54d3 Fix equire_admin_privileges Logic. (#2016) 2022-06-03 15:59:08 -07:00
0d14ca1ed1 pin protobuf (#1985) 2022-05-26 17:42:59 +00:00
91922ce01a enable python functions (#1907)
- queue_node_heartbeat
- queue_task_heartbeat

disable dotnet functions
- QueueNodeHeartbeat
- QueueTaskHeartbeat
2022-05-06 17:50:33 -07:00
5393dbab65 Replace queue_task_hearbeat (#1899)
* Replace queue_task_hearbeat

* dont rename statsFormat

* using hashsets for the helpers
2022-05-05 11:02:36 -07:00
3370c3df9a Replace queue_node_heatbeat (#1875)
* repalce queue_node_heatbeat

* Changing the name of the input queue

* [testing] hard code deployment of -net function

* explicitly disable other C# functions
2022-05-04 17:09:50 -07:00
b2399c4571 allow jobs with no log data to be schedule (#1893) 2022-05-04 20:09:19 +00:00
44059f20ca Adding Admin Checks to Node Operations. (#1779)
* Adding Admin Checks to Node Operations.

* Importing function.

* Changing naming convention.

* Fixing webhook events.

* Adding changes to scaleset init.
2022-04-27 11:31:43 -07:00
cb45c5685f add tool_name and onfuzz_version to CrashReport (#1635) 2022-04-18 23:56:07 +00:00
87eb606b35 Delete nodes when they're done (#1763)
* Delete nodes when they're done

* Missed a file

* Load node disposal strategy from env var

* Lint

* Fix subtle bug

* Deleting doesn't work, will 'decomission' nodes once they complete work

* Missed a file

* Remove logging line
2022-04-12 17:32:15 +00:00
8299d8fb57 Using existing auto scale settings isn't an error (#1745) 2022-04-06 12:41:58 +00:00
7add51fd3a Log redirection, service side (#1727)
* Setting the service side of the log management
- a log is created or reused when e create a job
- when scheduling the task we send the log location to the agent
The expected log structure looks liek
{fuzzContainer}/logs/{job_id}/{task_id}/{machine_id}/1.log

* regenerate doces

* including job_id in the container name

* regenerating docs
removing bad doc file
2022-03-29 18:47:20 +00:00
424ffdb4b5 Adding auto scale via cli (#1717)
* Initial implementation for adding auto scale via cli

* Remove unused argument

* Remove unused import

* I had a 👻extra line👻
2022-03-29 09:50:07 -04:00
24454e3681 pin click to fix black (#1726)
* pin click to fix black

* missed a couple

Co-authored-by: stas <statis@microsoft.com>
2022-03-28 12:06:13 -07:00
ce03394376 Handle instance being destroyed before updating scaling protection (#1719)
* Handle instance being destroyed before updating scaling protection

* Fix bug where we release protection too early
2022-03-28 10:14:39 -04:00
5c418eeb36 Add autoscaling diagnostics (#1708)
* Initial attempt

* Adding diagnostics works

* 🧹

* lint

* I wish the linter could auto fix these issues

* Lint
2022-03-21 13:25:21 +00:00
40b0e6685a Give function app resource group scoped contributor role (#1698)
* Give function app resource group scoped contributor role

* Reenable autoscaling

* We don't know what the minimum capacity for a sku is yet

* Lint
2022-03-09 13:07:22 -05:00
4d1c1f5713 Abstract out node disposal (#1686)
* Abstract node disposal strategy

* Cleanup + lint

* Handle possibile scalesets being in resize state

* Setting the size is still exposed via CLI, we don't want to break that functionality

* PR comments
2022-03-08 13:30:34 -05:00
7c507ab7c7 Remove dependency on onefuzz deployment role to unblock (#1693) 2022-03-04 18:51:57 +00:00
16166e1c14 Create autoscale resources for scaleset (#1661)
* Initial progress to adding a auto scale resource

* auto scale API is ready

* When creating a scaleset, add an autoscale resource to it as well

* Auto scale is correctly linked with scaleset

* 🧹

* Lint

* Cleaned up
2022-02-28 17:28:31 +00:00
5d8516bd70 Enable scale in protection on VMSS instances (#1647)
* draft attempt at adding scaling protection

* Service can now control scaling protection policy on VM instances

* Improve logging a bit

* draft attempt at adding scaling protection

* Service can now control scaling protection policy on VM instances

* Improve logging a bit

* Error message was missing info

* Linter

* Don't schedule work if we can't protect the node

* Last of the linter changes
2022-02-14 14:56:55 +00:00