140 Commits

Author SHA1 Message Date
George Pollard
66b990bb7e
Bring ErrorCode enums into sync (#3129)
ErrorCode enums on Python & C# side had gotten out of sync. This can cause the CLI to fail to parse responses.
2023-05-24 08:41:41 -04:00
George Pollard
2f478d6c0b
Expand valid scaleset names (#3045)
Scaleset names are now permitted to be any (valid) strings, instead of only GUIDs. When we generate a scaleset name it is now based upon the pool name; for example the pool `pool` might get a scaleset named `pool-3b24ba211cad4b078655914754485838`.

This should be backwards-compatible since GUIDs are [already serialized to table storage as strings](dddcfa4949/src/ApiService/ApiService/onefuzzlib/orm/EntityConverter.cs (L190-L191)), so this simply loosens the restrictions placed upon them.

Scaleset IDs now have a strong type in the same way as other IDs; this helps to avoid mixing them up with other strings. Because of this I found one bug in the scaleset search query logic due to Pool ID/VMSS ID confusion. As part of fixing this I've changed the scaleset search query to only return nodes from the table rather than querying Azure to find a list; this seems to be sufficient for the CLI.
2023-05-17 09:58:58 +12:00
Teo Voinea
cee37b5ece
Create migrating-to-events-2.0.md (#3098)
* Create migrating-to-events-2.0.md

* Update migrating-to-events-2.0.md
2023-05-11 12:27:30 -07:00
Cheick Keita
aa28550aad
update the ado logic to consume the list of existing items once (#3014)
* update the ado logic to consume the list of existing items once

* format

* Update src/ApiService/ApiService/onefuzzlib/notifications/Ado.cs

Co-authored-by: Teo Voinea <58236992+tevoinea@users.noreply.github.com>

* Adding a notification testing endpoint

* fix tests

* format

* regen docs

* update logic

* format

* fix dummy name

* mypy fix

* make mypy happy

* bandit fix

* renaming

* address PR Comment

---------

Co-authored-by: Teo Voinea <58236992+tevoinea@users.noreply.github.com>
2023-04-19 14:27:16 -07:00
Cheick Keita
6933521a1a
Adding validation command to the agent (#2948)
* WIP: Adding a validation command to the agent

* introducing a ValidationConfig

* refactoring

* adding verification code

* remove unused test

* format

* update dependencies

* adding a command to get the loading logs

* add print logs for linux

* clippy fix

* clippy on windows

* renaming stuff

* bug fix
2023-03-31 13:23:25 -07:00
Teo Voinea
ac789fabf2
Update ado.md (#2956) 2023-03-31 11:34:09 -07:00
George Pollard
3c3f12a7e4
Make ImageReference strongly-typed and checked up-front (#2369)
- Turn `ImageReference` into its own type so it is validated early on in request submission time, and we don't end up with malformed IDs, etc.
- Add in support for shared image galleries since that was easy enough to add while I'm doing this.
- Explicitly document which image sources are permitted and how to reference them with resource IDs.

This addresses/closes #1464 for the C# port. Also fixes #2927 which was recently reported.
2023-03-26 22:20:08 +00:00
Adam
1f67494334
Deployment fix for --auto_create_cli_app flag bug (#2921)
* Update .gitignore

* re-add sync-fork.yml

deleted after merge from origin/main

* Update README.md

TEST

* Update README.md

* Update sync-fork.yml

bump ver to 1.8

* updated deploy.py and configuration.py

* cleanup

* formatting

* linter cleanup

* linter cleanup 2

* better logging

* last linter issue

* remove extra app

* Updating getting started docs for config refactor

* Update docs/getting-started.md

Co-authored-by: Noah McGregor Harper <74685766+nharper285@users.noreply.github.com>

* update getting-started.md doc for config refactor

* update getting-started.md doc for config refactor

---------

Co-authored-by: Noah McGregor Harper <74685766+nharper285@users.noreply.github.com>
2023-03-10 15:26:40 -08:00
Teo Voinea
49543cfa14
Ipc between agent and task (#2912)
* .

* It doesn't work yet but we're making progress

* Added graceful shutdown and tests

* Small fix

* Fix crate issues

* test fix

* Fix build

* make clippy happy

* The order changed

* Use timeout in kill

* Almost done shutting down ipc

* It should all work now

* Update deny.toml

* Fix warning
2023-03-10 17:21:18 +00:00
Teo Voinea
6f66fcb9f8
Revert "Create 2 way IPC connection between agent and task" (#2910)
* Revert "Create 2 way IPC connection between agent and task (#2886)"

This reverts commit 091c870be6d9813cfceb60d61932f09c35f9bb67.

* Temporarily allow vulnerability since a new one just came out

* Temporarily allow vulnerability

* Update proxy.sh

* Update agent.sh

* Update deny.toml
2023-03-06 16:09:29 -05:00
Teo Voinea
091c870be6
Create 2 way IPC connection between agent and task (#2886)
* .

* It doesn't work yet but we're making progress

* Added graceful shutdown and tests

* Small fix

* Fix crate issues

* test fix

* Fix build

* make clippy happy

* The order changed

* Use timeout in kill
2023-03-03 14:08:34 -05:00
Joe Ranweiler
f12319b359
Document coverage crate and tool (#2904) 2023-03-02 11:50:13 -08:00
Cheick Keita
3d299ce51e
fix extra container intilization (#2887)
* fix extra container intilization

* fix extra url download

* fix extra dir parameter to the agent

* rename extra to extra_dir
2023-02-27 13:57:32 -08:00
Cheick Keita
b84896802c
Adding extra container to tasks (#2847)
* adding extra container to tasks

* setup expand

* build fix

* generate docs

* build fix

* build fix

* build fix

* format

* format

* build fix

* fix extra container references

* format

* Update "Needs Triage" label to the one we use. (#2845)

* Report extension errors (#2846)

Old failure message:
```
failed to launch extension
```

New failure message:

```
failed to launch extension(s): Errors for extension 'CustomScriptExtension':
:Error: ProvisioningState/failed/3 (Provisioning failed) - Failed to download all specified files. Exiting. Error Message: The remote server returned an error: (400) Bad Request.
```

* Sematically validate notification configs (#2850)

* Add new command

* Update remaining jinja templates and references to use scriban

* Add ado template validation

* Validate ado and github templates

* Remove unnecessary function

* Update src/ApiService/ApiService/OneFuzzTypes/Model.cs

Co-authored-by: Cheick Keita <kcheick@gmail.com>

---------

Co-authored-by: Cheick Keita <kcheick@gmail.com>

* adding extra container to integration tests

* adding doc

* update tests

* format

* build and clippy fix

* Update src/agent/onefuzz-task/src/tasks/report/generic.rs

Co-authored-by: Teo Voinea <58236992+tevoinea@users.noreply.github.com>

---------

Co-authored-by: Marc Greisen <mgreisen@microsoft.com>
Co-authored-by: George Pollard <gpollard@microsoft.com>
Co-authored-by: Teo Voinea <58236992+tevoinea@users.noreply.github.com>
2023-02-23 19:08:01 +00:00
Marc Greisen
dfb0db87c1
Update the document name. (#2882) 2023-02-22 14:38:45 -08:00
Joe Ranweiler
4caaf8fc32
Remove old libfuzzer dotnet template (#2875)
### Context
The original `libfuzzer dotnet` job template was a proof of concept that demonstrated how the `libfuzzer_fuzz` task could be used to express fuzzing via the (pre SharpFuzz 2.0) `libfuzzer-dotnet` tool. It (and its associated integration test) used a harness that linked an older version of SharpFuzz, and which is incompatible with LibFuzzerDotnetLoader (which requires SharpFuzz 2.0 or greater).

### Changes
- Rename `libfuzzer dotnet_dll` job template to `libfuzzer dotnet`, making it the _only_ `libfuzzer-dotnet` template
- Remove integration tests and docs for the old proof-of-concept job type

### Notice

This is a breaking change.

Closes #2874.
2023-02-21 10:54:36 +13:00
Joe Ranweiler
42c4f6204d
Remove rest of unused telemetry (#2863) 2023-02-20 06:21:29 -08:00
Joe Ranweiler
ddbc715b3f
Remove Z3 telemetry (#2860) 2023-02-17 14:32:52 -08:00
Joe Ranweiler
58f756de47
Switch over to new coverage task (#2741) 2023-02-15 12:41:19 -08:00
Teo Voinea
08bb0ec355
Update remaining jinja docs (#2838)
* Add new command

* Update remaining jinja templates and references to use scriban
2023-02-15 18:25:40 +00:00
Cheick Keita
f13f52ab71
Deprecating the job template feature (#2798)
* deprecating the job template feature

* removing the code

* format
2023-02-08 19:21:03 +00:00
Cheick Keita
d732028201
Add unmanaged nodes integration tests (#2780)
* Add docker file to the runtime tools

* fixes

* bug fixes

* more bug fixes and added doc

* don;t overwrite the RUST_LOG env var

* integration test for unmanaged nodes

* add unamanged parameters to launch()

* add ing object_id

* more bug fixes

* bug fixes

* chmod on the linux files in docker

* format

* cleanup merge

* added test_unmanaged command

* cleanup

* use a single image for the docker compose
remove images after the test

* docs and formatting

* format

* format

* format and bug fixes

* using windows server

* fix linux container
make the base image a paramter on windows
use the windows server base image on  windows server

* format

* bug fix

* more fixes

* allow reboot

* more fixes

* added more logging around the service principal creation

* format

* more logging

* change restart policy

* fix multi tenant domain

* more fixes

* exit instead of reboot when running inside docker

* remove comment

* build fix

* try_exist instead of exist

* save the docker logs

* bug_fix

* adding timeout

* fix timeout logic

* adding a build profile

* make all agent depend on the first one

* remove profile

* another fix

* restart agent 1

* Update docs/unmnaged-nodes.md

Co-authored-by: Teo Voinea <58236992+tevoinea@users.noreply.github.com>

---------

Co-authored-by: Teo Voinea <58236992+tevoinea@users.noreply.github.com>
2023-02-08 11:07:19 -08:00
Teo Voinea
c1f6dfc366
Validate scriban from cli (#2800)
* Add validate scriban endpoint to cli

* missed a file

* Lint -- I miss C#

* docs
2023-02-06 08:32:49 -05:00
Cheick Keita
bc57fa016c
Add dockerfile to the runtime tools (#2730)
* Add docker file to the runtime tools

* fixes

* bug fixes

* more bug fixes and added doc

* don;t overwrite the RUST_LOG env var

* integration test for unmanaged nodes

* add unamanged parameters to launch()

* add ing object_id

* more bug fixes

* bug fixes

* chmod on the linux files in docker

* format

* revert changes in  integration tests

* Apply suggestions from code review

Co-authored-by: Marc Greisen <mgreisen@microsoft.com>

* format and bug fix

* fix condition

---------

Co-authored-by: Marc Greisen <mgreisen@microsoft.com>
2023-01-30 20:33:11 -08:00
George Pollard
f90b53d833
Fix command in docs (#2779)
The `--container_type` argument is not named
2023-01-26 08:55:51 -05:00
Cheick Keita
0fb8bc4a86
Bug fixes and documentation (#2694) 2022-12-15 19:39:17 -08:00
Noah McGregor Harper
c277cd1ec3
Update References to Ubuntu 18.04 to 20.04 (#2535)
* Update References to 18.04 to 20.04

* Fixing Ubuntu Image.

* Fix image.
2022-10-28 10:17:57 -07:00
Joe Ranweiler
ee0cbd70a0
Enable dotnet fuzzing (#2273)
Add a new CLI job template, `libfuzzer dotnet_dll`, and supporting server-side definitions.
2022-10-19 14:17:27 -07:00
George Pollard
b88f46779e
Remove suggestion to reset IterationPath upon duplicate (#2533)
* Remove suggestion to reset IterationPath upon duplicate

* Update highlighting to jsonc
2022-10-19 20:34:16 +00:00
George Pollard
71a915c9cd
Add logo to readme (#2340)
* Add logo to readme
2022-10-11 02:06:19 +00:00
George Pollard
cdc104f966
Update documentation to specify appropriate RID (#2490)
Closes #2457
2022-10-07 01:03:18 +00:00
Noah McGregor Harper
3f35d81f4b
Adding New Default Image Config Value to IC. (#2434)
* Adding New Default Image Config Value to IC.

* Removing forced image setting.

* Updating Webhook Events.

* Removing typo.

* Updating webhook_events again.

* Syncing webhook events.

* Fixing check for os type.

* Fixing import.

* PR Suggestions.

* Fix C# Model Typo.

* Removing other refs to images.

* Removing remaining refs to images outside of models.

* Removing hardcoded image values from tests.

* Update Default Proxy and Repro Images.

Co-authored-by: Marc Greisen <mgreisen@microsoft.com>
2022-09-23 10:40:44 -07:00
Teo Voinea
ca7b6be43b
Refactor notification support (#2363)
* Add teams notifications

* .

* Fix compilation isues

* Checkpoint

* Added Ado

* Fix some TODOs

* Teams messages work! 🎉

* fmt

* Bug fix container url generator

* Some small ado changes

* 🧹

* PR comments

* Fix packages

* Get more detailed restore information to debug errors

* Maybe fixes this issue?

* Undo CI change
2022-09-14 15:07:52 +00:00
Teo Voinea
615dff6a64
Update teams.md link (#2370) 2022-09-12 12:20:10 +00:00
Stas
c175b4676a
enable running dot-net function on Windows to allow attaching remote debugger (#2344)
* enable running dot-net function on Windows to allow attaching remote debugger

* rename from 'use_windows' to 'host_dotnet_on_windows'

* instructions

* reformat deploy.py

Co-authored-by: stas <statis@microsoft.com>
2022-09-06 09:34:22 -07:00
Avishay Balter
dc43242e0c
Detecting the use of OneFuzz by recommending a .onefuzz file (#2236)
* getting started

* PR fixes

* getting started

* PR fixes

Co-authored-by: Marc Greisen <mgreisen@microsoft.com>
2022-08-11 16:26:23 -07:00
Noah McGregor Harper
ceb5d6d5f9
Update Default Windows Image (#2226)
* Update Function Scaling Policy and Diagnostic Settings.

* Adding changes to scaling policy

* UPdating cooldown.

* Changing time windows.

* Updating duration.

* Updating Out-of-Date Windows Image.

* Adding repro operations back.

* Removing bad files.
2022-08-08 11:40:55 -07:00
Teo Voinea
f37224e8bb
Add dotnet coverage task (#2062)
* checkpoint

* some more progress

* more progress

* More progress

* Now it's time to test it

* It works locally 🎉

* Attempting clean build

* fmt

* temporarily stub out macos

* missed a few

* please be the last one

* .

* .

* .

* noop change to unstuck actions

* .

* .

* Fix setup script

* Some fixes

* It works except for a race condition -- use a directory watcher to fix it

* It works end to end!

* Execute the commands using tokio's structs and timeout mechanism

* It works.... for real this time

* Undo timer changes

* Cleanup

* 🧹

* Fix import

* .

* PR comments

* Fix clippy

* Clippy whyyy

* Only check dotnet path once

* fmt

* Fix a couple more comments
2022-07-06 16:13:45 -04:00
George Pollard
1c679e48fe
More documentation (#2095)
Create an index (README) for the `docs` folder, and add a basic architecture diagram.
2022-06-30 00:37:48 +00:00
Joe Ranweiler
52ccf05a29
Remove deprecated libfuzzer_coverage task (#2021)
- Remove the ability to create or execute a `libfuzzer_coverage` task
- Preserve the enum variant in `onefuzztypes` to prevent errors when deserializing old data
- Remove doc references to `libfuzzer_coverage`
2022-06-13 12:38:35 -07:00
Noah McGregor Harper
9989189e60
Adding Node State to Node Heartbeat (#2024)
* Adding Node State to Node Heartbeat.

* Updating docs.

* Fixing webhook events.

* Formatting.

* Resetting type.

* Updting param.

* Setting to nodestate.
2022-06-13 10:13:57 -07:00
George Pollard
cc33427c94
Update Getting Started instructions (#2030) 2022-06-09 11:33:08 +12:00
Noah McGregor Harper
79cc5d54d3
Fix equire_admin_privileges Logic. (#2016) 2022-06-03 15:59:08 -07:00
Teo Voinea
b856d44bed
Rename agent to task (#1980)
* Rename agent to task

* Missed a rename
2022-05-26 20:41:47 +00:00
Joe Ranweiler
02477b207a
Update coverage filtering docs (#1950) 2022-05-16 12:22:12 -07:00
Cheick Keita
b080f5b164
Make tool_version, onefuzz_version, tool_name optinal (#1940) 2022-05-13 20:12:22 +00:00
Noah McGregor Harper
f5ce5b3d6f
Updating OneFuzz Threat Model (#1917)
* Updating OneFuzz Threat Model

* Update htm threat model.
2022-05-09 15:03:44 -07:00
Adam
793ce85cdf
updating deploy.py to push dotnet by default (#1888)
* updating deploy.py to push dotnet by default

* Updating requirements for Azure Function core tools CLI to 4

* updating docs for Azure Functions Core Tools v 4.x requirement

* Update src/utils/check-pr/README.md

Co-authored-by: Cheick Keita <chkeita@microsoft.com>
2022-05-04 18:44:27 +00:00
Noah McGregor Harper
44059f20ca
Adding Admin Checks to Node Operations. (#1779)
* Adding Admin Checks to Node Operations.

* Importing function.

* Changing naming convention.

* Fixing webhook events.

* Adding changes to scaleset init.
2022-04-27 11:31:43 -07:00
Stas
ae85d81d76
OneFuzz CLI Docker container (#1831)
* OneFuzz CLI Docker container

* Update docs/OneFuzz-Docker-CLI.md

Co-authored-by: Joe Ranweiler <joe@lemma.co>

Co-authored-by: stas <statis@microsoft.com>
Co-authored-by: Joe Ranweiler <joe@lemma.co>
2022-04-22 13:53:16 -07:00