Commit Graph

1778 Commits

Author SHA1 Message Date
cd18c6066d Update clap to remove suppressions (#2856)
A Friday afternoon jaunt.

To be merged after #2855; does the remaining work to close #2277 (and closes off some things that Dependabot was trying to upgrade).

Tested by manually running the commands; we don't have good coverage for this kind of stuff. OTOH, most of these commands are for the experimental local fuzzing mode, which is not fully supported yet. I did specifically test the `onefuzz-task managed` command which is the one used in production.

## Details

- Bump `clap` to 4.1.6
  - Remove `structopt` as this is subsumed by clap now
- Bump `envlogger` to 0.10 (removes problematic dependency)
- Set `default-features=false` on `proc-maps` (removes a feature which is only needed to support FreeBSD), and bump it to 0.3

The main changes migrating `clap` are:

- `value_t!` is gone; now use `matches.get_one::<T>`. If `T` is not `String` then a parser must have been registered on the `Arg` when it was created, with `arg.value_parser(value_parser!(T))`.
- `Command::with_name` and `Arg::with_name` are now called `new`.
- `Command` and `Subcommand` were unified, and `App` is removed.
- `arg.takes_value(true)` is gone; it is the default. For flags use `arg.action(ArgAction::SetTrue)` and then retrieve the flag value with `matches.get_flag`.

This code would be simplified a lot by using the `clap::Parser` on structs, but that requires reworking the code significantly as we cannot dynamically add/remove arguments the way that this is currently done.

## Also found

Found one bug while manually testing the `onefuzz-task local` commands; see comment below.
2023-02-20 10:45:15 +13:00
1b07b7df25 Bump clap in proxy to 4.0 (#2855)
* Upgrade clap in proxy-manager
2023-02-19 20:19:51 +00:00
d5cff033d9 Remove function allowlist (#2859) 2023-02-17 20:19:38 -08:00
1ac3fd4bed Config Refactor Part 2 - Change Opt Param Names & Set File Expiry (#2835)
* Remove Old Optional Parameters and Hardcoded Values.

* Set file to expire.

* Adding expiry.

* test sleep

* Tested expiry.

* Set expirty to 24hrs.

* Syntax error.

* Formatting.

* Changing optional.

* Adding new params.

* Removing arguments.

* Removing arguments.

* Changing param names.

* Update params.
2023-02-17 16:12:17 -08:00
ddbc715b3f Remove Z3 telemetry (#2860) 2023-02-17 14:32:52 -08:00
5061a3de29 Updating Files w/ New Flags (#2861) 2023-02-17 13:23:56 -08:00
b38ab5de9e Bump arraydeque from 0.4.5 to 0.5.1 in /src/agent (#2849)
* Bump arraydeque from 0.4.5 to 0.5.1 in /src/agent

Bumps [arraydeque](https://github.com/andylokandy/arraydeque) from 0.4.5 to 0.5.1.
- [Release notes](https://github.com/andylokandy/arraydeque/releases)
- [Commits](https://github.com/andylokandy/arraydeque/commits)

---
updated-dependencies:
- dependency-name: arraydeque
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* Fixups

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: George Pollard <gpollard@microsoft.com>
2023-02-16 22:04:08 +00:00
ea90eb387b Accept optional dir of coverage test inputs (#2853) 2023-02-16 12:59:48 -08:00
572671b58e Sematically validate notification configs (#2850)
* Add new command

* Update remaining jinja templates and references to use scriban

* Add ado template validation

* Validate ado and github templates

* Remove unnecessary function

* Update src/ApiService/ApiService/OneFuzzTypes/Model.cs

Co-authored-by: Cheick Keita <kcheick@gmail.com>

---------

Co-authored-by: Cheick Keita <kcheick@gmail.com>
2023-02-16 18:08:30 +00:00
124f50ded6 Report extension errors (#2846)
Old failure message:
```
failed to launch extension
```

New failure message:

```
failed to launch extension(s): Errors for extension 'CustomScriptExtension':
:Error: ProvisioningState/failed/3 (Provisioning failed) - Failed to download all specified files. Exiting. Error Message: The remote server returned an error: (400) Bad Request.
```
2023-02-16 16:04:59 +13:00
e0e6981f75 Update "Needs Triage" label to the one we use. (#2845) 2023-02-15 23:53:40 +00:00
f3524c37aa Fix error reporting for creating a Scaleset under a missing Pool (#2844) 2023-02-15 22:32:21 +00:00
5ea0128727 Add --notification_config support for dotnet templates (#2842) 2023-02-16 10:54:58 +13:00
58f756de47 Switch over to new coverage task (#2741) 2023-02-15 12:41:19 -08:00
2d59a2a5ca Support another case when adding AssignedTo to telemetry (#2829)
* Add new command

* Catch another case
2023-02-15 19:24:11 +00:00
32cdf8b738 Sets EnableScriban to false by default at deployment time (#2839) 2023-02-15 18:58:49 +00:00
fbdba5d76b Add the new fabricBot IssueResponder. For now only trigger to add the Needs Triage label. (#2822)
Update the Sync workflow so that it does not run when labels are added or removed.
2023-02-15 10:45:58 -08:00
08bb0ec355 Update remaining jinja docs (#2838)
* Add new command

* Update remaining jinja templates and references to use scriban
2023-02-15 18:25:40 +00:00
e9f5a6a2e7 Validate scriban on new notifications (#2834)
* Add new command

* Enforce scriban at notification creation time

* fmt

* missed when merging
2023-02-15 13:16:01 -05:00
21374b36e6 Defer setting coverage breakpoints (#2832) 2023-02-14 10:52:08 -08:00
0c58a59011 Fix coverage example build (#2831) 2023-02-14 07:54:19 -08:00
bb82580a1f Add function allowlist to coverage example exe (#2830) 2023-02-14 07:25:10 -08:00
307526fb64 [Breaking Change] Deploy with scriban only turned on (#2809)
* Deploy with scriban only turned on

* Improve description
2023-02-14 13:31:39 +00:00
0bc22a8073 Fixing bug in registration.py for creating CLI SP's. (#2828)
* Fixing bug in registration.py for creating CLI SP's.

* Formatting.
2023-02-10 15:11:35 -08:00
85fe434b3e Remove stray print statement (#2823) 2023-02-10 14:31:34 -08:00
a6814f0745 Release-6.4.0 (#2799)
* Release-6.4.0
---------
Co-authored-by: Noah McGregor Harper <74685766+nharper285@users.noreply.github.com>
2023-02-10 06:42:00 -08:00
2e6eef7654 Add notification get command (#2818)
* Add new command

* lint
2023-02-09 11:30:40 -05:00
a13bb2feb3 Interact with migrations via cli (#2816)
* Interact with migrations via cli

* Move command under notifications
2023-02-08 22:02:40 +00:00
f13f52ab71 Deprecating the job template feature (#2798)
* deprecating the job template feature

* removing the code

* format
2023-02-08 19:21:03 +00:00
d732028201 Add unmanaged nodes integration tests (#2780)
* Add docker file to the runtime tools

* fixes

* bug fixes

* more bug fixes and added doc

* don;t overwrite the RUST_LOG env var

* integration test for unmanaged nodes

* add unamanged parameters to launch()

* add ing object_id

* more bug fixes

* bug fixes

* chmod on the linux files in docker

* format

* cleanup merge

* added test_unmanaged command

* cleanup

* use a single image for the docker compose
remove images after the test

* docs and formatting

* format

* format

* format and bug fixes

* using windows server

* fix linux container
make the base image a paramter on windows
use the windows server base image on  windows server

* format

* bug fix

* more fixes

* allow reboot

* more fixes

* added more logging around the service principal creation

* format

* more logging

* change restart policy

* fix multi tenant domain

* more fixes

* exit instead of reboot when running inside docker

* remove comment

* build fix

* try_exist instead of exist

* save the docker logs

* bug_fix

* adding timeout

* fix timeout logic

* adding a build profile

* make all agent depend on the first one

* remove profile

* another fix

* restart agent 1

* Update docs/unmnaged-nodes.md

Co-authored-by: Teo Voinea <58236992+tevoinea@users.noreply.github.com>

---------

Co-authored-by: Teo Voinea <58236992+tevoinea@users.noreply.github.com>
2023-02-08 11:07:19 -08:00
f93c75556d Resolving bug in config refactor. (#2811)
* Resolving bug in config refactor.

* fixing isort.

* Reformatting api.

* debugging.

* fixing backend.

* Adding condition to param retrieval.

* Adding back save.

* Fixing condition check.

* Formatting.
2023-02-07 18:13:29 -08:00
fed7e6939d Update azure_* packages in agent (#2807)
* Bump azure_storage_blobs from 0.7.0 to 0.9.0 in /src/agent

Bumps [azure_storage_blobs](https://github.com/azure/azure-sdk-for-rust) from 0.7.0 to 0.9.0.
- [Release notes](https://github.com/azure/azure-sdk-for-rust/releases)
- [Changelog](https://github.com/Azure/azure-sdk-for-rust/blob/main/HISTORY.md)
- [Commits](https://github.com/azure/azure-sdk-for-rust/commits)

---
updated-dependencies:
- dependency-name: azure_storage_blobs
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* Update other Azure packages

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: George Pollard <gpollard@microsoft.com>
2023-02-07 22:12:56 +00:00
2ab07ef21b Upgrade openssl. (#2812)
https://rustsec.org/advisories/RUSTSEC-2023-0006
https://rustsec.org/advisories/RUSTSEC-2023-0007
https://rustsec.org/advisories/RUSTSEC-2023-0009
https://rustsec.org/advisories/RUSTSEC-2023-0010
2023-02-07 20:50:06 +00:00
c52b0bab79 Bump quick-xml from 0.26.0 to 0.27.1 in /src/agent (#2808)
Bumps [quick-xml](https://github.com/tafia/quick-xml) from 0.26.0 to 0.27.1.
- [Release notes](https://github.com/tafia/quick-xml/releases)
- [Changelog](https://github.com/tafia/quick-xml/blob/master/Changelog.md)
- [Commits](https://github.com/tafia/quick-xml/compare/v0.26.0...v0.27.1)

---
updated-dependencies:
- dependency-name: quick-xml
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-02-08 09:00:31 +13:00
c5700b4a34 Bump os_pipe from 1.0.0 to 1.1.3 in /src/agent (#2795)
Bumps [os_pipe](https://github.com/oconnor663/os_pipe.rs) from 1.0.0 to 1.1.3.
- [Release notes](https://github.com/oconnor663/os_pipe.rs/releases)
- [Commits](https://github.com/oconnor663/os_pipe.rs/compare/1.0.0...1.1.3)

---
updated-dependencies:
- dependency-name: os_pipe
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-02-07 12:50:47 +13:00
d6fed9dc7d Bump num_cpus from 1.13.1 to 1.15.0 in /src/agent (#2717)
Bumps [num_cpus](https://github.com/seanmonstar/num_cpus) from 1.13.1 to 1.15.0.
- [Release notes](https://github.com/seanmonstar/num_cpus/releases)
- [Changelog](https://github.com/seanmonstar/num_cpus/blob/master/CHANGELOG.md)
- [Commits](https://github.com/seanmonstar/num_cpus/compare/v1.13.1...v1.15.0)

---
updated-dependencies:
- dependency-name: num_cpus
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-02-06 20:39:31 +00:00
80e9b737e8 Add readonly_inputs to dotnet & dotnet_dll templates (#2740)
Closes #2739
2023-02-06 20:15:21 +00:00
c1f6dfc366 Validate scriban from cli (#2800)
* Add validate scriban endpoint to cli

* missed a file

* Lint -- I miss C#

* docs
2023-02-06 08:32:49 -05:00
8f418f692c fix default value of analyzer_exe (#2797) 2023-02-03 00:32:33 +00:00
3ccb599c19 Update tokio to fix RUSTSEC-2023-0005 (#2796)
Fixing build failure, see: https://rustsec.org/advisories/RUSTSEC-2023-0005
2023-02-02 23:49:27 +00:00
949c579deb Add cobertura output to coverage example (#2777)
This makes it very easy to generate Cobertura reports.

Replace `--source` flag argument with `--output={mod-off, source, cobertura}` (default is still `mod-off`).
2023-02-02 22:12:08 +00:00
73fac638d1 Scriban to jinja migration script (#2787)
* .

* fmt

* Testing time

* .

* fmt

* More tests

* More tests

* Cleanup

---------

Co-authored-by: George Pollard <gpollard@microsoft.com>
2023-02-02 21:29:55 +00:00
840ced57df Check agent binary dependencies in CI (#2792)
Check the list of dynamic dependencies of the agent binaries to ensure we don't accidentally take a dependency on some new or unknown library.
2023-02-02 16:12:26 -05:00
457eb1e906 Adding Notification Failure Webhook. (#2628)
* Adding Notification Failure Webhook.

* Typo.

* Add await.
2023-02-02 10:31:13 -08:00
291f9dfbcf Fix query to get he existing proxy (#2791) 2023-02-01 20:49:14 +00:00
f209c0b2f4 Trim output from monitored process before logging (#2782) 2023-02-01 02:06:54 +00:00
be43032bf5 Fixes & improvements to Expand behaviour (#2789)
- Recursively expand Paths before canonicalization; fixes #2387
- Detect and error on infinitely-expanding variables (due to self-reference or mutual recursion); this fixes an existing problem with list expansion as well
- Detect and report references to unknown replacements (e.g. typoes)
- Detect and report references to _known_ replacements which are not available

---

There is a behavioural change here which is that values which were previously unknown such as `{x}` were unexpanded; from now on these will be errors. It's possible that something somewhere is relying on this.
2023-01-31 23:55:53 +00:00
f402304084 Config Refactor Round 2. (#2771)
* Config Refactor Round 2.

* Adding docs.

* Fix file formatting.

* Removing.

* fixing imports.

* Removing.

* Fixing cli access token retrieval.

* Fixing authority check.

* Small edits.

* Removing duplicate.

* Adding uuid check.

* Possible to override with existing params.

* Allowing flags to override storage.

* Trying to fix config params.?

* Fixing.

* Set endpoint params via app function.

* Checking changes to params.

* Make tenant_domain default.

* Remove endoint params from models.

* UPdating docs.

* Setting

* Removing hardcoded values.

* Typo.

* Removing endpoint upload.

* Typo.

* Fixing typos.

* Fix error message about aad tenant.

* Responding to comments.

* Update src/ApiService/ApiService/UserCredentials.cs

Co-authored-by: Marc Greisen <mgreisen@microsoft.com>

---------

Co-authored-by: Marc Greisen <mgreisen@microsoft.com>
2023-01-31 15:03:38 -08:00
3d2fb65c14 Truncate for queue as well + bug fix (#2788) 2023-01-31 17:41:59 +00:00
bc57fa016c Add dockerfile to the runtime tools (#2730)
* Add docker file to the runtime tools

* fixes

* bug fixes

* more bug fixes and added doc

* don;t overwrite the RUST_LOG env var

* integration test for unmanaged nodes

* add unamanged parameters to launch()

* add ing object_id

* more bug fixes

* bug fixes

* chmod on the linux files in docker

* format

* revert changes in  integration tests

* Apply suggestions from code review

Co-authored-by: Marc Greisen <mgreisen@microsoft.com>

* format and bug fix

* fix condition

---------

Co-authored-by: Marc Greisen <mgreisen@microsoft.com>
2023-01-30 20:33:11 -08:00