2095 Commits

Author SHA1 Message Date
Adam
36a54f9139 Redo 8.7.1 (#3469)
* Redo-8.7.1-hotfix

---------

Co-authored-by: Cheick Keita <chkeita@microsoft.com>
2023-10-03 11:51:39 -07:00
Kanan B
2a88838160
Kanan/semantic validation (#3386)
* Throw exceptoins for missing area/iteration paths

* Comment out feature flag check for work item creation

* Revert "Comment out feature flag check for work item creation"

This reverts commit ad961694535d06ca1f607fe261b3624f36ad1166.

* Add some extra info to the exceptions to distinguish them

* Add logging when a validation error is encountered

* Add call to validate in NotificationsTest for testing

* Fix area and iteration path validation

* Update error messages

* Fix ArgumentNullException

* Add extra information to path check failure

* Request classification nodes with depth equal to number of parts in the path

* Pass tree structure as parameter instead of always using Areas

* Remove '[PAT]' from error messages

* Require project name in Area/IterationPath during validation

* PR comments

* Sneak in a one-line fix for empty comments

* Change context messsage in response from notiifcation test
2023-08-28 15:14:50 -07:00
Noah McGregor Harper
10bb79a08c
Deleting AutoScale table entries on vmss shutdown. (#3455)
* Deleting AutoScale table entries on vmss shutdown.

* FIx.

* Trying again

* Fixing delete.

* Updating autoscale to orm

* Removing comments.
2023-08-28 14:53:32 -07:00
Cheick Keita
c4d0cd754b
Revert the location of the setupFolder (#3460)
* Revert the location of the setupFolder

* install llvm version 16.0.6

* print version

* install llvm before building fuzzing tools

* revert changes in the script
2023-08-28 13:49:10 -07:00
Cheick Keita
34dc7e5e7d
install LLVm 16 before running dotnet-fuzzing-tools.ps1 (#3462) 2023-08-28 20:43:26 +00:00
Cheick Keita
ead63eb3ff
Remove the retention policy setting (#3452)
* Remove the configuration of the autoscale diagnostic

* only remove the retention policy

* set retention policy to false

* remove settings

* remove retentionPolicy from bicep

* format
2023-08-25 23:35:06 +00:00
Noah McGregor Harper
c49442facd
Use language features. (#3456) 2023-08-25 13:36:36 -07:00
Justin Moore
38cfa9772e
Downgrade some debug logs from warn to debug (#3450) 2023-08-25 16:41:27 +00:00
Noah McGregor Harper
6d8be1a834
Remove JobResult Retry Logging (#3439)
* Replace JobResult error with warning.

* Make log statement.

* Leaving exception.
2023-08-21 11:30:33 -07:00
Cheick Keita
ea4a1abb8a
Fix local path generation (#3432)
* Fix local path generation

* format

* fix test
2023-08-21 09:39:54 -07:00
Teo Voinea
8fc2f833fa
Capture required args and make it match cli (#3429)
* Capture required args and make it match cli

* Match libfuzzer basic template
2023-08-21 13:37:22 +00:00
Teo Voinea
a5517099e1
Only keep coverage, libfuzzer and template commands (#3428)
* Only keep coverage, libfuzzer and template commands

* Remove unused code
2023-08-21 13:16:17 +00:00
Adam
c0c5b8324f
Release 8.7.0 (#3403)
* Release-8.7.0
2023-08-18 13:11:26 -07:00
Noah McGregor Harper
804b2993b7
Job Result Round #2 (#3380)
* Attempting new background process.

* Formatting.

* Adding JobResultOperations.

* Fixing.

* Adding strorage tests.

* Still trying.

* Adding.

* Adding to rest of queue.

* Adding arg.

* Fixing.

* Updating job result.

* Adding cargo lock.

* Restoring queuetaskheartbeat.

* Fixing job format.

* Properly naming function.

* Adding to program.cs.

* Removing uning unneccessary code.

* More small fixes.

* Adding regular crash results.

* Resolving issues.

* More fixes.

* Debugging crashing input.

* Adding

* Remove

* Fixed.

* Handling other cases.

* Adding value to data.

* Adding values to queue messages.

* Adidng values.

* Adidng await.

* Adding runtimestats.

* Fixing runtime_stats.

* Updating types.

* Fixing types.

* Resolve.

* Resolve type.

* Updading onefuzz-result cargo.

* Responding to comments.

* More comments.

* Removing unused params.

* Respond to more comments.

* Updating cargo.

* Fixing return type.

* UPdating keys."

* UPdating return type.

* Updating JobResult with constructor.

* Remove debug logging.

* Adding warning.

* Making generic into TaskContext.

* Adding crash dump.

* Updating for crash dumps.

* Formatting.

* Updating to warn.

* Formatting.

* Formatting.

* Borrowing new client.
2023-08-18 10:01:32 -07:00
Teo Voinea
bdb2f1337d
Add mariner support (#3306)
* Updating setup.sh

* logger works

* Install omi

* syntax

* Add option to create mariner pool in checkpr

* .

* Need to install sudo

* .

* Include both logging extensions

* Revert because we already isntall the azure monitor for linux extension

* Downgrade type handler version

* Add data collection rules and fuzzing articacts for integration test

* TODOs

* Fix linux jobs getting sent to mariner

* Fix linux jobs going to marienr pool

* Fix pools

* Remove the old logging extension on linux

* try to retain syslog

* Value to be set was not clear

* Trying to route logs

* Maybe we need to specify properties even if we don't set anything

* Start adding a data collection association

* Create association

* update packages.lock

* .

* Remove auto config and add dependency map

* Bring back GCS autoconfig

* Undo DCR stuff

* Undo package version bump

* Fix up files

* Remove TODO
2023-08-18 13:52:53 +00:00
Adam
4e2b3fc648
Update default windows image for windows 11 (#3374)
* Update all win10 image references to win11 as default
2023-08-17 19:36:08 +00:00
Teo Voinea
98e986e3e7
Tevoinea/migrate task types to template (#3397)
* Add analysis task

* generic report

* generator task

* Libfuzzer crash report

* Add libfuzzer merge

* libfuzzer regression

* Add libfuzzer test input

* Cleanup

* Update schema

* Some light docs and example templates

* Add testinput task

* Reorganize the code

* Cleanup

* Improve test, update model
2023-08-17 13:36:38 +00:00
George Pollard
b45446ac52
Make crashdumps optional in config (#3409) 2023-08-16 21:33:20 -07:00
Cheick Keita
9420f519fa
Deserialize the coverage files directly into the output files (#3410)
* deserialize the coverage files directly into the output files

* reduce the amount of data structure cloning
individual save function for each formats

* cleanup

* unformatted json

* bring back spawn

* remove stray comment

* build fix

* fix build
2023-08-15 13:03:20 -07:00
George Pollard
9c4bc3eed6
Bump to Rust 1.71.1 (#3401) 2023-08-10 21:08:27 +00:00
George Pollard
020c4d5caa
Cargo updates (#3400)
* Cargo updates

* Fix build
2023-08-10 20:42:12 +00:00
George Pollard
ec6ce0f1b0
Switch to NSubstitute (#3402) 2023-08-10 09:04:30 -04:00
George Pollard
a364051923
Capture crash dumps from libfuzzer, when provided (#2793)
Enables capturing crashdumps generated by ASAN at point of failure.

This helps in several ways:
- provides a crash dump in the case that we cannot repro a failure later
- some people have stated that crash dumps would be more useful to their team than the repro VM
- we should be able to use these for automated submission to Watson or similar

---

Crash dumps are automatically collected (for libfuzzer) jobs, if we find any. They should be activated by enabling crash dumps in ASAN, via:

- On Linux: `ASAN_OPTIONS=disable_coredump=0:abort_on_error=1:unmap_shadow_on_exit=1`
  - OneFuzz will override the (Ubuntu) crash dump options to generate core dumps instead and then upload them.
- On Windows: `ASAN_SAVE_DUMPS=my_dump.dmp`
  - OneFuzz will look for any `*.dmp` files in the working directory and then upload them.

In both cases, the crash dump will be renamed to match the crashing input, if possible, and uploaded to a new `crashdumps` container.

---

Also updated: the “simple” LibFuzzer test has been updated to be compiled with `cl.exe` instead of `clang` on Windows, so that we are exercising the MSVC implementation of ASAN/LibFuzzer, and the CI image has been updated to `windows-2022`. The restriction to an old version of the Windows SDK has been removed.
2023-08-09 21:55:27 +00:00
dependabot[bot]
74ae105074
Bump symbolic from 10.2.0 to 12.1.1 in /src/agent (#3161)
* Bump symbolic from 10.2.0 to 12.1.1 in /src/agent

Bumps [symbolic](https://github.com/getsentry/symbolic) from 10.2.0 to 12.1.1.
- [Release notes](https://github.com/getsentry/symbolic/releases)
- [Changelog](https://github.com/getsentry/symbolic/blob/master/CHANGELOG.md)
- [Commits](https://github.com/getsentry/symbolic/compare/10.2.0...12.1.1)

---
updated-dependencies:
- dependency-name: symbolic
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

* Update backtrace to remove dupe gimli

* Update goblin to remove dupe

* Update symbolic again

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: George Pollard <gpollard@microsoft.com>
2023-08-09 21:27:14 +00:00
Adam
ef0cfbfc35
Update Azure CLI packages and pin azure-core to 1.28.0 to fix Az login bug (#3396)
* Updating Azure core and cli packages
2023-08-09 14:01:01 -07:00
dependabot[bot]
9c0a34452a
Bump base64 from 0.13.0 to 0.21.0 in /src/agent (#2870)
* Bump base64 from 0.13.0 to 0.21.0 in /src/agent

Bumps [base64](https://github.com/marshallpierce/rust-base64) from 0.13.0 to 0.21.0.
- [Release notes](https://github.com/marshallpierce/rust-base64/releases)
- [Changelog](https://github.com/marshallpierce/rust-base64/blob/master/RELEASE-NOTES.md)
- [Commits](https://github.com/marshallpierce/rust-base64/compare/v0.13.0...v0.21.0)

---
updated-dependencies:
- dependency-name: base64
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* Update to new API

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: George Pollard <gpollard@microsoft.com>
2023-08-09 00:28:28 +00:00
dependabot[bot]
6c38b7d1a6
Bump crossterm from 0.26.1 to 0.27.0 in /src/agent (#3391)
Bumps [crossterm](https://github.com/crossterm-rs/crossterm) from 0.26.1 to 0.27.0.
- [Release notes](https://github.com/crossterm-rs/crossterm/releases)
- [Changelog](https://github.com/crossterm-rs/crossterm/blob/master/CHANGELOG.md)
- [Commits](https://github.com/crossterm-rs/crossterm/compare/0.26.1...0.27.0)

---
updated-dependencies:
- dependency-name: crossterm
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-08-09 12:00:02 +12:00
George Pollard
58447f72f6
Migrate from winapi to windows-rs (#3050)
`windows-rs` is the newer, Microsoft-supported version of the API bindings; `winapi` hasn't been updated in some time. This allows us to remove some code, as `windows-rs` includes the `Sym*` functions that we had to previously defined ourselves in `dbghelp`.

Rather than port the `jobs` and `com` code I removed it, as it was unused.

The `check_hr` and `check_winapi` macros have been replaced by use of `.ok()?` on `BOOL` and `?` on `HRESULT` which now support standard error handling facilities.
2023-08-08 21:13:42 +00:00
Justin Moore
3851ab0877
Tolerate errors when attempting to fetch metadata for some CreateKinds (#3393)
Tolerate errors when attempting to fetch metadata for `CreateKind::Any` or `CreateKind::Other`
2023-08-09 08:49:29 +12:00
George Pollard
8980988fe0
Update Rust edition to 2021 across all crates (#3392) 2023-08-08 03:12:33 +00:00
George Pollard
d53646e474
Add snapshot test support for coverage implementation (#3368)
Add a snapshot-based test to our coverage implementation. This adds coverage for a bunch of code which did not previously have it, and allows us to easily see that coverage is being generated as expected.

A fix was made to the recording code to remove the use of `TargetAllowList` as it was easy to misconfigure `CoverageRecorder` when using it. The use of `TargetAllowList` is now only a container struct in our generic coverage task.
2023-08-07 17:31:22 -07:00
Stas
d464e34a6f
fix tag shadowing (#3388)
Co-authored-by: stas <statis@microsoft.com>
2023-08-03 17:21:36 -07:00
Teo Voinea
c6b91470d4
Update feature-flags.bicep (#3387) 2023-08-03 23:08:09 +00:00
Adam
01c18994bb
Release 8.6.3 (Hotfix) (#3385)
* Release-8.6.3-hotfix

---------

Co-authored-by: Teo Voinea <58236992+tevoinea@users.noreply.github.com>
2023-08-03 22:44:35 +00:00
Teo Voinea
ab189b1c85
Handle another duplicate field (#3383)
* Handle another duplicate field

* Make case insensitive
2023-08-03 11:44:17 -07:00
dependabot[bot]
dde7a09694
Bump certifi in /contrib/deploy-onefuzz-via-azure-devops (#3346)
Bumps [certifi](https://github.com/certifi/python-certifi) from 2023.5.7 to 2023.7.22.
- [Commits](https://github.com/certifi/python-certifi/compare/2023.05.07...2023.07.22)

---
updated-dependencies:
- dependency-name: certifi
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Adam <103067949+AdamL-Microsoft@users.noreply.github.com>
2023-08-02 16:12:09 -07:00
Adam
1beb8887e9
Release 8.6.2 (Hotfix) (#3375)
* Release-8.6.2-Hotfix

---------

Co-authored-by: Teo Voinea <58236992+tevoinea@users.noreply.github.com>
Co-authored-by: Cheick Keita <chkeita@microsoft.com>
2023-08-02 11:14:05 -07:00
Cheick Keita
4c6fb7a962
Forcefully shutdown the runtime before existing the main task (#3378) 2023-08-02 09:36:58 -07:00
George Pollard
bcf3cf941a
Remove srcview (#3376) 2023-08-01 15:14:49 -07:00
Teo Voinea
cfbcb166d8
Refactor AdoTemplate rendering (#3370)
* .

* Add tests and complete refactor

* Remove done todo

* Log the query we created
2023-08-01 15:01:31 -04:00
Teo Voinea
55aa7e5c02
Improve error output in libfuzzer verification (#3355) 2023-07-31 13:26:18 +00:00
Joe Ranweiler
733247ce5b
Add missing winapi features (#3351) 2023-07-30 23:56:27 +00:00
Adam
09fabb7793
Release 8.6.1 (Hotfix) (#3361)
* Release-8.6.1
---------
Co-authored-by: Teo Voinea <58236992+tevoinea@users.noreply.github.com>
2023-07-28 17:27:32 -07:00
Teo Voinea
02b74c62a5
Requeue ado notification on ff (#3358)
* Requeue ado notification on ff

* Fix test
2023-07-28 17:48:47 -04:00
Teo Voinea
895b8da886
Feature flag work item creation (#3353) 2023-07-28 13:27:15 -04:00
Adam
f144544844
Release 8.6.0 (#3348)
* Release 8.6.0
2023-07-27 20:50:31 +00:00
Cheick Keita
8cb761a565
Implement with_tasks option to expand the task information (#3343)
* Implement with_tasks option to expand the task information

* Added tests

* format
2023-07-26 16:00:53 -07:00
Noah McGregor Harper
fc4e698f9b
Replacing 'Upgrade' flag with 'Skip AAD'. (#3345)
* Replacing 'Upgrade' flag with 'Skip AAD'.

* Format.

* Just skipping role assignment."

* Fixing role.

* Updating help.
2023-07-26 11:26:20 -07:00
Kanan B
ed4c02a8e8
Look for azcopy.exe in AZCOPY if it's a directory (#3344)
* Look for azcopy.exe in AZCOPY if it's a directory

* Make file check in AZCOPY dir more concise

Co-authored-by: George Pollard <gpollard@microsoft.com>

* Add logic to support searching for azcopy on linux

---------

Co-authored-by: George Pollard <gpollard@microsoft.com>
2023-07-26 09:00:41 -07:00
Teo Voinea
6ea2faa0b0
Populate InputBlob when possible (#3342)
* Populate inputblob when possible

* PR comment
2023-07-26 12:56:13 +00:00