53 Commits

Author SHA1 Message Date
George Pollard
aa54a15427
Add extra_output container, rename extra container (#3064)
## Summary of the Pull Request

- **Breaking** (but as far as I know this feature is not yet in use): rename the `extra_container` to `extra_setup_container`.
- **Add**: the `extra_output_container`, which pushes its outputs continually.
  - We may also want a type of container which both pushes & pulls? See discussion below.
- **Improved**: if `onefuzz-task` fails upon launch, we will log its output for diagnosis (might close #3113)

---

Some thoughts for the future:

We might want to redesign the containers so that we have something like the following which is passed to the agent, and the agent doesn't need to know the specifics of the containers supplied:

```jsonc
{
    // ...
    "containers": {
        "extra_setup_dir": {
            "mode": "pull",
            "container_name": "yyy",
        },
        "extra_output_dir": {
            "mode": "push",
            "continuous": true, // keep pushing while job is running
            "container_name": "xxx"
        }
    }
}
```

At the moment the agent needs to know what each container is for, for each task type. A more generic and flexible method might be simpler overall.
2023-06-15 02:48:27 +00:00
George Pollard
f4ab22d570
Checkout compiler-rt sparsely (#3101) 2023-05-15 10:52:01 -07:00
George Pollard
256f261f2c
Parse .NET exception stacks in crash output (#2988) 2023-04-18 10:50:02 +12:00
Adam
3adb2fee36
Update integration test pool size (#2935)
* Upping VMSS count from 10 -> 20 on default tests
2023-03-23 16:26:46 -07:00
Cheick Keita
3d299ce51e
fix extra container intilization (#2887)
* fix extra container intilization

* fix extra url download

* fix extra dir parameter to the agent

* rename extra to extra_dir
2023-02-27 13:57:32 -08:00
Cheick Keita
b84896802c
Adding extra container to tasks (#2847)
* adding extra container to tasks

* setup expand

* build fix

* generate docs

* build fix

* build fix

* build fix

* format

* format

* build fix

* fix extra container references

* format

* Update "Needs Triage" label to the one we use. (#2845)

* Report extension errors (#2846)

Old failure message:
```
failed to launch extension
```

New failure message:

```
failed to launch extension(s): Errors for extension 'CustomScriptExtension':
:Error: ProvisioningState/failed/3 (Provisioning failed) - Failed to download all specified files. Exiting. Error Message: The remote server returned an error: (400) Bad Request.
```

* Sematically validate notification configs (#2850)

* Add new command

* Update remaining jinja templates and references to use scriban

* Add ado template validation

* Validate ado and github templates

* Remove unnecessary function

* Update src/ApiService/ApiService/OneFuzzTypes/Model.cs

Co-authored-by: Cheick Keita <kcheick@gmail.com>

---------

Co-authored-by: Cheick Keita <kcheick@gmail.com>

* adding extra container to integration tests

* adding doc

* update tests

* format

* build and clippy fix

* Update src/agent/onefuzz-task/src/tasks/report/generic.rs

Co-authored-by: Teo Voinea <58236992+tevoinea@users.noreply.github.com>

---------

Co-authored-by: Marc Greisen <mgreisen@microsoft.com>
Co-authored-by: George Pollard <gpollard@microsoft.com>
Co-authored-by: Teo Voinea <58236992+tevoinea@users.noreply.github.com>
2023-02-23 19:08:01 +00:00
George Pollard
6cd53b1db6
Fix .NET integration tests (#2879) 2023-02-22 12:05:45 +13:00
Joe Ranweiler
4caaf8fc32
Remove old libfuzzer dotnet template (#2875)
### Context
The original `libfuzzer dotnet` job template was a proof of concept that demonstrated how the `libfuzzer_fuzz` task could be used to express fuzzing via the (pre SharpFuzz 2.0) `libfuzzer-dotnet` tool. It (and its associated integration test) used a harness that linked an older version of SharpFuzz, and which is incompatible with LibFuzzerDotnetLoader (which requires SharpFuzz 2.0 or greater).

### Changes
- Rename `libfuzzer dotnet_dll` job template to `libfuzzer dotnet`, making it the _only_ `libfuzzer-dotnet` template
- Remove integration tests and docs for the old proof-of-concept job type

### Notice

This is a breaking change.

Closes #2874.
2023-02-21 10:54:36 +13:00
Cheick Keita
d732028201
Add unmanaged nodes integration tests (#2780)
* Add docker file to the runtime tools

* fixes

* bug fixes

* more bug fixes and added doc

* don;t overwrite the RUST_LOG env var

* integration test for unmanaged nodes

* add unamanged parameters to launch()

* add ing object_id

* more bug fixes

* bug fixes

* chmod on the linux files in docker

* format

* cleanup merge

* added test_unmanaged command

* cleanup

* use a single image for the docker compose
remove images after the test

* docs and formatting

* format

* format

* format and bug fixes

* using windows server

* fix linux container
make the base image a paramter on windows
use the windows server base image on  windows server

* format

* bug fix

* more fixes

* allow reboot

* more fixes

* added more logging around the service principal creation

* format

* more logging

* change restart policy

* fix multi tenant domain

* more fixes

* exit instead of reboot when running inside docker

* remove comment

* build fix

* try_exist instead of exist

* save the docker logs

* bug_fix

* adding timeout

* fix timeout logic

* adding a build profile

* make all agent depend on the first one

* remove profile

* another fix

* restart agent 1

* Update docs/unmnaged-nodes.md

Co-authored-by: Teo Voinea <58236992+tevoinea@users.noreply.github.com>

---------

Co-authored-by: Teo Voinea <58236992+tevoinea@users.noreply.github.com>
2023-02-08 11:07:19 -08:00
Noah McGregor Harper
7fc6fc3668
Reverting Enable/Disable App Function Changes - Force -net App Function To Only Run Agent Functions (#2660)
* Testing app function.

* Addding functions back.

* Fixing name.

* updating to string.

* Fixing bicep.

* Fixing function names.

* Update src/ApiService/ApiService/Functions/ReproVmss.cs

Co-authored-by: George Pollard <gpollard@microsoft.com>

* Adding route specifier.

* Update src/ApiService/ApiService/Functions/NodeAddSshKey.cs

Co-authored-by: George Pollard <gpollard@microsoft.com>

* Update src/ApiService/ApiService/Functions/WebhookLogs.cs

Co-authored-by: George Pollard <gpollard@microsoft.com>

* Update src/ApiService/ApiService/Functions/WebhookPing.cs

Co-authored-by: George Pollard <gpollard@microsoft.com>

* Fixing function names.

* Increasing pool_size

* Adding logging statements.

* Adding check for transient failure.

* Adding check to else.

* Fix var

* Adding retry logic to dbg_lnx

* FIxing retry.

* Maybe fixed integration test.

* Formatting.

* Adding try and fail logic.

* Changing error.

* Refactoring logic.

* Fixing logic.

* Logic.

* Removing bad comparison.

* Fixing.

* All ready.

* Switching back to call.

* Add.

* Fixing

* Adding quit back.

* Reversing cli changes.

* Removing bad file.

* Formatting.

Co-authored-by: George Pollard <gpollard@microsoft.com>
2022-12-02 11:14:13 -08:00
Noah McGregor Harper
e0634a3365
Deploy update (#2650)
* Remove Python App Function Deployment Code and  Code.

* Updating yml and zip names.

* Fixing ci.yml.

* Typo.

* Format

* Trying to remove python service.

* Updating directories.

* Removing flag.

* Format.

* Fixng api-service-net ref.

* Re-add requirement.

* Fixing refs in bicep.

* Specifying version.

* Removing dotnet refs in integration tests.

* Updating role assignment naming convention.

* Adding ignore.

* Update src/deny.toml

Co-authored-by: George Pollard <porges@porg.es>

* Update version.

* Removing onefuzztypes dependency.

* Switch app service plan to windows.

* Update test version.

* Changing version.

* Returning version.

* Trying to add onefuzz types back.

* Force pipenv version.

* Fix.

* Syntax.

* Renaming.

* Trying different version.

* Removing build step.

* Fixing bicep parameter.

* Retrying run with older version.

* Trying pipenv with another version.

* Forcing pipenv version in tox.

* Adding pipenv fix and updating version.

Co-authored-by: George Pollard <porges@porg.es>
2022-11-23 17:48:30 -08:00
George Pollard
894dcc62be
Update to .NET 7 (#2615)
Update service code to .NET 7.0.

Also update the version installed onto agent nodes via setup scripts to 7.0.100.
2022-11-18 10:39:45 +13:00
George Pollard
e50bc0b389
Update the libfuzzer-with-options test to use fuzzing_target_options (#2565)
This currently causes the coverage task to time out, ensure that the `runs` parameter is only passed to the fuzzing task and not the coverage task.
2022-10-26 00:03:28 +00:00
Joe Ranweiler
ee0cbd70a0
Enable dotnet fuzzing (#2273)
Add a new CLI job template, `libfuzzer dotnet_dll`, and supporting server-side definitions.
2022-10-19 14:17:27 -07:00
George Pollard
02869eba9e
Fix #1285 (#2353) 2022-09-08 08:36:38 +12:00
George Pollard
bd4dfdc592
Update chrono, remove time to fix audit failures (#2285)
Closes #1366 and incidentally closes #1266.

Requires using a not-yet-merged modification to the latest version of `appinsights-rs` to remove the `time` feature from the `chrono` dependency (https://github.com/dmolokanov/appinsights-rs/pull/280). 

There are changes to use the new (tokio-based) version of `appinsights-rs`, e.g., made `set_appinsights_clients` async to ensure it is always called from an async context, since the constructor for appinsights now invokes `Tokio::spawn`.
2022-09-07 00:34:31 +00:00
Noah McGregor Harper
163e324227
Specifying initial size for scaleset create in integration-tests. (#2307)
* Specifying initial size for scaleset create in integration-tests.

* Spell check.
2022-08-26 16:09:36 -07:00
Cheick Keita
1a63f195f3
porting the proxy state machine (#2286)
* porting the proxy state machine

* use async version

* is used

* rename base state to state

* fix auth
fix extension path

* ignore log info in check logs
2022-08-25 19:42:01 +00:00
George Pollard
a3f1d59f70
Set autocrlf by default, update affected files (#2261) 2022-08-17 13:07:20 +12:00
George Pollard
c653f2e8ef
Add support for dotnet_endpoint/dotnet_functions to integration-test.py (#2192)
This will allow testing client-facing .NET Functions during integration tests (`check-pr`).
2022-07-26 14:44:59 +12:00
Joe Ranweiler
69b11f60c2
Add GoodBad C# example project, integration test (#2148) 2022-07-12 12:07:30 -07:00
Teo Voinea
9cbbf86e36
Add the setup state to the unavailable list (#1731)
* Add the setup state to the unavailable list

* Add the init state to the unavailable list

* Fix integration test

* Wrong import
2022-04-01 14:00:25 -04:00
Joe Ranweiler
ce36de72e7
Generate debuginfo for windows-libfuzzer-load-library test target (#1684) 2022-03-11 09:40:51 -08:00
Marc Greisen
e2bd878f59
Release 5.1.0 (#1681)
* Release 5.1.0

* Update CHANGELOG.md

Co-authored-by: Joe Ranweiler <joe@lemma.co>

* Update CHANGELOG.md

Co-authored-by: Joe Ranweiler <joe@lemma.co>

* Update CHANGELOG.md

Co-authored-by: Joe Ranweiler <joe@lemma.co>

* Prevent deletion of the repro VM on failure for debugging.

Co-authored-by: Joe Ranweiler <joe@lemma.co>
2022-03-03 15:55:36 -08:00
Cheick Keita
50bac07d19
increase the polling period in functional tests (#1682)
* increase the polling period

* update timeout to 30s

* format
2022-03-01 18:10:02 -08:00
Cheick Keita
674444b7d7
Split integration tests into different steps (#1650)
Refactoring check-pr.py to extract the logic of downloading the binaries
refactoring integration-tets.py to split the logic of setup, launch, check_result and cleanup
2022-02-22 22:33:00 +00:00
Cheick Keita
ee4dfe922f
Allow authority to be specified in check-pr (#1473)
* fix interactive login in check-pr

* specify tenant domain in check-pr

* bug fix

* rename tenant to authority
2022-02-02 16:06:42 -08:00
Noah McGregor Harper
d358938895
Working in retry to test endpoint during integration test. (#1565) 2022-01-27 11:00:22 -08:00
Noah McGregor Harper
1de2cc841d
Check-pr creates and uses SP. (#1504)
* Check-pr creates and uses SP.

* flake8.

* flake8.

* Fixing var name.

* Fixing deploy.py

* Looking for client_secret

* Change to check_output.

* mypy fix.

* Fixing check-pr

* working version.

* lint

* Updating arg text.

* Removing redundant functionality.

* Changing register codepath and adding flag.

* Removing pycache file.

* Fixing unattended flag.

* Adding space.

* Fixing a few calls.

* Removing file.

* Removing python3.

* Removing old file.

* Adding wait into registration.py

* Formatting registration.py.

* Removing space.

* Adding retry logic to check-pr.

* Formatting.

* Retriggering.

* Retriggering.

* Calling sp_create and adding retry to authorize.

* Fixing syntax.

* Removing comments.

* Adding another retry.

* Retriggering.

* Retriggering.

* Retriggering.

* Trying to find error.

* Adding retry logic.

* Increasing sleep.

* Fixing formatting.

* Retriggering.

* Removing bad file.

* Trying out retry for logger.

* typevar issue?

* Re-adding.

* Retriggering.

* retriggering.

* Retriggering.

Co-authored-by: nharper285 <nharper285@gmail.com>
Co-authored-by: Cheick Keita <kcheick@gmail.com>
2021-12-17 09:44:04 -08:00
bmc-msft
09bdad4979
work around issues between released version of cargo-fuzz and nightly (#1284)
See rust-fuzz/cargo-fuzz#276

Co-authored-by: Brian Caswell <bmc@shmoo.com>
2021-09-27 11:29:31 -04:00
bmc-msft
9413377c90
handle fake crash reports generated by debugging tools in regression tasks (#1233) 2021-09-22 17:16:35 +00:00
bmc-msft
041d3cc204
ignore table exists errors from integration testing at the azure functions layer (#1096) 2021-07-21 13:29:13 -04:00
bmc-msft
e065b571f7
add format string bug to example integration test (#1087) 2021-07-20 05:00:45 -04:00
bmc-msft
3305732160
increase libfuzzer integration test -runs (#1081) 2021-07-16 17:29:37 -04:00
bmc-msft
1121ebc6e8
fix issue specifying target_options for libfuzzer targets (#1066) 2021-07-14 14:38:35 -04:00
bmc-msft
b9950c5526
update log messages to ease debugging (#988) 2021-06-14 15:18:03 -04:00
bmc-msft
cf3d904940
address formatting from black 21.4b0 (#831) 2021-04-26 12:35:16 -04:00
bmc-msft
794400adf1
ignore benign error messages when checking integration test logs (#775) 2021-04-06 15:35:54 -04:00
bmc-msft
ca12904684
add log checking to refactored integration check (#700)
In practice, Application Insights can take up to 3 minutes before something sent to it is available via KQL.

This PR logs a start and stop marker such that the integration tests only search for logs during the integration tests. This reduces the complexity when using the integration tests during the development process.

Note: this migrated the new functionality from #356 into the latest integration test tools.
2021-04-02 21:49:19 +00:00
bmc-msft
2417c37a60
add LoadLibrary integration test (#689) 2021-03-19 09:46:26 -04:00
bmc-msft
6e60a8cf10
add regression testing tasks (#664) 2021-03-18 15:37:19 -04:00
bmc-msft
f41f110af8
libfuzzer library integration tests (#681) 2021-03-17 16:18:10 -04:00
bmc-msft
f6a426cc07
enable long-running integration tests (#654) 2021-03-10 17:03:15 -05:00
bmc-msft
7fc725d012
add non-x86_64 architecture libfuzzer target support using qemu-user (#600) 2021-03-03 19:06:50 -05:00
bmc-msft
cc5965ebbf
add .gitignore to ignore libfuzzer-dotnet build artifacts (#564) 2021-02-19 09:32:26 +00:00
bmc-msft
f64a0dcc05
lint integration-test.py (#549) 2021-02-16 12:22:45 -05:00
bmc-msft
933fe6850c
libfuzzer-dotnet integration (#535) 2021-02-11 17:30:24 -05:00
bmc-msft
4f941d5ff2
add the task_id to the integration failure logs (#446) 2021-01-21 13:35:54 -05:00
bmc-msft
af2ef9f4fa
verify local debugger installs before launching repro (#436)
This only launches the repro test cases if you have the os-specific prerequisite debugger installed.
2021-01-18 23:30:53 +00:00
bmc-msft
75d2ffd7f4
lint test utils (#395) 2021-01-05 08:50:52 -05:00