* Add min_available_memory_mb to cli and service configs to be passed to agent
* Add min_available_memory_mb to task create and template parameters
* Add logging to indicate the configured min_available_memory
* Add missing parameter
* Remove unnecessary import
* Update webhook_events.py
* test change
* original version of docs.
* Update webhook_events.py again
* Don't start memory checking task when min_available_memory_mb is zero
* cargo fmt
* Remove comments
* Add min_available_memory_mb to libfuzzer's _create_tasks()
* Add a megabyte converted value for min available memory logs
* Use float instead of int for division
---------
Co-authored-by: Noah Harper <noharper@microsoft.com>
* Ignore regression update when the work item is in some states
* format
* formatting
* don't hide messages in the poison queue
* fix typo
* update regression logic
update test_template to support regression
* build fix
* mypy fix
* build fix
* move regression ignore state under ADODuplicateTemplate
* replace extend with append
* update set_tcp_keepalive
* mke mypy happy
* copy ADODuplicateTemplate.OnDuplicate.RegressionIgnoreStates
* Add field to ado config for checking duplicate work items
* Make duplicate fields nullable and add it to python models
* Update broken tests
* Update docs to include new ado_duplicate_fields property
Enables capturing crashdumps generated by ASAN at point of failure.
This helps in several ways:
- provides a crash dump in the case that we cannot repro a failure later
- some people have stated that crash dumps would be more useful to their team than the repro VM
- we should be able to use these for automated submission to Watson or similar
---
Crash dumps are automatically collected (for libfuzzer) jobs, if we find any. They should be activated by enabling crash dumps in ASAN, via:
- On Linux: `ASAN_OPTIONS=disable_coredump=0:abort_on_error=1:unmap_shadow_on_exit=1`
- OneFuzz will override the (Ubuntu) crash dump options to generate core dumps instead and then upload them.
- On Windows: `ASAN_SAVE_DUMPS=my_dump.dmp`
- OneFuzz will look for any `*.dmp` files in the working directory and then upload them.
In both cases, the crash dump will be renamed to match the crashing input, if possible, and uploaded to a new `crashdumps` container.
---
Also updated: the “simple” LibFuzzer test has been updated to be compiled with `cl.exe` instead of `clang` on Windows, so that we are exercising the MSVC implementation of ASAN/LibFuzzer, and the CI image has been updated to `windows-2022`. The restriction to an old version of the Windows SDK has been removed.
* allow tasks environment variables to be set
* build fix
* build fix
* build fix
* another fix
* ajusting the analysis paramters
* make tools dir optional in the analysis task
* bug fix
## Summary of the Pull Request
- **Breaking** (but as far as I know this feature is not yet in use): rename the `extra_container` to `extra_setup_container`.
- **Add**: the `extra_output_container`, which pushes its outputs continually.
- We may also want a type of container which both pushes & pulls? See discussion below.
- **Improved**: if `onefuzz-task` fails upon launch, we will log its output for diagnosis (might close#3113)
---
Some thoughts for the future:
We might want to redesign the containers so that we have something like the following which is passed to the agent, and the agent doesn't need to know the specifics of the containers supplied:
```jsonc
{
// ...
"containers": {
"extra_setup_dir": {
"mode": "pull",
"container_name": "yyy",
},
"extra_output_dir": {
"mode": "push",
"continuous": true, // keep pushing while job is running
"container_name": "xxx"
}
}
}
```
At the moment the agent needs to know what each container is for, for each task type. A more generic and flexible method might be simpler overall.
Scaleset names are now permitted to be any (valid) strings, instead of only GUIDs. When we generate a scaleset name it is now based upon the pool name; for example the pool `pool` might get a scaleset named `pool-3b24ba211cad4b078655914754485838`.
This should be backwards-compatible since GUIDs are [already serialized to table storage as strings](dddcfa4949/src/ApiService/ApiService/onefuzzlib/orm/EntityConverter.cs (L190-L191)), so this simply loosens the restrictions placed upon them.
Scaleset IDs now have a strong type in the same way as other IDs; this helps to avoid mixing them up with other strings. Because of this I found one bug in the scaleset search query logic due to Pool ID/VMSS ID confusion. As part of fixing this I've changed the scaleset search query to only return nodes from the table rather than querying Azure to find a list; this seems to be sufficient for the CLI.
* WIP: Adding a validation command to the agent
* introducing a ValidationConfig
* refactoring
* adding verification code
* remove unused test
* format
* update dependencies
* adding a command to get the loading logs
* add print logs for linux
* clippy fix
* clippy on windows
* renaming stuff
* bug fix
- Turn `ImageReference` into its own type so it is validated early on in request submission time, and we don't end up with malformed IDs, etc.
- Add in support for shared image galleries since that was easy enough to add while I'm doing this.
- Explicitly document which image sources are permitted and how to reference them with resource IDs.
This addresses/closes #1464 for the C# port. Also fixes#2927 which was recently reported.
* .
* It doesn't work yet but we're making progress
* Added graceful shutdown and tests
* Small fix
* Fix crate issues
* test fix
* Fix build
* make clippy happy
* The order changed
* Use timeout in kill
* Almost done shutting down ipc
* It should all work now
* Update deny.toml
* Fix warning
* Revert "Create 2 way IPC connection between agent and task (#2886)"
This reverts commit 091c870be6.
* Temporarily allow vulnerability since a new one just came out
* Temporarily allow vulnerability
* Update proxy.sh
* Update agent.sh
* Update deny.toml
* .
* It doesn't work yet but we're making progress
* Added graceful shutdown and tests
* Small fix
* Fix crate issues
* test fix
* Fix build
* make clippy happy
* The order changed
* Use timeout in kill
### Context
The original `libfuzzer dotnet` job template was a proof of concept that demonstrated how the `libfuzzer_fuzz` task could be used to express fuzzing via the (pre SharpFuzz 2.0) `libfuzzer-dotnet` tool. It (and its associated integration test) used a harness that linked an older version of SharpFuzz, and which is incompatible with LibFuzzerDotnetLoader (which requires SharpFuzz 2.0 or greater).
### Changes
- Rename `libfuzzer dotnet_dll` job template to `libfuzzer dotnet`, making it the _only_ `libfuzzer-dotnet` template
- Remove integration tests and docs for the old proof-of-concept job type
### Notice
This is a breaking change.
Closes#2874.
* Add docker file to the runtime tools
* fixes
* bug fixes
* more bug fixes and added doc
* don;t overwrite the RUST_LOG env var
* integration test for unmanaged nodes
* add unamanged parameters to launch()
* add ing object_id
* more bug fixes
* bug fixes
* chmod on the linux files in docker
* format
* cleanup merge
* added test_unmanaged command
* cleanup
* use a single image for the docker compose
remove images after the test
* docs and formatting
* format
* format
* format and bug fixes
* using windows server
* fix linux container
make the base image a paramter on windows
use the windows server base image on windows server
* format
* bug fix
* more fixes
* allow reboot
* more fixes
* added more logging around the service principal creation
* format
* more logging
* change restart policy
* fix multi tenant domain
* more fixes
* exit instead of reboot when running inside docker
* remove comment
* build fix
* try_exist instead of exist
* save the docker logs
* bug_fix
* adding timeout
* fix timeout logic
* adding a build profile
* make all agent depend on the first one
* remove profile
* another fix
* restart agent 1
* Update docs/unmnaged-nodes.md
Co-authored-by: Teo Voinea <58236992+tevoinea@users.noreply.github.com>
---------
Co-authored-by: Teo Voinea <58236992+tevoinea@users.noreply.github.com>
* Add docker file to the runtime tools
* fixes
* bug fixes
* more bug fixes and added doc
* don;t overwrite the RUST_LOG env var
* integration test for unmanaged nodes
* add unamanged parameters to launch()
* add ing object_id
* more bug fixes
* bug fixes
* chmod on the linux files in docker
* format
* revert changes in integration tests
* Apply suggestions from code review
Co-authored-by: Marc Greisen <mgreisen@microsoft.com>
* format and bug fix
* fix condition
---------
Co-authored-by: Marc Greisen <mgreisen@microsoft.com>
* enable running dot-net function on Windows to allow attaching remote debugger
* rename from 'use_windows' to 'host_dotnet_on_windows'
* instructions
* reformat deploy.py
Co-authored-by: stas <statis@microsoft.com>
