- [x] ability to specify a retention period on a container, which applies to newly-created blobs
- [x] specify default retention periods in templates from CLI side
There's a small breaking change to the Python JobHelper class.
* Fix sed checks for CLI versioning
* Fix.
* Fix.
* Changing build_cli
* Trying greater than
* Tring once more.
* Trying major minor
* trying to replace major minor
* Using major minor
Enables capturing crashdumps generated by ASAN at point of failure.
This helps in several ways:
- provides a crash dump in the case that we cannot repro a failure later
- some people have stated that crash dumps would be more useful to their team than the repro VM
- we should be able to use these for automated submission to Watson or similar
---
Crash dumps are automatically collected (for libfuzzer) jobs, if we find any. They should be activated by enabling crash dumps in ASAN, via:
- On Linux: `ASAN_OPTIONS=disable_coredump=0:abort_on_error=1:unmap_shadow_on_exit=1`
- OneFuzz will override the (Ubuntu) crash dump options to generate core dumps instead and then upload them.
- On Windows: `ASAN_SAVE_DUMPS=my_dump.dmp`
- OneFuzz will look for any `*.dmp` files in the working directory and then upload them.
In both cases, the crash dump will be renamed to match the crashing input, if possible, and uploaded to a new `crashdumps` container.
---
Also updated: the “simple” LibFuzzer test has been updated to be compiled with `cl.exe` instead of `clang` on Windows, so that we are exercising the MSVC implementation of ASAN/LibFuzzer, and the CI image has been updated to `windows-2022`. The restriction to an old version of the Windows SDK has been removed.
* Look for azcopy.exe in AZCOPY if it's a directory
* Make file check in AZCOPY dir more concise
Co-authored-by: George Pollard <gpollard@microsoft.com>
* Add logic to support searching for azcopy on linux
---------
Co-authored-by: George Pollard <gpollard@microsoft.com>
* Add support for regression reports to 'repro get_files'
* Make get_files logic more clear
* Format api.py with black
* Change crash_info default values for mypy
* Set crash_info['input_blob_container'] as a Container instead of string
* Add logic for handling a missing 'original_crash_test_result' or missing 'crash_report's
* Adjust code format in get_files
* Update logic to comply with mypy
* Fix missing dict index operation
* allow tasks environment variables to be set
* build fix
* build fix
* build fix
* another fix
* ajusting the analysis paramters
* make tools dir optional in the analysis task
* bug fix
* Start event retention policy
* .
* Correlate telemetry from cli to service and out
* Traces end to end
* Linting
* .
* Fix build failures
* Trying to fix python dependency error
* .
* Lets let pip figure it out
* .
* Modified the wrong file
* .
* .
* .
* .
* .
* .
* This is the one
* fix lints?
* I _love_ python
* ...
* Undo some unnecessary changes
* Works again
* PR comments
## Summary of the Pull Request
- **Breaking** (but as far as I know this feature is not yet in use): rename the `extra_container` to `extra_setup_container`.
- **Add**: the `extra_output_container`, which pushes its outputs continually.
- We may also want a type of container which both pushes & pulls? See discussion below.
- **Improved**: if `onefuzz-task` fails upon launch, we will log its output for diagnosis (might close#3113)
---
Some thoughts for the future:
We might want to redesign the containers so that we have something like the following which is passed to the agent, and the agent doesn't need to know the specifics of the containers supplied:
```jsonc
{
// ...
"containers": {
"extra_setup_dir": {
"mode": "pull",
"container_name": "yyy",
},
"extra_output_dir": {
"mode": "push",
"continuous": true, // keep pushing while job is running
"container_name": "xxx"
}
}
}
```
At the moment the agent needs to know what each container is for, for each task type. A more generic and flexible method might be simpler overall.
Scaleset names are now permitted to be any (valid) strings, instead of only GUIDs. When we generate a scaleset name it is now based upon the pool name; for example the pool `pool` might get a scaleset named `pool-3b24ba211cad4b078655914754485838`.
This should be backwards-compatible since GUIDs are [already serialized to table storage as strings](dddcfa4949/src/ApiService/ApiService/onefuzzlib/orm/EntityConverter.cs (L190-L191)), so this simply loosens the restrictions placed upon them.
Scaleset IDs now have a strong type in the same way as other IDs; this helps to avoid mixing them up with other strings. Because of this I found one bug in the scaleset search query logic due to Pool ID/VMSS ID confusion. As part of fixing this I've changed the scaleset search query to only return nodes from the table rather than querying Azure to find a list; this seems to be sufficient for the CLI.
* Start reworking events
* Trying things out
* .
* Add versioning, create events container in deploy script
* Improve gitignore
* Making the types line up
* Lets see how it works
* .
* Add CLI for querying event
* Linting
* Update src/ApiService/ApiService/Functions/Events.cs
Co-authored-by: George Pollard <porges@porg.es>
* almost done testing
* Added test
---------
Co-authored-by: George Pollard <porges@porg.es>
Because `--check_fuzzer_help` is a positive flag (defaults to `True`), there is no way to change it to `False`, because specifying it on the command line sets it to `True`. Change the flag to a negative one instead, named `--no_check_fuzzer_help`.
### Context
The original `libfuzzer dotnet` job template was a proof of concept that demonstrated how the `libfuzzer_fuzz` task could be used to express fuzzing via the (pre SharpFuzz 2.0) `libfuzzer-dotnet` tool. It (and its associated integration test) used a harness that linked an older version of SharpFuzz, and which is incompatible with LibFuzzerDotnetLoader (which requires SharpFuzz 2.0 or greater).
### Changes
- Rename `libfuzzer dotnet_dll` job template to `libfuzzer dotnet`, making it the _only_ `libfuzzer-dotnet` template
- Remove integration tests and docs for the old proof-of-concept job type
### Notice
This is a breaking change.
Closes#2874.
* Add docker file to the runtime tools
* fixes
* bug fixes
* more bug fixes and added doc
* don;t overwrite the RUST_LOG env var
* integration test for unmanaged nodes
* add unamanged parameters to launch()
* add ing object_id
* more bug fixes
* bug fixes
* chmod on the linux files in docker
* format
* cleanup merge
* added test_unmanaged command
* cleanup
* use a single image for the docker compose
remove images after the test
* docs and formatting
* format
* format
* format and bug fixes
* using windows server
* fix linux container
make the base image a paramter on windows
use the windows server base image on windows server
* format
* bug fix
* more fixes
* allow reboot
* more fixes
* added more logging around the service principal creation
* format
* more logging
* change restart policy
* fix multi tenant domain
* more fixes
* exit instead of reboot when running inside docker
* remove comment
* build fix
* try_exist instead of exist
* save the docker logs
* bug_fix
* adding timeout
* fix timeout logic
* adding a build profile
* make all agent depend on the first one
* remove profile
* another fix
* restart agent 1
* Update docs/unmnaged-nodes.md
Co-authored-by: Teo Voinea <58236992+tevoinea@users.noreply.github.com>
---------
Co-authored-by: Teo Voinea <58236992+tevoinea@users.noreply.github.com>
