- [x] ability to specify a retention period on a container, which applies to newly-created blobs
- [x] specify default retention periods in templates from CLI side
There's a small breaking change to the Python JobHelper class.
* Starting integration tests
* Ready to test the test
* Parametrize test
* checkpoint
* Test works
* Run integration tests in pipeline
* fmt
* .
* -p
* Install clang
* quotes not required in yaml?
* Hopefully fixed windows?
* Try without killondrop
* lint
* small test
* another test
* Reuse core name
* Wrong step
* bump tokio?
* Try with rust
* make build happy
* Bump pete and small clean up
* Clean up and make the test pass regularly
* fix broken ci
* Lower the poll timeout
* Set the timeout in a nicer way
* fix windows
* fmt
* Include and copy pdbs
* Ignore if pdb is missing on linux
* It takes too long for coverage to be generated
* lint
* Only warn on missing coverage since it's flaky
* Fix windows build
* Small clean up
* Try lowering the poll delay
* fix coverage
* PR comments
* .
* Apparently make is missing?
* Remove aggressive step skipping in CI
* Updating setup.sh
* logger works
* Install omi
* syntax
* Add option to create mariner pool in checkpr
* .
* Need to install sudo
* .
* Include both logging extensions
* Revert because we already isntall the azure monitor for linux extension
* Downgrade type handler version
* Add data collection rules and fuzzing articacts for integration test
* TODOs
* Fix linux jobs getting sent to mariner
* Fix linux jobs going to marienr pool
* Fix pools
* Remove the old logging extension on linux
* try to retain syslog
* Value to be set was not clear
* Trying to route logs
* Maybe we need to specify properties even if we don't set anything
* Start adding a data collection association
* Create association
* update packages.lock
* .
* Remove auto config and add dependency map
* Bring back GCS autoconfig
* Undo DCR stuff
* Undo package version bump
* Fix up files
* Remove TODO
Enables capturing crashdumps generated by ASAN at point of failure.
This helps in several ways:
- provides a crash dump in the case that we cannot repro a failure later
- some people have stated that crash dumps would be more useful to their team than the repro VM
- we should be able to use these for automated submission to Watson or similar
---
Crash dumps are automatically collected (for libfuzzer) jobs, if we find any. They should be activated by enabling crash dumps in ASAN, via:
- On Linux: `ASAN_OPTIONS=disable_coredump=0:abort_on_error=1:unmap_shadow_on_exit=1`
- OneFuzz will override the (Ubuntu) crash dump options to generate core dumps instead and then upload them.
- On Windows: `ASAN_SAVE_DUMPS=my_dump.dmp`
- OneFuzz will look for any `*.dmp` files in the working directory and then upload them.
In both cases, the crash dump will be renamed to match the crashing input, if possible, and uploaded to a new `crashdumps` container.
---
Also updated: the “simple” LibFuzzer test has been updated to be compiled with `cl.exe` instead of `clang` on Windows, so that we are exercising the MSVC implementation of ASAN/LibFuzzer, and the CI image has been updated to `windows-2022`. The restriction to an old version of the Windows SDK has been removed.
Many jobs uploading output into the same artifact causes spurious failures unless the job which consumes the artifact depends upon all of the previous jobs. So, split the uploads into separate artifacts so that the CLI job (which is the main one that fails) can depend upon only a subset of the jobs.
Also add caching to the AFL/AFLPP builds.
* Start event retention policy
* .
* Correlate telemetry from cli to service and out
* Traces end to end
* Linting
* .
* Fix build failures
* Trying to fix python dependency error
* .
* Lets let pip figure it out
* .
* Modified the wrong file
* .
* .
* .
* .
* .
* .
* This is the one
* fix lints?
* I _love_ python
* ...
* Undo some unnecessary changes
* Works again
* PR comments
Caches are getting too big and we are exceeding the 10GB limit, leading to cache churning.
1. Try to make the caches smaller by using `Swatinem/rust-cache`, which is smarter about what gets cached.
- After doing this it turns out we don't really need `sccache` any more, it has very little impact upon compile times as the cache hit ratio is low. So remove it, to reduce complexity of build and size of build caches.
2. Also fix artifact caching which had been broken by a version format change (4956cf5406).
The coverage code is not currently exercised by any test.
* Add a test to the `coverage` example so that it can run in PR builds.
* Specify `--all-targets` so that example tests are run.
* Install and use [`nextest`](https://nexte.st/) instead of the standard test runner.
* This will parallelize test runs across binaries so the overall test run is faster.
* Make sleep duration of agent configurable and reduce it in the test run so that it doesn't wait for 30 seconds.
Rather than specifying this in `ci.yml`, specify it in the `global.json` file. This allows us to share the version amongst several workflows (and ADO pipelines) without needing to synchronize them.
### Context
The original `libfuzzer dotnet` job template was a proof of concept that demonstrated how the `libfuzzer_fuzz` task could be used to express fuzzing via the (pre SharpFuzz 2.0) `libfuzzer-dotnet` tool. It (and its associated integration test) used a harness that linked an older version of SharpFuzz, and which is incompatible with LibFuzzerDotnetLoader (which requires SharpFuzz 2.0 or greater).
### Changes
- Rename `libfuzzer dotnet_dll` job template to `libfuzzer dotnet`, making it the _only_ `libfuzzer-dotnet` template
- Remove integration tests and docs for the old proof-of-concept job type
### Notice
This is a breaking change.
Closes#2874.
Using [`cargo-deny`](https://embarkstudios.github.io/cargo-deny/) to ensure that disallowed dependencies removed in #2423 do not accidentally make their way back in.
`cargo-deny` subsumes the `cargo-audit` functionality, so switch to the `cargo-deny` version.
Setting this up required explicitly stating the license which was not in some of our `Cargo.toml` files.
Node.js 12 actions are deprecated, as well as `::set-output`
- Update `actions/checkout` to v3
- Update `actions/upload-artifact` to v3
- Update `actions/setup-python` to v4
- Update `actions/cache` to v3
- `set-output` on stdout is deprecated, update to `$GITHOUT_OUTPUT` method
- Change from `actions-rs/toolchain` (unsupported) to `dtolnay/rust-toolchain`
The only warning remaining after this is the one about Ubuntu 18.04.
To comply with Microsoft security policies, we must use only approved crypto libraries.
Notably, `ring` is not on the approved libraries list. We should use the platform crypto libraries instead. The official guidance is:
- on Windows, schannel and friends
- on Linux, the OpenSSL 1.1.1 series if available, otherwise OpenSSL 3.0. OpenSSL _must_ be dynamically and not statically linked so that distro-provided updates can be used.
Following these guidelines strictly would mean that we have to build distro/version specific binaries for Linux. Instead we have an exception which allows us to statically link against the OpenSSL 1.1.1 series, allowing us to have a (somewhat) portable binary.
OpenSSL 1.1.1. can be statically linked by switching to the `native-tls-vendored` feature of `reqwest`. (Verified by inspection that `ring` and `rustls` are no longer in `Cargo.lock` files.)
So the result is:
- on Windows, we use schannel
- on Linux we statically link against 1.1.1
It is up to us to ensure we keep up-to-date with OpenSSL issues and releases. The version can be checked by looking at the version of `openssl-src` that we depend upon. Currently this is `111.22.0+1.1.1q`. This is the latest version; also check the [Vulnerabilities page](https://www.openssl.org/news/vulnerabilities-1.1.1.html).
---
Verified by inspection that `rustls` and `ring` are no longer present in `Cargo.lock`.
Current dynamic library dependencies are:
```console
$ ldd ./onefuzz-agent
linux-vdso.so.1 (0x00007ffd9ba60000)
libunwind.so.8 => /lib/x86_64-linux-gnu/libunwind.so.8 (0x00007ff5ab5e5000)
librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007ff5ac1d1000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007ff5ac1cc000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007ff5ab4fe000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007ff5ac1c7000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007ff5ab2d4000)
/lib64/ld-linux-x86-64.so.2 (0x00007ff5ac1dc000)
liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007ff5ab2a9000)
```
Use Codecov to show coverage reports, so we get highlighted versions of the files where it is easy to see missing coverage.
- Setup Rust coverage using [`cargo-llvm-cov`](https://github.com/taiki-e/cargo-llvm-cov).
- Add the `ci/agent.sh` build script to the agent artifact cache key, since it wasn't there before.
- Don't run Rust tests in `--release` mode (have been meaning to change this so doing it at the same time).
There is some subtlety about putting the coverage result into the cached agent artifact, so that when we reuse the agent artifact we can still upload the coverage information for it to Codecov. Without this it would look like the coverage had dropped.
