Closes#3156.
From now on, we will store additional information about containers in a table (`ContainerInformation`). This is populated when a container is created, or lazily when looking for a container we have not yet stored in the table.
Reads from this table are also cached in memory for a short period of time (currently 10 minutes).
The table is currently used to speed up lookups for containers across storage accounts; we round-robin creation of containers across multiple storage accounts to spread load, so when we read from a container we need to find what storage account it lives in.
The `ContainerInformation` table could also be used in future to store information about access control (see, e.g. #1419).
See #3316. This helps diagnosing problems when running the service.
Read all configuration variables in the service upon startup to detect problems as early as possible. This also allows us to remove a bunch of nullability annotations and makes the config easier to use, as well as enforcing stronger types like `Url` and `ResourceIdentifier`.
* Bump notify from 5.1.0 to 6.0.1 in /src/agent
Bumps [notify](https://github.com/notify-rs/notify) from 5.1.0 to 6.0.1.
- [Release notes](https://github.com/notify-rs/notify/releases)
- [Changelog](https://github.com/notify-rs/notify/blob/main/CHANGELOG.md)
- [Commits](https://github.com/notify-rs/notify/compare/notify-5.1.0...notify-6.0.1)
---
updated-dependencies:
- dependency-name: notify
dependency-type: direct:production
update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
* Update for new notify API which has renames
* Add tests for expected behaviour
* Disable crossbeam support as recommended for use with Tokio
* Fix Windows behaviour
* Pre-filter events
* Log when ignoring unknown rename event
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: George Pollard <gpollard@microsoft.com>
* Trim LibFuzzerDotNetLoader
* Trim more, update invocations
* Undo trimming
* Also reduce size of SharpFuzz
* Undo Sharpfuzz changes; they break it for some reason
Starts to address #3158.
Rather than storing the UPN and then removing it later, do not store it in the first place.
I found one problem while working on this which was that `UserInfo` was not actually stored on `Job`, so address that as well. This was because properties which are not constructor parameters are not stored by the ORM, so I've also removed other examples from `Task` and `Node` and have them only on the response types.
---
Also started to simplify the ORM roundtripping tests; remove the custom equality comparison and instead simply compare JSON-serialized values. Rely on the FsCheck automatic generation of values where it works, rather than writing a custom Arbitrary which does the same thing.
Many jobs uploading output into the same artifact causes spurious failures unless the job which consumes the artifact depends upon all of the previous jobs. So, split the uploads into separate artifacts so that the CLI job (which is the main one that fails) can depend upon only a subset of the jobs.
Also add caching to the AFL/AFLPP builds.
* Trim System.Title if length is > 128
* Format Ado.cs
* Add System.Title to Repro Steps if it needs to be trimmed
* Use Convert.ToHexString() instead of an Aggregate
* Add the full length System.Title to the Scriban template instead of inserting it into repro steps
* Upgrade transient dependency for component governance
* allow tasks environment variables to be set
* build fix
* build fix
* build fix
* another fix
* ajusting the analysis paramters
* make tools dir optional in the analysis task
* bug fix
I noticed these were getting very big and impacting our CI time due to copying huge artifacts. Presumably this would be slowing down copying in the live environment as well.
Two changes have been made:
- Use `debug=1` instead of `debug=true` (equivalent to `debug=2`); this should be sufficient for our needs
- On Linux, compress debug information after building
| Binary | Before | with `debug=1` | after compression |
|--|--:|--:|--:|
| onefuzz-agent (Linux) | 170 MB | 83 MB | 30 MB |
| onefuzz-task (Linux) | 284 MB | 134 MB | 46 MB |
| onefuzz_agent.pdb (Windows) | 89 MB | 42 MB | — |
| onefuzz_task.pdb (Windows) | 150 MB | 63 MB | — |
| onefuzz-deployment.zip | 364 MB | 286 MB | 285 MB |
Overall the compressed `release-artifacts` reduced from 374 → 297 MB.
Build speed improvements (latest build on `main` vs this PR):
| Step | Before | After |
|--|--:|--:|
| agent upload-artifact | 26s/32s/45s | 16s/15s/20s |
| package download-artifact | 1m 57s | 26 s |
| package upload-artifact | 2m 8s | 1m 35s |
* Bump tokio from 1.28.0 to 1.29.0 in /src/agent
Bumps [tokio](https://github.com/tokio-rs/tokio) from 1.28.0 to 1.29.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases)
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.28.0...tokio-1.29.0)
---
updated-dependencies:
- dependency-name: tokio
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* Update dependencies list for ARM64
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: George Pollard <gpollard@microsoft.com>
* Start event retention policy
* .
* Correlate telemetry from cli to service and out
* Traces end to end
* Linting
* .
* Fix build failures
* Trying to fix python dependency error
* .
* Lets let pip figure it out
* .
* Modified the wrong file
* .
* .
* .
* .
* .
* .
* This is the one
* fix lints?
* I _love_ python
* ...
* Undo some unnecessary changes
* Works again
* PR comments
* log exception as "error" since we are retrying anyways
* log exception when sending of webhook runs out of attempts
---------
Co-authored-by: stas <statis@microsoft.com>
