Commit Graph

2050 Commits

Author SHA1 Message Date
85437df253 Bump PyGithub (#2569)
Co-authored-by: Teo Voinea <Teodor.Voinea@microsoft.com>
2022-10-27 16:04:52 +00:00
76cde40234 Better logging with webhooks (#2568)
Co-authored-by: Teo Voinea <Teodor.Voinea@microsoft.com>
2022-10-27 11:53:22 -04:00
c086f249d7 fix path onfuzz config path (#2567) 2022-10-26 16:22:05 -07:00
49ca5b5248 ETag should be updated in Update and Replace as well (#2562) 2022-10-26 22:42:34 +00:00
9299357133 Don't log an error if we delete a Repro and it is already missing (#2563) 2022-10-26 22:04:42 +00:00
81e4b1542d Include computer name in AgentRegistration & decode Instance ID from it (#2557)
Addressing #2550 but in a slightly different way:

1. Always include the machine name in the agent registration message. From the machine name we can extract the instance ID.
2. Use the stored value in preference to fetching it every time.
3. To back-fill existing nodes that do not have the value stored, update it in AcquireScaleInProtection if it is not already present.
2022-10-27 10:01:17 +13:00
0e8876b9f1 Exponential backoff for failed notifications (#2555)
* .

* 🧹

* Do backoff on main queue, kick off to poison queue after sufficient retries

* PR comments

Co-authored-by: Teo Voinea <Teodor.Voinea@microsoft.com>
2022-10-26 12:46:07 -04:00
e50bc0b389 Update the libfuzzer-with-options test to use fuzzing_target_options (#2565)
This currently causes the coverage task to time out, ensure that the `runs` parameter is only passed to the fuzzing task and not the coverage task.
2022-10-26 00:03:28 +00:00
72d0c6f25e Use the vitual env folder to store the config if it exists (#2561)
* Use the vitual env folder to store the config if it exists

* address PR comment

* fix

* mypy fix

* mypy fix
2022-10-25 16:30:33 -07:00
59f4402eb0 Extend default coverage timeout to 120s (#2556)
As of #2529 we now fail if the coverage recording times out. The default timeout (5s) does not appear to be long enough for even simple tasks to run, so extend it to 120s.
2022-10-25 00:43:10 +00:00
db1a712ed3 Lower log level on region overlap (#2559) 2022-10-24 16:15:02 -07:00
cafba18c85 fix onefuzz status pool (#2551) 2022-10-21 15:18:46 -07:00
f302e31ed2 Downgrade instance not found exception (#2549)
* Don't log exception when node no longer exists in azure

* fmt

Co-authored-by: Teo Voinea <Teodor.Voinea@microsoft.com>
2022-10-21 15:17:05 -04:00
9600c5c873 fix exception message formatting (#2546)
Co-authored-by: stas <statis@microsoft.com>
2022-10-21 15:32:57 +00:00
d10740ab4c fix onefuzz address (#2543) 2022-10-20 17:02:07 -07:00
66dc7f591d Handle nulls better in scaleset shutdown (#2538)
* Handle nulls better

* Populate data if it's not

* fmt

* Fix bug where scaleset won't go to 0

Co-authored-by: Teo Voinea <Teodor.Voinea@microsoft.com>
2022-10-20 23:47:28 +00:00
f313a37c9c Ignoring the scanning log file when reporting an issue with azcopy (#2536)
* Ignoring the scanning log file when reporting an issue with azcopy

* fix comment
2022-10-20 22:18:35 +00:00
e83a18be41 Fix notifications bugs (#2534)
* Fix bug where field key is not serialized properly

* Log if we match multiple active WIs, re-open if we only find duplicates

* Wrap up testing

* Update naming to clarify duplicates are talking about work items, not notification config

Co-authored-by: Teo Voinea <Teodor.Voinea@microsoft.com>
2022-10-20 13:26:20 -04:00
c10cb08fcd Remove use of unapproved crypto libraries (#2423)
To comply with Microsoft security policies, we must use only approved crypto libraries.

Notably, `ring` is not on the approved libraries list. We should use the platform crypto libraries instead. The official guidance is:

- on Windows, schannel and friends
- on Linux, the OpenSSL 1.1.1 series if available, otherwise OpenSSL 3.0. OpenSSL _must_ be dynamically and not statically linked so that distro-provided updates can be used.

Following these guidelines strictly would mean that we have to build distro/version specific binaries for Linux. Instead we have an exception which allows us to statically link against the OpenSSL 1.1.1 series, allowing us to have a (somewhat) portable binary.

OpenSSL 1.1.1. can be statically linked by switching to the `native-tls-vendored` feature of `reqwest`.  (Verified by inspection that `ring` and `rustls` are no longer in `Cargo.lock` files.)

So the result is:

- on Windows, we use schannel
- on Linux we statically link against 1.1.1

It is up to us to ensure we keep up-to-date with OpenSSL issues and releases. The version can be checked by looking at the version of `openssl-src` that we depend upon. Currently this is `111.22.0+1.1.1q`. This is the latest version; also check the [Vulnerabilities page](https://www.openssl.org/news/vulnerabilities-1.1.1.html).

--- 

Verified by inspection that `rustls` and `ring` are no longer present in `Cargo.lock`.

Current dynamic library dependencies are:

```console
$ ldd ./onefuzz-agent 
        linux-vdso.so.1 (0x00007ffd9ba60000)
        libunwind.so.8 => /lib/x86_64-linux-gnu/libunwind.so.8 (0x00007ff5ab5e5000)
        librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007ff5ac1d1000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007ff5ac1cc000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007ff5ab4fe000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007ff5ac1c7000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007ff5ab2d4000)
        /lib64/ld-linux-x86-64.so.2 (0x00007ff5ac1dc000)
        liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007ff5ab2a9000)
```
2022-10-19 22:39:00 +00:00
ee0cbd70a0 Enable dotnet fuzzing (#2273)
Add a new CLI job template, `libfuzzer dotnet_dll`, and supporting server-side definitions.
2022-10-19 14:17:27 -07:00
b88f46779e Remove suggestion to reset IterationPath upon duplicate (#2533)
* Remove suggestion to reset IterationPath upon duplicate

* Update highlighting to jsonc
2022-10-19 20:34:16 +00:00
c46c6bee39 Produce an error if coverage recording failed due to timeout (#2529)
Closes #2520.

After investigation the underlying cause here is that the process is getting killed due to timeout, but we don't report the timeout or produce an error. Modify the coverage code so that it fails if timeout is hit.
2022-10-18 21:48:36 +00:00
b3fd6d5ed0 Bump process_control from 3.1.0 to 4.0.0 in /src/agent (#2377)
* Bump process_control from 3.1.0 to 4.0.0 in /src/agent

Bumps [process_control](https://github.com/dylni/process_control) from 3.1.0 to 4.0.0.
- [Release notes](https://github.com/dylni/process_control/releases)
- [Commits](https://github.com/dylni/process_control/compare/3.1.0...4.0.0)

---
updated-dependencies:
- dependency-name: process_control
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

* Update code

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: George Pollard <gpollard@microsoft.com>
2022-10-18 15:57:08 +13:00
d36ecf9b46 Release 5.16.0 (#2516)
* Release 5.16.0

* Improve scaling protection logging (#2518)

* Improve scaling protection logging

* Fix a bug where we're always protecting from scaling actions

* We never want to protect from scaleset actions

Co-authored-by: Noah McGregor Harper <74685766+nharper285@users.noreply.github.com>
2022-10-17 12:25:44 -07:00
d95342b8b8 Add deleteAll to the orm (#2519)
* Add deleteAll to the orm

* address pr comments
2022-10-17 10:43:51 -07:00
8a4b46e4f0 Improve scaling protection logging (#2518)
* Improve scaling protection logging

* Fix a bug where we're always protecting from scaling actions

* We never want to protect from scaleset actions
2022-10-14 12:50:25 -04:00
3c35b3445a Convert ADO traces to customEvents and update notificationInfo (#2508)
* Adding Project and Name to ADO Notification.

* Convert ADO traces to events and update notificationInfo.

* Updating event formatting.

* Fix typos.

* Fixing event name.

* Update the logic update the scaleinProtection (#2505)

* Update the logic update the scaleinProtection

* build fix

* Add SrcView::try_insert function (#2510)

* Add SrcView::try_insert function

* Make try_insert return bool indicating success

* Rustfmt

* Set test to be ignored (no compile)

* Bump srcview version

Co-authored-by: Sutton Bradley <suttonb@microsoft.com>
Co-authored-by: George Pollard <gpollard@microsoft.com>

* Remove old code.

* Updating filename

* Updating event types.

* Fixing logging.

* Removing tag from event name.

* Setting new item.

* remove extra code.

Co-authored-by: Cheick Keita <chkeita@microsoft.com>
Co-authored-by: Sutton Bradley <sbradley4646@gmail.com>
Co-authored-by: Sutton Bradley <suttonb@microsoft.com>
Co-authored-by: George Pollard <gpollard@microsoft.com>
2022-10-13 15:58:40 -07:00
51b8dc05b1 Add $schema to cgmanifest.json (#2320)
Co-authored-by: Jamie Magee <jamie.magee@microsoft.com>
2022-10-13 20:49:09 +00:00
f37e9db299 Fix github issue query (#2514) 2022-10-13 15:35:52 +00:00
58b3d6f3cb Fix github client header value (#2513) 2022-10-13 07:25:29 -07:00
542cf75c1e Add SrcView::try_insert function (#2510)
* Add SrcView::try_insert function

* Make try_insert return bool indicating success

* Rustfmt

* Set test to be ignored (no compile)

* Bump srcview version

Co-authored-by: Sutton Bradley <suttonb@microsoft.com>
Co-authored-by: George Pollard <gpollard@microsoft.com>
2022-10-12 21:50:22 +00:00
d519f0a44b Update the logic update the scaleinProtection (#2505)
* Update the logic update the scaleinProtection

* build fix
2022-10-12 11:04:08 -07:00
9e5006fc40 Update the logic for checking if a blob exist before uploading (#2503)
* Update the logic for checking if a blob exist before uploading

* remove unused ref

* update the status code check
2022-10-12 03:03:12 +00:00
0fab487d76 Only fetch VM InstanceView data when required (#2506)
* Only fetch VM InstanceView data when reuqired

* Missed one
2022-10-12 15:34:27 +13:00
7596cf4bd0 Update Rust azure_* dependencies (#2500) 2022-10-12 10:09:42 +13:00
71a915c9cd Add logo to readme (#2340)
* Add logo to readme
2022-10-11 02:06:19 +00:00
94b6fe0a7c change the verbosity of the azcopy logs (#2498) 2022-10-11 01:25:39 +00:00
76caef47e2 Python: Upgrade rsa dependency (#2470)
* Upgrade `rsa` dependency

* Adjust version to match Github advisory
2022-10-11 13:43:19 +13:00
41fc3b22c0 Update Logging Statement to Warning and Include Http Info (#2484)
* Convert Error to Warning on Poll State Transition Failure.

* Adding Result information.
2022-10-10 10:49:41 -07:00
cdc104f966 Update documentation to specify appropriate RID (#2490)
Closes #2457
2022-10-07 01:03:18 +00:00
053153fa44 Release 5.15.1 (#2493) 2022-10-06 11:56:10 -07:00
80b1122a8d Undo tab change from previous PR (#2492) 2022-10-05 17:32:33 +00:00
124c62756c Migrating notification templates (#2486)
* Add jinja template migration

* Support migrating our most common jinja templates to scriban on the fly

* Fix tests
2022-10-05 14:48:20 +00:00
f6a680bb3d Attempt to fix integration-tests-linux failure (#2487) 2022-10-05 00:29:08 +00:00
7809b40e74 Update to Rust 1.64 (#2488)
[Release notes.](https://blog.rust-lang.org/2022/09/22/Rust-1.64.0.html)

Probably one of the more important things is:
> performance improvements of 10-20% for compiling Rust code on Windows
2022-10-04 22:31:53 +00:00
7529a184cf Release 5.15.0 (#2474)
* Release 5.15.0

* Update CHANGELOG.md

Co-authored-by: Joe Ranweiler <joe@lemma.co>

* Adding PR.

* Update CHANGELOG.md

Co-authored-by: Marc Greisen <mgreisen@microsoft.com>

* Adding state machine pr.

* UPdating changelog.

* Update CHANGELOG.md

Co-authored-by: Joe Ranweiler <joe@lemma.co>

* Adding cache PR.

Co-authored-by: Joe Ranweiler <joe@lemma.co>
Co-authored-by: Marc Greisen <mgreisen@microsoft.com>
5.15.0
2022-10-04 14:13:47 -07:00
489579a971 Adding missing caching from python code (#2467)
* bringing back some more caching

* more caching

* formatting

* use a record instead of a string as the key to the cache entry
2022-10-03 16:05:52 -07:00
ef5682c282 Debug failing check pr (#2476)
* add more logs

* bug fix

* more logs

* another fix

* fix integration tests

* do not log error when vm deletion is in progress

* addressing comments

* .

* ..

Co-authored-by: stas <statis@microsoft.com>
2022-10-03 10:40:10 -07:00
e77a87a782 Enable backtraces for agent (#2437) 2022-09-30 10:13:09 -07:00
4662df3e39 Cache VMSS VM InstanceID lookups (#2464)
* Cache VMSS VM InstanceID lookups

* Adding an expiration time to the cache

* make the TTL 10 min

* properly add entries to the cache

Co-authored-by: Cheick Keita <chkeita@microsoft.com>
2022-09-30 16:22:09 +00:00