1588 Commits

Author SHA1 Message Date
Teo Voinea
f302e31ed2
Downgrade instance not found exception (#2549)
* Don't log exception when node no longer exists in azure

* fmt

Co-authored-by: Teo Voinea <Teodor.Voinea@microsoft.com>
2022-10-21 15:17:05 -04:00
Stas
9600c5c873
fix exception message formatting (#2546)
Co-authored-by: stas <statis@microsoft.com>
2022-10-21 15:32:57 +00:00
Cheick Keita
d10740ab4c
fix onefuzz address (#2543) 2022-10-20 17:02:07 -07:00
Teo Voinea
66dc7f591d
Handle nulls better in scaleset shutdown (#2538)
* Handle nulls better

* Populate data if it's not

* fmt

* Fix bug where scaleset won't go to 0

Co-authored-by: Teo Voinea <Teodor.Voinea@microsoft.com>
2022-10-20 23:47:28 +00:00
Cheick Keita
f313a37c9c
Ignoring the scanning log file when reporting an issue with azcopy (#2536)
* Ignoring the scanning log file when reporting an issue with azcopy

* fix comment
2022-10-20 22:18:35 +00:00
Teo Voinea
e83a18be41
Fix notifications bugs (#2534)
* Fix bug where field key is not serialized properly

* Log if we match multiple active WIs, re-open if we only find duplicates

* Wrap up testing

* Update naming to clarify duplicates are talking about work items, not notification config

Co-authored-by: Teo Voinea <Teodor.Voinea@microsoft.com>
2022-10-20 13:26:20 -04:00
George Pollard
c10cb08fcd
Remove use of unapproved crypto libraries (#2423)
To comply with Microsoft security policies, we must use only approved crypto libraries.

Notably, `ring` is not on the approved libraries list. We should use the platform crypto libraries instead. The official guidance is:

- on Windows, schannel and friends
- on Linux, the OpenSSL 1.1.1 series if available, otherwise OpenSSL 3.0. OpenSSL _must_ be dynamically and not statically linked so that distro-provided updates can be used.

Following these guidelines strictly would mean that we have to build distro/version specific binaries for Linux. Instead we have an exception which allows us to statically link against the OpenSSL 1.1.1 series, allowing us to have a (somewhat) portable binary.

OpenSSL 1.1.1. can be statically linked by switching to the `native-tls-vendored` feature of `reqwest`.  (Verified by inspection that `ring` and `rustls` are no longer in `Cargo.lock` files.)

So the result is:

- on Windows, we use schannel
- on Linux we statically link against 1.1.1

It is up to us to ensure we keep up-to-date with OpenSSL issues and releases. The version can be checked by looking at the version of `openssl-src` that we depend upon. Currently this is `111.22.0+1.1.1q`. This is the latest version; also check the [Vulnerabilities page](https://www.openssl.org/news/vulnerabilities-1.1.1.html).

--- 

Verified by inspection that `rustls` and `ring` are no longer present in `Cargo.lock`.

Current dynamic library dependencies are:

```console
$ ldd ./onefuzz-agent 
        linux-vdso.so.1 (0x00007ffd9ba60000)
        libunwind.so.8 => /lib/x86_64-linux-gnu/libunwind.so.8 (0x00007ff5ab5e5000)
        librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007ff5ac1d1000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007ff5ac1cc000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007ff5ab4fe000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007ff5ac1c7000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007ff5ab2d4000)
        /lib64/ld-linux-x86-64.so.2 (0x00007ff5ac1dc000)
        liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007ff5ab2a9000)
```
2022-10-19 22:39:00 +00:00
Joe Ranweiler
ee0cbd70a0
Enable dotnet fuzzing (#2273)
Add a new CLI job template, `libfuzzer dotnet_dll`, and supporting server-side definitions.
2022-10-19 14:17:27 -07:00
George Pollard
b88f46779e
Remove suggestion to reset IterationPath upon duplicate (#2533)
* Remove suggestion to reset IterationPath upon duplicate

* Update highlighting to jsonc
2022-10-19 20:34:16 +00:00
George Pollard
c46c6bee39
Produce an error if coverage recording failed due to timeout (#2529)
Closes #2520.

After investigation the underlying cause here is that the process is getting killed due to timeout, but we don't report the timeout or produce an error. Modify the coverage code so that it fails if timeout is hit.
2022-10-18 21:48:36 +00:00
dependabot[bot]
b3fd6d5ed0
Bump process_control from 3.1.0 to 4.0.0 in /src/agent (#2377)
* Bump process_control from 3.1.0 to 4.0.0 in /src/agent

Bumps [process_control](https://github.com/dylni/process_control) from 3.1.0 to 4.0.0.
- [Release notes](https://github.com/dylni/process_control/releases)
- [Commits](https://github.com/dylni/process_control/compare/3.1.0...4.0.0)

---
updated-dependencies:
- dependency-name: process_control
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

* Update code

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: George Pollard <gpollard@microsoft.com>
2022-10-18 15:57:08 +13:00
Marc Greisen
d36ecf9b46
Release 5.16.0 (#2516)
* Release 5.16.0

* Improve scaling protection logging (#2518)

* Improve scaling protection logging

* Fix a bug where we're always protecting from scaling actions

* We never want to protect from scaleset actions

Co-authored-by: Noah McGregor Harper <74685766+nharper285@users.noreply.github.com>
2022-10-17 12:25:44 -07:00
Cheick Keita
d95342b8b8
Add deleteAll to the orm (#2519)
* Add deleteAll to the orm

* address pr comments
2022-10-17 10:43:51 -07:00
Teo Voinea
8a4b46e4f0
Improve scaling protection logging (#2518)
* Improve scaling protection logging

* Fix a bug where we're always protecting from scaling actions

* We never want to protect from scaleset actions
2022-10-14 12:50:25 -04:00
Noah McGregor Harper
3c35b3445a
Convert ADO traces to customEvents and update notificationInfo (#2508)
* Adding Project and Name to ADO Notification.

* Convert ADO traces to events and update notificationInfo.

* Updating event formatting.

* Fix typos.

* Fixing event name.

* Update the logic update the scaleinProtection (#2505)

* Update the logic update the scaleinProtection

* build fix

* Add SrcView::try_insert function (#2510)

* Add SrcView::try_insert function

* Make try_insert return bool indicating success

* Rustfmt

* Set test to be ignored (no compile)

* Bump srcview version

Co-authored-by: Sutton Bradley <suttonb@microsoft.com>
Co-authored-by: George Pollard <gpollard@microsoft.com>

* Remove old code.

* Updating filename

* Updating event types.

* Fixing logging.

* Removing tag from event name.

* Setting new item.

* remove extra code.

Co-authored-by: Cheick Keita <chkeita@microsoft.com>
Co-authored-by: Sutton Bradley <sbradley4646@gmail.com>
Co-authored-by: Sutton Bradley <suttonb@microsoft.com>
Co-authored-by: George Pollard <gpollard@microsoft.com>
2022-10-13 15:58:40 -07:00
Jamie Magee
51b8dc05b1
Add $schema to cgmanifest.json (#2320)
Co-authored-by: Jamie Magee <jamie.magee@microsoft.com>
2022-10-13 20:49:09 +00:00
Teo Voinea
f37e9db299
Fix github issue query (#2514) 2022-10-13 15:35:52 +00:00
Teo Voinea
58b3d6f3cb
Fix github client header value (#2513) 2022-10-13 07:25:29 -07:00
Sutton Bradley
542cf75c1e
Add SrcView::try_insert function (#2510)
* Add SrcView::try_insert function

* Make try_insert return bool indicating success

* Rustfmt

* Set test to be ignored (no compile)

* Bump srcview version

Co-authored-by: Sutton Bradley <suttonb@microsoft.com>
Co-authored-by: George Pollard <gpollard@microsoft.com>
2022-10-12 21:50:22 +00:00
Cheick Keita
d519f0a44b
Update the logic update the scaleinProtection (#2505)
* Update the logic update the scaleinProtection

* build fix
2022-10-12 11:04:08 -07:00
Cheick Keita
9e5006fc40
Update the logic for checking if a blob exist before uploading (#2503)
* Update the logic for checking if a blob exist before uploading

* remove unused ref

* update the status code check
2022-10-12 03:03:12 +00:00
George Pollard
0fab487d76
Only fetch VM InstanceView data when required (#2506)
* Only fetch VM InstanceView data when reuqired

* Missed one
2022-10-12 15:34:27 +13:00
George Pollard
7596cf4bd0
Update Rust azure_* dependencies (#2500) 2022-10-12 10:09:42 +13:00
George Pollard
71a915c9cd
Add logo to readme (#2340)
* Add logo to readme
2022-10-11 02:06:19 +00:00
Cheick Keita
94b6fe0a7c
change the verbosity of the azcopy logs (#2498) 2022-10-11 01:25:39 +00:00
George Pollard
76caef47e2
Python: Upgrade rsa dependency (#2470)
* Upgrade `rsa` dependency

* Adjust version to match Github advisory
2022-10-11 13:43:19 +13:00
Noah McGregor Harper
41fc3b22c0
Update Logging Statement to Warning and Include Http Info (#2484)
* Convert Error to Warning on Poll State Transition Failure.

* Adding Result information.
2022-10-10 10:49:41 -07:00
George Pollard
cdc104f966
Update documentation to specify appropriate RID (#2490)
Closes #2457
2022-10-07 01:03:18 +00:00
Marc Greisen
053153fa44
Release 5.15.1 (#2493) 2022-10-06 11:56:10 -07:00
Teo Voinea
80b1122a8d
Undo tab change from previous PR (#2492) 2022-10-05 17:32:33 +00:00
Teo Voinea
124c62756c
Migrating notification templates (#2486)
* Add jinja template migration

* Support migrating our most common jinja templates to scriban on the fly

* Fix tests
2022-10-05 14:48:20 +00:00
George Pollard
f6a680bb3d
Attempt to fix integration-tests-linux failure (#2487) 2022-10-05 00:29:08 +00:00
George Pollard
7809b40e74
Update to Rust 1.64 (#2488)
[Release notes.](https://blog.rust-lang.org/2022/09/22/Rust-1.64.0.html)

Probably one of the more important things is:
> performance improvements of 10-20% for compiling Rust code on Windows
2022-10-04 22:31:53 +00:00
Noah McGregor Harper
7529a184cf
Release 5.15.0 (#2474)
* Release 5.15.0

* Update CHANGELOG.md

Co-authored-by: Joe Ranweiler <joe@lemma.co>

* Adding PR.

* Update CHANGELOG.md

Co-authored-by: Marc Greisen <mgreisen@microsoft.com>

* Adding state machine pr.

* UPdating changelog.

* Update CHANGELOG.md

Co-authored-by: Joe Ranweiler <joe@lemma.co>

* Adding cache PR.

Co-authored-by: Joe Ranweiler <joe@lemma.co>
Co-authored-by: Marc Greisen <mgreisen@microsoft.com>
5.15.0
2022-10-04 14:13:47 -07:00
Cheick Keita
489579a971
Adding missing caching from python code (#2467)
* bringing back some more caching

* more caching

* formatting

* use a record instead of a string as the key to the cache entry
2022-10-03 16:05:52 -07:00
Stas
ef5682c282
Debug failing check pr (#2476)
* add more logs

* bug fix

* more logs

* another fix

* fix integration tests

* do not log error when vm deletion is in progress

* addressing comments

* .

* ..

Co-authored-by: stas <statis@microsoft.com>
2022-10-03 10:40:10 -07:00
Joe Ranweiler
e77a87a782
Enable backtraces for agent (#2437) 2022-09-30 10:13:09 -07:00
George Pollard
4662df3e39
Cache VMSS VM InstanceID lookups (#2464)
* Cache VMSS VM InstanceID lookups

* Adding an expiration time to the cache

* make the TTL 10 min

* properly add entries to the cache

Co-authored-by: Cheick Keita <chkeita@microsoft.com>
2022-09-30 16:22:09 +00:00
Stas
969701aa7d
Lowercase webhooks digest header value (#2471)
Co-authored-by: stas <statis@microsoft.com>
2022-09-29 19:20:24 -07:00
George Pollard
2155c48b99
Allow worker loops to continue after errors (#2469)
During `TimerWorkers` if updating one entity fails and throws an exception we will abandon the whole update. Instead log the error and continue to attempt to process the remaining entities. This will allow us to make progress even if one entity is stuck.
2022-09-30 12:40:38 +13:00
George Pollard
4c5023cb55
CLI: Retry on connection reset (#2468)
In our integration test run we are seeing some connection-reset errors which causes the CLI operation to fail.

To fix this:
1. Set TCP-KeepAlive to keep Azure load balancer connections alive longer than the default timeout (4 minutes).
2. Treat ConnectionResetError as retryable.
2022-09-30 11:23:29 +13:00
Joe Ranweiler
41f973184e
Prefix target_exe with setup dir at use sites (#2405) 2022-09-29 13:47:04 -07:00
George Pollard
0c4cd5414d
C#: Fix UpdateConfigs (#2463) 2022-09-29 07:00:04 +00:00
George Pollard
1ec9b13e55
Disable PoolName validation (#2459)
* Add comment

* Disable test
2022-09-29 04:18:11 +00:00
George Pollard
8f4cf9d3b6
Correct pool transitions (#2462) 2022-09-29 17:10:39 +13:00
Stas
0e2f651a35
fix null ref exception (#2460)
* fix null ref exception

* write out <null> if builder or message are null

Co-authored-by: stas <statis@microsoft.com>
2022-09-28 14:44:32 -07:00
George Pollard
9180215a10
Permit periods in Pool names (#2452) 2022-09-28 21:05:42 +00:00
Teo Voinea
0e9e32a934
EnsureNotNull doesn't support our custom formatter (#2458) 2022-09-28 19:18:51 +00:00
Teo Voinea
9a042724d7
Can create ado notifications (#2456)
* Can create ado notifications

* Missed a small issue
2022-09-28 11:04:39 -04:00
George Pollard
872c1070fc
Fix: Node state getting reset to init (#2454)
When `isNew` was passed, then the creation should fail if there is a `Node` that already exists. Instead, the existing `Node` was being overwritten.
2022-09-28 16:53:40 +13:00