1882 Commits

Author SHA1 Message Date
Cheick Keita
d10740ab4c
fix onefuzz address (#2543) 2022-10-20 17:02:07 -07:00
Teo Voinea
66dc7f591d
Handle nulls better in scaleset shutdown (#2538)
* Handle nulls better

* Populate data if it's not

* fmt

* Fix bug where scaleset won't go to 0

Co-authored-by: Teo Voinea <Teodor.Voinea@microsoft.com>
2022-10-20 23:47:28 +00:00
Cheick Keita
f313a37c9c
Ignoring the scanning log file when reporting an issue with azcopy (#2536)
* Ignoring the scanning log file when reporting an issue with azcopy

* fix comment
2022-10-20 22:18:35 +00:00
Teo Voinea
e83a18be41
Fix notifications bugs (#2534)
* Fix bug where field key is not serialized properly

* Log if we match multiple active WIs, re-open if we only find duplicates

* Wrap up testing

* Update naming to clarify duplicates are talking about work items, not notification config

Co-authored-by: Teo Voinea <Teodor.Voinea@microsoft.com>
2022-10-20 13:26:20 -04:00
George Pollard
c10cb08fcd
Remove use of unapproved crypto libraries (#2423)
To comply with Microsoft security policies, we must use only approved crypto libraries.

Notably, `ring` is not on the approved libraries list. We should use the platform crypto libraries instead. The official guidance is:

- on Windows, schannel and friends
- on Linux, the OpenSSL 1.1.1 series if available, otherwise OpenSSL 3.0. OpenSSL _must_ be dynamically and not statically linked so that distro-provided updates can be used.

Following these guidelines strictly would mean that we have to build distro/version specific binaries for Linux. Instead we have an exception which allows us to statically link against the OpenSSL 1.1.1 series, allowing us to have a (somewhat) portable binary.

OpenSSL 1.1.1. can be statically linked by switching to the `native-tls-vendored` feature of `reqwest`.  (Verified by inspection that `ring` and `rustls` are no longer in `Cargo.lock` files.)

So the result is:

- on Windows, we use schannel
- on Linux we statically link against 1.1.1

It is up to us to ensure we keep up-to-date with OpenSSL issues and releases. The version can be checked by looking at the version of `openssl-src` that we depend upon. Currently this is `111.22.0+1.1.1q`. This is the latest version; also check the [Vulnerabilities page](https://www.openssl.org/news/vulnerabilities-1.1.1.html).

--- 

Verified by inspection that `rustls` and `ring` are no longer present in `Cargo.lock`.

Current dynamic library dependencies are:

```console
$ ldd ./onefuzz-agent 
        linux-vdso.so.1 (0x00007ffd9ba60000)
        libunwind.so.8 => /lib/x86_64-linux-gnu/libunwind.so.8 (0x00007ff5ab5e5000)
        librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007ff5ac1d1000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007ff5ac1cc000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007ff5ab4fe000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007ff5ac1c7000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007ff5ab2d4000)
        /lib64/ld-linux-x86-64.so.2 (0x00007ff5ac1dc000)
        liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007ff5ab2a9000)
```
2022-10-19 22:39:00 +00:00
Joe Ranweiler
ee0cbd70a0
Enable dotnet fuzzing (#2273)
Add a new CLI job template, `libfuzzer dotnet_dll`, and supporting server-side definitions.
2022-10-19 14:17:27 -07:00
George Pollard
c46c6bee39
Produce an error if coverage recording failed due to timeout (#2529)
Closes #2520.

After investigation the underlying cause here is that the process is getting killed due to timeout, but we don't report the timeout or produce an error. Modify the coverage code so that it fails if timeout is hit.
2022-10-18 21:48:36 +00:00
dependabot[bot]
b3fd6d5ed0
Bump process_control from 3.1.0 to 4.0.0 in /src/agent (#2377)
* Bump process_control from 3.1.0 to 4.0.0 in /src/agent

Bumps [process_control](https://github.com/dylni/process_control) from 3.1.0 to 4.0.0.
- [Release notes](https://github.com/dylni/process_control/releases)
- [Commits](https://github.com/dylni/process_control/compare/3.1.0...4.0.0)

---
updated-dependencies:
- dependency-name: process_control
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

* Update code

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: George Pollard <gpollard@microsoft.com>
2022-10-18 15:57:08 +13:00
Cheick Keita
d95342b8b8
Add deleteAll to the orm (#2519)
* Add deleteAll to the orm

* address pr comments
2022-10-17 10:43:51 -07:00
Teo Voinea
8a4b46e4f0
Improve scaling protection logging (#2518)
* Improve scaling protection logging

* Fix a bug where we're always protecting from scaling actions

* We never want to protect from scaleset actions
2022-10-14 12:50:25 -04:00
Noah McGregor Harper
3c35b3445a
Convert ADO traces to customEvents and update notificationInfo (#2508)
* Adding Project and Name to ADO Notification.

* Convert ADO traces to events and update notificationInfo.

* Updating event formatting.

* Fix typos.

* Fixing event name.

* Update the logic update the scaleinProtection (#2505)

* Update the logic update the scaleinProtection

* build fix

* Add SrcView::try_insert function (#2510)

* Add SrcView::try_insert function

* Make try_insert return bool indicating success

* Rustfmt

* Set test to be ignored (no compile)

* Bump srcview version

Co-authored-by: Sutton Bradley <suttonb@microsoft.com>
Co-authored-by: George Pollard <gpollard@microsoft.com>

* Remove old code.

* Updating filename

* Updating event types.

* Fixing logging.

* Removing tag from event name.

* Setting new item.

* remove extra code.

Co-authored-by: Cheick Keita <chkeita@microsoft.com>
Co-authored-by: Sutton Bradley <sbradley4646@gmail.com>
Co-authored-by: Sutton Bradley <suttonb@microsoft.com>
Co-authored-by: George Pollard <gpollard@microsoft.com>
2022-10-13 15:58:40 -07:00
Jamie Magee
51b8dc05b1
Add $schema to cgmanifest.json (#2320)
Co-authored-by: Jamie Magee <jamie.magee@microsoft.com>
2022-10-13 20:49:09 +00:00
Teo Voinea
f37e9db299
Fix github issue query (#2514) 2022-10-13 15:35:52 +00:00
Teo Voinea
58b3d6f3cb
Fix github client header value (#2513) 2022-10-13 07:25:29 -07:00
Sutton Bradley
542cf75c1e
Add SrcView::try_insert function (#2510)
* Add SrcView::try_insert function

* Make try_insert return bool indicating success

* Rustfmt

* Set test to be ignored (no compile)

* Bump srcview version

Co-authored-by: Sutton Bradley <suttonb@microsoft.com>
Co-authored-by: George Pollard <gpollard@microsoft.com>
2022-10-12 21:50:22 +00:00
Cheick Keita
d519f0a44b
Update the logic update the scaleinProtection (#2505)
* Update the logic update the scaleinProtection

* build fix
2022-10-12 11:04:08 -07:00
Cheick Keita
9e5006fc40
Update the logic for checking if a blob exist before uploading (#2503)
* Update the logic for checking if a blob exist before uploading

* remove unused ref

* update the status code check
2022-10-12 03:03:12 +00:00
George Pollard
0fab487d76
Only fetch VM InstanceView data when required (#2506)
* Only fetch VM InstanceView data when reuqired

* Missed one
2022-10-12 15:34:27 +13:00
George Pollard
7596cf4bd0
Update Rust azure_* dependencies (#2500) 2022-10-12 10:09:42 +13:00
Cheick Keita
94b6fe0a7c
change the verbosity of the azcopy logs (#2498) 2022-10-11 01:25:39 +00:00
George Pollard
76caef47e2
Python: Upgrade rsa dependency (#2470)
* Upgrade `rsa` dependency

* Adjust version to match Github advisory
2022-10-11 13:43:19 +13:00
Noah McGregor Harper
41fc3b22c0
Update Logging Statement to Warning and Include Http Info (#2484)
* Convert Error to Warning on Poll State Transition Failure.

* Adding Result information.
2022-10-10 10:49:41 -07:00
Teo Voinea
80b1122a8d
Undo tab change from previous PR (#2492) 2022-10-05 17:32:33 +00:00
Teo Voinea
124c62756c
Migrating notification templates (#2486)
* Add jinja template migration

* Support migrating our most common jinja templates to scriban on the fly

* Fix tests
2022-10-05 14:48:20 +00:00
George Pollard
7809b40e74
Update to Rust 1.64 (#2488)
[Release notes.](https://blog.rust-lang.org/2022/09/22/Rust-1.64.0.html)

Probably one of the more important things is:
> performance improvements of 10-20% for compiling Rust code on Windows
2022-10-04 22:31:53 +00:00
Cheick Keita
489579a971
Adding missing caching from python code (#2467)
* bringing back some more caching

* more caching

* formatting

* use a record instead of a string as the key to the cache entry
2022-10-03 16:05:52 -07:00
Stas
ef5682c282
Debug failing check pr (#2476)
* add more logs

* bug fix

* more logs

* another fix

* fix integration tests

* do not log error when vm deletion is in progress

* addressing comments

* .

* ..

Co-authored-by: stas <statis@microsoft.com>
2022-10-03 10:40:10 -07:00
Joe Ranweiler
e77a87a782
Enable backtraces for agent (#2437) 2022-09-30 10:13:09 -07:00
George Pollard
4662df3e39
Cache VMSS VM InstanceID lookups (#2464)
* Cache VMSS VM InstanceID lookups

* Adding an expiration time to the cache

* make the TTL 10 min

* properly add entries to the cache

Co-authored-by: Cheick Keita <chkeita@microsoft.com>
2022-09-30 16:22:09 +00:00
Stas
969701aa7d
Lowercase webhooks digest header value (#2471)
Co-authored-by: stas <statis@microsoft.com>
2022-09-29 19:20:24 -07:00
George Pollard
2155c48b99
Allow worker loops to continue after errors (#2469)
During `TimerWorkers` if updating one entity fails and throws an exception we will abandon the whole update. Instead log the error and continue to attempt to process the remaining entities. This will allow us to make progress even if one entity is stuck.
2022-09-30 12:40:38 +13:00
George Pollard
4c5023cb55
CLI: Retry on connection reset (#2468)
In our integration test run we are seeing some connection-reset errors which causes the CLI operation to fail.

To fix this:
1. Set TCP-KeepAlive to keep Azure load balancer connections alive longer than the default timeout (4 minutes).
2. Treat ConnectionResetError as retryable.
2022-09-30 11:23:29 +13:00
Joe Ranweiler
41f973184e
Prefix target_exe with setup dir at use sites (#2405) 2022-09-29 13:47:04 -07:00
George Pollard
0c4cd5414d
C#: Fix UpdateConfigs (#2463) 2022-09-29 07:00:04 +00:00
George Pollard
1ec9b13e55
Disable PoolName validation (#2459)
* Add comment

* Disable test
2022-09-29 04:18:11 +00:00
George Pollard
8f4cf9d3b6
Correct pool transitions (#2462) 2022-09-29 17:10:39 +13:00
Stas
0e2f651a35
fix null ref exception (#2460)
* fix null ref exception

* write out <null> if builder or message are null

Co-authored-by: stas <statis@microsoft.com>
2022-09-28 14:44:32 -07:00
George Pollard
9180215a10
Permit periods in Pool names (#2452) 2022-09-28 21:05:42 +00:00
Teo Voinea
0e9e32a934
EnsureNotNull doesn't support our custom formatter (#2458) 2022-09-28 19:18:51 +00:00
Teo Voinea
9a042724d7
Can create ado notifications (#2456)
* Can create ado notifications

* Missed a small issue
2022-09-28 11:04:39 -04:00
George Pollard
872c1070fc
Fix: Node state getting reset to init (#2454)
When `isNew` was passed, then the creation should fail if there is a `Node` that already exists. Instead, the existing `Node` was being overwritten.
2022-09-28 16:53:40 +13:00
Stas
476c99a998
use InterpolatedStringHandler to move values to CustomDimensions Tags instead of keeping them in the error message (#2450)
* use InterpolatedStringHandler to move values to CustomDimensions Tags instead of keeping them in the error message

* log blob save raw response failure

* add StringBuilder to CSharpExtensions

Co-authored-by: stas <statis@microsoft.com>
2022-09-27 15:22:29 -07:00
George Pollard
b3748e4283
Handle 404 in Queue.RemoveFirstMessage (#2451) 2022-09-28 11:01:02 +13:00
dependabot[bot]
e6d3b39d1a
Bump protobuf from 3.20.0 to 3.20.2 in /src/api-service/__app__ (#2446)
Bumps [protobuf](https://github.com/protocolbuffers/protobuf) from 3.20.0 to 3.20.2.
- [Release notes](https://github.com/protocolbuffers/protobuf/releases)
- [Changelog](https://github.com/protocolbuffers/protobuf/blob/main/generate_changelog.py)
- [Commits](https://github.com/protocolbuffers/protobuf/compare/v3.20.0...v3.20.2)

---
updated-dependencies:
- dependency-name: protobuf
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-09-27 12:42:55 +13:00
George Pollard
5ee4cd045d
Add Roslyn analyzer to check results are used (#2443)
As seen in #2441, it is easy to drop return values of updated entities accidentally.

This PR adds a Roslyn Analyzer which will detect when return values are unused. To explicitly ignore a value you can drop it with `_ = …;`

Closes #2442.
2022-09-26 22:26:06 +00:00
Cheick Keita
bb2e8ad05e
Fix logic to retrieve partitionKey and rowKey (#2447)
* Fix logic to retrieve partitionKey and rowKey

* Moved key getters to EntityConverter and added unit test
2022-09-26 22:01:00 +00:00
Noah McGregor Harper
3f35d81f4b
Adding New Default Image Config Value to IC. (#2434)
* Adding New Default Image Config Value to IC.

* Removing forced image setting.

* Updating Webhook Events.

* Removing typo.

* Updating webhook_events again.

* Syncing webhook events.

* Fixing check for os type.

* Fixing import.

* PR Suggestions.

* Fix C# Model Typo.

* Removing other refs to images.

* Removing remaining refs to images outside of models.

* Removing hardcoded image values from tests.

* Update Default Proxy and Repro Images.

Co-authored-by: Marc Greisen <mgreisen@microsoft.com>
2022-09-23 10:40:44 -07:00
Stas
dc2c4649c8
do not loose proxy objects when setting state (#2441)
Co-authored-by: stas <statis@microsoft.com>
2022-09-23 13:29:22 +12:00
Stas
e1851b0af4
Add more logs (#2438)
* add logs

* avoid relying on exceptions for logic flow control

* add logs to agent commands

* add more logs and fix error logging when table writes fail

* move machine ID to CustomDimensions

* log insert errors

* Log Delete failures

* more logs

* more logs

* more logs

* More logs (I think that's it there is no more...)

Co-authored-by: stas <statis@microsoft.com>
2022-09-23 11:28:37 +12:00
Teo Voinea
4f9682d3cf
Do not fail task on notification failure (#2435)
* Do not fail task on notification failure

* Need to throw on the last iteration in order for it to go to poison queue

* lint
2022-09-22 21:05:07 +00:00