Commit Graph

1767 Commits

Author SHA1 Message Date
fb8a4e36f1 Add a registration command to allow an application registration to connect to the Onefuzz (#2571) 2022-11-03 16:41:38 +00:00
6cc7caf67b Handle duplicate containers across accounts (#2596) 2022-11-03 01:21:35 +00:00
f1f6400dad Improve AppInsights setup (#2597)
Switch to using the package provided by Azure Functions to set up Application Insights.

Using the provided setup method `AddApplicationInsightsTelemetry` will configure the dependency collector by default as well as set up some other things that we haven’t done manually, including application version, which would be very useful.

This also means that `operation_Id` is populated more consistently which permits joining `traces` or `dependencies` on the `requests` table.

This also means that the end-to-end data in AI is now correct, so the chart works:
![MicrosoftTeams-image](https://user-images.githubusercontent.com/12575/199629195-f442c9b4-2bb3-4f78-a532-3936cf6bf74f.png)
2022-11-03 01:09:51 +00:00
22cb201805 Fix the notification get request (#2591)
* Fix the notification get request

* remove unused import
2022-11-02 17:56:50 -07:00
e6dd1ac35b Only ignore known issue (#2595) 2022-11-02 20:27:45 +00:00
d655fcd5c4 Optimize the output side of srcview (#2515) 2022-11-01 17:59:37 -07:00
323106b48b Temporarily disable pipenv check call (#2594) 2022-11-01 17:48:34 -07:00
33e2d5c954 Delete nodes on decommission instead of just releasing scale in protection (#2586)
Co-authored-by: Teo Voinea <Teodor.Voinea@microsoft.com>
2022-11-01 08:48:42 -04:00
8be6e3c946 do not break the for loop, and cache all instance ids (#2548)
Co-authored-by: stas <statis@microsoft.com>
2022-10-29 11:43:57 -07:00
d7d55c77fc create the config_path folder if it does not exist (#2583) 2022-10-28 17:18:49 -07:00
36265895ee Reduce fetches to VMSS (#2577)
* Remove one GET from ListInstanceIds

* Only invoke ListInstanceIds if needed

* Format code

* Downgrade Exception log to Verbose

* Insert InstanceID during CleanupNodes

Co-authored-by: Cheick Keita <chkeita@microsoft.com>
2022-10-28 19:45:27 +00:00
4b305d0e55 Release-5.17.0 (#2537)
* Release-5.17.0
2022-10-28 10:41:04 -07:00
c277cd1ec3 Update References to Ubuntu 18.04 to 20.04 (#2535)
* Update References to 18.04 to 20.04

* Fixing Ubuntu Image.

* Fix image.
2022-10-28 10:17:57 -07:00
539861b573 Fix Github actions warnings (#2576)
Node.js 12 actions are deprecated, as well as `::set-output` 

- Update `actions/checkout` to v3
- Update `actions/upload-artifact` to v3
- Update `actions/setup-python` to v4
- Update `actions/cache` to v3
- `set-output` on stdout is deprecated, update to `$GITHOUT_OUTPUT` method
- Change from `actions-rs/toolchain` (unsupported) to `dtolnay/rust-toolchain`

The only warning remaining after this is the one about Ubuntu 18.04.
2022-10-27 17:21:30 -07:00
fc2f29dc6f Reduce number of ARM calls in ListVmss (#2539)
1. `ListVmss` was invoking `GetVirtualMachineScaleSetAsync` and then `GetVirtualMachineScaleSetVms`; instead we can get the VMSS resource directly and invoke `GetVirtualMachineScaleSetVms` on it.

2. I also removed some unneeded invocations of `AsAsyncEnumerable` which might be turning async enumerables into sync enumerables.

3. Do the same `HasData` check in the instance ID lookup that we were already doing in `ListVmss`.

After this change (#3) we no longer do any `GET`s against individual VMSS VMs during the `check-pr` run: 
![image](https://user-images.githubusercontent.com/12575/198410236-250ed8d7-f7a1-4c2d-8bf4-2ebd1c1a0f7a.png)
2022-10-27 22:40:34 +00:00
228fe99bc1 Update to released version of appinsights-rs (#2573) 2022-10-27 14:01:31 -07:00
7232981d95 Use records to unpack the request parameters in AgentRegistration (#2570) 2022-10-27 10:26:34 -07:00
85437df253 Bump PyGithub (#2569)
Co-authored-by: Teo Voinea <Teodor.Voinea@microsoft.com>
2022-10-27 16:04:52 +00:00
76cde40234 Better logging with webhooks (#2568)
Co-authored-by: Teo Voinea <Teodor.Voinea@microsoft.com>
2022-10-27 11:53:22 -04:00
c086f249d7 fix path onfuzz config path (#2567) 2022-10-26 16:22:05 -07:00
49ca5b5248 ETag should be updated in Update and Replace as well (#2562) 2022-10-26 22:42:34 +00:00
9299357133 Don't log an error if we delete a Repro and it is already missing (#2563) 2022-10-26 22:04:42 +00:00
81e4b1542d Include computer name in AgentRegistration & decode Instance ID from it (#2557)
Addressing #2550 but in a slightly different way:

1. Always include the machine name in the agent registration message. From the machine name we can extract the instance ID.
2. Use the stored value in preference to fetching it every time.
3. To back-fill existing nodes that do not have the value stored, update it in AcquireScaleInProtection if it is not already present.
2022-10-27 10:01:17 +13:00
0e8876b9f1 Exponential backoff for failed notifications (#2555)
* .

* 🧹

* Do backoff on main queue, kick off to poison queue after sufficient retries

* PR comments

Co-authored-by: Teo Voinea <Teodor.Voinea@microsoft.com>
2022-10-26 12:46:07 -04:00
e50bc0b389 Update the libfuzzer-with-options test to use fuzzing_target_options (#2565)
This currently causes the coverage task to time out, ensure that the `runs` parameter is only passed to the fuzzing task and not the coverage task.
2022-10-26 00:03:28 +00:00
72d0c6f25e Use the vitual env folder to store the config if it exists (#2561)
* Use the vitual env folder to store the config if it exists

* address PR comment

* fix

* mypy fix

* mypy fix
2022-10-25 16:30:33 -07:00
59f4402eb0 Extend default coverage timeout to 120s (#2556)
As of #2529 we now fail if the coverage recording times out. The default timeout (5s) does not appear to be long enough for even simple tasks to run, so extend it to 120s.
2022-10-25 00:43:10 +00:00
db1a712ed3 Lower log level on region overlap (#2559) 2022-10-24 16:15:02 -07:00
cafba18c85 fix onefuzz status pool (#2551) 2022-10-21 15:18:46 -07:00
f302e31ed2 Downgrade instance not found exception (#2549)
* Don't log exception when node no longer exists in azure

* fmt

Co-authored-by: Teo Voinea <Teodor.Voinea@microsoft.com>
2022-10-21 15:17:05 -04:00
9600c5c873 fix exception message formatting (#2546)
Co-authored-by: stas <statis@microsoft.com>
2022-10-21 15:32:57 +00:00
d10740ab4c fix onefuzz address (#2543) 2022-10-20 17:02:07 -07:00
66dc7f591d Handle nulls better in scaleset shutdown (#2538)
* Handle nulls better

* Populate data if it's not

* fmt

* Fix bug where scaleset won't go to 0

Co-authored-by: Teo Voinea <Teodor.Voinea@microsoft.com>
2022-10-20 23:47:28 +00:00
f313a37c9c Ignoring the scanning log file when reporting an issue with azcopy (#2536)
* Ignoring the scanning log file when reporting an issue with azcopy

* fix comment
2022-10-20 22:18:35 +00:00
e83a18be41 Fix notifications bugs (#2534)
* Fix bug where field key is not serialized properly

* Log if we match multiple active WIs, re-open if we only find duplicates

* Wrap up testing

* Update naming to clarify duplicates are talking about work items, not notification config

Co-authored-by: Teo Voinea <Teodor.Voinea@microsoft.com>
2022-10-20 13:26:20 -04:00
c10cb08fcd Remove use of unapproved crypto libraries (#2423)
To comply with Microsoft security policies, we must use only approved crypto libraries.

Notably, `ring` is not on the approved libraries list. We should use the platform crypto libraries instead. The official guidance is:

- on Windows, schannel and friends
- on Linux, the OpenSSL 1.1.1 series if available, otherwise OpenSSL 3.0. OpenSSL _must_ be dynamically and not statically linked so that distro-provided updates can be used.

Following these guidelines strictly would mean that we have to build distro/version specific binaries for Linux. Instead we have an exception which allows us to statically link against the OpenSSL 1.1.1 series, allowing us to have a (somewhat) portable binary.

OpenSSL 1.1.1. can be statically linked by switching to the `native-tls-vendored` feature of `reqwest`.  (Verified by inspection that `ring` and `rustls` are no longer in `Cargo.lock` files.)

So the result is:

- on Windows, we use schannel
- on Linux we statically link against 1.1.1

It is up to us to ensure we keep up-to-date with OpenSSL issues and releases. The version can be checked by looking at the version of `openssl-src` that we depend upon. Currently this is `111.22.0+1.1.1q`. This is the latest version; also check the [Vulnerabilities page](https://www.openssl.org/news/vulnerabilities-1.1.1.html).

--- 

Verified by inspection that `rustls` and `ring` are no longer present in `Cargo.lock`.

Current dynamic library dependencies are:

```console
$ ldd ./onefuzz-agent 
        linux-vdso.so.1 (0x00007ffd9ba60000)
        libunwind.so.8 => /lib/x86_64-linux-gnu/libunwind.so.8 (0x00007ff5ab5e5000)
        librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007ff5ac1d1000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007ff5ac1cc000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007ff5ab4fe000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007ff5ac1c7000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007ff5ab2d4000)
        /lib64/ld-linux-x86-64.so.2 (0x00007ff5ac1dc000)
        liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007ff5ab2a9000)
```
2022-10-19 22:39:00 +00:00
ee0cbd70a0 Enable dotnet fuzzing (#2273)
Add a new CLI job template, `libfuzzer dotnet_dll`, and supporting server-side definitions.
2022-10-19 14:17:27 -07:00
b88f46779e Remove suggestion to reset IterationPath upon duplicate (#2533)
* Remove suggestion to reset IterationPath upon duplicate

* Update highlighting to jsonc
2022-10-19 20:34:16 +00:00
c46c6bee39 Produce an error if coverage recording failed due to timeout (#2529)
Closes #2520.

After investigation the underlying cause here is that the process is getting killed due to timeout, but we don't report the timeout or produce an error. Modify the coverage code so that it fails if timeout is hit.
2022-10-18 21:48:36 +00:00
b3fd6d5ed0 Bump process_control from 3.1.0 to 4.0.0 in /src/agent (#2377)
* Bump process_control from 3.1.0 to 4.0.0 in /src/agent

Bumps [process_control](https://github.com/dylni/process_control) from 3.1.0 to 4.0.0.
- [Release notes](https://github.com/dylni/process_control/releases)
- [Commits](https://github.com/dylni/process_control/compare/3.1.0...4.0.0)

---
updated-dependencies:
- dependency-name: process_control
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

* Update code

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: George Pollard <gpollard@microsoft.com>
2022-10-18 15:57:08 +13:00
d36ecf9b46 Release 5.16.0 (#2516)
* Release 5.16.0

* Improve scaling protection logging (#2518)

* Improve scaling protection logging

* Fix a bug where we're always protecting from scaling actions

* We never want to protect from scaleset actions

Co-authored-by: Noah McGregor Harper <74685766+nharper285@users.noreply.github.com>
2022-10-17 12:25:44 -07:00
d95342b8b8 Add deleteAll to the orm (#2519)
* Add deleteAll to the orm

* address pr comments
2022-10-17 10:43:51 -07:00
8a4b46e4f0 Improve scaling protection logging (#2518)
* Improve scaling protection logging

* Fix a bug where we're always protecting from scaling actions

* We never want to protect from scaleset actions
2022-10-14 12:50:25 -04:00
3c35b3445a Convert ADO traces to customEvents and update notificationInfo (#2508)
* Adding Project and Name to ADO Notification.

* Convert ADO traces to events and update notificationInfo.

* Updating event formatting.

* Fix typos.

* Fixing event name.

* Update the logic update the scaleinProtection (#2505)

* Update the logic update the scaleinProtection

* build fix

* Add SrcView::try_insert function (#2510)

* Add SrcView::try_insert function

* Make try_insert return bool indicating success

* Rustfmt

* Set test to be ignored (no compile)

* Bump srcview version

Co-authored-by: Sutton Bradley <suttonb@microsoft.com>
Co-authored-by: George Pollard <gpollard@microsoft.com>

* Remove old code.

* Updating filename

* Updating event types.

* Fixing logging.

* Removing tag from event name.

* Setting new item.

* remove extra code.

Co-authored-by: Cheick Keita <chkeita@microsoft.com>
Co-authored-by: Sutton Bradley <sbradley4646@gmail.com>
Co-authored-by: Sutton Bradley <suttonb@microsoft.com>
Co-authored-by: George Pollard <gpollard@microsoft.com>
2022-10-13 15:58:40 -07:00
51b8dc05b1 Add $schema to cgmanifest.json (#2320)
Co-authored-by: Jamie Magee <jamie.magee@microsoft.com>
2022-10-13 20:49:09 +00:00
f37e9db299 Fix github issue query (#2514) 2022-10-13 15:35:52 +00:00
58b3d6f3cb Fix github client header value (#2513) 2022-10-13 07:25:29 -07:00
542cf75c1e Add SrcView::try_insert function (#2510)
* Add SrcView::try_insert function

* Make try_insert return bool indicating success

* Rustfmt

* Set test to be ignored (no compile)

* Bump srcview version

Co-authored-by: Sutton Bradley <suttonb@microsoft.com>
Co-authored-by: George Pollard <gpollard@microsoft.com>
2022-10-12 21:50:22 +00:00
d519f0a44b Update the logic update the scaleinProtection (#2505)
* Update the logic update the scaleinProtection

* build fix
2022-10-12 11:04:08 -07:00
9e5006fc40 Update the logic for checking if a blob exist before uploading (#2503)
* Update the logic for checking if a blob exist before uploading

* remove unused ref

* update the status code check
2022-10-12 03:03:12 +00:00