1882 Commits

Author SHA1 Message Date
Cheick Keita
3878793147
Trim whitespaces from options passed to executables (#2151)
* Trim whitespaces from options passed to executables

* check for None

* refactor to a function
2022-07-11 23:37:48 +00:00
Teo Voinea
7d704a9688
Fix clippy (#2152) 2022-07-11 19:00:51 -04:00
Noah McGregor Harper
28e42ab8e8
Update Function Scaling Policy and Diagnostic Settings. (#2140)
* Update Function Scaling Policy and Diagnostic Settings.

* Adding changes to scaling policy

* UPdating cooldown.

* Changing time windows.

* Updating duration.
2022-07-08 12:03:14 -07:00
Joe Ranweiler
34a8d721bb
Add LibFuzzerDotnetLoader docs (#2142) 2022-07-08 08:24:56 -07:00
George Pollard
e37f56dd05
Remove some allow(clippy::uninit_vec) (#2111)
Closes #1595. Leaving one instance of `allow(clippy::uninit_vec)` in `pipe_handle.rs` as it is more difficult to remove. (Maybe there's  another crate we could use for pipes?)

To satisfy clippy, instead of creating the vec and then resizing it straight away (exposing uninitialized data), write into the `spare_capacity_mut()` (which is `&mut [MaybeUninit<_>]`) and then resize it afterwards.
2022-07-08 01:24:11 +00:00
Joe Ranweiler
6bf77a6230
Add logging (#2141)
- Add logging
- Catch terminal exceptions and log as errors before re-throwing
- Nest some exceptions
- Warn when falling back to non-zero cost delegate for legacy compat
2022-07-07 17:41:53 -07:00
Cheick Keita
ff1d088fa1
include state filter when searching for expired jobs and tasks (#2138) 2022-07-07 14:53:51 -07:00
Joe Ranweiler
edf537137b
Add multi-var target spec, improve errors (#2136) 2022-07-07 11:20:45 -07:00
Cheick Keita
aa2e76e7e9
Fix execution of TimerProxy (#2133)
* Fix execution of timerProxy
- fixed behavior of Container.SaveBlob (file was not being overwritten)
- added support for default value in the entity converter
- fix scaleset size data type
- added initialization of Subnet
- renamed TestHook Info to _Info to prevent clashing of name function mame
- added target framework to ApiServiceProject to help local debugging

* fix unit tests

* fix unit tests

* remove unused property

* fix typo

* removing partial class TimerProxy
2022-07-07 09:25:51 -07:00
Teo Voinea
0f0fd8ee9c
Enable TimerRepro (#2108) 2022-07-07 15:25:20 +00:00
Joe Ranweiler
478788f607
Remove stubbed support for macOS builds (#2135)
Closes #2134.
2022-07-06 23:23:47 +00:00
Teo Voinea
f37224e8bb
Add dotnet coverage task (#2062)
* checkpoint

* some more progress

* more progress

* More progress

* Now it's time to test it

* It works locally 🎉

* Attempting clean build

* fmt

* temporarily stub out macos

* missed a few

* please be the last one

* .

* .

* .

* noop change to unstuck actions

* .

* .

* Fix setup script

* Some fixes

* It works except for a race condition -- use a directory watcher to fix it

* It works end to end!

* Execute the commands using tokio's structs and timeout mechanism

* It works.... for real this time

* Undo timer changes

* Cleanup

* 🧹

* Fix import

* .

* PR comments

* Fix clippy

* Clippy whyyy

* Only check dotnet path once

* fmt

* Fix a couple more comments
2022-07-06 16:13:45 -04:00
Joe Ranweiler
bc48069a94
Import LibFuzzerDotnetLoader, build .NET fuzzing tools (#2126) 2022-07-06 11:51:47 -07:00
Teo Voinea
4d7a01e157
Fixed (#2122) 2022-07-05 20:32:04 +00:00
Cheick Keita
52b760b89a
Porting divergent code introduced in #2116 (#2121)
Make the log sas url last as long as the job duration
2022-07-05 11:35:42 -07:00
Cheick Keita
61fc091f88
Make the log sas url last as long as the job duration (#2116) 2022-07-02 01:40:02 +00:00
Teo Voinea
262ad689f3
Enable C# functions (#2093)
Enables 2 more C# refactor functions in code. They're ready to be tested in canary.

Improves the precommit script in devcontainer.
2022-07-01 03:57:36 +00:00
George Pollard
a9fe5e6c17
Refactor agent_registration to C# (#2107)
Port `agent_registration` from Python to C#.
2022-07-01 02:38:52 +00:00
dependabot[bot]
2453127298
Bump tokio from 1.18.2 to 1.19.2 in /src/proxy-manager (#2022)
Bumps [tokio](https://github.com/tokio-rs/tokio) from 1.18.2 to 1.19.2.
- [Release notes](https://github.com/tokio-rs/tokio/releases)
- [Commits](https://github.com/tokio-rs/tokio/commits)

---
updated-dependencies:
- dependency-name: tokio
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-07-01 14:00:05 +12:00
Cheick Keita
8d8a8b01d2
remove unused Poolname validation (#2094) 2022-06-30 23:23:53 +00:00
Cheick Keita
870a51a1e4
Enabling TimerProxy (#2089)
* Testing TimerProzy

* Update src/deployment/deploy.py
2022-06-30 22:47:37 +00:00
George Pollard
a017b10d17
Add correct routes and auth to "agents" Functions (#2109)
The routes and auth were missing from the "agents" functions.
2022-06-30 22:05:07 +00:00
Peleus Uhley
fb0e055ca9
Add azcopy upload support and switch to the default max_concurrency (#1556)
* Add azcopy copy support and switch to the default max_concurrency

* Modify upload_file_data to use upload_file for consistency

* Fix lint issues

* Apply black formatting rules

* Apply isort formatting rules

Co-authored-by: Stas <stishkin@live.com>
2022-06-30 11:30:52 -07:00
Cheick Keita
ed69cb48b5
Fix Racing condition in monitor.rs (#2105)
* Fix Racing condition in monitor.rs

* address pr comment

* address pr comment

* emmiting a warning instead of stopping the monitoring
2022-06-30 09:29:51 -07:00
George Pollard
ea60410df9
Enable C# functions in code & fill out missing functionality (#2084)
C# HTTP functions won’t take effect by default so it is safe to enable them in code. Also implement required authentication code.
2022-06-30 01:33:12 +00:00
Cheick Keita
63bff8bedc
Add caching to Storage.cs (#2102)
* Add caching to Storage.cs

* make Storage disposable

* make Storage disposable
2022-06-29 23:40:31 +00:00
George Pollard
7e0565dac1
Treat warnings as errors (#2103) 2022-06-29 21:25:18 +00:00
Stas
30daae2215
Potential solution for TLS errors in OneFuzz (#2087)
* proposed fix from here:

https://github.com/Azure/azure-functions-durable-python/issues/194#issuecomment-710670377

* Update src/api-service/__app__/__init__.py

Co-authored-by: George Pollard <porges@porg.es>

Co-authored-by: stas <statis@microsoft.com>
Co-authored-by: George Pollard <porges@porg.es>
Co-authored-by: George Pollard <gpollard@microsoft.com>
2022-06-29 08:24:40 -07:00
George Pollard
2a6a0d9996
Raise AnalysisLevel to 6.0-Recommended, fix warnings (#2086)
Raise the .NET Code Analysis Level to `6.0-Minimum` and then `6.0-Recommended`.
2022-06-28 21:27:33 +00:00
George Pollard
3c3aeaa569
Add test for StatefulOrm static constructor (#2092)
`StatefulOrm` performs checks in the static constructor. Make sure that all these checks pass.
2022-06-29 08:19:44 +12:00
George Pollard
f6fb980cb9
Revert "Update appinsights to released version (#2033)" (#2091)
#2033 appears to be breaking agent since we are getting nested tokio runtimes. Revert it!
2022-06-27 22:53:32 +00:00
George Pollard
e947f35996
Use GetRequiredService instead of GetService + check (#2085)
A small tidy-up.
2022-06-27 22:19:52 +00:00
George Pollard
2eae1793ad
Integration tests project (#2083)
Move integration tests into their own project. This makes it easier to run unit tests if you don't want to (or don't have) `azurite` running, since you can `dotnet test` just that directory.

Also add a check into the `AzuriteStorage` class that will abort the entire test run if `azurite` is not running, which is simpler than reading lots of socket exceptions.
2022-06-27 21:14:51 +00:00
George Pollard
0259759407
Update appinsights to released version (#2033)
I noticed that there was a note to update `appinsights` to a released version once one was available, and now one is.

The `close` function was changed to be `async` in the latest version, so `try_flush_and_close` must also become async. Luckily, the three places we use this already have an async context available.
2022-06-27 04:23:12 +00:00
Cheick Keita
d61fe48a55
Migrate timer_task part 3 (#2066)
* Adding unit tests for the scheduler

* formatting

* fix unit test

* bug fixes

* proting a couple more missong functions

* more bug fixes

* fixing queries and serilization

* fix sas url generation

* fix condition

* [testing] enabling timer_tasks for testing

* Update src/ApiService/Tests/SchedulerTests.cs

Co-authored-by: George Pollard <porges@porg.es>

* address PR comments

* build fix

* address PR comment

* removing renamed function

* resolve merge

* Update src/ApiService/Tests/Integration/AzuriteStorage.cs

Co-authored-by: George Pollard <porges@porg.es>

* - Added verification of the state transition functions
- disabled validation on PoolName

* Update src/deployment/deploy.py

Co-authored-by: George Pollard <porges@porg.es>
Co-authored-by: George Pollard <gpollard@microsoft.com>
2022-06-24 16:22:08 +00:00
George Pollard
fb9af4b811
Implement the download Function for C# (#2076)
Port the `download` function to the C# service.
2022-06-23 15:48:24 +12:00
George Pollard
4eec0bfc45
Implement the node C# function (#2072)
1. Ports the `node` function from Python to C#.
2. Adds a missing authentication check.
3. Add validated string type `PoolName` for consistency with Python version.
2022-06-23 13:44:14 +12:00
George Pollard
1eeefce85c
Implement info Function in C# (#2061)
Adds implementation of the `info` function. 

Added support for blobs in the function integration test stuff.

Simplified usage of integration test classes by removing the account name parameter.
2022-06-23 13:06:40 +12:00
George Pollard
4cb7bba2c9
Add underscore naming convention to .editorconfig (#2070) 2022-06-22 15:47:10 +12:00
George Pollard
bbc060bbed
Add null analysis attributes to result types (#2069) 2022-06-21 23:41:32 +00:00
Stas
8f30801c48
Add nsg config to check-pr (#2067)
Co-authored-by: Noah McGregor Harper <74685766+nharper285@users.noreply.github.com>
2022-06-21 12:32:02 -07:00
George Pollard
4ecd43fc0f
Convert agent_events to C# & add tests (#2032)
### Function implementation

Added implementation of `agent_events` function and some basic tests.

Fixed several issues encountered in `TestOperations`.

### Additional JsonConverter

The existing Python code for `agent_events` figures out which class to use only based upon the shape of the input, without a discriminator. I have added an additional `JsonConverter` named `SubclassConverter<T>` which will pick a subclass of `T` to deserialize into based upon what properties exist in the JSON.

### Enum helpers

Converted some enum helpers to extension methods to make them a bit more readable.
2022-06-16 22:28:09 +00:00
George Pollard
cdec3c9e8d
Test infrastructure for C# Azure Function testing (#2055)
Add support for function tests and the ability to run them against either real Azure Storage or the Azurite emulator.

See follow-up PR #2032 for actual usage.
2022-06-16 21:50:02 +00:00
Cheick Keita
e9147ba9a7
Add context to every future in the supervisor task (#1976)
* Add context to every futur in the supervisor task

* adding more context info for the sync operations

* adding more logging in the monitoring of the process pipe
2022-06-16 12:01:22 -07:00
George Pollard
2225cb3fb8
Update onefuzz-agent to clap 3 (#2049)
Mostly a maintenance update (dependabot was trying to do this and failing), but it does have a small improvement in that the `debug node_event state_update` help:

```console
$ target/debug/onefuzz-agent debug node_event state_update --help
onefuzz-agent-debug-node_event-state_update 0.1.0

USAGE:
    onefuzz-agent debug node_event state_update <state>

FLAGS:
    -h, --help       Prints help information
    -V, --version    Prints version information

ARGS:
    <state>    
```

Now displays the applicable enum values:

```console
$ target/debug/onefuzz-agent debug node_event state_update --help
onefuzz-agent-debug-node_event-state_update 

USAGE:
    onefuzz-agent debug node_event state_update <STATE>

ARGS:
    <STATE>    [possible values: init, free, setting-up, rebooting, ready, busy, done]

OPTIONS:
    -h, --help    Print help information
```

`structopt` dependency removed as it was integrated into `clap`.

Updating the other projects that use `clap` is more involved as they use the lower-level interface.
2022-06-16 14:15:40 +12:00
Cheick Keita
e73aa705fd
fix typo (#2054) 2022-06-15 12:56:23 -07:00
Cheick Keita
286ebcc07b
Adding Node State to Node Heartbeat (#2053) 2022-06-15 18:30:10 +00:00
Cheick Keita
8b826187a5
flushing the logs when the task is done (#2017)
* flushing the logs when the task is done

* removed teh flush logevent
flushig is now done by sending a message to the task_logger directly
moved the initialization of the logger to cmd.rs

* build fix

* include a duration in the flush message

* exit with a done state instead of panicing
2022-06-15 17:53:59 +00:00
George Pollard
ab1e422d34
Attempt to improve sccacheability (#2010)
Attempt to speed up CI builds:
1. Disable incremental builds when using `sccache` (incremental can't be cached)
2. Set `--locked` on all invocations of cargo
3. Add `restore-keys` for Github cache action
2022-06-14 21:49:57 +00:00
Teo Voinea
447f1813b1
Enable queue_signalr_events (#2047) 2022-06-14 18:25:16 +00:00