Commit Graph

1141 Commits

Author SHA1 Message Date
22faa1b5db instance config (#1791)
* instance config

* address PR comments

* make logs scoped

* make logs scoped

Co-authored-by: stas <statis@microsoft.com>
2022-04-14 14:20:28 -07:00
b03d420804 Allow some safe functions with HANDLE arguments (#1792) 2022-04-14 12:29:36 -07:00
9d8d3327d2 Migrating QueueTaskHeartbeat (#1777)
* Migrating QueueTaskHeartbeat

* changing the name of the input queue

* rename type alias Tasks to Async

* Fix property casing

* fixing types

* Removing IStorageProvider

* fix function name

* address PR comments
2022-04-14 00:19:44 +00:00
31b9163514 Release 5.3.0 (#1767)
* Release 5.3.0

* Update CHANGELOG.md

Co-authored-by: Joe Ranweiler <joe@lemma.co>

* Update CHANGELOG.md

Co-authored-by: Joe Ranweiler <joe@lemma.co>

* Update CHANGELOG.md

Co-authored-by: Joe Ranweiler <joe@lemma.co>

* Update CHANGELOG.md

Co-authored-by: Joe Ranweiler <joe@lemma.co>

* Update CHANGELOG.md

Co-authored-by: Joe Ranweiler <joe@lemma.co>

* Update comments

Co-authored-by: Joe Ranweiler <joe@lemma.co>
2022-04-13 15:51:49 -07:00
febaf6e057 make logger immutable (#1783)
Co-authored-by: stas <statis@microsoft.com>
2022-04-12 11:24:48 -07:00
87eb606b35 Delete nodes when they're done (#1763)
* Delete nodes when they're done

* Missed a file

* Load node disposal strategy from env var

* Lint

* Fix subtle bug

* Deleting doesn't work, will 'decomission' nodes once they complete work

* Missed a file

* Remove logging line
2022-04-12 17:32:15 +00:00
faaa5d2d78 bump dependencies to latest (#1782)
remove/implement "cached todo" from some methods,

Co-authored-by: stas <statis@microsoft.com>
2022-04-12 09:48:34 -07:00
50637d4791 switch to our logger (#1780)
* switch to our logger

* preserve correlation id when calling CorpusAccounts

Co-authored-by: Stas Tishkin <statis@microsoft.com>
2022-04-12 08:32:55 -07:00
75039a96eb [C# Port] Adding new Proxy Update Queue Function. (#1757)
* Adding QueueProxyUpdate.

* Setting to serializer.

* Updates.

* Updating with new ORM model and [model]Operation.

* Fixing return type.

* Working on changes.

* Tested and ready for review.

* Formatting.

* Removing test code.

* Update src/ApiService/Tests/OrmTest.cs

Co-authored-by: Cheick Keita <chkeita@microsoft.com>

* Fixing tests.

* Fixing tests again.

* Asserting null in tests.

* Adding null param.

* Removing whitespace.

* syntax error.

Co-authored-by: Cheick Keita <chkeita@microsoft.com>
2022-04-11 13:32:43 -07:00
5aceb25843 Finishing implementation of SendEvent (#1765) 2022-04-11 09:32:09 -07:00
9323d78c19 Bump serde from 1.0.130 to 1.0.136 in /src/agent (#1618)
Bumps [serde](https://github.com/serde-rs/serde) from 1.0.130 to 1.0.136.
- [Release notes](https://github.com/serde-rs/serde/releases)
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.130...v1.0.136)

---
updated-dependencies:
- dependency-name: serde
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Marc Greisen <mgreisen@microsoft.com>
2022-04-08 11:44:55 -07:00
d95d46a420 Add formatting step to the build And apply default formatting (#1762)
* Add formatting step to the build

* apply formatting

* Adding .editorconfigfile and option to remove "unused using directive errors"
2022-04-07 21:44:54 +00:00
3d527908fb Update Cobertura Output (#1728)
* Updated cobertura to have directory as package and filename as class name

* Format

* Update to get line-rate

* Update test

* Format

* clean up

* Remove unneeded typing

* Unduplicating work, part 1

* Unduplicate code part 2

* Unduplicate code part 3

* Unduplicate code part 4

* Added in panic message

* Fixed lint errors

* Fixed lint error

* Changing line rate computation function, also changing 32-bit data types to 64-bit

* Create and use new Line Values type

* Pass in slice rather than vector

* Refactor

* Format

* Initial handling of panic

* Updated panic handling for path parent

* Update test to reflect invalid file format checking

* Fixed path error message

* Fixed match statment

* Make paths in test consistent

* Inital attempt to properly deal with paths on different OSs

* Rework of how to deal with different paths on differents OSs

* Updated path handling

* format

* Seperate out function to get directory

* Seperate out conversion to new path

* Updated path coversion to make it into standarized format

* Add in test XML file to compare to

* Adding in unit tests with differing path formats

* Improved type usage

* Adding some additional unit tests to test specific path conversion scenarios

* Commiting suggested changes

* Renaming tests for discovery

* Split out tests

* Determining issue with converting windows to posix paths

* Format

* Determining issue with converting windows to posix paths, update

* Manually converting windows paths to posix paths

* Remove unneeded test

* Initial code for comparing parsed xml

* revert and fix replace method

* make paths lowercase

* make class name only file name

* format
2022-04-07 13:14:00 -07:00
627b401c87 Continue port of QueueNodeHearbeat (#1761)
[WIP ] continue port of  QueueNodeHearbeat
2022-04-07 11:55:12 -07:00
5004b7902a Refactor signal events azure function (#1760) 2022-04-07 11:09:53 -04:00
c5f60dcaec Refactor queue file changes (#1755)
* Untested refactor

* Untested refactor

* It works with some test messages

* Use custom deserializer

* Clean up warnings
2022-04-06 20:37:52 +00:00
8299d8fb57 Using existing auto scale settings isn't an error (#1745) 2022-04-06 12:41:58 +00:00
a603b17436 Re enable Nullable on the test project (#1753) 2022-04-06 07:58:37 -04:00
13b2a6ef80 Add dotnet project to the Continuous integration (#1750) 2022-04-05 23:06:22 +00:00
9eb7f019a9 merge models (#1749)
Co-authored-by: stas <statis@microsoft.com>
2022-04-05 14:54:11 -07:00
2877676f2b partial port of user_credentials.py (#1744)
Co-authored-by: stas <statis@microsoft.com>
2022-04-05 13:47:57 -07:00
78f8e3215f Starting migration of QueueNodeHearbeat (#1742)
* Add support for etag and timestamp
Introducing EntityBase
Starting migration of QueueNodeHearbeat

* rename namespaces

* upgrade Microsoft.Azure.Functions.Worker to 1.6.0
Added support when name contains underscore tot the case converter

* Support for not renaming enum fields

* bug fixes

* Arm client created in the contructor
added null check
2022-04-05 20:08:46 +00:00
0c3d9fcad2 Add CLI command to download agent logs (#1723)
* It does some things

* Download logs from job config

* Lint

* Make mypy happy

* Update to handle the new logs path

* progress

* A job might not have logs set in config

* Mypy wanted a type annotation
2022-04-05 15:35:15 -04:00
be27d430cd Bump structopt from 0.3.25 to 0.3.26 in /src/agent (#1617)
Bumps [structopt](https://github.com/TeXitoi/structopt) from 0.3.25 to 0.3.26.
- [Release notes](https://github.com/TeXitoi/structopt/releases)
- [Changelog](https://github.com/TeXitoi/structopt/blob/master/CHANGELOG.md)
- [Commits](https://github.com/TeXitoi/structopt/compare/v0.3.25...v0.3.26)

---
updated-dependencies:
- dependency-name: structopt
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Marc Greisen <mgreisen@microsoft.com>
2022-04-04 15:12:26 -07:00
26911c82db Setting up the ORM (#1738)
* setting up orm

* Add unit tests

* Added support for object serilization
added tests

* moving stuff aroung

* Fix handling of enum flags cases
removing dependency to CaseExtension
Adding more tests

* add support for property rename

* caching EntityInfo

* remove access to environment variable

* formatting

* renaming dbName to columnName
2022-04-04 16:58:52 +00:00
fbff3fc4af Logging (#1737)
* Logging

* Doing dependency injection

* expose GetLoggers for better testing

Co-authored-by: stas <statis@microsoft.com>
2022-04-02 15:36:08 -07:00
f11e79de4b Starting work... - add http client - add package lock - set namespace to Microsoft.OneFuzz.Service - add static class to get environment variables (#1736)
Co-authored-by: stas <statis@microsoft.com>
2022-04-01 13:20:08 -07:00
9cbbf86e36 Add the setup state to the unavailable list (#1731)
* Add the setup state to the unavailable list

* Add the init state to the unavailable list

* Fix integration test

* Wrong import
2022-04-01 14:00:25 -04:00
20d3df0a11 Deploy dotnet Azure function alongside Python Azure function (#1733)
Co-authored-by: stas <statis@microsoft.com>
2022-04-01 09:42:06 -07:00
e2d554da13 ApiService solution in C# (#1734) 2022-04-01 01:14:12 +00:00
2ffeadfa35 switch to bicep template only and bicep refactor (#1732)
* switch to bicep template only and bicep refactor

* correct monitorAccount name

Co-authored-by: stas <statis@microsoft.com>
2022-03-31 13:01:02 -07:00
dc354cffe3 port arm template to bicep (#1724)
* port template to bicep

* Update src/deployment/azuredeploy.bicep

Co-authored-by: Teo Voinea <58236992+tevoinea@users.noreply.github.com>

* port template to bicep

* adding type annotation

* apply changes from #1679

Co-authored-by: stas <statis@microsoft.com>
Co-authored-by: Teo Voinea <58236992+tevoinea@users.noreply.github.com>
2022-03-31 08:18:44 -07:00
a2e87c6158 Upload logs of the agent (#1721)
* Setting the service side of the log management
- a log is created or reused when e create a job
- when scheduling the task we send the log location to the agent
The expected log structure looks like
{fuzzContainer}/logs/{job_id}/{task_id}/{machine_id}/1.log
2022-03-30 15:20:42 -07:00
7add51fd3a Log redirection, service side (#1727)
* Setting the service side of the log management
- a log is created or reused when e create a job
- when scheduling the task we send the log location to the agent
The expected log structure looks liek
{fuzzContainer}/logs/{job_id}/{task_id}/{machine_id}/1.log

* regenerate doces

* including job_id in the container name

* regenerating docs
removing bad doc file
2022-03-29 18:47:20 +00:00
424ffdb4b5 Adding auto scale via cli (#1717)
* Initial implementation for adding auto scale via cli

* Remove unused argument

* Remove unused import

* I had a 👻extra line👻
2022-03-29 09:50:07 -04:00
5e31ba5b18 Consolidating Log Analytics References & Definitions (#1679)
* Consolidating Log Analytics References & Definitons.

* Updating variable name.

* Adding vm insights var name.

* removing bad files.

* Bad file.

* Fixing var.

* Adding new variables for all resources names.

* Removing autoscale changes.

Co-authored-by: Hayley Call <Hayley.Call@microsoft.com>
2022-03-28 13:34:38 -07:00
24454e3681 pin click to fix black (#1726)
* pin click to fix black

* missed a couple

Co-authored-by: stas <statis@microsoft.com>
2022-03-28 12:06:13 -07:00
387d5446ab Bump num_cpus from 1.13.0 to 1.13.1 in /src/agent (#1548)
Bumps [num_cpus](https://github.com/seanmonstar/num_cpus) from 1.13.0 to 1.13.1.
- [Release notes](https://github.com/seanmonstar/num_cpus/releases)
- [Changelog](https://github.com/seanmonstar/num_cpus/blob/master/CHANGELOG.md)
- [Commits](https://github.com/seanmonstar/num_cpus/compare/v1.13.0...v1.13.1)

---
updated-dependencies:
- dependency-name: num_cpus
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Marc Greisen <mgreisen@microsoft.com>
2022-03-28 10:00:11 -07:00
ce03394376 Handle instance being destroyed before updating scaling protection (#1719)
* Handle instance being destroyed before updating scaling protection

* Fix bug where we release protection too early
2022-03-28 10:14:39 -04:00
5c418eeb36 Add autoscaling diagnostics (#1708)
* Initial attempt

* Adding diagnostics works

* 🧹

* lint

* I wish the linter could auto fix these issues

* Lint
2022-03-21 13:25:21 +00:00
9d3b0e0f92 Add Linux dynamic library checks using ldd/LD_DEBUG (#1718) 2022-03-17 17:26:31 -07:00
9000a234e9 Add crate to debug missing dynamic libraries on Windows (#1713)
Add a CLI tool and library code to debug missing dynamic library errors on Windows.

The implementation manually edits the registry global flags for an image file to temporarily enable loader snaps, runs the target under our custom debugger to collect the debug output strings, then parses them for informative loading errors. It does not depend on the presence of `gflags.exe`.

This detects both dynamic linking (and thus process startup) errors, as well as dynamic loading (`LoadLibrary`) errors. It can report multiple missing dynamically-linked libraries.
2022-03-16 16:11:26 -07:00
194e7d0e9e Move the event grid topic creation and subscription to the deployment template (#1591)
* move the event grid subscription to the template

* change the name of the new subscription to prevent deleting the wrong subscription

* refactoring

* mypy fix

* format

* format

* remove old event grid before arm deployment

* fix deply

* attempt to fix check-pr issue

* fix interactive login in check-pr

* move the event grid subscription to the tempalte

* change the name of the new subscription to prevent deleting the wrong subscription

* refactoring

* mypy fix

* format

* format

* remove old event grid before arm deployment

* using resource Id

* fix type

* fix location

* revert changes in registration.py

* build fix attempt

* build fix

* revert ci changes

* remove file

* address comment

* address PR comments

* naming

* fix deplyment
2022-03-15 10:48:42 -07:00
ce36de72e7 Generate debuginfo for windows-libfuzzer-load-library test target (#1684) 2022-03-11 09:40:51 -08:00
c1ffec32f4 Reduce log level in ASAN parsing (#1705) 2022-03-10 14:03:58 -08:00
99091e7035 Release 5.2.0 (#1701)
* 5.2.0

* Update CHANGELOG.md

Co-authored-by: Joe Ranweiler <joe@lemma.co>

Co-authored-by: Joe Ranweiler <joe@lemma.co>
5.2.0
2022-03-10 11:30:20 -08:00
d4c92d497f Add type definition (#1703) 2022-03-09 23:52:06 +00:00
40b0e6685a Give function app resource group scoped contributor role (#1698)
* Give function app resource group scoped contributor role

* Reenable autoscaling

* We don't know what the minimum capacity for a sku is yet

* Lint
2022-03-09 13:07:22 -05:00
fa8589a3d6 Bump backoff from 0.3.0 to 0.4.0 in /src/agent, resolve dependabot block (#1589)
* Inital changes needed for backoff 0.4 to work

* Update backoff versions, fix BackoffError:Transient fields, other uses

* Format

* Removed redundant field name

* Improved backoff update changes

* Update backoff update

* Revert

* Changed to using Error::transient function
2022-03-08 16:07:07 -08:00
b22955de9b Upgrade regex crate (#1699)
* Bump to non vulnerable regex version

* Removed unnecessary cargo.toml change
2022-03-08 19:17:29 +00:00