Commit Graph

311 Commits

Author SHA1 Message Date
6e60a8cf10 add regression testing tasks (#664) 2021-03-18 15:37:19 -04:00
7ebdeac537 Added UserInfo Filter Logging Function (#661)
## Summary of the Pull Request

_What is this about?_
Due to our GDPR privacy requirements, we decided that it would be best to completely purge personal identifiable information from our AppInsights telemetry and logging. Instead of just removing all of the logging statements with personal info, I created a filter function that logs telemetry after it's been run through a recursive scrubbing function. This PR includes this new scrubbing function. 

## PR Checklist
* [x] Applies to work item: #660
* [ ] CLA signed. If not, go over [here](https://cla.opensource.microsoft.com/microsoft/onefuzz) and sign the CLI.
* [ ] Tests added/passed
* [ ] Requires documentation to be updated
* [x] I've discussed this with core contributors already. If not checked, I'm ready to accept this work might be rejected in favor of a different grand plan. Issue number where discussion took place: #xxx

## Info on Pull Request

_What does this include?_
Includes changes to events.py in onefuzzlib. I've implemented functionality - log_event() - to recursively check Event structures for UserInfo before logging to AppInsights. 

## Validation Steps Performed
I run local tests using a script I created with test events. 

_How does someone test & validate?_
I can provide local testing script. If that is insufficient, I can write a unit test that will run against this code.
2021-03-15 23:56:00 +00:00
a3fdc74c53 handle exception related to manually deleted scalesets (#672)
If a user manually deletes a scaleset managed by OneFuzz, then `get_vmss_size` returns None.

When this happens, `Scaleset.shutdown` generates an exception from the `logging.info` call on line 573.

This PR handles this edge condition.
2021-03-15 14:18:59 +00:00
6888fc8fb8 send EventTaskFailed and EventTaskStopped once the task is stopped (#651)
As is, these events are sent once the task enters the state `stopping`.
However, the tasks can still be running on the VMs which can be
confusing.
2021-03-12 01:48:28 +00:00
14c7d5e4d9 mark dependant tasks failed upon failure (#650)
Fix #644
2021-03-11 22:24:43 +00:00
b4ceb263e0 stop jobs once all tasks are stopped (#649)
Fixed #643
2021-03-09 20:09:18 +00:00
0a3812d8bc Add job stopped task info (#648) 2021-03-09 10:06:06 -05:00
23dc8ad301 explain the source of task failures related notifications (#635) 2021-03-06 13:35:09 +00:00
92c1d0a7a1 only set VM passwords on Windows (#620) 2021-03-03 21:27:56 +00:00
78d6adf555 upgrade azure-storage-blob to 12.8.0 (#625)
Note, this makes use of the new feature, `ContainerClient.exists()` which models our existing mechanism for doing container existence checking.
2021-03-03 10:33:23 +00:00
04fc41597e require target_exe to be a canonicalized relative path (#613) 2021-03-02 19:06:02 -05:00
4489036d9f add node & task heartbeat events (#621)
This adds node & task heartbeats and makes the event data available as a structured data in the logs.
2021-03-02 22:04:39 +00:00
7f66eeee0d handle OperationNotAllowed errors when creating VMSS (#614) 2021-03-02 16:14:10 -05:00
c537458ade update azure-mgmt-compute to 19.0.0 (#611) 2021-03-02 18:44:26 +00:00
296ba2ee23 update azure-mgmt-storage to 17.0.0 (#612) 2021-03-02 18:00:07 +00:00
32681b2611 update azure-mgmt-resource (#607) 2021-03-02 08:35:07 +00:00
06f45f338c Update Task Heartbeat to include Job_id (#594) 2021-02-26 13:36:10 -05:00
8600a44f1f fix bool queries (#597)
This addresses broken queries used for identifying outdated nodes.
2021-02-26 16:51:05 +00:00
daef1637f8 update jinja2 (#595) 2021-02-26 09:19:10 -05:00
fb482e357e don't schedule work to a node if the scaleset or pool is shutting down (#583) 2021-02-23 13:33:41 -05:00
cebb84b9e7 handle error condition when creating a container that is being deleted (#582)
When users try to create a container immediately after deleting it, Azure will fail saying the deletion is in-progress.

catching ResourceExistsError during create handles this error.
2021-02-22 01:49:07 +00:00
feb80ecb54 allow nodes with multiple tasks to continue on task stop (#567)
As is, when multiple tasks are running on a single node, if any one of them stops, the node gets reimaged.

This changes the behavior such that when a node with multiple tasks has one task stop, the other tasks will continue.
2021-02-19 23:54:26 +00:00
6ba5795f36 update proxy port ranges to avoid current blocks (#552) 2021-02-19 17:50:09 -05:00
4de19ffe5e stop jobs that do not start within 30 days (#565)
If a job does not start within 30 days, stop the job and mark all of the tasks as `failed`.
2021-02-19 21:23:35 +00:00
305c23a4d9 add instance information to webhooks (#577)
Fixes #574
2021-02-19 21:00:51 +00:00
8ce4638b8a clarify scaleset logging (#568) 2021-02-19 19:36:16 +00:00
4992b494f1 add task config to all task events (#580) 2021-02-19 14:10:48 -05:00
872a5ddc14 add details to exceptions generated during report render failures (#576) 2021-02-19 13:48:49 -05:00
3a7bc95316 import local relative paths (#579) 2021-02-19 12:29:35 -05:00
929d9ce496 make user triggered reimaging happen immediately (#566) 2021-02-18 14:08:25 -05:00
279629292f handle SkuNotAvailable errors when creating VM Scalesets (#557) 2021-02-17 16:52:37 -05:00
bdcab6eb08 handle tokens from x-ms-token-aad-id-token (#531) 2021-02-10 12:41:15 -05:00
8ee7fae240 use the cached Azure Identity instance for storage operations (#526) 2021-02-09 12:20:12 -05:00
91a3690551 fix logging to show corpus accounts found, not a function ref (#524) 2021-02-09 11:55:52 -05:00
5114332ea0 clarify proxy log messages (#520) 2021-02-08 17:39:26 -05:00
8c9f65c0be add missing scaleset nodes (#518) 2021-02-08 13:50:08 -05:00
fd995718c8 standardize on unix file endings (#516)
The majority of our source files have `\n` line endings.  This updates the few files that use `\r\n` to use `\n`.
2021-02-06 14:14:26 +00:00
1d74379a70 use the primitive types in more places (#514) 2021-02-05 13:10:37 -05:00
51f4eea069 update nodes and scalesets in a consistent order (#512) 2021-02-05 12:51:38 -05:00
e3dfcb8b95 Scalesets that are about to be deleted don't need updated configs (#511) 2021-02-05 09:53:29 -05:00
3cb055d331 clarify message upon service & agent version mismatch (#510) 2021-02-04 19:58:45 -05:00
a02e084522 split out node, scaleset, and pool code (#507) 2021-02-04 19:07:49 -05:00
c5bb0f0588 Update Proxy heartbeat & logging (#502) 2021-02-04 15:38:17 -05:00
1adee4ea83 update pools before nodes or scalesets (#503) 2021-02-04 13:26:19 -05:00
e0cd3d535b rename proxy queue endpoint (#504) 2021-02-04 12:45:36 -05:00
0a4278110d update azure-mgmt-compute to 18.2.0 (#499) 2021-02-04 12:34:29 +00:00
5e2e9448df add security auditing of python code using Bandit during CICD (#491) 2021-02-01 16:51:03 -05:00
0f70ffa3e2 try pushing updates to scaleset configs frequently until the push succeeds (#489) 2021-02-01 10:09:40 -05:00
a46f7b4193 expose supervisor tasks that are fully self-contained fuzzing tasks in the service (#474)
Exposes the functionality added in #454 to the service & CLI.

Fixes #439
2021-01-29 00:01:59 +00:00
14fc1ca51f remove unused Event generation from the pre-2.0.0 SignalR integration (#477)
Remove a vestige of the adhoc events used by the previous SingalR integration for container updates.
2021-01-28 21:56:31 +00:00