Commit Graph

143 Commits

Author SHA1 Message Date
cf3d904940 address formatting from black 21.4b0 (#831) 2021-04-26 12:35:16 -04:00
c5e0163068 catch VM SKU & VM Image generation mismatch failures (#803) 2021-04-14 14:34:12 -04:00
c8572cd55a remove timestamp from WebhookMessageLog model (#804) 2021-04-14 12:49:46 -04:00
627463d94b only record the first failure if a task has multiple failures (#797) 2021-04-13 17:36:56 -04:00
470e95c833 add Timestamp to multiple models (#796)
Expose the Azure storage table's "Timestamp" for the models where Timestamp should be user-accessible and remove the Timestamp field from models that did not sign up for it.

The behavior where Timestamp is only set by Azure Storage is kept.
2021-04-13 19:03:25 +00:00
39464dc606 simplify removing UserInfo from events prior to logging (#795) 2021-04-13 15:47:08 +00:00
46b8bdccbc add TaskConfig to crash_reported and regression_reported events (#793)
resolves #757 and #758
2021-04-13 10:24:12 +00:00
80b3533f83 Report the setup failure in the task when available (#781) 2021-04-09 08:57:56 -04:00
e21eafd135 clarify telemetry key names at the service level (#769) 2021-04-05 15:23:03 -04:00
ca12904684 add log checking to refactored integration check (#700)
In practice, Application Insights can take up to 3 minutes before something sent to it is available via KQL.

This PR logs a start and stop marker such that the integration tests only search for logs during the integration tests. This reduces the complexity when using the integration tests during the development process.

Note: this migrated the new functionality from #356 into the latest integration test tools.
2021-04-02 21:49:19 +00:00
7e5cf780a6 Added support for multi tenant authentication (#746)
## Summary of the Pull Request

_What is this about?_

## PR Checklist
* [x] Applies to work item: #562 
* [x] CLA signed. If not, go over [here](https://cla.opensource.microsoft.com/microsoft/onefuzz) and sign the CLI.
* [x] Tests added/passed
* [ ] Requires documentation to be updated
* [x] I've discussed this with core contributors already. If not checked, I'm ready to accept this work might be rejected in favor of a different grand plan. Issue number where discussion took place: #xxx

## Info on Pull Request

The end-to-end changes needed to have onefuzz deployed with multi-tenant authentication.

## Validation Steps Performed

_How does someone test & validate?_
2021-04-02 14:39:20 +00:00
3096f99e86 enable using ephemeral disks by default (#461) 2021-03-30 18:48:44 -04:00
3eb7c8643b set expect_crash_on_failure default to False on libFuzzer tasks (#748) 2021-03-30 21:51:15 +00:00
92b5139a0a Removing UserInfo from notifications logging (#724) 2021-03-23 18:47:05 -04:00
1706a91291 Removing UserInfo from 'created task' logging (#725) 2021-03-23 18:45:18 -04:00
516b1e000e expose minimized_stack_depth functionality in the CLI/API (#715) 2021-03-23 10:09:34 -04:00
6e60a8cf10 add regression testing tasks (#664) 2021-03-18 15:37:19 -04:00
7ebdeac537 Added UserInfo Filter Logging Function (#661)
## Summary of the Pull Request

_What is this about?_
Due to our GDPR privacy requirements, we decided that it would be best to completely purge personal identifiable information from our AppInsights telemetry and logging. Instead of just removing all of the logging statements with personal info, I created a filter function that logs telemetry after it's been run through a recursive scrubbing function. This PR includes this new scrubbing function. 

## PR Checklist
* [x] Applies to work item: #660
* [ ] CLA signed. If not, go over [here](https://cla.opensource.microsoft.com/microsoft/onefuzz) and sign the CLI.
* [ ] Tests added/passed
* [ ] Requires documentation to be updated
* [x] I've discussed this with core contributors already. If not checked, I'm ready to accept this work might be rejected in favor of a different grand plan. Issue number where discussion took place: #xxx

## Info on Pull Request

_What does this include?_
Includes changes to events.py in onefuzzlib. I've implemented functionality - log_event() - to recursively check Event structures for UserInfo before logging to AppInsights. 

## Validation Steps Performed
I run local tests using a script I created with test events. 

_How does someone test & validate?_
I can provide local testing script. If that is insufficient, I can write a unit test that will run against this code.
2021-03-15 23:56:00 +00:00
a3fdc74c53 handle exception related to manually deleted scalesets (#672)
If a user manually deletes a scaleset managed by OneFuzz, then `get_vmss_size` returns None.

When this happens, `Scaleset.shutdown` generates an exception from the `logging.info` call on line 573.

This PR handles this edge condition.
2021-03-15 14:18:59 +00:00
6888fc8fb8 send EventTaskFailed and EventTaskStopped once the task is stopped (#651)
As is, these events are sent once the task enters the state `stopping`.
However, the tasks can still be running on the VMs which can be
confusing.
2021-03-12 01:48:28 +00:00
14c7d5e4d9 mark dependant tasks failed upon failure (#650)
Fix #644
2021-03-11 22:24:43 +00:00
b4ceb263e0 stop jobs once all tasks are stopped (#649)
Fixed #643
2021-03-09 20:09:18 +00:00
0a3812d8bc Add job stopped task info (#648) 2021-03-09 10:06:06 -05:00
23dc8ad301 explain the source of task failures related notifications (#635) 2021-03-06 13:35:09 +00:00
92c1d0a7a1 only set VM passwords on Windows (#620) 2021-03-03 21:27:56 +00:00
78d6adf555 upgrade azure-storage-blob to 12.8.0 (#625)
Note, this makes use of the new feature, `ContainerClient.exists()` which models our existing mechanism for doing container existence checking.
2021-03-03 10:33:23 +00:00
04fc41597e require target_exe to be a canonicalized relative path (#613) 2021-03-02 19:06:02 -05:00
7f66eeee0d handle OperationNotAllowed errors when creating VMSS (#614) 2021-03-02 16:14:10 -05:00
8600a44f1f fix bool queries (#597)
This addresses broken queries used for identifying outdated nodes.
2021-02-26 16:51:05 +00:00
fb482e357e don't schedule work to a node if the scaleset or pool is shutting down (#583) 2021-02-23 13:33:41 -05:00
cebb84b9e7 handle error condition when creating a container that is being deleted (#582)
When users try to create a container immediately after deleting it, Azure will fail saying the deletion is in-progress.

catching ResourceExistsError during create handles this error.
2021-02-22 01:49:07 +00:00
feb80ecb54 allow nodes with multiple tasks to continue on task stop (#567)
As is, when multiple tasks are running on a single node, if any one of them stops, the node gets reimaged.

This changes the behavior such that when a node with multiple tasks has one task stop, the other tasks will continue.
2021-02-19 23:54:26 +00:00
6ba5795f36 update proxy port ranges to avoid current blocks (#552) 2021-02-19 17:50:09 -05:00
4de19ffe5e stop jobs that do not start within 30 days (#565)
If a job does not start within 30 days, stop the job and mark all of the tasks as `failed`.
2021-02-19 21:23:35 +00:00
305c23a4d9 add instance information to webhooks (#577)
Fixes #574
2021-02-19 21:00:51 +00:00
8ce4638b8a clarify scaleset logging (#568) 2021-02-19 19:36:16 +00:00
4992b494f1 add task config to all task events (#580) 2021-02-19 14:10:48 -05:00
872a5ddc14 add details to exceptions generated during report render failures (#576) 2021-02-19 13:48:49 -05:00
3a7bc95316 import local relative paths (#579) 2021-02-19 12:29:35 -05:00
929d9ce496 make user triggered reimaging happen immediately (#566) 2021-02-18 14:08:25 -05:00
279629292f handle SkuNotAvailable errors when creating VM Scalesets (#557) 2021-02-17 16:52:37 -05:00
bdcab6eb08 handle tokens from x-ms-token-aad-id-token (#531) 2021-02-10 12:41:15 -05:00
8ee7fae240 use the cached Azure Identity instance for storage operations (#526) 2021-02-09 12:20:12 -05:00
91a3690551 fix logging to show corpus accounts found, not a function ref (#524) 2021-02-09 11:55:52 -05:00
5114332ea0 clarify proxy log messages (#520) 2021-02-08 17:39:26 -05:00
8c9f65c0be add missing scaleset nodes (#518) 2021-02-08 13:50:08 -05:00
1d74379a70 use the primitive types in more places (#514) 2021-02-05 13:10:37 -05:00
e3dfcb8b95 Scalesets that are about to be deleted don't need updated configs (#511) 2021-02-05 09:53:29 -05:00
3cb055d331 clarify message upon service & agent version mismatch (#510) 2021-02-04 19:58:45 -05:00
a02e084522 split out node, scaleset, and pool code (#507) 2021-02-04 19:07:49 -05:00