Commit Graph

984 Commits

Author SHA1 Message Date
a1502b7d58 move to using rustls (#1029) 2021-06-30 12:27:43 -04:00
59ad1f9b9b save the agents in a per-$(uname) directory (#1030) 2021-06-30 11:33:05 -04:00
2f71689d48 initial support for running the agent locally on OSX (#1016) 2021-06-29 16:20:23 -04:00
b5388d360c add min_length specifications to Github Issue Template configs (#1019) 2021-06-25 18:04:25 +00:00
bb7419186f bump github actions cache key IDs (#1024) 2021-06-25 13:32:48 -04:00
6a0c8e656b release 2.23.1 (#1025) 2.23.1 2021-06-25 12:19:19 -04:00
883c93aaf4 ensure VM IDs are unique before calling Azure reimage/delete APIs (#1023) 2021-06-25 11:54:52 -04:00
880039a617 release 2.23.0 (#1020) 2.23.0 2021-06-24 16:25:08 -04:00
5f72cc20b8 Fix issue when deserializing GithubIssueTemplate with hidden secret (#1008) 2021-06-24 13:50:54 -04:00
4c9a9f2948 address issues from latest clippy (#1018) 2021-06-24 12:13:16 -04:00
27b434e996 fix debug report (#1011)
The debug report created by the command `onefuzz debug  notification job <job id>`  is causing a crash in the regression task 
```
error running task: libfuzzer regression

Caused by:
    0: handling crash reports
    1: unable to parse crash report: fake-crash-sample.json
    2: unable to parse report: task_unique_reports_2/fake-crash-sample.json - "{\"input_url\": null, \"input_blob\": {\"account\": \"fuzz27ee6imdmr5gy\", \"container\": \"oft-crashes-cecbd958a1f257688f9768edaaf6c94d\", \"name\": \"fake-crash-sample\"}, \"executable\": \"fuzz.exe\", \"crash_type\": \"fake crash report\", \"crash_site\": \"fake crash site\", \"call_stack\": [\"#0 fake\", \"#1 call\", \"#2 stack\"], \"call_stack_sha256\": \"0000000000000000000000000000000000000000000000000000000000000000\", \"input_sha256\": \"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\", \"asan_log\": \"fake asan log\", \"task_id\": \"b1107de0-c3cb-43ff-ab68-5accc579f4d4\", \"job_id\": \"afa45e3e-9a75-4a47-8d59-ef3154599fc7\", \"scariness_score\": null, \"scariness_description\": null, \"minimized_stack\": null, \"minimized_stack_sha256\": null, \"minimized_stack_function_names\": null, \"minimized_stack_function_names_sha256\": null}"
```
2021-06-24 00:57:58 +00:00
10d2e3e366 update azure-keyvault-secrets to 4.3.0 (#1012) 2021-06-23 18:27:32 -04:00
5f8e423265 remove nodes from db upon reimage (#1005)
The flag `Node.reimage_queued` is intended to stop nodes from reimaging repeatedly.  

In #970, in order to work around Azure API failures, this flag was cycled if the node was already set to cleanup.  Unfortunately, reimaging can sometimes take a significant amount of time, causing this change to get nodes multiple times.

Instead of using `reimage_queued` as a flag, this PR deletes the node from the storage table upon reimage.  When the node registers OR the next time through `Scaleset.cleanup_nodes`, the Node will be recreated automatically, whichever comes first.
2021-06-23 22:25:15 +00:00
7c549e7fbb use the refreshed token when retrying after auth failure (#1017) 2021-06-23 18:03:51 -04:00
50652c2e48 mark tasks as failed when the node is being reimaged due to heartbeat issues (#1015) 2021-06-23 16:39:47 -04:00
97cba8eda1 update third-party Rust dependencies (#1009) 2021-06-22 11:51:38 -04:00
778f09efaf normalize log retention (#1007) 2021-06-22 07:59:17 -04:00
f450e937a8 update rust dependencies (#1004) 2021-06-18 12:58:13 -04:00
c8862c7f18 add cooloff period for rapidly exiting libFuzzer targets (#1002) 2021-06-18 11:40:36 -04:00
888c0459ad release 2.22.0 (#1001) 2.22.0 2021-06-17 17:54:47 -04:00
cb895d09c7 fix GithubIssueTemplate deserialization (#990) 2021-06-17 13:02:28 -04:00
2ff9c47e0c Explicitly track debuggee thread state to detect thread exit/debug event races (#1000)
- Define an enum to track the debugger's current understanding of debuggee thread state
- Update our private suspend/resume methods to update and return the current state
- Detect thread exit/debug event races in suspend/resume calls
2021-06-16 14:45:43 -07:00
a58df95573 reduce log retention duration to 30 days (#997) 2021-06-16 17:12:53 -04:00
5aba12716a fix SyncedDir.remote_url issue with relative paths (#996) 2021-06-16 14:13:06 -04:00
5efc4f959c add context to errors in syncdir (#995) 2021-06-16 12:49:24 -04:00
c71ce580bd updates from upstream libclusterfuzz (#992) 2021-06-15 16:32:13 -04:00
0d709ed5f8 fix markdown link (#989) 2021-06-14 16:40:04 -04:00
93cfedbd2e fix scaleset identity name in the documentation (#987) 2021-06-14 16:10:14 -04:00
ef5eb12abe set azcopy retry notice to debug rather than warn (#986) 2021-06-14 15:38:41 -04:00
b9950c5526 update log messages to ease debugging (#988) 2021-06-14 15:18:03 -04:00
4472d584ac handle serialization of secrets sent from the CLI (#985)
This normalizes the SecretData serialization from the client to address #981.

When serializing objects sent to the service with secrets, we would turn it into a SecretData

We use SecretData to convert this:
`{"auth": {"user": "A", "personal_access_token": "B"}}`
to this:
`"auth": { "secret": { "url": "https://KEYVAULT-URL" }}`

Currently, in the case we have a SecretData we've not yet saved, the serialized form looks like this:
`{"auth": { "secret": {"user": "A", "personal_access_token": "B"}}}`

This PR simplifies the client side serialization to this:
`{"auth": {"user": "A", "personal_access_token": "B"}}`
2021-06-12 14:39:14 +00:00
bcdae2d5cb Check scaleset size for missing nodes (#984) 2021-06-11 18:47:21 -04:00
c0b3a409e4 add monkeypatch to hotfix pydantic Union issues (#982)
Until Pydantic supports discriminated or "smart" unions, we need to work around the coercion issue impacting unions in our models.

This reuses the "smart union" implementation from https://github.com/samuelcolvin/pydantic/pull/2092
2021-06-11 22:00:06 +00:00
a0bce1a538 fix markdown link (#983) 2021-06-11 08:09:28 -04:00
75cc11af3a release 2.21.0 (#980) 2.21.0 2021-06-10 15:12:21 -04:00
1a059f2c40 Add context to coverage tracing errors (#979) 2021-06-10 12:41:32 -04:00
b39f0964c9 Retry coverage recording (#978) 2021-06-10 10:40:32 -04:00
3ec32160cb fix service authentication refresh errors (#976) 2021-06-09 19:11:18 -04:00
2be1edd9dc handle reimaging failures by resetting reimage_queued (#970)
In a previous commit, reimage_queued was added to prevent reimaging a node while it is reimaging.  However, this means reimaging failures due to Azure issues don't finish reimaging.

This will reset the this flag allowing the node to reimage in the following cleanup cycle.
2021-06-09 18:58:56 +00:00
4f6e4dbbec add types-requests (#973) 2021-06-09 13:01:17 -04:00
da931b3a5c address issues raised from latest mypy (#972) 2021-06-09 12:04:24 -04:00
3794bb0c68 move to wiremock for testing (#971) 2021-06-09 12:04:10 -04:00
78d5a58965 allow updates to a scaleset in the resize state (#969) 2021-06-09 13:10:43 +00:00
af39d25a7d reimage/delete expired nodes even with the debug_keep_node flag (#968)
Fixes #965
2021-06-08 17:37:10 +00:00
ed289c9a3c handle scaleset resize exceptions (#967) 2021-06-08 09:30:36 -04:00
2d377d1423 add onefuzz debug job rerun JOB_ID command (#960)
This allows the specification of an alternate pool or duration when rerunning an existing job.
2021-06-07 20:53:34 +00:00
52f11cc097 release 2.20.0 (#964) 2.20.0 2021-06-07 12:48:33 -04:00
c0e605146c remove RequestBuilder abstraction in onefuzz-supervisor coordinator (#963) 2021-06-03 22:28:54 -04:00
6b7906dfb1 libfuzzer fuzzing task perf improvements (#941)
- resuse the regex to parse the output of libfuzzer
- added a cancellation notification to report_fuzzer_sys_info.
   ~~The code seems to be actively waiting this function and consuming some cpu time~~
   The notification allows us to reduce the time waiting for the fuzzing loop to terminate. 

## Summary of the Pull Request

_What is this about?_

## PR Checklist
* [ ] Applies to work item: #xxx
* [ ] CLA signed. If not, go over [here](https://cla.opensource.microsoft.com/microsoft/onefuzz) and sign the CLI.
* [ ] Tests added/passed
* [ ] Requires documentation to be updated
* [ ] I've discussed this with core contributors already. If not checked, I'm ready to accept this work might be rejected in favor of a different grand plan. Issue number where discussion took place: #xxx

## Info on Pull Request

_What does this include?_

## Validation Steps Performed

_How does someone test & validate?_
2021-06-04 00:14:15 +00:00
2c72bd590f Add generic coverage task (#763)
**Todo:**
- [x] Finalize format for coverage file(s)
- [x] Add service support
- [x] Integration test
- [x] Merge #926 
- [x] Merge #929
2021-06-03 23:36:00 +00:00