Commit Graph

719 Commits

Author SHA1 Message Date
97cba8eda1 update third-party Rust dependencies (#1009) 2021-06-22 11:51:38 -04:00
778f09efaf normalize log retention (#1007) 2021-06-22 07:59:17 -04:00
f450e937a8 update rust dependencies (#1004) 2021-06-18 12:58:13 -04:00
c8862c7f18 add cooloff period for rapidly exiting libFuzzer targets (#1002) 2021-06-18 11:40:36 -04:00
888c0459ad release 2.22.0 (#1001) 2.22.0 2021-06-17 17:54:47 -04:00
cb895d09c7 fix GithubIssueTemplate deserialization (#990) 2021-06-17 13:02:28 -04:00
2ff9c47e0c Explicitly track debuggee thread state to detect thread exit/debug event races (#1000)
- Define an enum to track the debugger's current understanding of debuggee thread state
- Update our private suspend/resume methods to update and return the current state
- Detect thread exit/debug event races in suspend/resume calls
2021-06-16 14:45:43 -07:00
a58df95573 reduce log retention duration to 30 days (#997) 2021-06-16 17:12:53 -04:00
5aba12716a fix SyncedDir.remote_url issue with relative paths (#996) 2021-06-16 14:13:06 -04:00
5efc4f959c add context to errors in syncdir (#995) 2021-06-16 12:49:24 -04:00
c71ce580bd updates from upstream libclusterfuzz (#992) 2021-06-15 16:32:13 -04:00
0d709ed5f8 fix markdown link (#989) 2021-06-14 16:40:04 -04:00
93cfedbd2e fix scaleset identity name in the documentation (#987) 2021-06-14 16:10:14 -04:00
ef5eb12abe set azcopy retry notice to debug rather than warn (#986) 2021-06-14 15:38:41 -04:00
b9950c5526 update log messages to ease debugging (#988) 2021-06-14 15:18:03 -04:00
4472d584ac handle serialization of secrets sent from the CLI (#985)
This normalizes the SecretData serialization from the client to address #981.

When serializing objects sent to the service with secrets, we would turn it into a SecretData

We use SecretData to convert this:
`{"auth": {"user": "A", "personal_access_token": "B"}}`
to this:
`"auth": { "secret": { "url": "https://KEYVAULT-URL" }}`

Currently, in the case we have a SecretData we've not yet saved, the serialized form looks like this:
`{"auth": { "secret": {"user": "A", "personal_access_token": "B"}}}`

This PR simplifies the client side serialization to this:
`{"auth": {"user": "A", "personal_access_token": "B"}}`
2021-06-12 14:39:14 +00:00
bcdae2d5cb Check scaleset size for missing nodes (#984) 2021-06-11 18:47:21 -04:00
c0b3a409e4 add monkeypatch to hotfix pydantic Union issues (#982)
Until Pydantic supports discriminated or "smart" unions, we need to work around the coercion issue impacting unions in our models.

This reuses the "smart union" implementation from https://github.com/samuelcolvin/pydantic/pull/2092
2021-06-11 22:00:06 +00:00
a0bce1a538 fix markdown link (#983) 2021-06-11 08:09:28 -04:00
75cc11af3a release 2.21.0 (#980) 2.21.0 2021-06-10 15:12:21 -04:00
1a059f2c40 Add context to coverage tracing errors (#979) 2021-06-10 12:41:32 -04:00
b39f0964c9 Retry coverage recording (#978) 2021-06-10 10:40:32 -04:00
3ec32160cb fix service authentication refresh errors (#976) 2021-06-09 19:11:18 -04:00
2be1edd9dc handle reimaging failures by resetting reimage_queued (#970)
In a previous commit, reimage_queued was added to prevent reimaging a node while it is reimaging.  However, this means reimaging failures due to Azure issues don't finish reimaging.

This will reset the this flag allowing the node to reimage in the following cleanup cycle.
2021-06-09 18:58:56 +00:00
4f6e4dbbec add types-requests (#973) 2021-06-09 13:01:17 -04:00
da931b3a5c address issues raised from latest mypy (#972) 2021-06-09 12:04:24 -04:00
3794bb0c68 move to wiremock for testing (#971) 2021-06-09 12:04:10 -04:00
78d5a58965 allow updates to a scaleset in the resize state (#969) 2021-06-09 13:10:43 +00:00
af39d25a7d reimage/delete expired nodes even with the debug_keep_node flag (#968)
Fixes #965
2021-06-08 17:37:10 +00:00
ed289c9a3c handle scaleset resize exceptions (#967) 2021-06-08 09:30:36 -04:00
2d377d1423 add onefuzz debug job rerun JOB_ID command (#960)
This allows the specification of an alternate pool or duration when rerunning an existing job.
2021-06-07 20:53:34 +00:00
52f11cc097 release 2.20.0 (#964) 2.20.0 2021-06-07 12:48:33 -04:00
c0e605146c remove RequestBuilder abstraction in onefuzz-supervisor coordinator (#963) 2021-06-03 22:28:54 -04:00
6b7906dfb1 libfuzzer fuzzing task perf improvements (#941)
- resuse the regex to parse the output of libfuzzer
- added a cancellation notification to report_fuzzer_sys_info.
   ~~The code seems to be actively waiting this function and consuming some cpu time~~
   The notification allows us to reduce the time waiting for the fuzzing loop to terminate. 

## Summary of the Pull Request

_What is this about?_

## PR Checklist
* [ ] Applies to work item: #xxx
* [ ] CLA signed. If not, go over [here](https://cla.opensource.microsoft.com/microsoft/onefuzz) and sign the CLI.
* [ ] Tests added/passed
* [ ] Requires documentation to be updated
* [ ] I've discussed this with core contributors already. If not checked, I'm ready to accept this work might be rejected in favor of a different grand plan. Issue number where discussion took place: #xxx

## Info on Pull Request

_What does this include?_

## Validation Steps Performed

_How does someone test & validate?_
2021-06-04 00:14:15 +00:00
2c72bd590f Add generic coverage task (#763)
**Todo:**
- [x] Finalize format for coverage file(s)
- [x] Add service support
- [x] Integration test
- [x] Merge #926 
- [x] Merge #929
2021-06-03 23:36:00 +00:00
60d1853dcc add retry logic to authorize application (#959) 2021-06-03 17:45:26 -04:00
e28389c343 release 2.19.0 (#955) 2.19.0 2021-06-02 16:42:51 -04:00
0e164107bb remove unused adal import (#954) 2021-06-02 14:00:10 -04:00
c46abbcec3 stabilize onefuzz jobs containers download (#953) 2021-06-02 13:29:20 -04:00
1822acf943 add onefuzz jobs containers delete JOB_ID (#949)
Addresses #943
2021-06-02 17:04:51 +00:00
a92c84d42a work around issue with discriminated typed unions (#939)
We're experiencing a bug where Unions of sub-models are getting downcast, which causes a loss of information.  

As an example, EventScalesetCreated was getting downcast to EventScalesetDeleted.  I have not figured out why, nor can I replicate it locally to minimize the bug send upstream, but I was able to reliably replicate it on the service.

While working through this issue, I noticed that deserialization of SignalR events was frequently wrong, leaving things like tasks as "init" in `status top`.

Both of these issues are Unions of models with a type field, so it's likely these are related.
2021-06-02 16:40:58 +00:00
3d191c3c5d use deploy logging context for app creation logs (#952) 2021-06-02 12:15:47 -04:00
60ae07c34f handle azure-storage deleting nonexistent containers (#948) 2021-06-02 15:11:33 +00:00
74c0e0a5c5 Udpate adal error handling during deployment (#947) 2021-06-02 10:51:57 -04:00
134c947ec1 Bump urllib3 in /contrib/deploy-onefuzz-via-azure-devops (#950) 2021-06-02 09:23:37 -04:00
1a60bd8a15 capture azcopy log file (#945) 2021-06-01 17:12:37 -04:00
e7669138c4 add an optional initial delay for heartbeats (#937)
One of the difficulties in crash repro as task is a race condition where we the client tries to connect before the cdb is running.

This makes it such that we can use the heartbeat to identify if the task has started before connecting in.

NOTE:  In this PR, it's always set to None.  See #830 for it's actual usage.  However, I split out the PR for easier review.
2021-06-01 20:08:53 +00:00
f878064c8f update rust dependencies (#942) 2021-06-01 13:23:04 -04:00
b761908409 send NodeCommandStopIfFree on node shutdown (#940)
If we move to shutdown a single node but it's not doing work, it will
wait until it picks up work to shutdown.  This shortcuts that.
2021-06-01 15:03:33 +00:00
0a6021bfa1 prevent object id collision in hide_secrets (#936)
this fixes an issue related to object id reuse that can occur making the
object identification cache fail.  Instead, this simplifies the
hide_secrets to always recurse and use setattr to always set the value
based on the recursion.

Note, the object id reuse issue was seen in the
`events.filter_event_recurse` development and this was the fix for the
id reuse there.

Python documentation states:

id(object):

Return the “identity” of an object. This is an integer (or long integer)
which is guaranteed to be unique and constant for this object during its
lifetime. Two objects with non-overlapping lifetimes may have the same
id() value.
2021-05-27 08:28:02 -04:00