Commit Graph

2095 Commits

Author SHA1 Message Date
4f6e4dbbec add types-requests (#973) 2021-06-09 13:01:17 -04:00
da931b3a5c address issues raised from latest mypy (#972) 2021-06-09 12:04:24 -04:00
3794bb0c68 move to wiremock for testing (#971) 2021-06-09 12:04:10 -04:00
78d5a58965 allow updates to a scaleset in the resize state (#969) 2021-06-09 13:10:43 +00:00
af39d25a7d reimage/delete expired nodes even with the debug_keep_node flag (#968)
Fixes #965
2021-06-08 17:37:10 +00:00
ed289c9a3c handle scaleset resize exceptions (#967) 2021-06-08 09:30:36 -04:00
2d377d1423 add onefuzz debug job rerun JOB_ID command (#960)
This allows the specification of an alternate pool or duration when rerunning an existing job.
2021-06-07 20:53:34 +00:00
52f11cc097 release 2.20.0 (#964) 2.20.0 2021-06-07 12:48:33 -04:00
c0e605146c remove RequestBuilder abstraction in onefuzz-supervisor coordinator (#963) 2021-06-03 22:28:54 -04:00
6b7906dfb1 libfuzzer fuzzing task perf improvements (#941)
- resuse the regex to parse the output of libfuzzer
- added a cancellation notification to report_fuzzer_sys_info.
   ~~The code seems to be actively waiting this function and consuming some cpu time~~
   The notification allows us to reduce the time waiting for the fuzzing loop to terminate. 

## Summary of the Pull Request

_What is this about?_

## PR Checklist
* [ ] Applies to work item: #xxx
* [ ] CLA signed. If not, go over [here](https://cla.opensource.microsoft.com/microsoft/onefuzz) and sign the CLI.
* [ ] Tests added/passed
* [ ] Requires documentation to be updated
* [ ] I've discussed this with core contributors already. If not checked, I'm ready to accept this work might be rejected in favor of a different grand plan. Issue number where discussion took place: #xxx

## Info on Pull Request

_What does this include?_

## Validation Steps Performed

_How does someone test & validate?_
2021-06-04 00:14:15 +00:00
2c72bd590f Add generic coverage task (#763)
**Todo:**
- [x] Finalize format for coverage file(s)
- [x] Add service support
- [x] Integration test
- [x] Merge #926 
- [x] Merge #929
2021-06-03 23:36:00 +00:00
60d1853dcc add retry logic to authorize application (#959) 2021-06-03 17:45:26 -04:00
e28389c343 release 2.19.0 (#955) 2.19.0 2021-06-02 16:42:51 -04:00
0e164107bb remove unused adal import (#954) 2021-06-02 14:00:10 -04:00
c46abbcec3 stabilize onefuzz jobs containers download (#953) 2021-06-02 13:29:20 -04:00
1822acf943 add onefuzz jobs containers delete JOB_ID (#949)
Addresses #943
2021-06-02 17:04:51 +00:00
a92c84d42a work around issue with discriminated typed unions (#939)
We're experiencing a bug where Unions of sub-models are getting downcast, which causes a loss of information.  

As an example, EventScalesetCreated was getting downcast to EventScalesetDeleted.  I have not figured out why, nor can I replicate it locally to minimize the bug send upstream, but I was able to reliably replicate it on the service.

While working through this issue, I noticed that deserialization of SignalR events was frequently wrong, leaving things like tasks as "init" in `status top`.

Both of these issues are Unions of models with a type field, so it's likely these are related.
2021-06-02 16:40:58 +00:00
3d191c3c5d use deploy logging context for app creation logs (#952) 2021-06-02 12:15:47 -04:00
60ae07c34f handle azure-storage deleting nonexistent containers (#948) 2021-06-02 15:11:33 +00:00
74c0e0a5c5 Udpate adal error handling during deployment (#947) 2021-06-02 10:51:57 -04:00
134c947ec1 Bump urllib3 in /contrib/deploy-onefuzz-via-azure-devops (#950) 2021-06-02 09:23:37 -04:00
1a60bd8a15 capture azcopy log file (#945) 2021-06-01 17:12:37 -04:00
e7669138c4 add an optional initial delay for heartbeats (#937)
One of the difficulties in crash repro as task is a race condition where we the client tries to connect before the cdb is running.

This makes it such that we can use the heartbeat to identify if the task has started before connecting in.

NOTE:  In this PR, it's always set to None.  See #830 for it's actual usage.  However, I split out the PR for easier review.
2021-06-01 20:08:53 +00:00
f878064c8f update rust dependencies (#942) 2021-06-01 13:23:04 -04:00
b761908409 send NodeCommandStopIfFree on node shutdown (#940)
If we move to shutdown a single node but it's not doing work, it will
wait until it picks up work to shutdown.  This shortcuts that.
2021-06-01 15:03:33 +00:00
0a6021bfa1 prevent object id collision in hide_secrets (#936)
this fixes an issue related to object id reuse that can occur making the
object identification cache fail.  Instead, this simplifies the
hide_secrets to always recurse and use setattr to always set the value
based on the recursion.

Note, the object id reuse issue was seen in the
`events.filter_event_recurse` development and this was the fix for the
id reuse there.

Python documentation states:

id(object):

Return the “identity” of an object. This is an integer (or long integer)
which is guaranteed to be unique and constant for this object during its
lifetime. Two objects with non-overlapping lifetimes may have the same
id() value.
2021-05-27 08:28:02 -04:00
d557fc16c6 mark tasks that are stopped that never started with an error (#935) 2021-05-26 18:42:21 -04:00
87eb8ac80a Add timeout when recording Linux coverage (#929)
- Simplify the Linux block coverage `Recorder` API 
- Add a (mandatory) timeout to recording runs
2021-05-26 12:22:28 -07:00
d2084f70b1 Rename helper method for setup-relative files (#926)
We are going to use this for more than just target exes, and in fact, it applies to any file that must occur in the setup directory/container.
2021-05-26 18:49:52 +00:00
2f1fc61f01 always show message details in status top (#933) 2021-05-26 14:22:24 -04:00
c107a04cf9 fix issue deleting proxy from storage tables (#932) 2021-05-26 13:33:22 -04:00
8b74d08d3d fix deleting nodes with expired heartbeats (#930) 2021-05-26 13:06:44 -04:00
269a529f93 save more context in the failure log (#931) 2021-05-26 12:12:55 -04:00
13dc0dd330 only set the SSH key permissions on Windows once (#928) 2021-05-26 11:33:39 -04:00
eaf770fa22 Update coverage example (#919) 2021-05-25 08:24:38 -04:00
21c193624e release 2.18.0 (#920) 2.18.0 2021-05-24 15:26:21 -04:00
7952f16032 basic list proxy functionality (#905) 2021-05-24 13:24:22 -04:00
2241dcc7a4 update azure-mgmt-resource to 18.0.0 (#903) 2021-05-24 16:33:06 +00:00
db4c03ea56 Catch argparse type errors (#911)
Catch argparse argument type errors and display them in a simplified fashion.

As an example, this is how the output is rendered.
```
❯ onefuzz template libfuzzer basic --target_env  ASAN_OPTIONS=allocator_may_return_null=1 a a a linux
unable to parse arguments: unable to parse value as a key=value pair: 'a'   
❯
```

Note, this builds upon #910.
2021-05-24 16:04:59 +00:00
2c9a73dad9 Only use pseudo-handle for coverage dbghelp context (#918)
When recording coverage, don't try to use live process handles as the context for the `dbghelp` symbol handler in the module feature cache. Instead, use the default pseudo-handle for the semantically static PDB search, and the target process handle for any queries about the target and its address space.
2021-05-24 08:37:13 -07:00
700e55cfc4 Fix and improve PDB search (#913)
- Fix a bug in PDB search where we treated the absence of the PE-specified file as a hard error
- In the return type of `find_pdb_path()`, distinguish between "no file found" and a search error
2021-05-24 08:11:52 -07:00
a103985c0d Fix multi proxy race condition (#909)
Refactored PR of #904 for easier review.  Once #908 is reviewed & merged, this will be easier to review.
2021-05-22 06:50:08 +00:00
6e5f7e4d4c encode proxy name as base58 to allow full deletion of resources (#907) 2021-05-21 20:54:17 -04:00
e7197f1407 explain key=value argument parsing failures (#910) 2021-05-21 13:44:34 -04:00
a4bb670fb2 add proxy_state_updated events (#908) 2021-05-21 12:47:54 -04:00
2f81c44f01 Refactoring proxy lifetime to only shutdown when proxy is out-of-date. (#839)
## Summary of the Pull Request

_What is this about?_
We'd like to refactor the proxy lifecycle to only delete when the proxy is out-of-date - i.e. when the proxy is older than 7 days or a mismatched version. I've changed two files, proxy.py and timer_daily\init.py to check for the version and timestamp before stopping a live proxy. 

## PR Checklist
* [ ] Applies to work item: #xxx
* [ ] CLA signed. If not, go over [here](https://cla.opensource.microsoft.com/microsoft/onefuzz) and sign the CLI.
* [ ] Tests added/passed
* [ ] Requires documentation to be updated
* [x] I've discussed this with core contributors already. If not checked, I'm ready to accept this work might be rejected in favor of a different grand plan. Issue number where discussion took place: #xxx

## Info on Pull Request

_What does this include?_
Changes to two files: 
proxy.py: 
- get_or_create() edited to check if timestamp is >7 days.
- Created is_outdated() to check version and timestamp for out-of-date proxy. 
timer_daily/init.py
- Proxy check now includes is_outdated() before determining if a proxy should be shutdown. 

## Validation Steps Performed
Deploying test instance to determine if proxy lives past a single day.
2021-05-20 14:33:29 +00:00
2b67c7b02f add setup_dir to generator task (#901)
fixes #848
2021-05-19 16:10:13 +00:00
776e8fa909 Add setup arguments to enable specifying alt tenants on the CLI (#900)
This enables specifying the endpoint configuration for alternate tenants purely on the command line.

Previously, on a single tenant you could use the following:
```
onefuzz --endpoint https://INSTANCE.azurewebsites.net info get
```

For multi-tenant installs, we need to expose more than just endpoint.

This enables:
```
onefuzz --endpoint https://INSTANCE.azurewebsites.net --client_id CLIENT_ID  --authority https://login.microsoftonline.com/common --tenant_domain TENANT_DOMAIN info get
```
2021-05-19 07:38:34 +00:00
ff140a6b1b Stop tasks on nodes before deleting task queues (#801) 2021-05-17 18:59:13 +00:00
194ea49f03 Use assign User.Read from Microsoft graph instead of AAD Graph (#894)
Replace the permission in the application registration. 
closes #877
2021-05-17 18:42:08 +00:00