This works around an issue in AAD service principal creation. The race condition in AAD shows up as:
```
INFO:deploy:checking if RBAC already exists
INFO:deploy:creating Application registration
INFO:deploy:creating service principal
Traceback (most recent call last):
File "deploy.py", line 926, in <module>
main()
File "deploy.py", line 920, in main
state[1](client)
File "deploy.py", line 303, in setup_rbac
client.service_principals.create(service_principal_params)
File "/tmp/tmpp2x7ybfg/deploy-venv/lib/python3.8/site-packages/azure/graphrbac/operations/service_principals_operations.py", line 87, in create
raise models.GraphErrorException(self._deserialize, response)
azure.graphrbac.models.graph_error_py3.GraphErrorException: When using this permission, the backing application of the service principal being created must in the local tenant
```
The azure-cli has the same issue, as seen in https://github.com/Azure/azure-cli/issues/14767
This builds upon #591 to expand the stack minimization to crash reporting mechanisms.
Example (see #703 for an example without the new functionality):
```
$ onefuzz-agent local test-input /tmp/fuzz.exe /etc/passwd
{
"crash_report": {
"input_sha256": "a35b3ce1038750e9175a6dcd3f64c8d4e85720affb12cc11f5d0b6889274d06e",
"executable": "/tmp/fuzz.exe",
"crash_type": "SIGABRT",
"crash_site": "0x7ffff7e0d18b in gsignal+0xcb (/usr/lib/x86_64-linux-gnu/libc-2.31.so+0x4618b)",
"call_stack": [
"#0 0x7ffff7e0d18b in gsignal+0xcb (/usr/lib/x86_64-linux-gnu/libc-2.31.so+0x4618b)",
"#1 0x7ffff7dec859 in abort+0x12b (/usr/lib/x86_64-linux-gnu/libc-2.31.so+0x25859)",
"#2 0x7ffff7e573ee in <unknown> (/usr/lib/x86_64-linux-gnu/libc-2.31.so+0x903ee)",
"#3 0x7ffff7ef9b4a in __fortify_fail+0x2a (/usr/lib/x86_64-linux-gnu/libc-2.31.so+0x132b4a)",
"#4 0x7ffff7ef83e6 in __chk_fail+0x16 (/usr/lib/x86_64-linux-gnu/libc-2.31.so+0x1313e6)",
"#5 0x7ffff7ef7e09 in __strncpy_chk+0x19 (/usr/lib/x86_64-linux-gnu/libc-2.31.so+0x130e09)",
"#6 0x400a54 in from_file+0xa4 (/tmp/fuzz.exe+0xa54)",
"#7 0x7ffff7dee0b3 in __libc_start_main+0xf3 (/usr/lib/x86_64-linux-gnu/libc-2.31.so+0x270b3)",
"#8 0x40077a in _start+0x2a (/tmp/fuzz.exe+0x77a)"
],
"call_stack_sha256": "99625a7c103136e02910b65c7b60f1bbd1a7612242d6838da52d968369039409",
"minimized_stack": [
"__fortify_fail",
"__chk_fail",
"from_file"
],
"minimized_stack_sha256": "237f13bfa384c6c2bc06369099373efbb36995a9ad00fd5469d354b5fc672ba1",
"minimized_stack_function_names": [
"__fortify_fail",
"__chk_fail",
"from_file"
],
"minimized_stack_function_names_sha256": "237f13bfa384c6c2bc06369099373efbb36995a9ad00fd5469d354b5fc672ba1",
"asan_log": "",
"task_id": "00000000-0000-0000-0000-000000000000",
"job_id": "00000000-0000-0000-0000-000000000000"
}
}
$
```
Adds `test-input` and `test-input-libfuzzer`, which print the CrashTestResult in json form.
While many of the existing tasks make sense running in a managed loop, crash report generation is something that having a single one-off is useful.
Example:
```
$ onefuzz-agent local test-input /tmp/fuzz.exe /tmp/crash.txt
{
"crash_report": {
"input_sha256": "a35b3ce1038750e9175a6dcd3f64c8d4e85720affb12cc11f5d0b6889274d06e",
"executable": "/tmp/fuzz.exe",
"crash_type": "SIGABRT",
"crash_site": "0x7f0d9d4ad18b in gsignal+0xcb (/usr/lib/x86_64-linux-gnu/libc-2.31.so+0x4618b)",
"call_stack": [
"#0 0x7f0d9d4ad18b in gsignal+0xcb (/usr/lib/x86_64-linux-gnu/libc-2.31.so+0x4618b)",
"#1 0x7f0d9d48c859 in abort+0x12b (/usr/lib/x86_64-linux-gnu/libc-2.31.so+0x25859)",
"#2 0x7f0d9d4f73ee in <unknown> (/usr/lib/x86_64-linux-gnu/libc-2.31.so+0x903ee)",
"#3 0x7f0d9d599b4a in __fortify_fail+0x2a (/usr/lib/x86_64-linux-gnu/libc-2.31.so+0x132b4a)",
"#4 0x7f0d9d5983e6 in __chk_fail+0x16 (/usr/lib/x86_64-linux-gnu/libc-2.31.so+0x1313e6)",
"#5 0x7f0d9d597e09 in __strncpy_chk+0x19 (/usr/lib/x86_64-linux-gnu/libc-2.31.so+0x130e09)",
"#6 0x400a54 in from_file+0xa4 (/tmp/fuzz.exe+0xa54)",
"#7 0x7f0d9d48e0b3 in __libc_start_main+0xf3 (/usr/lib/x86_64-linux-gnu/libc-2.31.so+0x270b3)",
"#8 0x40077a in _start+0x2a (/tmp/fuzz.exe+0x77a)"
],
"call_stack_sha256": "6906234fb235690cc2843a1a55f49ff68b424e54bec55f9b8258415d97b3e638",
"task_id": "00000000-0000-0000-0000-000000000000",
"job_id": "00000000-0000-0000-0000-000000000000"
}
}
$
```
For a given entry in a call stack, this parses out the following: line, function name, function offset, source file name, source file line, module path, and module offset.
Additionally, this provides a code-generated libclusterfuzz port of the regular expressions used for stack minimization.
For an example of the minimization, instead of:
```json
[
"#0 0x56512a9c1418 in __sanitizer_print_stack_trace /b/s/w/ir/cache/builder/src/third_party/llvm/compiler-rt/lib/asan/asan_stack.cpp:86:3",
"#1 0x56512aaaa42d in fuzzer::PrintStackTrace() third_party/libFuzzer/src/FuzzerUtil.cpp:205:5",
"#2 0x56512aa6a85e in fuzzer::Fuzzer::CrashCallback() third_party/libFuzzer/src/FuzzerLoop.cpp:232:3",
"#3 0x56512aa6a7df in fuzzer::Fuzzer::StaticCrashSignalCallback() third_party/libFuzzer/src/FuzzerLoop.cpp:203:6",
"#4 0x56512aaab948 in fuzzer::CrashHandler(int, siginfo_t*, void*) third_party/libFuzzer/src/FuzzerUtilPosix.cpp:46:3",
"#5 0x7f1ee3f0188f (/lib/x86_64-linux-gnu/libpthread.so.0+0x1288f)",
"#6 0x56512a9e5aa1 in Json::OurReader::parse(char const*, char const*, Json::Value&, bool) third_party/jsoncpp/source/src/lib_json/json_reader.cpp:1062:10",
"#7 0x56512a9eedb4 in Json::OurCharReader::parse(char const*, char const*, Json::Value*, std::__Cr::basic_string<char, std::__Cr::char_traits<char>, std::__Cr::allocator<char> >*) third_party/jsoncpp/source/src/lib_json/json_reader.cpp:1899:23",
"#8 0x56512a9e03a3 in LLVMFuzzerTestOneInput third_party/jsoncpp/fuzzers/json_fuzzer.cc:39:24",
"#9 0x56512aa6d0cf in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) third_party/libFuzzer/src/FuzzerLoop.cpp:556:15",
"#10 0x56512aa3b7da in fuzzer::RunOneTest(fuzzer::Fuzzer*, char const*, unsigned long) third_party/libFuzzer/src/FuzzerDriver.cpp:292:6",
"#11 0x56512aa4108a in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) third_party/libFuzzer/src/FuzzerDriver.cpp:774:9","#12 0x56512aa821ac in main third_party/libFuzzer/src/FuzzerMain.cpp:19:10",
"#13 0x7f1ee3361b96 in __libc_start_main /build/glibc-OTsEL5/glibc-2.27/csu/../csu/libc-start.c:310",
]
```
The minimized call stack is:
```json
[
"Json::OurReader::parse(char const*, char const*, Json::Value&, bool)",
"Json::OurCharReader::parse(char const*, char const*, Json::Value*, std::__Cr::basic_string<char, std::__Cr::char_traits<char>, std::__Cr::allocator<char> >*)",
"json_fuzzer.cc"
]
```
This also provides a naïve function name list, which comes close to Clusterfuzz's function identification.
This would result in:
```json
[
"Json::OurReader::parse",
"Json::OurCharReader::parse",
"json_fuzzer.cc"
]
```
Lastly, for our `stack hash` functionality used by the crash reporting task, those now provide the ability to specify the number of frames to include when building the hash.
## Summary of the Pull Request
Add PS function to install VC Redistributables
## PR Checklist
* [X] Applies to work item: 556
* [X] CLA signed. If not, go over [here](https://cla.opensource.microsoft.com/microsoft/onefuzz) and sign the CLI.
* [X] Tests added/passed
* [X] Requires documentation to be updated
* [X] I've discussed this with core contributors already. If not checked, I'm ready to accept this work might be rejected in favor of a different grand plan. Issue number where discussion took place: #xxx
## Info on Pull Request
Added another PS function invoked from the setup.ps script
## Validation Steps Performed
Tested the change and verified by Jordyn Puryear
## Summary of the Pull Request
This is a refactoring of the local debugging.
- The input queue as been abstracted and can now be locally
- The SyncDir can now sync a local directory
- Added the ability to monitor a local directory with a queue
## Reviewers Notes
The most import part of the PR are
- The queue abstraction and implementations
- src/agent/storage-queue/src/azure_queue.rs
- src/agent/storage-queue/src/local_queue.rs
- src/agent/storage-queue/src/lib.rs
- Changes to support local files in
- src/agent/onefuzz/src/syncdir.rs
- Example of how those abstractions are used to link task together
- src/agent/onefuzz-agent/src/local/libfuzzer_fuzz.rs
- src/agent/onefuzz-agent/src/local/common.rs
## Validation Steps Performed
_How does someone test & validate?_
## Summary of the Pull Request
_What is this about?_
Due to our GDPR privacy requirements, we decided that it would be best to completely purge personal identifiable information from our AppInsights telemetry and logging. Instead of just removing all of the logging statements with personal info, I created a filter function that logs telemetry after it's been run through a recursive scrubbing function. This PR includes this new scrubbing function.
## PR Checklist
* [x] Applies to work item: #660
* [ ] CLA signed. If not, go over [here](https://cla.opensource.microsoft.com/microsoft/onefuzz) and sign the CLI.
* [ ] Tests added/passed
* [ ] Requires documentation to be updated
* [x] I've discussed this with core contributors already. If not checked, I'm ready to accept this work might be rejected in favor of a different grand plan. Issue number where discussion took place: #xxx
## Info on Pull Request
_What does this include?_
Includes changes to events.py in onefuzzlib. I've implemented functionality - log_event() - to recursively check Event structures for UserInfo before logging to AppInsights.
## Validation Steps Performed
I run local tests using a script I created with test events.
_How does someone test & validate?_
I can provide local testing script. If that is insufficient, I can write a unit test that will run against this code.
If a user manually deletes a scaleset managed by OneFuzz, then `get_vmss_size` returns None.
When this happens, `Scaleset.shutdown` generates an exception from the `logging.info` call on line 573.
This PR handles this edge condition.
