This fixes retrying on bad status codes.
Note, this adds support to specify specific error codes as "successful". This is important for the If-None-Match conditional upload case (blob uploading). The response we get back is 409 (Conflict).
Previously, we would "fail fast" the 409. However, what we want is "409 is basically success here" and every other call 409 should be a failure.
In practice, Application Insights can take up to 3 minutes before something sent to it is available via KQL.
This PR logs a start and stop marker such that the integration tests only search for logs during the integration tests. This reduces the complexity when using the integration tests during the development process.
Note: this migrated the new functionality from #356 into the latest integration test tools.
## Summary of the Pull Request
_What is this about?_
## PR Checklist
* [x] Applies to work item: #562
* [x] CLA signed. If not, go over [here](https://cla.opensource.microsoft.com/microsoft/onefuzz) and sign the CLI.
* [x] Tests added/passed
* [ ] Requires documentation to be updated
* [x] I've discussed this with core contributors already. If not checked, I'm ready to accept this work might be rejected in favor of a different grand plan. Issue number where discussion took place: #xxx
## Info on Pull Request
The end-to-end changes needed to have onefuzz deployed with multi-tenant authentication.
## Validation Steps Performed
_How does someone test & validate?_
Clean up the interface of the Linux `Recorder` struct, and make it more consistent with the Windows version. Hold a mutable ref to a `ModuleCache` to enable in-memory cache re-use.
This works around an issue in AAD service principal creation. The race condition in AAD shows up as:
```
INFO:deploy:checking if RBAC already exists
INFO:deploy:creating Application registration
INFO:deploy:creating service principal
Traceback (most recent call last):
File "deploy.py", line 926, in <module>
main()
File "deploy.py", line 920, in main
state[1](client)
File "deploy.py", line 303, in setup_rbac
client.service_principals.create(service_principal_params)
File "/tmp/tmpp2x7ybfg/deploy-venv/lib/python3.8/site-packages/azure/graphrbac/operations/service_principals_operations.py", line 87, in create
raise models.GraphErrorException(self._deserialize, response)
azure.graphrbac.models.graph_error_py3.GraphErrorException: When using this permission, the backing application of the service principal being created must in the local tenant
```
The azure-cli has the same issue, as seen in https://github.com/Azure/azure-cli/issues/14767
This builds upon #591 to expand the stack minimization to crash reporting mechanisms.
Example (see #703 for an example without the new functionality):
```
$ onefuzz-agent local test-input /tmp/fuzz.exe /etc/passwd
{
"crash_report": {
"input_sha256": "a35b3ce1038750e9175a6dcd3f64c8d4e85720affb12cc11f5d0b6889274d06e",
"executable": "/tmp/fuzz.exe",
"crash_type": "SIGABRT",
"crash_site": "0x7ffff7e0d18b in gsignal+0xcb (/usr/lib/x86_64-linux-gnu/libc-2.31.so+0x4618b)",
"call_stack": [
"#0 0x7ffff7e0d18b in gsignal+0xcb (/usr/lib/x86_64-linux-gnu/libc-2.31.so+0x4618b)",
"#1 0x7ffff7dec859 in abort+0x12b (/usr/lib/x86_64-linux-gnu/libc-2.31.so+0x25859)",
"#2 0x7ffff7e573ee in <unknown> (/usr/lib/x86_64-linux-gnu/libc-2.31.so+0x903ee)",
"#3 0x7ffff7ef9b4a in __fortify_fail+0x2a (/usr/lib/x86_64-linux-gnu/libc-2.31.so+0x132b4a)",
"#4 0x7ffff7ef83e6 in __chk_fail+0x16 (/usr/lib/x86_64-linux-gnu/libc-2.31.so+0x1313e6)",
"#5 0x7ffff7ef7e09 in __strncpy_chk+0x19 (/usr/lib/x86_64-linux-gnu/libc-2.31.so+0x130e09)",
"#6 0x400a54 in from_file+0xa4 (/tmp/fuzz.exe+0xa54)",
"#7 0x7ffff7dee0b3 in __libc_start_main+0xf3 (/usr/lib/x86_64-linux-gnu/libc-2.31.so+0x270b3)",
"#8 0x40077a in _start+0x2a (/tmp/fuzz.exe+0x77a)"
],
"call_stack_sha256": "99625a7c103136e02910b65c7b60f1bbd1a7612242d6838da52d968369039409",
"minimized_stack": [
"__fortify_fail",
"__chk_fail",
"from_file"
],
"minimized_stack_sha256": "237f13bfa384c6c2bc06369099373efbb36995a9ad00fd5469d354b5fc672ba1",
"minimized_stack_function_names": [
"__fortify_fail",
"__chk_fail",
"from_file"
],
"minimized_stack_function_names_sha256": "237f13bfa384c6c2bc06369099373efbb36995a9ad00fd5469d354b5fc672ba1",
"asan_log": "",
"task_id": "00000000-0000-0000-0000-000000000000",
"job_id": "00000000-0000-0000-0000-000000000000"
}
}
$
```
