This fixes retrying on bad status codes.
Note, this adds support to specify specific error codes as "successful". This is important for the If-None-Match conditional upload case (blob uploading). The response we get back is 409 (Conflict).
Previously, we would "fail fast" the 409. However, what we want is "409 is basically success here" and every other call 409 should be a failure.
In practice, Application Insights can take up to 3 minutes before something sent to it is available via KQL.
This PR logs a start and stop marker such that the integration tests only search for logs during the integration tests. This reduces the complexity when using the integration tests during the development process.
Note: this migrated the new functionality from #356 into the latest integration test tools.
## Summary of the Pull Request
_What is this about?_
## PR Checklist
* [x] Applies to work item: #562
* [x] CLA signed. If not, go over [here](https://cla.opensource.microsoft.com/microsoft/onefuzz) and sign the CLI.
* [x] Tests added/passed
* [ ] Requires documentation to be updated
* [x] I've discussed this with core contributors already. If not checked, I'm ready to accept this work might be rejected in favor of a different grand plan. Issue number where discussion took place: #xxx
## Info on Pull Request
The end-to-end changes needed to have onefuzz deployed with multi-tenant authentication.
## Validation Steps Performed
_How does someone test & validate?_
Clean up the interface of the Linux `Recorder` struct, and make it more consistent with the Windows version. Hold a mutable ref to a `ModuleCache` to enable in-memory cache re-use.
This works around an issue in AAD service principal creation. The race condition in AAD shows up as:
```
INFO:deploy:checking if RBAC already exists
INFO:deploy:creating Application registration
INFO:deploy:creating service principal
Traceback (most recent call last):
File "deploy.py", line 926, in <module>
main()
File "deploy.py", line 920, in main
state[1](client)
File "deploy.py", line 303, in setup_rbac
client.service_principals.create(service_principal_params)
File "/tmp/tmpp2x7ybfg/deploy-venv/lib/python3.8/site-packages/azure/graphrbac/operations/service_principals_operations.py", line 87, in create
raise models.GraphErrorException(self._deserialize, response)
azure.graphrbac.models.graph_error_py3.GraphErrorException: When using this permission, the backing application of the service principal being created must in the local tenant
```
The azure-cli has the same issue, as seen in https://github.com/Azure/azure-cli/issues/14767
