This adds a retry wrapper around azcopy.
something to note: tokio's Command doesn't allow for clone, which makes this unfortunately difficult to generalize to any command.
## Info on Pull Request
This is a quick fix for multi tenant authentication. After doing some more testing, I realized I was missing a parameter in `azuredeploy.json`. So added missing code from previous [PR ](https://github.com/microsoft/onefuzz/pull/563/files).
## Validation Steps Performed
I have tested these changes and was successfully able to deploy in both single tenant and multi tenant environments.
## Summary of the Pull Request
This PR add a UI to the local run.
- The UI currently monitors the logs and some of the directory created (the rest will be wired in a coming PR)
- pressing 'q' will quit the PR
- By default, the job directory is deleted when the ui quits unless the parameter 'keep_job_dir' is specified
This fixes retrying on bad status codes.
Note, this adds support to specify specific error codes as "successful". This is important for the If-None-Match conditional upload case (blob uploading). The response we get back is 409 (Conflict).
Previously, we would "fail fast" the 409. However, what we want is "409 is basically success here" and every other call 409 should be a failure.
In practice, Application Insights can take up to 3 minutes before something sent to it is available via KQL.
This PR logs a start and stop marker such that the integration tests only search for logs during the integration tests. This reduces the complexity when using the integration tests during the development process.
Note: this migrated the new functionality from #356 into the latest integration test tools.
## Summary of the Pull Request
_What is this about?_
## PR Checklist
* [x] Applies to work item: #562
* [x] CLA signed. If not, go over [here](https://cla.opensource.microsoft.com/microsoft/onefuzz) and sign the CLI.
* [x] Tests added/passed
* [ ] Requires documentation to be updated
* [x] I've discussed this with core contributors already. If not checked, I'm ready to accept this work might be rejected in favor of a different grand plan. Issue number where discussion took place: #xxx
## Info on Pull Request
The end-to-end changes needed to have onefuzz deployed with multi-tenant authentication.
## Validation Steps Performed
_How does someone test & validate?_
Clean up the interface of the Linux `Recorder` struct, and make it more consistent with the Windows version. Hold a mutable ref to a `ModuleCache` to enable in-memory cache re-use.
This works around an issue in AAD service principal creation. The race condition in AAD shows up as:
```
INFO:deploy:checking if RBAC already exists
INFO:deploy:creating Application registration
INFO:deploy:creating service principal
Traceback (most recent call last):
File "deploy.py", line 926, in <module>
main()
File "deploy.py", line 920, in main
state[1](client)
File "deploy.py", line 303, in setup_rbac
client.service_principals.create(service_principal_params)
File "/tmp/tmpp2x7ybfg/deploy-venv/lib/python3.8/site-packages/azure/graphrbac/operations/service_principals_operations.py", line 87, in create
raise models.GraphErrorException(self._deserialize, response)
azure.graphrbac.models.graph_error_py3.GraphErrorException: When using this permission, the backing application of the service principal being created must in the local tenant
```
The azure-cli has the same issue, as seen in https://github.com/Azure/azure-cli/issues/14767
