Commit Graph

35 Commits

Author SHA1 Message Date
78d6adf555 upgrade azure-storage-blob to 12.8.0 (#625)
Note, this makes use of the new feature, `ContainerClient.exists()` which models our existing mechanism for doing container existence checking.
2021-03-03 10:33:23 +00:00
7f66eeee0d handle OperationNotAllowed errors when creating VMSS (#614) 2021-03-02 16:14:10 -05:00
cebb84b9e7 handle error condition when creating a container that is being deleted (#582)
When users try to create a container immediately after deleting it, Azure will fail saying the deletion is in-progress.

catching ResourceExistsError during create handles this error.
2021-02-22 01:49:07 +00:00
279629292f handle SkuNotAvailable errors when creating VM Scalesets (#557) 2021-02-17 16:52:37 -05:00
8ee7fae240 use the cached Azure Identity instance for storage operations (#526) 2021-02-09 12:20:12 -05:00
91a3690551 fix logging to show corpus accounts found, not a function ref (#524) 2021-02-09 11:55:52 -05:00
1d74379a70 use the primitive types in more places (#514) 2021-02-05 13:10:37 -05:00
5e2e9448df add security auditing of python code using Bandit during CICD (#491) 2021-02-01 16:51:03 -05:00
0f70ffa3e2 try pushing updates to scaleset configs frequently until the push succeeds (#489) 2021-02-01 10:09:40 -05:00
5027745ee2 simplify get/delete for scalesets (#468) 2021-01-26 14:43:14 -05:00
165257e989 update python prereqs (#427)
Updates the following libraries in the service:
* azure-core
* azure-functions
* azure-identity
* azure-keyvault-keys
* azure-keyvault-secrets
* azure-mgmt-compute
* azure-mgmt-core
* azure-mgmt-loganalytics
* azure-mgmt-network
* azure-mgmt-resource
* azure-mgmt-storage
* azure-mgmt-subscription
* azure-storage-blob
* azure-storage-queue
* pydantic
* requests
* jsonpatch

Removes the following libraries in the service:
* azure-cli-core
* azure-cli-nspkg
* azure-mgmt-cosmosdb
* azure-servicebus

Updates the following libraries in the CLI:
* requests
* semver
* asciimatics
* pydantic
* tenacity

Updates the following libraries in onefuzztypes:
* pydantic

The primary "legacy" libraries are [azure-graphrbac](https://pypi.org/project/azure-graphrbac/) and azure-cosmosdb-table.  The former has not been updated to use azure-identity yet. The later is being rewritten as [azure-data-tables](https://pypi.org/project/azure-data-tables/), but is still in early beta.
2021-01-25 20:53:40 +00:00
3f2883d38e Storing secrets in azure keyvault (#326) 2021-01-25 11:12:07 -05:00
e799eb03cd Shorten the expiry window for the work queue SAS URLs assigned at node registration (#416)
The underlying impact is that nodes must re-register on a more frequent basis.

Nodes find out they are out-of-date is during registration and immediately prior to starting a new set of work.  Requiring nodes re-register on a shortened cycle provides more opportunities for nodes to get re-imaged.

Additionally, this addresses an issue handling the SAS URL expiry in a more clean fashion in the supervisor.
2021-01-07 12:34:26 +00:00
3b26ffef65 support multiple corpus accounts (#334)
Add support for sharding across multiple storage accounts for blob containers used for corpus management.

Things to note:

1. Additional storage accounts must be in the same resource group, support the "blob" endpoint, and have the tag `storage_type` with the value `corpus`.  A utility is provided (`src/utils/add-corpus-storage-accounts`), which adds storage accounts. 
2. If any secondary storage accounts exist, they are used by default for containers.
3. Storage account names are cached in memory the Azure Function instance forever.   Upon adding new storage accounts, the app needs to be restarted to pick up the new accounts.
2021-01-06 23:11:39 +00:00
9b3ccf37ea use the correct instrumentation key (#355) 2020-12-01 18:44:10 -05:00
7f97c142ed add the instrumentation key to Info (#353) 2020-12-01 11:13:06 -05:00
b2b4a06afa Address typing issues hidden by memoization.caching (#322) 2020-11-18 15:08:40 -05:00
e47e89609a Use Storage Account types, rather than account_id (#320)
We need to move to supporting data sharding.

One of the steps towards that is stop passing around `account_id`, rather we need to specify the type of storage we need.
2020-11-18 14:06:14 +00:00
ba59230187 fix create_vmss log message (#293) 2020-11-11 12:17:42 -05:00
e638908aac Add application-insights debug cli (#281) 2020-11-11 06:17:43 -05:00
bbee84ab1f Storing the user assigned managed identity in the scaleset table (#255) 2020-11-05 18:36:59 -05:00
b5578381ce default TTL for queued messages to infinite (#259) 2020-11-04 15:41:05 -05:00
6c598773dd add instance_id generated at install time (#245) 2020-11-02 14:27:51 -05:00
154be220ae Enable User assigned managed identity for scalesets (#219) 2020-10-29 13:53:11 -04:00
d4c584342a address multiple issues found by pylint (#206) 2020-10-26 12:24:50 -04:00
8a62830f2b reduce how often list_containers is called (#196) 2020-10-23 09:43:45 -04:00
d769072343 cache tokens in memory forever (#195) 2020-10-22 19:13:59 -04:00
0a560cefba fix format strings (#189) 2020-10-22 12:15:39 -04:00
ed370a0d8c using multiple pgs (#121) 2020-10-08 16:08:38 -04:00
46325ea490 add '--endpoint' to 'repro_cmd' for integrations (#113) 2020-10-07 12:11:34 -04:00
e308a4ae1e refactor node state to fully put the agent in charge (#90) 2020-10-03 02:43:04 -04:00
f72543dde2 cleanup logging in multiple functions (#73) 2020-10-01 23:31:08 -04:00
534d99bb87 Set log levels in Azure Functions by hand for 3rd party libraries (#63) 2020-10-01 14:57:09 -04:00
d363a22e39 fix deleting repro VMs (#36) 2020-09-29 09:57:52 -04:00
d3a0b292e6 initial public release 2020-09-18 12:21:04 -04:00