Commit Graph

2095 Commits

Author SHA1 Message Date
4abfbeca56 mark connection refused errors as an ephemeral error (#506) 2021-02-04 15:27:27 -05:00
a50eb94db5 Making input_tester and expand immutable (#500) 2021-02-04 15:18:15 -05:00
220fe46f9d disable check_debugger by default (#505) 2021-02-04 14:23:51 -05:00
1adee4ea83 update pools before nodes or scalesets (#503) 2021-02-04 13:26:19 -05:00
e0cd3d535b rename proxy queue endpoint (#504) 2021-02-04 12:45:36 -05:00
cdfdc2be84 split telemetry into it's own crate (#501)
Splits out telemetry crate such that it can be reused by other components (specifically the proxy-manager) easily.
2021-02-04 14:46:35 +00:00
0a4278110d update azure-mgmt-compute to 18.2.0 (#499) 2021-02-04 12:34:29 +00:00
3094ebb846 Release 2.1.1 (#496) 2.1.1 2021-02-02 16:02:12 -05:00
02721f3ed9 address clippy issues in agent (#490) 2021-02-02 14:41:27 -05:00
e60d697040 add telemetry stats utility (#493) 2021-02-02 13:56:34 +00:00
5e2e9448df add security auditing of python code using Bandit during CICD (#491) 2021-02-01 16:51:03 -05:00
0f70ffa3e2 try pushing updates to scaleset configs frequently until the push succeeds (#489) 2021-02-01 10:09:40 -05:00
e894a523ab Release 2.1.0 (#487) 2.1.0 2021-01-29 16:26:00 -05:00
cc1c79b27d Fix bug when checking if the default cli application exists (#488) 2021-01-29 15:32:17 -05:00
0a1021447b Return an error when download_input fails (#485) 2021-01-29 17:13:36 +00:00
5acb59e5b9 Add task_id & job_id to variable expansion (#481)
Fixes #479 

Note, this is built on top of #480
2021-01-29 15:35:59 +00:00
9c7eb33149 Expand tools_dir accessibility in variable expansion (#480)
Fixes #478
2021-01-29 11:33:23 +00:00
bcf42485eb sync new inputs found by supervisor tasks (#484) 2021-01-28 21:33:30 -05:00
a46f7b4193 expose supervisor tasks that are fully self-contained fuzzing tasks in the service (#474)
Exposes the functionality added in #454 to the service & CLI.

Fixes #439
2021-01-29 00:01:59 +00:00
14fc1ca51f remove unused Event generation from the pre-2.0.0 SignalR integration (#477)
Remove a vestige of the adhoc events used by the previous SingalR integration for container updates.
2021-01-28 21:56:31 +00:00
f155ad625f reimage long-lived nodes (#476)
This helps keep nodes on scalesets that use `latest` OS image SKUs reasonably up-to-date with OS patches without disrupting running fuzzing tasks with patch reboot cycles.

In combination with the already-merged #416, this PR closes #414.
2021-01-28 20:36:40 +00:00
98b6626a84 use license rather than license_file for python packages (#472)
Use the correct field for specifying the license for the onefuzz python pacakges following [setup tools documentation](https://packaging.python.org/guides/distributing-packages-using-setuptools/#license)
2021-01-28 17:56:29 +00:00
07760bf967 Update expansion documentation to describe nested options as implemented (#475)
This expands the documentation to demonstrate how nested lists are handled in practice, as well as to provide examples for how to deal with the nested list expansion.
2021-01-28 17:12:48 +00:00
2e44a5a22f update azure-mgmt-network to the latest version (#471) 2021-01-28 07:04:52 +00:00
81e3ea9895 force set-versions.sh to run in Bash (#473)
Co-authored-by: Brian Caswell <bmc@shmoo.com>
2021-01-27 09:06:12 -08:00
24685ca8df Updating Windows Default Image from RS5-Pro to 20H2-Pro (#469)
RS5-Pro is no longer updated in the Azure Marketplace. In order to ensure the Windows 10 VMs are regularly updated, we need to switch the default image to 20H2-Pro, which is regularly maintained.
2021-01-27 13:46:46 +00:00
cfcf493a23 add context to command failures (#466)
Fixes #465
2021-01-26 21:29:59 +00:00
de67c9db63 Release 2.0.0 (#464) 2.0.0 2021-01-26 15:41:29 -05:00
5027745ee2 simplify get/delete for scalesets (#468) 2021-01-26 14:43:14 -05:00
165257e989 update python prereqs (#427)
Updates the following libraries in the service:
* azure-core
* azure-functions
* azure-identity
* azure-keyvault-keys
* azure-keyvault-secrets
* azure-mgmt-compute
* azure-mgmt-core
* azure-mgmt-loganalytics
* azure-mgmt-network
* azure-mgmt-resource
* azure-mgmt-storage
* azure-mgmt-subscription
* azure-storage-blob
* azure-storage-queue
* pydantic
* requests
* jsonpatch

Removes the following libraries in the service:
* azure-cli-core
* azure-cli-nspkg
* azure-mgmt-cosmosdb
* azure-servicebus

Updates the following libraries in the CLI:
* requests
* semver
* asciimatics
* pydantic
* tenacity

Updates the following libraries in onefuzztypes:
* pydantic

The primary "legacy" libraries are [azure-graphrbac](https://pypi.org/project/azure-graphrbac/) and azure-cosmosdb-table.  The former has not been updated to use azure-identity yet. The later is being rewritten as [azure-data-tables](https://pypi.org/project/azure-data-tables/), but is still in early beta.
2021-01-25 20:53:40 +00:00
31ea71e8b6 use the unique-string based keyvault names (#462) 2021-01-25 15:02:12 -05:00
4bc90a7564 set max stdout/stderr size (#460) 2021-01-25 13:07:35 -05:00
3ae70cea6b add keyvault management to administrative role (#389)
This is a prerequisite for #326
2021-01-25 16:40:36 +00:00
3f2883d38e Storing secrets in azure keyvault (#326) 2021-01-25 11:12:07 -05:00
dc31ffc92b add support for fully self-contained fuzzers (#454) 2021-01-22 18:20:22 -05:00
e4ecf7e230 remove early-exit from cleanup_nodes that broke dead node cleanup (#458) 2021-01-22 18:04:50 -05:00
42337587ad Update QueueClient with latest SAS URL upon renewing registration (#459) 2021-01-22 17:41:42 -05:00
ec982c68c5 Update the registration logic to print manual steps when adal authentication fails (#447)
Mitigate the deployment issue related to the conditional access policy.
The registration logic is updated to use the old rbac python library when possible. 
The deployment will print some manual step for operations that cannot be automated
2021-01-22 22:21:43 +00:00
2f3139cda1 unify node resetting & deleting into delete/recreate (#450) 2021-01-22 22:04:44 +00:00
c0a4b0dba4 remove workaround for an issue addressed in latest mypy (#455) 2021-01-22 14:00:35 -05:00
3c76baa3bb Use goblin for more direct ELF handling (#453)
- Switch to using `goblin` for both ELF and PE parsing
- Refactor block entry point recovery, with better documentation
- Fix a broken example binary

Co-authored-by: bmc-msft <41130664+bmc-msft@users.noreply.github.com>
2021-01-21 15:00:19 -08:00
4f941d5ff2 add the task_id to the integration failure logs (#446) 2021-01-21 13:35:54 -05:00
95bd4ac308 add unit test to verify functionality of nested list expansion (#448) 2021-01-21 12:05:17 -05:00
e6dec041b2 move to using machine_id rather than node_id (#451)
Handle unifying onto machine_id for NodeMessage.
2021-01-21 16:22:16 +00:00
7e6415b15f ensure consistency in doc generation by always sorting json keys (#449) 2021-01-21 15:28:11 +00:00
b499b9b17d always set SignalR events at the end of handlers (#445)
Addresses an issue where events meant for SignalR do not get sent to the service when an App Service instance spins down before a timer event fires.
2021-01-21 10:31:02 +00:00
f3d81566e3 add expanding {tools_dir} to supervisor_options and supervisor_env (#444)
Fixes #443
2021-01-20 20:34:25 +00:00
fd956380d4 experimental "local fuzzing" support (#405)
This PR adds an experimental "local" mode for the agent, starting with `libfuzzer`.  For tasks that poll a queue, in local mode, they just monitor a directory for new files.

Supported commands: 
* libfuzzer-fuzz (models the `libfuzzer-fuzz` task)
* libfuzzer-coverage (models the `libfuzzer-coverage` task)
* libfuzzer-crash-report (models the `libfuzzer-crash-report` task)
* libfuzzer (models the `libfuzzer basic` job template, running libfuzzer-fuzz and libfuzzer-crash-report tasks concurrently, where any files that show up in `crashes_dir` are automatically turned into reports, and optionally runs the coverage task which runs the coverage data exporter for each file that shows up in `inputs_dir`).

Under the hood, there are a handful of changes required to the rest of the system to enable this feature.
1. `SyncedDir` URLs are now optional.  In local mode, these no longer make sense.   (We've discussed moving management of `SyncedDirs` to the Supervisor.  This is tangential to that effort.)
2. `InputPoller` uses a `tempdir` rather than abusing `task_id` for temporary directory naming.
3. Moved the `agent` to only use a single tokio runtime, rather than one for each of the subcommands.
4. Sets the default log level to `info`.  (RUST_LOG can still be used as is).

Note, this removes the `onefuzz-agent debug` commands for the tasks that are now exposed via `onefuzz-agent local`, as these provide a more featureful version of the debug tasks.
2021-01-20 03:33:25 +00:00
af2ef9f4fa verify local debugger installs before launching repro (#436)
This only launches the repro test cases if you have the os-specific prerequisite debugger installed.
2021-01-18 23:30:53 +00:00
efaa381899 Add missing prerequisites to the Getting Started guide (#435)
Fixes #434
2021-01-16 09:07:13 +00:00