ExternalVendorCode/onefuzz
1
0
Fork 0
mirror of https://github.com/microsoft/onefuzz.git synced 2025-06-21 05:42:14 +00:00
Code Issues Packages Projects Releases Wiki Activity

Declarative templates (#266)

Browse Source
This commit is contained in:
bmc-msft
2020-11-17 16:00:09 -05:00
committed by GitHub
parent ce3356d597
commit 64bd389eb7
40 changed files with 2111 additions and 69 deletions
Download Patch File Download Diff File Expand all files Collapse all files

221
docs/declarative-templates.md Normal file
View File

@ -0,0 +1,221 @@
# Declarative Job Templates
Provide the ability to maintain job templates, akin to `onefuzz template
libfuzzer basic` at the service level. The templates include a job
definition, an arbitrary set of tasks, and an arbitrary set of notification
configs. The templates are managed at the service level, with job-submission
time updates using a declarative syntax based on `jsonpatch`.
The SDK makes use of template configs, provided by the service, to
dynamically build Python methods for each template. This process enables the
automatic argument parser generated by type signatures to automatically
create the CLI subcommands for the template.
## User Experience
* On `onefuzz login` (or `onefuzz job_templates refresh`), cache the existing set of templates
* Users can see the supported templates via `onefuzz job_templates submit --help`
* Users can submit jobs via `onefuzz job_templates submit libfuzzer OSNAME project name build pool --target_exe fuzz.exe`
* Template configs are refreshed automatically if they are older than 24 hours.
Future work:
* submitting jobs by config. Not everything is easy to express via argparse,
such as values that begin with `-`. In order to support this, it should be
trivial to expose a method that takes a json file and submits it.
## Admin Experience
Administrators can manage their own templates via `onefuzz job_templates
manage [create,delete,list,update]`.
If the runtime configuration for the template changes, users will need to
refresh their cache to pull the runtime configuration.
## Implementation Details
A declarative job template includes:
* a Job (JobConfig, as used by `onefuzz jobs create`)
* a list of Tasks (TaskConfig, which is used by `onefuzz tasks create`)
* a list of Notifications (NotificationConfig + container type, akin to what is used by `onefuzz notifications create`)
* a list of required and optional form fields used to update the aforementioned JobConfig, TaskConfig, and NotificationConfig entries at runtime.
The form fields allow for 'add' or 'replace' of basic field data using [jsonpatch](http://jsonpatch.com) semantics.
## Example Form Fields
This following field named `target_workers`, which is required to be an `int`,
will optionally (if the request includes it) replace the `target_workers` value
of in the first task in the template.
```python
UserField(
name="target_workers",
required=False,
type=UserFieldType.Int,
help="The number of workers to use for this task",
locations=[
UserFieldLocation(
op=UserFieldOperation.replace,
path="/tasks/0/task/target_workers",
),
],
)
```
## Allowed Data Types
The data types allowed in configuring arbitrary components in the JobTemplate are:
* `bool`
* `int`
* `str`
* `Dict[str, str]`
* `List[str]`
## Referring to Tasks
In existing procedural templates, some tasks require that other tasks are
running before they may be scheduled. For example, in the `libfuzzer`
procedural template, the `libfuzzer_crash_report` task has a `libfuzzer_fuzz`
task prerequisite. This is specified via the prerequisite's `task_id`, which is
a random server-assigned UUID.
In procedural templates, the dependency task is simply created before its
dependent.
To support such a reference in `OnefuzzTemplate`, specify the prerequisite task
by the `u128` representation index in to the list of tasks. Example, to refer
to the first task, use:
```python
TaskConfig(
prereq_tasks=[UUID(int=0)],
...
)
```
## Hardcoded vs Runtime-specified Container Names
To support differentiating _always use "afl-linux" for tools_ vs _ask
what container to use for setup_, if the container name is blank in the
template, it will be provided as part of the `JobTemplateConfig` and in the
resulting `JobTemplateRequest`.
## Specifying Notifications in the Template
The existing templates support adding a notification config on the command
line, via `--notification_config`, but the existing templates themselves do
not include default notifications.
Declarative job templates include optional support to configure notifications
as part of the template, rather than requiring the user provide the
configuration.
Example declarative job template that specifies using the aforementioned
NotificationConfig for the `unique_reports` containers used in the Job.
```python
JobTemplateNotification(
container_type=ContainerType.unique_reports,
notification=NotificationConfig(config=TeamsTemplate(url="https://contoso.com/webhook-url-here")),
)
```
## Differences from Existing Templates
* Declaratively specifying the allowed values for enums, such as StatsFormat,
is not supported. Fields must currently use Str, which evaluates to Enum
value during template rendering, is functional.
* Existing templates automatically differentiate between Windows and Linux
tasks. This does not support differentiating between platforms automatically.
As such, there is a new required parameter to specify the OS.
## From the CLI
```
❯ onefuzz job_templates submit libfuzzer --help
usage: onefuzz job_templates submit libfuzzer [-h] [-v] [--format {json,raw}] [--query QUERY] [--target_exe TARGET_EXE]
[--duration DURATION] [--target_workers TARGET_WORKERS] [--vm_count VM_COUNT]
[--target_options [TARGET_OPTIONS [TARGET_OPTIONS ...]]]
[--target_env str=str [str=str ...]] [--reboot_after_setup]
[--check_retry_count CHECK_RETRY_COUNT] [--target_timeout TARGET_TIMEOUT]
[--tags str=str [str=str ...]] [--readonly_inputs_dir READONLY_INPUTS_DIR]
[--setup_dir SETUP_DIR] [--inputs_dir INPUTS_DIR]
[--container_names ContainerType=str [ContainerType=str ...]]
OS project name build pool_name
positional arguments:
OS Specify the OS to use in the job. accepted OS: windows, linux
project Name of the Project
name Name of the Target
build Name of the Target
pool_name Execute the task on the specified pool
optional arguments:
-h, --help show this help message and exit
-v, --verbose increase output verbosity
--format {json,raw} output format
--query QUERY JMESPath query string. See http://jmespath.org/ for more information and examples.
--target_exe TARGET_EXE
Path to the target executable (default: fuzz.exe)
--duration DURATION Number of hours to execute the task (default: 24)
--target_workers TARGET_WORKERS
Number of instances of the libfuzzer target on each VM
--vm_count VM_COUNT Number of VMs to use for fuzzing (default: 2)
--target_options [TARGET_OPTIONS [TARGET_OPTIONS ...]]
Command line options for the target
--target_env str=str [str=str ...]
Environment variables for the target
--reboot_after_setup After executing the setup script, reboot the VM (Default: False. Sets value to True)
--check_retry_count CHECK_RETRY_COUNT
Number of times to retry a crash to verify reproducability
--target_timeout TARGET_TIMEOUT
Number of seconds to timeout during reproduction
--tags str=str [str=str ...]
User provided metadata for the tasks
--readonly_inputs_dir READONLY_INPUTS_DIR
Local path to the readonly_inputs directory
--setup_dir SETUP_DIR
Local path to the setup directory
--inputs_dir INPUTS_DIR
Local path to the inputs directory
--container_names ContainerType=str [ContainerType=str ...]
custom container names (eg: setup=my-setup-container)
```
## From the SDK
From the perspective of the SDK, this looks _very_ similar to the existing templates. All of the arguments from the request are converted into named arguments with appropriate type signatures.
```
❯ python
Python 3.8.2 (default, Jul 16 2020, 14:00:26)
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from onefuzz.api import Onefuzz
>>> a = Onefuzz()
>>> help(a.job_templates.submit.libfuzzer)
Help on method func in module onefuzz.job_templates.main:
func(platform: onefuzztypes.enums.OS, *, project: str, name: str, build: str, pool_name: str, target_exe: <function NewType.<locals>.new_type at 0x7f7909338040> = 'fuzz.exe', duration: int = 24, target_workers: Union[int, NoneType], vm_count: int = 2, target_options: Union[List[str], NoneType], target_env: Union[Dict[str, str], NoneType], reboot_after_setup: bool = False, check_retry_count: Union[int, NoneType], target_timeout: Union[int, NoneType], tags: Union[Dict[str, str], NoneType], readonly_inputs_dir: Union[Directory, NoneType], setup_dir: Union[Directory, NoneType], inputs_dir: Union[Directory, NoneType], container_names: Union[Dict[onefuzztypes.enums.ContainerType, str], NoneType] = None) -> onefuzztypes.models.Job method of onefuzz.job_templates.main.TemplateHandler instance
Launch 'libfuzzer' job
:param Platform platform: Specify the OS to use in the job.
:param str project: Name of the Project
:param str name: Name of the Target
:param str build: Name of the Target
:param str pool_name: Execute the task on the specified pool
:param str target_exe: Path to the target executable
:param int duration: Number of hours to execute the task
:param int target_workers: Number of instances of the libfuzzer target on each VM
:param int vm_count: Number of VMs to use for fuzzing
:param list target_options: Command line options for the target
:param dict target_env: Environment variables for the target
:param bool reboot_after_setup: After executing the setup script, reboot the VM
:param int check_retry_count: Number of times to retry a crash to verify reproducability
:param int target_timeout: Number of seconds to timeout during reproduction
:param dict tags: User provided metadata for the tasks
:param Directory readonly_inputs_dir: Local path to the readonly_inputs directory
:param Directory setup_dir: Local path to the setup directory
:param Directory inputs_dir: Local path to the inputs directory
:param dict container_names: custom container names (eg: setup=my-setup-container)
```

6
docs/webhook_events.md
View File

@ -581,7 +581,8 @@ Each event will be submitted via HTTP POST to the user provided URL.
467,
468,
469,
470
470,
471
]
},
"Error": {
@ -1124,7 +1125,8 @@ Each event will be submitted via HTTP POST to the user provided URL.
467,
468,
469,
470
470,
471
]
},
"Error": {

3
src/agent/win-util/src/handle.rs
View File

@ -1,3 +1,6 @@
// Copyright (c) Microsoft Corporation.
// Licensed under the MIT License.
use winapi::{
shared::minwindef::{FALSE, LPHANDLE},
um::{

42
src/api-service/__app__/job_templates/__init__.py Normal file
View File

@ -0,0 +1,42 @@
#!/usr/bin/env python
#
# Copyright (c) Microsoft Corporation.
# Licensed under the MIT License.
import azure.functions as func
from onefuzztypes.job_templates import JobTemplateRequest
from onefuzztypes.models import Error
from ..onefuzzlib.job_templates.templates import JobTemplateIndex
from ..onefuzzlib.request import not_ok, ok, parse_request
from ..onefuzzlib.user_credentials import parse_jwt_token
def get(req: func.HttpRequest) -> func.HttpResponse:
configs = JobTemplateIndex.get_configs()
return ok(configs)
def post(req: func.HttpRequest) -> func.HttpResponse:
request = parse_request(JobTemplateRequest, req)
if isinstance(request, Error):
return not_ok(request, context="JobTemplateRequest")
user_info = parse_jwt_token(req)
if isinstance(user_info, Error):
return not_ok(user_info, context="JobTemplateRequest")
job = JobTemplateIndex.execute(request, user_info)
if isinstance(job, Error):
return not_ok(job, context="JobTemplateRequest")
return ok(job)
def main(req: func.HttpRequest) -> func.HttpResponse:
if req.method == "GET":
return get(req)
elif req.method == "POST":
return post(req)
else:
raise Exception("invalid method")

20
src/api-service/__app__/job_templates/function.json Normal file
View File

@ -0,0 +1,20 @@
{
"scriptFile": "__init__.py",
"bindings": [
{
"authLevel": "anonymous",
"type": "httpTrigger",
"direction": "in",
"name": "req",
"methods": [
"get",
"post"
]
},
{
"type": "http",
"direction": "out",
"name": "$return"
}
]
}

74
src/api-service/__app__/job_templates_manage/__init__.py Normal file
View File

@ -0,0 +1,74 @@
#!/usr/bin/env python
#
# Copyright (c) Microsoft Corporation.
# Licensed under the MIT License.
import azure.functions as func
from onefuzztypes.enums import ErrorCode
from onefuzztypes.job_templates import (
JobTemplateCreate,
JobTemplateDelete,
JobTemplateUpdate,
)
from onefuzztypes.models import Error
from onefuzztypes.responses import BoolResult
from ..onefuzzlib.job_templates.templates import JobTemplateIndex
from ..onefuzzlib.request import not_ok, ok, parse_request
def get(req: func.HttpRequest) -> func.HttpResponse:
templates = JobTemplateIndex.get_index()
return ok(templates)
def post(req: func.HttpRequest) -> func.HttpResponse:
request = parse_request(JobTemplateCreate, req)
if isinstance(request, Error):
return not_ok(request, context="JobTemplateCreate")
entry = JobTemplateIndex(name=request.name, template=request.template)
result = entry.save(new=True)
if isinstance(result, Error):
return not_ok(result, context="JobTemplateCreate")
return ok(BoolResult(result=True))
def patch(req: func.HttpRequest) -> func.HttpResponse:
request = parse_request(JobTemplateUpdate, req)
if isinstance(request, Error):
return not_ok(request, context="JobTemplateUpdate")
entry = JobTemplateIndex.get(request.name)
if entry is None:
return not_ok(
Error(code=ErrorCode.UNABLE_TO_UPDATE, errors=["no such job template"]),
context="JobTemplateUpdate",
)
entry.template = request.template
entry.save()
return ok(BoolResult(result=True))
def delete(req: func.HttpRequest) -> func.HttpResponse:
request = parse_request(JobTemplateDelete, req)
if isinstance(request, Error):
return not_ok(request, context="JobTemplateDelete")
entry = JobTemplateIndex.get(request.name)
return ok(BoolResult(result=entry is not None))
def main(req: func.HttpRequest) -> func.HttpResponse:
if req.method == "GET":
return get(req)
elif req.method == "POST":
return post(req)
elif req.method == "DELETE":
return delete(req)
elif req.method == "PATCH":
return patch(req)
else:
raise Exception("invalid method")

23
src/api-service/__app__/job_templates_manage/function.json Normal file
View File

@ -0,0 +1,23 @@
{
"scriptFile": "__init__.py",
"bindings": [
{
"authLevel": "anonymous",
"type": "httpTrigger",
"direction": "in",
"name": "req",
"methods": [
"get",
"post",
"delete",
"patch"
],
"route": "job_templates/manage"
},
{
"type": "http",
"direction": "out",
"name": "$return"
}
]
}

18
src/api-service/__app__/notifications/__init__.py
View File

@ -6,11 +6,9 @@
import logging
import azure.functions as func
from onefuzztypes.enums import ErrorCode
from onefuzztypes.models import Error
from onefuzztypes.requests import NotificationCreate, NotificationGet
from ..onefuzzlib.azure.containers import container_exists
from ..onefuzzlib.notifications.main import Notification
from ..onefuzzlib.request import not_ok, ok, parse_request
@ -29,19 +27,11 @@ def post(req: func.HttpRequest) -> func.HttpResponse:
if isinstance(request, Error):
return not_ok(request, context="notification create")
if not container_exists(request.container):
return not_ok(
Error(code=ErrorCode.INVALID_REQUEST, errors=["invalid container"]),
context=request.container,
)
entry = Notification.create(container=request.container, config=request.config)
if isinstance(entry, Error):
return not_ok(entry, context="notification create")
existing = Notification.get_existing(request.container, request.config)
if existing is not None:
return ok(existing)
item = Notification(container=request.container, config=request.config)
item.save()
return ok(item)
return ok(entry)
def delete(req: func.HttpRequest) -> func.HttpResponse:

0
src/api-service/__app__/onefuzzlib/job_templates/__init__.py Normal file
View File

14
src/api-service/__app__/onefuzzlib/job_templates/defaults/__init__.py Normal file
View File

@ -0,0 +1,14 @@
#!/usr/bin/env python
#
# Copyright (c) Microsoft Corporation.
# Licensed under the MIT License.
from .afl import afl_linux, afl_windows
from .libfuzzer import libfuzzer_linux, libfuzzer_windows
TEMPLATES = {
"afl_windows": afl_windows,
"afl_linux": afl_linux,
"libfuzzer_linux": libfuzzer_linux,
"libfuzzer_windows": libfuzzer_windows,
}

271
src/api-service/__app__/onefuzzlib/job_templates/defaults/afl.py Normal file
View File

@ -0,0 +1,271 @@
#!/usr/bin/env python
#
# Copyright (c) Microsoft Corporation.
# Licensed under the MIT License.
from uuid import UUID
from onefuzztypes.enums import (
OS,
ContainerType,
TaskType,
UserFieldOperation,
UserFieldType,
)
from onefuzztypes.job_templates import JobTemplate, UserField, UserFieldLocation
from onefuzztypes.models import (
JobConfig,
TaskConfig,
TaskContainers,
TaskDetails,
TaskPool,
)
from onefuzztypes.primitives import Container, PoolName
from .common import (
DURATION_HELP,
POOL_HELP,
REBOOT_HELP,
RETRY_COUNT_HELP,
TAGS_HELP,
TARGET_EXE_HELP,
TARGET_OPTIONS_HELP,
VM_COUNT_HELP,
)
afl_linux = JobTemplate(
os=OS.linux,
job=JobConfig(project="", name=Container(""), build="", duration=1),
tasks=[
TaskConfig(
job_id=(UUID(int=0)),
task=TaskDetails(
type=TaskType.generic_supervisor,
duration=1,
target_exe="fuzz.exe",
target_env={},
target_options=[],
supervisor_exe="",
supervisor_options=[],
supervisor_input_marker="@@",
),
pool=TaskPool(count=1, pool_name=PoolName("")),
containers=[
TaskContainers(
name=Container("afl-container-name"), type=ContainerType.tools
),
TaskContainers(name=Container(""), type=ContainerType.setup),
TaskContainers(name=Container(""), type=ContainerType.crashes),
TaskContainers(name=Container(""), type=ContainerType.inputs),
],
tags={},
),
TaskConfig(
job_id=UUID(int=0),
prereq_tasks=[UUID(int=0)],
task=TaskDetails(
type=TaskType.generic_crash_report,
duration=1,
target_exe="fuzz.exe",
target_env={},
target_options=[],
check_debugger=True,
),
pool=TaskPool(count=1, pool_name=PoolName("")),
containers=[
TaskContainers(name=Container(""), type=ContainerType.setup),
TaskContainers(name=Container(""), type=ContainerType.crashes),
TaskContainers(name=Container(""), type=ContainerType.no_repro),
TaskContainers(name=Container(""), type=ContainerType.reports),
TaskContainers(name=Container(""), type=ContainerType.unique_reports),
],
tags={},
),
],
notifications=[],
user_fields=[
UserField(
name="pool_name",
help=POOL_HELP,
type=UserFieldType.Str,
required=True,
locations=[
UserFieldLocation(
op=UserFieldOperation.replace,
path="/tasks/0/pool/pool_name",
),
UserFieldLocation(
op=UserFieldOperation.replace,
path="/tasks/1/pool/pool_name",
),
],
),
UserField(
name="duration",
help=DURATION_HELP,
type=UserFieldType.Int,
default=24,
locations=[
UserFieldLocation(
op=UserFieldOperation.replace,
path="/tasks/0/task/duration",
),
UserFieldLocation(
op=UserFieldOperation.replace,
path="/tasks/1/task/duration",
),
UserFieldLocation(op=UserFieldOperation.replace, path="/job/duration"),
],
),
UserField(
name="target_exe",
help=TARGET_EXE_HELP,
type=UserFieldType.Str,
default="fuzz.exe",
locations=[
UserFieldLocation(
op=UserFieldOperation.replace,
path="/tasks/0/task/target_exe",
),
UserFieldLocation(
op=UserFieldOperation.replace,
path="/tasks/1/task/target_exe",
),
],
),
UserField(
name="target_options",
help=TARGET_OPTIONS_HELP,
type=UserFieldType.ListStr,
locations=[
UserFieldLocation(
op=UserFieldOperation.replace,
path="/tasks/0/task/target_options",
),
UserFieldLocation(
op=UserFieldOperation.replace,
path="/tasks/1/task/target_options",
),
],
),
UserField(
name="supervisor_exe",
help="Path to the AFL executable",
type=UserFieldType.Str,
default="{tools_dir}/afl-fuzz",
locations=[
UserFieldLocation(
op=UserFieldOperation.replace,
path="/tasks/0/task/supervisor_exe",
),
],
),
UserField(
name="supervisor_options",
help="AFL command line options",
type=UserFieldType.ListStr,
default=[
"-d",
"-i",
"{input_corpus}",
"-o",
"{runtime_dir}",
"--",
"{target_exe}",
"{target_options}",
],
locations=[
UserFieldLocation(
op=UserFieldOperation.replace,
path="/tasks/0/task/supervisor_options",
),
],
),
UserField(
name="supervisor_env",
help="Enviornment variables for AFL",
type=UserFieldType.DictStr,
locations=[
UserFieldLocation(
op=UserFieldOperation.replace,
path="/tasks/0/task/supervisor_env",
),
],
),
UserField(
name="vm_count",
help=VM_COUNT_HELP,
type=UserFieldType.Int,
default=2,
locations=[
UserFieldLocation(
op=UserFieldOperation.replace,
path="/tasks/0/pool/count",
),
],
),
UserField(
name="check_retry_count",
help=RETRY_COUNT_HELP,
type=UserFieldType.Int,
locations=[
UserFieldLocation(
op=UserFieldOperation.replace,
path="/tasks/1/task/check_retry_count",
),
],
),
UserField(
name="afl_container",
help=(
"Name of the AFL storage container (use "
"this to specify alternate builds of AFL)"
),
type=UserFieldType.Str,
default="afl-linux",
locations=[
UserFieldLocation(
op=UserFieldOperation.replace,
path="/tasks/0/containers/0/name",
),
],
),
UserField(
name="reboot_after_setup",
help=REBOOT_HELP,
type=UserFieldType.Bool,
default=False,
locations=[
UserFieldLocation(
op=UserFieldOperation.replace,
path="/tasks/0/task/reboot_after_setup",
),
UserFieldLocation(
op=UserFieldOperation.replace,
path="/tasks/1/task/reboot_after_setup",
),
],
),
UserField(
name="tags",
help=TAGS_HELP,
type=UserFieldType.DictStr,
locations=[
UserFieldLocation(
op=UserFieldOperation.add,
path="/tasks/0/tags",
),
UserFieldLocation(
op=UserFieldOperation.add,
path="/tasks/1/tags",
),
],
),
],
)
afl_windows = afl_linux.copy(deep=True)
afl_windows.os = OS.windows
for user_field in afl_windows.user_fields:
if user_field.name == "afl_container":
user_field.default = "afl-windows"

14
src/api-service/__app__/onefuzzlib/job_templates/defaults/common.py Normal file
View File

@ -0,0 +1,14 @@
#!/usr/bin/env python
#
# Copyright (c) Microsoft Corporation.
# Licensed under the MIT License.
POOL_HELP = "Execute the task on the specified pool"
DURATION_HELP = "Number of hours to execute the task"
TARGET_EXE_HELP = "Path to the target executable"
TARGET_OPTIONS_HELP = "Command line options for the target"
VM_COUNT_HELP = "Number of VMs to use for fuzzing"
RETRY_COUNT_HELP = "Number of times to retry a crash to verify reproducability"
REBOOT_HELP = "After executing the setup script, reboot the VM"
TAGS_HELP = "User provided metadata for the tasks"

285
src/api-service/__app__/onefuzzlib/job_templates/defaults/libfuzzer.py Normal file
View File

@ -0,0 +1,285 @@
#!/usr/bin/env python
#
# Copyright (c) Microsoft Corporation.
# Licensed under the MIT License.
from uuid import UUID
from onefuzztypes.enums import (
OS,
ContainerType,
TaskType,
UserFieldOperation,
UserFieldType,
)
from onefuzztypes.job_templates import JobTemplate, UserField, UserFieldLocation
from onefuzztypes.models import (
JobConfig,
TaskConfig,
TaskContainers,
TaskDetails,
TaskPool,
)
from onefuzztypes.primitives import Container, PoolName
from .common import (
DURATION_HELP,
POOL_HELP,
REBOOT_HELP,
RETRY_COUNT_HELP,
TAGS_HELP,
TARGET_EXE_HELP,
TARGET_OPTIONS_HELP,
VM_COUNT_HELP,
)
libfuzzer_linux = JobTemplate(
os=OS.linux,
job=JobConfig(project="", name=Container(""), build="", duration=1),
tasks=[
TaskConfig(
job_id=UUID(int=0),
task=TaskDetails(
type=TaskType.libfuzzer_fuzz,
duration=1,
target_exe="fuzz.exe",
target_env={},
target_options=[],
),
pool=TaskPool(count=1, pool_name=PoolName("")),
containers=[
TaskContainers(name=Container(""), type=ContainerType.setup),
TaskContainers(name=Container(""), type=ContainerType.crashes),
TaskContainers(name=Container(""), type=ContainerType.inputs),
],
tags={},
),
TaskConfig(
job_id=UUID(int=0),
prereq_tasks=[UUID(int=0)],
task=TaskDetails(
type=TaskType.libfuzzer_crash_report,
duration=1,
target_exe="fuzz.exe",
target_env={},
target_options=[],
),
pool=TaskPool(count=1, pool_name=PoolName("")),
containers=[
TaskContainers(name=Container(""), type=ContainerType.setup),
TaskContainers(name=Container(""), type=ContainerType.crashes),
TaskContainers(name=Container(""), type=ContainerType.no_repro),
TaskContainers(name=Container(""), type=ContainerType.reports),
TaskContainers(name=Container(""), type=ContainerType.unique_reports),
],
tags={},
),
TaskConfig(
job_id=UUID(int=0),
prereq_tasks=[UUID(int=0)],
task=TaskDetails(
type=TaskType.libfuzzer_coverage,
duration=1,
target_exe="fuzz.exe",
target_env={},
target_options=[],
),
pool=TaskPool(count=1, pool_name=PoolName("")),
containers=[
TaskContainers(name=Container(""), type=ContainerType.setup),
TaskContainers(name=Container(""), type=ContainerType.readonly_inputs),
TaskContainers(name=Container(""), type=ContainerType.coverage),
],
tags={},
),
],
notifications=[],
user_fields=[
UserField(
name="pool_name",
help=POOL_HELP,
type=UserFieldType.Str,
required=True,
locations=[
UserFieldLocation(
op=UserFieldOperation.replace,
path="/tasks/0/pool/pool_name",
),
UserFieldLocation(
op=UserFieldOperation.replace,
path="/tasks/1/pool/pool_name",
),
UserFieldLocation(
op=UserFieldOperation.replace,
path="/tasks/2/pool/pool_name",
),
],
),
UserField(
name="target_exe",
help=TARGET_EXE_HELP,
type=UserFieldType.Str,
default="fuzz.exe",
locations=[
UserFieldLocation(
op=UserFieldOperation.replace,
path="/tasks/0/task/target_exe",
),
UserFieldLocation(
op=UserFieldOperation.replace,
path="/tasks/1/task/target_exe",
),
UserFieldLocation(
op=UserFieldOperation.replace,
path="/tasks/2/task/target_exe",
),
],
),
UserField(
name="duration",
help=DURATION_HELP,
type=UserFieldType.Int,
default=24,
locations=[
UserFieldLocation(
op=UserFieldOperation.replace,
path="/tasks/0/task/duration",
),
UserFieldLocation(
op=UserFieldOperation.replace,
path="/tasks/1/task/duration",
),
UserFieldLocation(
op=UserFieldOperation.replace,
path="/tasks/2/task/duration",
),
UserFieldLocation(op=UserFieldOperation.replace, path="/job/duration"),
],
),
UserField(
name="target_workers",
help="Number of instances of the libfuzzer target on each VM",
type=UserFieldType.Int,
locations=[
UserFieldLocation(
op=UserFieldOperation.replace,
path="/tasks/0/task/target_workers",
),
],
),
UserField(
name="vm_count",
help=VM_COUNT_HELP,
type=UserFieldType.Int,
default=2,
locations=[
UserFieldLocation(
op=UserFieldOperation.replace,
path="/tasks/0/pool/count",
),
],
),
UserField(
name="target_options",
help=TARGET_OPTIONS_HELP,
type=UserFieldType.ListStr,
locations=[
UserFieldLocation(
op=UserFieldOperation.replace,
path="/tasks/0/task/target_options",
),
UserFieldLocation(
op=UserFieldOperation.replace,
path="/tasks/1/task/target_options",
),
UserFieldLocation(
op=UserFieldOperation.replace,
path="/tasks/2/task/target_options",
),
],
),
UserField(
name="target_env",
help="Environment variables for the target",
type=UserFieldType.DictStr,
locations=[
UserFieldLocation(
op=UserFieldOperation.replace,
path="/tasks/0/task/target_env",
),
UserFieldLocation(
op=UserFieldOperation.replace,
path="/tasks/1/task/target_env",
),
UserFieldLocation(
op=UserFieldOperation.replace,
path="/tasks/2/task/target_env",
),
],
),
UserField(
name="reboot_after_setup",
help=REBOOT_HELP,
type=UserFieldType.Bool,
default=False,
locations=[
UserFieldLocation(
op=UserFieldOperation.replace,
path="/tasks/0/task/reboot_after_setup",
),
UserFieldLocation(
op=UserFieldOperation.replace,
path="/tasks/1/task/reboot_after_setup",
),
UserFieldLocation(
op=UserFieldOperation.replace,
path="/tasks/2/task/reboot_after_setup",
),
],
),
UserField(
name="check_retry_count",
help=RETRY_COUNT_HELP,
type=UserFieldType.Int,
locations=[
UserFieldLocation(
op=UserFieldOperation.replace,
path="/tasks/1/task/check_retry_count",
),
],
),
UserField(
name="target_timeout",
help="Number of seconds to timeout during reproduction",
type=UserFieldType.Int,
locations=[
UserFieldLocation(
op=UserFieldOperation.replace,
path="/tasks/1/task/target_timeout",
),
],
),
UserField(
name="tags",
help=TAGS_HELP,
type=UserFieldType.DictStr,
locations=[
UserFieldLocation(
op=UserFieldOperation.add,
path="/tasks/0/tags",
),
UserFieldLocation(
op=UserFieldOperation.add,
path="/tasks/1/tags",
),
UserFieldLocation(
op=UserFieldOperation.add,
path="/tasks/2/tags",
),
],
),
],
)
libfuzzer_windows = libfuzzer_linux.copy(deep=True)
libfuzzer_windows.os = OS.windows

131
src/api-service/__app__/onefuzzlib/job_templates/render.py Normal file
View File

@ -0,0 +1,131 @@
#!/usr/bin/env python
#
# Copyright (c) Microsoft Corporation.
# Licensed under the MIT License.
import json
from typing import Dict, List
from jsonpatch import apply_patch
from memoization import cached
from onefuzztypes.enums import ContainerType, ErrorCode, UserFieldType
from onefuzztypes.job_templates import (
TEMPLATE_BASE_FIELDS,
JobTemplate,
JobTemplateConfig,
JobTemplateField,
JobTemplateRequest,
TemplateUserData,
UserField,
)
from onefuzztypes.models import Error, Result
def template_container_types(template: JobTemplate) -> List[ContainerType]:
return list(set(c.type for t in template.tasks for c in t.containers if not c.name))
@cached
def build_input_config(name: str, template: JobTemplate) -> JobTemplateConfig:
user_fields = [
JobTemplateField(
name=x.name,
type=x.type,
required=x.required,
default=x.default,
help=x.help,
)
for x in TEMPLATE_BASE_FIELDS + template.user_fields
]
containers = template_container_types(template)
return JobTemplateConfig(
os=template.os,
name=name,
user_fields=user_fields,
containers=containers,
)
def build_patches(
data: TemplateUserData, field: UserField
) -> List[Dict[str, TemplateUserData]]:
patches = []
if field.type == UserFieldType.Bool and not isinstance(data, bool):
raise Exception("invalid bool field")
if field.type == UserFieldType.Int and not isinstance(data, int):
raise Exception("invalid int field")
if field.type == UserFieldType.Str and not isinstance(data, str):
raise Exception("invalid str field")
if field.type == UserFieldType.DictStr and not isinstance(data, dict):
raise Exception("invalid DictStr field")
if field.type == UserFieldType.ListStr and not isinstance(data, list):
raise Exception("invalid ListStr field")
for location in field.locations:
patches.append(
{
"op": location.op.name,
"path": location.path,
"value": data,
}
)
return patches
def _fail(why: str) -> Error:
return Error(code=ErrorCode.INVALID_REQUEST, errors=[why])
def render(request: JobTemplateRequest, template: JobTemplate) -> Result[JobTemplate]:
patches = []
seen = set()
for name in request.user_fields:
for field in TEMPLATE_BASE_FIELDS + template.user_fields:
if field.name == name:
if name in seen:
return _fail(f"duplicate specification: {name}")
seen.add(name)
if name not in seen:
return _fail(f"extra field: {name}")
for field in TEMPLATE_BASE_FIELDS + template.user_fields:
if field.name not in request.user_fields:
if field.required:
return _fail(f"missing required field: {field.name}")
else:
# optional fields can be missing
continue
patches += build_patches(request.user_fields[field.name], field)
raw = json.loads(template.json())
updated = apply_patch(raw, patches)
rendered = JobTemplate.parse_obj(updated)
used_containers = []
for task in rendered.tasks:
for task_container in task.containers:
if task_container.name:
# only need to fill out containers with names
continue
for entry in request.containers:
if entry.type != task_container.type:
continue
task_container.name = entry.name
used_containers.append(entry)
if not task_container.name:
return _fail(f"missing container definition {task_container.type}")
for entry in request.containers:
if entry not in used_containers:
return _fail(f"unused container in request: {entry}")
return rendered

104
src/api-service/__app__/onefuzzlib/job_templates/templates.py Normal file
View File

@ -0,0 +1,104 @@
#!/usr/bin/env python
#
# Copyright (c) Microsoft Corporation.
# Licensed under the MIT License.
from typing import List, Optional, Tuple
from onefuzztypes.enums import ErrorCode
from onefuzztypes.job_templates import JobTemplateConfig
from onefuzztypes.job_templates import JobTemplateIndex as BASE_INDEX
from onefuzztypes.job_templates import JobTemplateRequest
from onefuzztypes.models import Error, Result, UserInfo
from ..jobs import Job
from ..notifications.main import Notification
from ..orm import ORMMixin
from ..tasks.config import TaskConfigError, check_config
from ..tasks.main import Task
from .defaults import TEMPLATES
from .render import build_input_config, render
class JobTemplateIndex(BASE_INDEX, ORMMixin):
@classmethod
def key_fields(cls) -> Tuple[str, Optional[str]]:
return ("name", None)
@classmethod
def get_index(cls) -> List[BASE_INDEX]:
entries = [BASE_INDEX(name=x.name, template=x.template) for x in cls.search()]
# if the local install has replaced the built-in templates, skip over them
for name, template in TEMPLATES.items():
if any(x.name == name for x in entries):
continue
entries.append(BASE_INDEX(name=name, template=template))
return entries
@classmethod
def get_configs(cls) -> List[JobTemplateConfig]:
configs = [build_input_config(x.name, x.template) for x in cls.get_index()]
return configs
@classmethod
def execute(cls, request: JobTemplateRequest, user_info: UserInfo) -> Result[Job]:
index = cls.get(request.name)
if index is None:
if request.name not in TEMPLATES:
return Error(
code=ErrorCode.INVALID_REQUEST,
errors=["no such template: %s" % request.name],
)
base_template = TEMPLATES[request.name]
else:
base_template = index.template
template = render(request, base_template)
if isinstance(template, Error):
return template
try:
for task_config in template.tasks:
check_config(task_config)
if task_config.pool is None:
return Error(
code=ErrorCode.INVALID_REQUEST, errors=["pool not defined"]
)
except TaskConfigError as err:
return Error(code=ErrorCode.INVALID_REQUEST, errors=[str(err)])
for notification_config in template.notifications:
for task_container in request.containers:
if task_container.type == notification_config.container_type:
notification = Notification.create(
task_container.name, notification_config.notification.config
)
if isinstance(notification, Error):
return notification
job = Job(config=template.job)
job.save()
tasks: List[Task] = []
for task_config in template.tasks:
task_config.job_id = job.job_id
if task_config.prereq_tasks:
# pydantic verifies prereq_tasks in u128 form are index refs to
# previously generated tasks
task_config.prereq_tasks = [
tasks[x.int].task_id for x in task_config.prereq_tasks
]
task = Task.create(
config=task_config, job_id=job.job_id, user_info=user_info
)
if isinstance(task, Error):
return task
tasks.append(task)
return job

5
src/api-service/__app__/onefuzzlib/notifications/github_issues.py
View File

@ -1,3 +1,8 @@
#!/usr/bin/env python
#
# Copyright (c) Microsoft Corporation.
# Licensed under the MIT License.
import logging
from typing import List, Optional

31
src/api-service/__app__/onefuzzlib/notifications/main.py
View File

@ -4,7 +4,7 @@
# Licensed under the MIT License.
import logging
from typing import Dict, List, Optional, Sequence, Tuple, Union
from typing import Dict, List, Optional, Sequence, Tuple
from uuid import UUID
from memoization import cached
@ -15,11 +15,16 @@ from onefuzztypes.models import (
Error,
GithubIssueTemplate,
NotificationTemplate,
Result,
TeamsTemplate,
)
from onefuzztypes.primitives import Container, Event
from ..azure.containers import get_container_metadata, get_file_sas_url
from ..azure.containers import (
container_exists,
get_container_metadata,
get_file_sas_url,
)
from ..azure.creds import get_fuzz_storage
from ..azure.queue import send_message
from ..dashboard import add_event
@ -34,7 +39,7 @@ from .teams import notify_teams
class Notification(models.Notification, ORMMixin):
@classmethod
def get_by_id(cls, notification_id: UUID) -> Union[Error, "Notification"]:
def get_by_id(cls, notification_id: UUID) -> Result["Notification"]:
notifications = cls.search(query={"notification_id": [notification_id]})
if not notifications:
return Error(
@ -63,6 +68,26 @@ class Notification(models.Notification, ORMMixin):
def key_fields(cls) -> Tuple[str, str]:
return ("notification_id", "container")
@classmethod
def create(
cls, container: Container, config: NotificationTemplate
) -> Result["Notification"]:
if not container_exists(container):
return Error(code=ErrorCode.INVALID_REQUEST, errors=["invalid container"])
existing = cls.get_existing(container, config)
if existing is not None:
return existing
entry = cls(container=container, config=config)
entry.save()
logging.info(
"created notification. notification_id:%s container:%s",
entry.notification_id,
entry.container,
)
return entry
@cached(ttl=10)
def get_notifications(container: Container) -> List[Notification]:

4
src/api-service/__app__/onefuzzlib/orm.py
View File

@ -309,7 +309,9 @@ class ORMMixin(ModelMixin):
try:
self.etag = client.insert_entity(self.table_name(), raw)
except AzureConflictHttpError:
return Error(code=ErrorCode.UNABLE_TO_CREATE, errors=["row exists"])
return Error(
code=ErrorCode.UNABLE_TO_CREATE, errors=["entry already exists"]
)
elif self.etag and require_etag:
self.etag = client.replace_entity(
self.table_name(), raw, if_match=self.etag

8
src/api-service/__app__/onefuzzlib/tasks/main.py
View File

@ -64,6 +64,14 @@ class Task(BASE_TASK, ORMMixin):
user_info=user_info,
)
)
logging.info(
"created task. job_id:%s task_id:%s type:%s user:%s",
task.job_id,
task.task_id,
task.config.task.type.name,
user_info,
)
return task
def save_exclude(self) -> Optional[MappingIntStrAny]:

5
src/api-service/__app__/onefuzzlib/webhooks.py
View File

@ -1,3 +1,8 @@
#!/usr/bin/env python
#
# Copyright (c) Microsoft Corporation.
# Licensed under the MIT License.
import datetime
import hmac
import logging

1
src/api-service/__app__/requirements.txt
View File

@ -33,5 +33,6 @@ requests~=2.24.0
memoization~=0.3.1
github3.py~=1.3.0
typing-extensions~=3.7.4.3
jsonpatch==1.26
# onefuzz types version is set during build
onefuzztypes==0.0.0

15
src/api-service/dev-deploy.sh
View File

@ -6,8 +6,8 @@
APP_DIR=$(dirname $0)
if [ "$#" -ne 1 ]; then
echo "usage: $0 <TARGET>"
if [ "$#" -lt 1 ]; then
echo "usage: $0 <TARGET> [<VERSION>]"
exit 1
fi
@ -15,8 +15,10 @@ set -ex
TARGET=${1}
pushd ${APP_DIR}
VERSION=$(../ci/get-version.sh)
../ci/set-versions.sh
VERSION=${2:-$(../ci/get-version.sh)}
../ci/set-versions.sh $VERSION
# clean up any previously built onefuzztypes packages
rm -f __app__/onefuzztypes*.whl
@ -25,12 +27,15 @@ rm -f __app__/onefuzztypes*.whl
rm -rf local-pytypes
cp -r ../pytypes local-pytypes
pushd local-pytypes
rm -f dist/*
rm -rf dist build
python setup.py sdist bdist_wheel
cp dist/*.whl ../__app__
popd
rm -r local-pytypes
# deploy a the instance with the locally built onefuzztypes
pushd __app__
uuidgen > onefuzzlib/build.id

3
src/api-service/mypy.ini
View File

@ -34,3 +34,6 @@ ignore_missing_imports = True
[mypy-github3.*]
ignore_missing_imports = True
[mypy-jsonpatch.*]
ignore_missing_imports = True

2
src/ci/get-version.sh
View File

@ -23,6 +23,6 @@ else
if $(git diff --quiet); then
echo ${BASE_VERSION}-${GIT_HASH}
else
echo ${BASE_VERSION}-${GIT_HASH}.localchanges
echo ${BASE_VERSION}-${GIT_HASH}localchanges
fi
fi

3
src/cli/mypy.ini
View File

@ -43,3 +43,6 @@ ignore_missing_imports = True
[mypy-docstring_parser.*]
ignore_missing_imports = True
[mypy-jsonpatch.*]
ignore_missing_imports = True

15
src/cli/onefuzz/api.py
View File

@ -46,7 +46,7 @@ UUID_RE = r"^[a-f0-9]{8}-?[a-f0-9]{4}-?[a-f0-9]{4}-?[a-f0-9]{4}-?[a-f0-9]{12}\Z"
class PreviewFeature(Enum):
pass
job_templates = "job_templates"
def is_uuid(value: str) -> bool:
@ -1446,16 +1446,24 @@ class Onefuzz:
self.nodes = Node(self)
self.webhooks = Webhooks(self)
if self._backend.is_feature_enabled(PreviewFeature.job_templates.name):
self.job_templates = JobTemplates(self)
# these are externally developed cli modules
self.template = Template(self, self.logger)
self.debug = Debug(self, self.logger)
self.status = Status(self, self.logger)
self.utils = Utils(self, self.logger)
self.__setup__()
def __setup__(self, endpoint: Optional[str] = None) -> None:
if endpoint:
self._backend.config.endpoint = endpoint
if self._backend.is_feature_enabled(PreviewFeature.job_templates.name):
self.job_templates._load_cache()
def licenses(self) -> object:
""" Return third-party licenses used by this package """
stream = pkg_resources.resource_stream(__name__, "data/licenses.json")
@ -1473,6 +1481,10 @@ class Onefuzz:
# Rather than interacting MSAL directly, call a simple API which
# actuates the login process
self.info.get()
# TODO: once job templates are out of preview, this should be enabled
if self._backend.is_feature_enabled(PreviewFeature.job_templates.name):
self.job_templates.refresh()
return "succeeded"
def config(
@ -1671,5 +1683,6 @@ class Onefuzz:
from .debug import Debug # noqa: E402
from .job_templates.main import JobTemplates # noqa: E402
from .status.cmd import Status # noqa: E402
from .template import Template # noqa: E402

5
src/cli/onefuzz/backend.py
View File

@ -39,8 +39,9 @@ from tenacity.wait import wait_random
_ACCESSTOKENCACHE_UMASK = 0o077
DEFAULT_CONFIG_PATH = os.path.join("~", ".cache", "onefuzz", "config.json")
DEFAULT_TOKEN_PATH = os.path.join("~", ".cache", "onefuzz", "access_token.json")
ONEFUZZ_BASE_PATH = os.path.join("~", ".cache", "onefuzz")
DEFAULT_CONFIG_PATH = os.path.join(ONEFUZZ_BASE_PATH, "config.json")
DEFAULT_TOKEN_PATH = os.path.join(ONEFUZZ_BASE_PATH, "access_token.json")
LOGGER = logging.getLogger("nsv-backend")

44
src/cli/onefuzz/cli.py
View File

@ -72,20 +72,6 @@ def call_func(func: Callable, args: argparse.Namespace) -> Any:
return func(**myargs)
class AsDict(argparse.Action):
def __call__(
self,
parser: argparse.ArgumentParser, # noqa: F841 - unused args required by argparse
namespace: argparse.Namespace,
values: Union[str, Sequence[Any], None],
option_string: str = None, # noqa: F841 - unused args required by argparse
) -> None:
if values is None:
return
as_dict: Dict[str, str] = {k: v for k, v in (x.split("=", 1) for x in values)}
setattr(namespace, self.dest, as_dict)
def arg_bool(arg: str) -> bool:
acceptable = ["true", "false"]
if arg not in acceptable:
@ -152,7 +138,7 @@ def add_base(parser: argparse.ArgumentParser) -> None:
def enum_help(entry: Type[Enum]) -> str:
return "accepted %s: %s" % (entry.__name__, ", ".join(entry.__members__))
return "accepted %s: %s" % (entry.__name__, ", ".join([x.name for x in entry]))
def tuple_help(entry: Any) -> str:
@ -160,7 +146,7 @@ def tuple_help(entry: Any) -> str:
for item in entry:
if inspect.isclass(item) and issubclass(item, Enum):
doc.append(
"accepted %s: %s." % (item.__name__, ", ".join(item.__members__))
"accepted %s: %s." % (item.__name__, ", ".join([x.name for x in item]))
)
return " ".join(doc)
@ -174,7 +160,6 @@ class Builder:
Container: {"type": str},
File: {"type": arg_file},
Directory: {"type": arg_dir},
Dict[str, str]: {"action": AsDict, "nargs": "+", "metavar": "key=val"},
}
self.api_types = tuple(api_types)
self.top_level = argparse.ArgumentParser(add_help=False)
@ -303,6 +288,27 @@ class Builder:
return None
def build_dict_parser(self, annotation: Any) -> Dict[str, Any]:
(key_arg, val_arg) = get_arg(annotation)
class AsDictCustom(argparse.Action):
def __call__(
self,
_parser: argparse.ArgumentParser, # noqa: F841 - unused args required by argparse
namespace: argparse.Namespace,
values: Union[str, Sequence[Any], None],
option_string: str = None, # noqa: F841 - unused args required by argparse
) -> None:
if values is None:
return
as_dict: Dict[str, str] = {
key_arg(k): val_arg(v) for k, v in (x.split("=", 1) for x in values)
}
setattr(namespace, self.dest, as_dict)
metavar = "%s=%s" % (key_arg.__name__, val_arg.__name__)
return {"action": AsDictCustom, "nargs": "+", "metavar": metavar}
def parse_annotation(
self,
name: str,
@ -333,6 +339,10 @@ class Builder:
result["nargs"] = "*"
return result
if is_a(annotation, (dict, Dict)):
result.update(self.build_dict_parser(annotation))
return result
if is_a(annotation, (tuple, Tuple)):
types = get_arg(annotation)

0
src/cli/onefuzz/job_templates/__init__.py Normal file
View File

132
src/cli/onefuzz/job_templates/builder.py Normal file
View File

@ -0,0 +1,132 @@
#!/usr/bin/env python
#
# Copyright (c) Microsoft Corporation.
# Licensed under the MIT License.
import logging
from inspect import Parameter, signature
from typing import Any, Dict, List, Optional
from onefuzztypes.enums import ContainerType, UserFieldType
from onefuzztypes.job_templates import JobTemplateConfig, JobTemplateRequestParameters
from onefuzztypes.models import Job
from onefuzztypes.primitives import Directory, File
from ..api import PreviewFeature
from .handlers import TemplateSubmitHandler
LOGGER = logging.getLogger("job-templates")
TYPES = {
UserFieldType.Str: str,
UserFieldType.Int: int,
UserFieldType.ListStr: List[str],
UserFieldType.DictStr: Dict[str, str],
UserFieldType.Bool: bool,
}
NAMES = {
UserFieldType.Str: "str",
UserFieldType.Int: "int",
UserFieldType.ListStr: "list",
UserFieldType.DictStr: "dict",
UserFieldType.Bool: "bool",
}
def container_type_name(container_type: ContainerType) -> str:
return container_type.name + "_dir"
def config_to_params(config: JobTemplateConfig) -> List[Parameter]:
params: List[Parameter] = []
for entry in config.user_fields:
default = entry.default if entry.default is not None else Parameter.empty
annotation: Any = None
if entry.name == "target_exe":
annotation = File
else:
annotation = TYPES[entry.type]
is_optional = entry.default is None and entry.required is False
if is_optional:
annotation = Optional[annotation]
param = Parameter(
entry.name,
Parameter.KEYWORD_ONLY,
annotation=annotation,
default=default,
)
params.append(param)
for container in config.containers:
if container not in ContainerType.user_config():
continue
param = Parameter(
container_type_name(container),
Parameter.KEYWORD_ONLY,
annotation=Optional[Directory],
default=Parameter.empty,
)
params.append(param)
if config.containers:
param = Parameter(
"container_names",
Parameter.KEYWORD_ONLY,
annotation=Optional[Dict[ContainerType, str]],
default=None,
)
params.append(param)
param = Parameter(
"parameters",
Parameter.KEYWORD_ONLY,
annotation=Optional[JobTemplateRequestParameters],
default=Parameter.empty,
)
params.append(param)
return params
def build_template_doc(config: JobTemplateConfig) -> str:
docs = [
f"Launch '{config.name}' job",
"",
]
for entry in config.user_fields:
line = f":param {NAMES[entry.type]} {entry.name}: {entry.help}"
docs.append(line)
for container in config.containers:
if container not in ContainerType.user_config():
continue
line = f":param Directory {container_type_name(container)}: Local path to the {container.name} directory"
docs.append(line)
if config.containers:
line = ":param dict container_names: custom container names (eg: setup=my-setup-container)"
docs.append(line)
return "\n".join(docs)
def build_template_func(config: JobTemplateConfig) -> Any:
def func(self: TemplateSubmitHandler, **kwargs: Any) -> Job:
self.onefuzz._warn_preview(PreviewFeature.job_templates)
return self._execute(config, kwargs)
sig = signature(func)
params = [sig.parameters["self"]] + config_to_params(config)
sig = sig.replace(parameters=tuple(params))
func.__signature__ = sig # type: ignore
func.__doc__ = build_template_doc(config)
return func

61
src/cli/onefuzz/job_templates/cache.py Normal file
View File

@ -0,0 +1,61 @@
#!/usr/bin/env python
#
# Copyright (c) Microsoft Corporation.
# Licensed under the MIT License.
import datetime
import json
import logging
import os
from typing import Dict, List, Optional
from onefuzztypes.job_templates import JobTemplateConfig
from pydantic import BaseModel, Field
from ..backend import ONEFUZZ_BASE_PATH
TEMPLATE_CACHE = os.path.expanduser(os.path.join(ONEFUZZ_BASE_PATH, "templates.json"))
class EndpointCache(BaseModel):
timestamp: datetime.datetime
configs: List[JobTemplateConfig]
class CachedTemplates(BaseModel):
entries: Dict[str, EndpointCache] = Field(default_factory=dict)
@classmethod
def add(cls, endpoint: str, configs: List[JobTemplateConfig]) -> None:
cache = cls.load()
cache.entries[endpoint] = EndpointCache(
timestamp=datetime.datetime.utcnow(), configs=configs
)
cache.save()
@classmethod
def get(cls, endpoint: str) -> Optional[EndpointCache]:
cache = cls.load()
return cache.entries.get(endpoint)
@classmethod
def load(cls) -> "CachedTemplates":
if not os.path.exists(TEMPLATE_CACHE):
entry = cls()
entry.save()
return entry
try:
with open(TEMPLATE_CACHE, "r") as handle:
raw = json.load(handle)
return cls.parse_obj(raw)
except Exception as err:
logging.warning("unable to load template cache: %s", err)
entry = cls()
entry.save()
return entry
def save(self) -> None:
with open(TEMPLATE_CACHE, "w") as handle:
handle.write(self.json())

156
src/cli/onefuzz/job_templates/handlers.py Normal file
View File

@ -0,0 +1,156 @@
#!/usr/bin/env python
#
# Copyright (c) Microsoft Corporation.
# Licensed under the MIT License.
import os
from typing import Any, Dict, List
from onefuzztypes.enums import ContainerType
from onefuzztypes.job_templates import JobTemplateConfig, JobTemplateRequest
from onefuzztypes.models import Job, TaskContainers
from ..api import Endpoint
from ..templates import _build_container_name
def container_type_name(container_type: ContainerType) -> str:
return container_type.name + "_dir"
class TemplateSubmitHandler(Endpoint):
""" Submit Job Template """
_endpoint = "job_templates"
def _process_containers(
self, request: JobTemplateRequest, args: Dict[str, Any]
) -> None:
""" Create containers based on the argparse args """
for container in request.containers:
directory_arg = container_type_name(container.type)
self.onefuzz.logger.info("creating container: %s", container.name)
self.onefuzz.containers.create(
container.name, metadata={"container_type": container.type.name}
)
if directory_arg in args and args[directory_arg] is not None:
self.onefuzz.logger.info(
"uploading %s to %s", args[directory_arg], container.name
)
self.onefuzz.containers.files.upload_dir(
container.name, args[directory_arg]
)
elif container.type == ContainerType.setup and "target_exe" in args:
# This is isn't "declarative", but models our existing paths for
# templates.
target_exe = args["target_exe"]
if target_exe is None:
continue
self.onefuzz.logger.info(
"uploading %s to %s", target_exe, container.name
)
self.onefuzz.containers.files.upload_file(container.name, target_exe)
pdb_path = os.path.splitext(target_exe)[0] + ".pdb"
if os.path.exists(pdb_path):
self.onefuzz.containers.files.upload_file(container.name, pdb_path)
def _define_missing_containers(
self, config: JobTemplateConfig, request: JobTemplateRequest
) -> None:
for container_type in config.containers:
seen = False
for container in request.containers:
if container_type == container.type:
seen = True
if not seen:
assert isinstance(request.user_fields["project"], str)
assert isinstance(request.user_fields["name"], str)
assert isinstance(request.user_fields["build"], str)
container_name = _build_container_name(
self.onefuzz,
container_type,
request.user_fields["project"],
request.user_fields["name"],
request.user_fields["build"],
config.os,
)
request.containers.append(
TaskContainers(name=container_name, type=container_type)
)
def _submit(self, request: JobTemplateRequest) -> Job:
self.onefuzz.logger.debug("submitting request: %s", request)
return self._req_model(
"POST", Job, data=request, alternate_endpoint=self._endpoint
)
def _execute_request(
self,
config: JobTemplateConfig,
request: JobTemplateRequest,
args: Dict[str, Any],
) -> Job:
self._define_missing_containers(config, request)
self._process_containers(request, args)
return self._submit(request)
def _convert_container_args(
self, config: JobTemplateConfig, args: Dict[str, Any]
) -> List[TaskContainers]:
""" Convert the job template into a list of containers """
containers = []
container_names = args["container_names"]
if container_names is None:
container_names = {}
for container_type in config.containers:
if container_type in container_names:
container_name = container_names[container_type]
containers.append(
TaskContainers(name=container_name, type=container_type)
)
return containers
def _convert_args(
self, config: JobTemplateConfig, args: Dict[str, Any]
) -> JobTemplateRequest:
""" convert arguments from argparse into a JobTemplateRequest """
user_fields = {}
for field in config.user_fields:
value = None
if field.name in args:
value = args[field.name]
elif field.name in args["parameters"]:
value = args["parameters"][field.name]
elif field.required:
raise Exception("missing field: %s" % field.name)
if field.name == "target_exe" and isinstance(value, str):
value = os.path.basename(value)
if value is not None:
user_fields[field.name] = value
containers = self._convert_container_args(config, args)
request = JobTemplateRequest(
name=config.name, user_fields=user_fields, containers=containers
)
return request
def _execute(
self,
config: JobTemplateConfig,
args: Dict[str, Any],
) -> Job:
""" Convert argparse args into a JobTemplateRequest and submit it """
self.onefuzz.logger.debug("building: %s", config.name)
request = self._convert_args(config, args)
return self._execute_request(config, request, args)

108
src/cli/onefuzz/job_templates/main.py Normal file
View File

@ -0,0 +1,108 @@
#!/usr/bin/env python
#
# Copyright (c) Microsoft Corporation.
# Licensed under the MIT License.
import datetime
import logging
from typing import List, Optional
from onefuzztypes.job_templates import JobTemplateConfig
from ..api import Endpoint, Onefuzz, PreviewFeature
from .builder import build_template_func
from .cache import CachedTemplates
from .handlers import TemplateSubmitHandler
from .manage import Manage
LOGGER = logging.getLogger("job-templates")
def load_templates(templates: List[JobTemplateConfig]) -> None:
handlers = {
TemplateSubmitHandler: build_template_func,
}
for handler in handlers:
for name in dir(handler):
if name.startswith("_"):
continue
delattr(handler, name)
for template in templates:
setattr(handler, template.name, handlers[handler](template))
class JobTemplates(Endpoint):
""" Job Templates """
endpoint = "job_templates"
def __init__(self, onefuzz: Onefuzz):
super().__init__(onefuzz)
self.manage = Manage(onefuzz)
self.submit = TemplateSubmitHandler(onefuzz)
def info(self, name: str) -> Optional[JobTemplateConfig]:
""" Display information for a Job Template """
self.onefuzz._warn_preview(PreviewFeature.job_templates)
endpoint = self.onefuzz._backend.config.endpoint
if endpoint is None:
return None
entry = CachedTemplates.get(endpoint)
if entry is None:
return None
for config in entry.configs:
if config.name == name:
return config
return None
def list(self) -> Optional[List[str]]:
""" List available Job Templates """
self.onefuzz._warn_preview(PreviewFeature.job_templates)
endpoint = self.onefuzz._backend.config.endpoint
if endpoint is None:
return None
entry = CachedTemplates.get(endpoint)
if entry is None:
return None
return [x.name for x in entry.configs]
def _load_cache(self) -> None:
endpoint = self.onefuzz._backend.config.endpoint
if endpoint is None:
return
yesterday = datetime.datetime.utcnow() - datetime.timedelta(hours=24)
entry = CachedTemplates.get(endpoint)
if not entry or entry.timestamp < yesterday:
self.refresh()
return
load_templates(entry.configs)
def refresh(self) -> None:
""" Update available templates """
self.onefuzz._warn_preview(PreviewFeature.job_templates)
self.onefuzz.logger.info("refreshing job template cache")
endpoint = self.onefuzz._backend.config.endpoint
if endpoint is None:
return None
templates = self._req_model_list("GET", JobTemplateConfig)
for template in templates:
self.onefuzz.logger.info("updated template definition: %s", template.name)
CachedTemplates.add(endpoint, templates)
load_templates(templates)

63
src/cli/onefuzz/job_templates/manage.py Normal file
View File

@ -0,0 +1,63 @@
#!/usr/bin/env python
#
# Copyright (c) Microsoft Corporation.
# Licensed under the MIT License.
from typing import List
from onefuzztypes.job_templates import (
JobTemplate,
JobTemplateCreate,
JobTemplateDelete,
JobTemplateIndex,
JobTemplateUpdate,
)
from onefuzztypes.responses import BoolResult
from ..api import Endpoint, PreviewFeature
class Manage(Endpoint):
""" Manage Job Templates """
endpoint = "job_templates/manage"
def list(self) -> List[JobTemplateIndex]:
""" List templates """
self.onefuzz._warn_preview(PreviewFeature.job_templates)
self.onefuzz.logger.debug("listing job templates")
return self._req_model_list("GET", JobTemplateIndex)
def create(self, domain: str, name: str, template: JobTemplate) -> BoolResult:
""" Create a Job Template """
self.onefuzz._warn_preview(PreviewFeature.job_templates)
self.onefuzz.logger.debug("creating job templates")
return self._req_model(
"POST",
BoolResult,
data=JobTemplateCreate(domain=domain, name=name, template=template),
)
def update(self, domain: str, name: str, template: JobTemplate) -> BoolResult:
""" Update an existing Job Template """
self.onefuzz._warn_preview(PreviewFeature.job_templates)
self.onefuzz.logger.debug("update job templates")
return self._req_model(
"POST",
BoolResult,
data=JobTemplateUpdate(domain=domain, name=name, template=template),
)
def delete(self, domain: str, name: str) -> BoolResult:
""" Delete a Job Template """
self.onefuzz._warn_preview(PreviewFeature.job_templates)
self.onefuzz.logger.debug("delete job templates")
return self._req_model(
"DELETE",
BoolResult,
data=JobTemplateDelete(domain=domain, name=name),
)

46
src/cli/onefuzz/templates/__init__.py
View File

@ -24,6 +24,33 @@ class StoppedEarly(Exception):
pass
def _build_container_name(
onefuzz: "Onefuzz",
container_type: ContainerType,
project: str,
name: str,
build: str,
platform: OS,
) -> Container:
if container_type == ContainerType.setup:
guid = onefuzz.utils.namespaced_guid(
project,
name,
build=build,
platform=platform.name,
)
else:
guid = onefuzz.utils.namespaced_guid(project, name)
return Container(
"oft-%s-%s"
% (
container_type.name.replace("_", "-"),
guid.hex,
)
)
class JobHelper:
def __init__(
self,
@ -80,22 +107,13 @@ class JobHelper:
"""
for container_type in types:
if container_type == ContainerType.setup:
guid = self.onefuzz.utils.namespaced_guid(
self.containers[container_type] = _build_container_name(
self.onefuzz,
container_type,
self.project,
self.name,
build=self.build,
platform=self.platform.name,
)
else:
guid = self.onefuzz.utils.namespaced_guid(self.project, self.name)
self.containers[container_type] = Container(
"oft-%s-%s"
% (
container_type.name.replace("_", "-"),
guid.hex,
)
self.build,
self.platform,
)
def create_containers(self) -> None:

3
src/proxy-manager/build.rs
View File

@ -1,3 +1,6 @@
// Copyright (c) Microsoft Corporation.
// Licensed under the MIT License.
use std::env;
use std::error::Error;
use std::fs::File;

18
src/pytypes/onefuzztypes/enums.py
View File

@ -217,6 +217,10 @@ class ContainerType(Enum):
cls.unique_inputs,
]
@classmethod
def user_config(cls) -> List["ContainerType"]:
return [cls.setup, cls.inputs, cls.readonly_inputs]
class StatsFormat(Enum):
AFL = "AFL"
@ -244,6 +248,7 @@ class ErrorCode(Enum):
TASK_FAILED = 468
INVALID_NODE = 469
NOTIFICATION_FAILURE = 470
UNABLE_TO_UPDATE = 471
class HeartbeatType(Enum):
@ -372,3 +377,16 @@ class WebhookMessageState(Enum):
retrying = "retrying"
succeeded = "succeeded"
failed = "failed"
class UserFieldOperation(Enum):
add = "add"
replace = "replace"
class UserFieldType(Enum):
Bool = "Bool"
Int = "Int"
Str = "Str"
DictStr = "DictStr"
ListStr = "ListStr"

185
src/pytypes/onefuzztypes/job_templates.py Normal file
View File

@ -0,0 +1,185 @@
#!/usr/bin/env python
#
# Copyright (c) Microsoft Corporation.
# Licensed under the MIT License.
from typing import Dict, List, Optional, Union
from pydantic import BaseModel, Field, root_validator, validator
from .enums import OS, ContainerType, UserFieldOperation, UserFieldType
from .models import JobConfig, NotificationConfig, TaskConfig, TaskContainers
from .primitives import File
from .requests import BaseRequest
from .responses import BaseResponse
from .validators import check_template_name
class UserFieldLocation(BaseModel):
op: UserFieldOperation
path: str
TemplateUserData = Union[bool, int, str, Dict[str, str], List[str], File]
TemplateUserFields = Dict[str, TemplateUserData]
class UserField(BaseModel):
name: str
type: UserFieldType
locations: List[UserFieldLocation]
required: bool = Field(default=False)
default: Optional[TemplateUserData]
help: str
@validator("locations", allow_reuse=True)
def check_locations(cls, value: List) -> List:
if len(value) == 0:
raise ValueError("must provide at least one location")
return value
class JobTemplateNotification(BaseModel):
container_type: ContainerType
notification: NotificationConfig
class JobTemplate(BaseModel):
os: OS
job: JobConfig
tasks: List[TaskConfig]
notifications: List[JobTemplateNotification]
user_fields: List[UserField]
@root_validator()
def check_task_prereqs(cls, data: Dict) -> Dict:
for idx, task in enumerate(data["tasks"]):
# prereq_tasks must refer to previously defined tasks, using the u128
# representation of the UUID as an index
if task.prereq_tasks:
for prereq in task.prereq_tasks:
if prereq.int >= idx:
raise Exception(f"invalid task reference: {idx} - {prereq}")
return data
@root_validator()
def check_fields(cls, data: Dict) -> Dict:
seen = set()
seen_path = set()
for entry in TEMPLATE_BASE_FIELDS + data["user_fields"]:
# field names, which are sent to the user for filling out, must be
# specified once and only once
if entry.name in seen:
raise Exception(f"duplicate field found: {entry.name}")
seen.add(entry.name)
# location.path, the location in the json doc that is modified,
# must be specified once and only once
for location in entry.locations:
if location.path in seen_path:
raise Exception(f"duplicate path found: {location.path}")
seen_path.add(location.path)
if entry.name in ["platform"]:
raise Exception(f"reserved field name: {entry.name}")
return data
class JobTemplateIndex(BaseResponse):
name: str
template: JobTemplate
_validate_name: classmethod = validator("name", allow_reuse=True)(
check_template_name
)
class JobTemplateField(BaseModel):
name: str
help: str
type: UserFieldType
required: bool
default: Optional[TemplateUserData]
class JobTemplateConfig(BaseResponse):
os: OS
name: str
user_fields: List[JobTemplateField]
containers: List[ContainerType]
TEMPLATE_BASE_FIELDS = [
UserField(
name="project",
help="Name of the Project",
type=UserFieldType.Str,
required=True,
locations=[
UserFieldLocation(
op=UserFieldOperation.replace,
path="/job/project",
),
],
),
UserField(
name="name",
help="Name of the Target",
type=UserFieldType.Str,
required=True,
locations=[
UserFieldLocation(
op=UserFieldOperation.replace,
path="/job/name",
),
],
),
UserField(
name="build",
help="Name of the Target",
type=UserFieldType.Str,
required=True,
locations=[
UserFieldLocation(
op=UserFieldOperation.replace,
path="/job/build",
),
],
),
]
class JobTemplateCreate(BaseRequest):
name: str
template: JobTemplate
_verify_name: classmethod = validator("name", allow_reuse=True)(check_template_name)
class JobTemplateDelete(BaseRequest):
name: str
_verify_name: classmethod = validator("name", allow_reuse=True)(check_template_name)
class JobTemplateUpdate(BaseRequest):
name: str
template: JobTemplate
_verify_name: classmethod = validator("name", allow_reuse=True)(check_template_name)
class JobTemplateRequest(BaseRequest):
name: str
user_fields: TemplateUserFields
containers: List[TaskContainers]
_validate_name: classmethod = validator("name", allow_reuse=True)(
check_template_name
)
class JobTemplateRequestParameters(BaseRequest):
user_fields: TemplateUserFields

30
src/pytypes/onefuzztypes/validators.py
View File

@ -5,16 +5,34 @@
from string import ascii_letters, digits
ALPHA_NUM = ascii_letters + digits
ALPHA_NUM_DASH = ALPHA_NUM + "-"
ALPHA_NUM_UNDERSCORE = ALPHA_NUM + "_"
def check_alnum_dash(value: str) -> str:
accepted = ascii_letters + digits + "-"
if not all(x in accepted for x in value):
def check_value(value: str, charset: str) -> str:
if not all(x in charset for x in value):
raise ValueError("invalid value: %s" % value)
return value
def check_alnum(value: str) -> str:
accepted = ascii_letters + digits
if not all(x in accepted for x in value):
return check_value(value, ALPHA_NUM)
def check_alnum_dash(value: str) -> str:
return check_value(value, ALPHA_NUM_DASH)
def check_alnum_underscore(value: str) -> str:
return check_value(value, ALPHA_NUM_UNDERSCORE)
def check_template_name(value: str) -> str:
if not value:
raise ValueError("invalid value: %s" % value)
return value
if value[0] not in ALPHA_NUM:
raise ValueError("invalid value: %s" % value)
return check_alnum_underscore(value)

5
src/pytypes/onefuzztypes/webhooks.py
View File

@ -1,3 +1,8 @@
#!/usr/bin/env python
#
# Copyright (c) Microsoft Corporation.
# Licensed under the MIT License.
from typing import List, Optional, Union
from uuid import UUID, uuid4