onefuzz/docs/deprecated/declarative-templates.md

Declarative Job Templates -- Deprecated

Provide the ability to maintain job templates, akin to onefuzz template libfuzzer basic at the service level. The templates include a job definition, an arbitrary set of tasks, and an arbitrary set of notification configs. The templates are managed at the service level, with job-submission time updates using a declarative syntax based on jsonpatch.

The SDK makes use of template configs, provided by the service, to dynamically build Python methods for each template. This process enables the automatic argument parser generated by type signatures to automatically create the CLI subcommands for the template.

User Experience

• On onefuzz login (or onefuzz job_templates refresh), cache the existing set of templates
• Users can see the supported templates via onefuzz job_templates submit --help
• Users can submit jobs via onefuzz job_templates submit libfuzzer OSNAME project name build pool --target_exe fuzz.exe
• Template configs are refreshed automatically if they are older than 24 hours.

Future work:

• submitting jobs by config. Not everything is easy to express via argparse, such as values that begin with -. In order to support this, it should be trivial to expose a method that takes a json file and submits it.

Admin Experience

Administrators can manage their own templates via onefuzz job_templates manage [list,get,update,delete].

If the runtime configuration for the template changes, users will need to refresh their cache to pull the runtime configuration.

Implementation Details

A declarative job template includes:

• a Job (JobConfig, as used by onefuzz jobs create)
• a list of Tasks (TaskConfig, which is used by onefuzz tasks create)
• a list of Notifications (NotificationConfig + container type, akin to what is used by onefuzz notifications create)
• a list of required and optional form fields used to update the aforementioned JobConfig, TaskConfig, and NotificationConfig entries at runtime.

The form fields allow for 'add' or 'replace' of basic field data using jsonpatch semantics.

Example Form Fields

This following field named target_workers, which is required to be an int, will optionally (if the request includes it) replace the target_workers value of in the first task in the template.

UserField(
    name="target_workers",
    required=False,
    type=UserFieldType.Int,
    help="The number of workers to use for this task",
    locations=[
        UserFieldLocation(
            op=UserFieldOperation.replace,
            path="/tasks/0/task/target_workers",
        ),
    ],
)

Allowed Data Types

The data types allowed in configuring arbitrary components in the JobTemplate are:

• bool
• int
• str
• Dict[str, str]
• List[str]

Referring to Tasks

In existing procedural templates, some tasks require that other tasks are running before they may be scheduled. For example, in the libfuzzer procedural template, the libfuzzer_crash_report task has a libfuzzer_fuzz task prerequisite. This is specified via the prerequisite's task_id, which is a random server-assigned UUID.

In procedural templates, the dependency task is simply created before its dependent.

To support such a reference in OnefuzzTemplate, specify the prerequisite task by the u128 representation index in to the list of tasks. Example, to refer to the first task, use:

TaskConfig(
    prereq_tasks=[UUID(int=0)],
    ...
)

Hardcoded vs Runtime-specified Container Names

To support differentiating always use "afl-linux" for tools vs ask what container to use for setup, if the container name is blank in the template, it will be provided as part of the JobTemplateConfig and in the resulting JobTemplateRequest.

Specifying Notifications in the Template

The existing templates support adding a notification config on the command line, via --notification_config, but the existing templates themselves do not include default notifications.

Declarative job templates include optional support to configure notifications as part of the template, rather than requiring the user provide the configuration.

Example declarative job template that specifies using the aforementioned NotificationConfig for the unique_reports containers used in the Job.

JobTemplateNotification(
    container_type=ContainerType.unique_reports,
    notification=NotificationConfig(config=TeamsTemplate(url="https://contoso.com/webhook-url-here")),
)

Differences from Existing Templates

• Declaratively specifying the allowed values for enums, such as StatsFormat, is not supported. Fields must currently use Str, which evaluates to Enum value during template rendering, is functional.
• Existing templates automatically differentiate between Windows and Linux tasks. This does not support differentiating between platforms automatically. As such, there is a new required parameter to specify the OS.

From the CLI

❯ onefuzz job_templates submit libfuzzer --help
usage: onefuzz job_templates submit libfuzzer [-h] [-v] [--format {json,raw}] [--query QUERY] [--target_exe TARGET_EXE]
                                              [--duration DURATION] [--target_workers TARGET_WORKERS] [--vm_count VM_COUNT]
                                              [--target_options [TARGET_OPTIONS [TARGET_OPTIONS ...]]]
                                              [--target_env str=str [str=str ...]] [--reboot_after_setup]
                                              [--check_retry_count CHECK_RETRY_COUNT] [--target_timeout TARGET_TIMEOUT]
                                              [--tags str=str [str=str ...]] [--readonly_inputs_dir READONLY_INPUTS_DIR]
                                              [--setup_dir SETUP_DIR] [--inputs_dir INPUTS_DIR]
                                              [--container_names ContainerType=str [ContainerType=str ...]]
                                              OS project name build pool_name

positional arguments:
  OS                    Specify the OS to use in the job. accepted OS: windows, linux
  project               Name of the Project
  name                  Name of the Target
  build                 Name of the Target
  pool_name             Execute the task on the specified pool

optional arguments:
  -h, --help            show this help message and exit
  -v, --verbose         increase output verbosity
  --format {json,raw}   output format
  --query QUERY         JMESPath query string. See http://jmespath.org/ for more information and examples.
  --target_exe TARGET_EXE
                        Path to the target executable (default: fuzz.exe)
  --duration DURATION   Number of hours to execute the task (default: 24)
  --target_workers TARGET_WORKERS
                        Number of instances of the libfuzzer target on each VM
  --vm_count VM_COUNT   Number of VMs to use for fuzzing (default: 2)
  --target_options [TARGET_OPTIONS [TARGET_OPTIONS ...]]
                        Command line options for the target
  --target_env str=str [str=str ...]
                        Environment variables for the target
  --reboot_after_setup  After executing the setup script, reboot the VM (Default: False. Sets value to True)
  --check_retry_count CHECK_RETRY_COUNT
                        Number of times to retry a crash to verify reproducability
  --target_timeout TARGET_TIMEOUT
                        Number of seconds to timeout during reproduction
  --tags str=str [str=str ...]
                        User provided metadata for the tasks
  --readonly_inputs_dir READONLY_INPUTS_DIR
                        Local path to the readonly_inputs directory
  --setup_dir SETUP_DIR
                        Local path to the setup directory
  --inputs_dir INPUTS_DIR
                        Local path to the inputs directory
  --container_names ContainerType=str [ContainerType=str ...]
                        custom container names (eg: setup=my-setup-container)

From the SDK

From the perspective of the SDK, this looks very similar to the existing templates. All of the arguments from the request are converted into named arguments with appropriate type signatures.

❯ python
Python 3.8.2 (default, Jul 16 2020, 14:00:26)
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from onefuzz.api import Onefuzz
>>> a = Onefuzz()
>>> help(a.job_templates.submit.libfuzzer)
Help on method func in module onefuzz.job_templates.main:

func(platform: onefuzztypes.enums.OS, *, project: str, name: str, build: str, pool_name: str, target_exe: <function NewType.<locals>.new_type at 0x7f7909338040> = 'fuzz.exe', duration: int = 24, target_workers: Union[int, NoneType], vm_count: int = 2, target_options: Union[List[str], NoneType], target_env: Union[Dict[str, str], NoneType], reboot_after_setup: bool = False, check_retry_count: Union[int, NoneType], target_timeout: Union[int, NoneType], tags: Union[Dict[str, str], NoneType], readonly_inputs_dir: Union[Directory, NoneType], setup_dir: Union[Directory, NoneType], inputs_dir: Union[Directory, NoneType], container_names: Union[Dict[onefuzztypes.enums.ContainerType, str], NoneType] = None) -> onefuzztypes.models.Job method of onefuzz.job_templates.main.TemplateHandler instance
    Launch 'libfuzzer' job

    :param Platform platform: Specify the OS to use in the job.
    :param str project: Name of the Project
    :param str name: Name of the Target
    :param str build: Name of the Target
    :param str pool_name: Execute the task on the specified pool
    :param str target_exe: Path to the target executable
    :param int duration: Number of hours to execute the task
    :param int target_workers: Number of instances of the libfuzzer target on each VM
    :param int vm_count: Number of VMs to use for fuzzing
    :param list target_options: Command line options for the target
    :param dict target_env: Environment variables for the target
    :param bool reboot_after_setup: After executing the setup script, reboot the VM
    :param int check_retry_count: Number of times to retry a crash to verify reproducability
    :param int target_timeout: Number of seconds to timeout during reproduction
    :param dict tags: User provided metadata for the tasks
    :param Directory readonly_inputs_dir: Local path to the readonly_inputs directory
    :param Directory setup_dir: Local path to the setup directory
    :param Directory inputs_dir: Local path to the inputs directory
    :param dict container_names: custom container names (eg: setup=my-setup-container)