ExternalVendorCode/onefuzz
1
0
Fork 0
mirror of https://github.com/microsoft/onefuzz.git synced 2025-06-16 11:58:09 +00:00
Code Issues Packages Projects Releases Wiki Activity

Add generic_analysis example that performs coverage analysis (#1072)

Browse Source 
This adds an example script and tool that enables LLVM source-based coverage using the `generic_analysis` task.

This provides:
1. sample python script that launches the template and then the analysis task
1. sample `analysis_exe` wrapper script that launches the LLVM coverage tools
1. sample libfuzzer target for the example
1. walk through submitting the jobs and inspecting the results
This commit is contained in:
bmc-msft
2021-07-21 13:12:24 -04:00
committed by GitHub
parent f151591322
commit 198d765cb4
9 changed files with 470 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1
docs/custom-analysis.md
View File

@ -26,3 +26,4 @@ template for any Windows targets.
See also:
* [Command Replacements](command-replacements.md)
* [Example to collect LLVM Source-Based Coverage using custom analysis](../src/cli/examples/llvm-source-coverage/README.md)

2
src/cli/examples/llvm-source-coverage/.gitignore vendored Normal file
View File

@ -0,0 +1,2 @@
setup/fuzz.exe
setup/fuzz-coverage.exe

134
src/cli/examples/llvm-source-coverage/README.md Normal file
View File

@ -0,0 +1,134 @@
# Collecting Source Coverage using Analysis Tasks
The `generic_analysis` task can be used to perform a user-defined analysis of a target executable for every test input from some storage container.
Running an application compiled with [LLVM's source-based code coverage](https://clang.llvm.org/docs/SourceBasedCodeCoverage.html) with each input can be used to generate source based coverage information.
This example demonstrates using `generic_analysis` and the LLVM source coverage tools to provide source-based coverage on every input for a job. For more information, see [Custom Analysis Tasks](../../../../docs/custom-analysis.md)
* [source-coverage-libfuzzer.py](source-coverage-libfuzzer.py): A wrapper that will launch a standard `libfuzzer basic` job *with* a source-based coverage task. (used below)
* [source-coverage.py](source-coverage.py): A wrapper that will launch a new job comprised of a source-based coverage task
* [setup](setup): a basic libFuzzer target that builds with and without source coverage enabled
* [tools/source-coverage.sh](tools/source-coverage.sh): a script that wraps `llvm-profdata` and `llvm-cov` to perform the source analysis
This example generates the following data in the `analysis` container:
* inputs/`SHA256_OF_INPUT`.profraw: the "raw" coverage data for each input analyzed
* coverage.profdata: The merged coverage data using `llvm-profdata`
* coverage.report: The `JSON` report of the merged coverage data provided by `llvm-cov export`
* coverage.lcov : The `lcov` report of the merged coverage data provided by `llvm-cov export --format lcov`
```
❯ # build our libfuzzer
❯ cd setup/
❯ ls
Makefile simple.c
❯ make
clang -g3 -fsanitize=fuzzer -fsanitize=address simple.c -o fuzz.exe
clang -g3 -fsanitize=fuzzer -fprofile-instr-generate -fcoverage-mapping simple.c -o fuzz-coverage.exe
❯ cd ..
❯ # submit our basic job with an additional analysis task
❯ ./source-coverage-libfuzzer.py setup/ setup/fuzz.exe ./setup/fuzz-coverage.exe coverage-example 1 1 linux-1 ./tools/
INFO:onefuzz:creating libfuzzer from template
INFO:onefuzz:creating job (runtime: 24 hours)
INFO:onefuzz:created job: 61bc5c7c-d24f-4ebc-9bac-bec8fe040ade
INFO:onefuzz:using container: oft-setup-d1100b49a03c5a9483f140cee0676b87
INFO:onefuzz:using container: oft-inputs-6f3b76e7e841532bb7714375f564d483
INFO:onefuzz:using container: oft-crashes-6f3b76e7e841532bb7714375f564d483
INFO:onefuzz:using container: oft-reports-6f3b76e7e841532bb7714375f564d483
INFO:onefuzz:using container: oft-unique-reports-6f3b76e7e841532bb7714375f564d483
INFO:onefuzz:using container: oft-unique-inputs-6f3b76e7e841532bb7714375f564d483
INFO:onefuzz:using container: oft-no-repro-6f3b76e7e841532bb7714375f564d483
INFO:onefuzz:using container: oft-coverage-d1100b49a03c5a9483f140cee0676b87
INFO:onefuzz:using container: oft-regression-reports-06bdcba10b5f5e45bdb38ed924856426
INFO:onefuzz:uploading setup dir `setup/`
INFO:onefuzz:creating libfuzzer_regression task
INFO:onefuzz:creating libfuzzer task
INFO:onefuzz:creating coverage task
INFO:onefuzz:creating libfuzzer_crash_report task
INFO:onefuzz:done creating tasks
INFO:onefuzz:using container: oft-setup-d1100b49a03c5a9483f140cee0676b87
INFO:onefuzz:using container: oft-analysis-6f3b76e7e841532bb7714375f564d483
INFO:onefuzz:using container: oft-inputs-6f3b76e7e841532bb7714375f564d483
INFO:onefuzz:using container: oft-tools-6f3b76e7e841532bb7714375f564d483
INFO:onefuzz:Creating generic_analysis task
job:{
"timestamp": null,
"job_id": "61bc5c7c-d24f-4ebc-9bac-bec8fe040ade",
"state": "init",
"config": {
"project": "coverage-example",
"name": "1",
"build": "1",
"duration": 24
},
"error": null,
"end_time": null,
"task_info": null,
"user_info": {
"application_id": "00000000-0000-0000-0000-000000000000",
"object_id": "00000000-0000-0000-0000-000000000000",
"upn": "example@contoso.com"
}
}
❯ # a little while later, check on the status of our job
❯ onefuzz
job: 61bc5c7c-d24f-4ebc-9bac-bec8fe040ade
project:coverage-example name:1 build:1
tasks:
50e1b076 target:fuzz-coverage.exe state:running type:generic_analysis
63880445 target:fuzz.exe state:stopped type:libfuzzer_regression
77fb6177 target:fuzz.exe state:running type:coverage
a8e3338c target:fuzz.exe state:running type:libfuzzer_crash_report
aae7ba1b target:fuzz.exe state:running type:libfuzzer_fuzz
containers:
setup count:4 name:oft-setup-d1100b49a03c5a9483f140cee0676b87
analysis count:14 name:oft-analysis-6f3b76e7e841532bb7714375f564d483
tools count:1 name:oft-tools-6f3b76e7e841532bb7714375f564d483
crashes count:11 name:oft-inputs-6f3b76e7e841532bb7714375f564d483
crashes count:4 name:oft-crashes-6f3b76e7e841532bb7714375f564d483
unique_reports count:3 name:oft-unique-reports-6f3b76e7e841532bb7714375f564d483
regression_reports count:0 name:oft-regression-reports-06bdcba10b5f5e45bdb38ed924856426
coverage count:1 name:oft-coverage-d1100b49a03c5a9483f140cee0676b87
readonly_inputs count:11 name:oft-inputs-6f3b76e7e841532bb7714375f564d483
reports count:4 name:oft-reports-6f3b76e7e841532bb7714375f564d483
no_repro count:0 name:oft-no-repro-6f3b76e7e841532bb7714375f564d483
inputs count:11 name:oft-inputs-6f3b76e7e841532bb7714375f564d483
❯ # lets check on the results of the analysis thus far
❯ onefuzz containers files list oft-analysis-6f3b76e7e841532bb7714375f564d483
{
"files": [
"coverage.lcov",
"coverage.profdata",
"coverage.report",
"inputs/06a7e66b4ddb9d43b9007e20f351c8076a2f5c5c13ec6d683e1307eeee472f7a.profraw",
"inputs/075de2b906dbd7066da008cab735bee896370154603579a50122f9b88545bd45.profraw",
"inputs/0fc4f9bfb1e6850b77e130904c0d5f8d0bfabe9a658efee7c4c41ad0015bff22.profraw",
"inputs/15dab3cc1c78958bc8c6d959cf708c2062e8327d3db873c2629b243c7e1a1759.profraw",
"inputs/3ebe1b59762a1c8020c1efe3747dd07f0e30617ed60b4e6a5bee16b6ea421dd0.profraw",
"inputs/594e519ae499312b29433b7dd8a97ff068defcba9755b6d5d00e84c524d67b06.profraw",
"inputs/75558b9c2275acb05f57066ce1199be864c7affffece0b952edac02e785bbc9f.profraw",
"inputs/bc9b8634ef85180578a9b501c901ce394ccd9087096fa4f298e4fc3752e60804.profraw",
"inputs/c6b27b6743b120d83d5cc1d37b0f51acddcb69ff544763e7552efb7b575bac38.profraw",
"inputs/c8bc644c4ddaaeafdb76142b72577e1f923b6797d87d254025f2fdf2b8225540.profraw",
"inputs/e5e1b99e66064d2e9414a37158465eb4fdc1a8120b9fa8e10e9301b5fc25bc98.profraw"
]
}
❯ # this parses the report and checks that it's an coverage json report as we expect
❯ 1f containers files get oft-analysis-6f3b76e7e841532bb7714375f564d483 coverage.report | jq .type
"llvm.coverage.json.export"
❯ # now let's inspect the merged lcov file
❯ 1f containers files get oft-analysis-6f3b76e7e841532bb7714375f564d483 coverage.lcov |head -n 10
SF:/home/USERNAME/onefuzz/src/cli/examples/llvm-source-coverage/setup/simple.c
FN:8,LLVMFuzzerTestOneInput
FNDA:6,LLVMFuzzerTestOneInput
FNF:1
FNH:1
DA:8,6
DA:9,6
DA:10,6
DA:11,6
DA:12,1
❯
```

1
src/cli/examples/llvm-source-coverage/inputs/input.txt Normal file
View File

@ -0,0 +1 @@
0000

20
src/cli/examples/llvm-source-coverage/setup/Makefile Normal file
View File

@ -0,0 +1,20 @@
CC=clang
CFLAGS=-g3 -fsanitize=fuzzer -fsanitize=address
CFLAGS_COV=-g3 -fsanitize=fuzzer -fprofile-instr-generate -fcoverage-mapping
all: fuzz.exe fuzz-coverage.exe
fuzz.exe: simple.c
$(CC) $(CFLAGS) $< -o $@
fuzz-coverage.exe: simple.c
$(CC) $(CFLAGS_COV) $< -o $@
test: fuzz.exe
./fuzz.exe -runs=100 ../inputs
.PHONY: clean
clean:
rm -f fuzz.exe fuzz-coverage.exe

69
src/cli/examples/llvm-source-coverage/setup/simple.c Normal file
View File

@ -0,0 +1,69 @@
// Copyright (c) Microsoft Corporation. All rights reserved.
// Licensed under the MIT License.
#include <stdint.h>
#include <stdlib.h>
int LLVMFuzzerTestOneInput(const uint8_t *data, size_t len) {
int cnt = 0;
if (len < 4) {
return 0;
}
if (data[0] == 'x') { cnt++; }
if (data[1] == 'y') { cnt++; }
if (data[2] == 'z') { cnt++; }
if (cnt >= 3) {
switch (data[3]) {
case '0': {
// segv
int *p = NULL; *p = 123;
break;
}
case '1': {
// stack-buffer-underflow
int* p = &cnt - 32; for (int i = 0; i < 32; i++) { *(p + i) = 0; }
break;
}
case '2': {
// stack-buffer-overflow
int* p = &cnt + 32; for (int i = 0; i < 32; i++) { *(p - i) = 0; }
break;
}
case '3': {
// bad-free
int *p = &cnt; free(p);
break;
}
case '4': {
// double-free
int* p = (int *) malloc(sizeof(int)); free(p); free(p);
break;
}
case '5': {
// heap-use-after-free
int* p = (int *) malloc(sizeof(int)); free(p); *p = 123;
break;
}
case '6': {
// heap-buffer-overflow
int* p = (int *) malloc(8 * sizeof(int)); for (int i = 0; i < 32; i++) { *(p + i) = 0; }
break;
}
case '7': {
// fpe
int x = 0; int y = 123 / x;
break;
}
case '8': {
abort();
break;
}
}
}
return 0;
}

104
src/cli/examples/llvm-source-coverage/source-coverage-libfuzzer.py Executable file
View File

@ -0,0 +1,104 @@
#!/usr/bin/env python
#
# Copyright (c) Microsoft Corporation.
# Licensed under the MIT License.
import argparse
import logging
from onefuzztypes.enums import ContainerType, TaskType
from onefuzz.api import Onefuzz
from onefuzz.templates import JobHelper
def main() -> None:
parser = argparse.ArgumentParser(
formatter_class=argparse.ArgumentDefaultsHelpFormatter
)
parser.add_argument("setup_dir", type=str, help="Target setup directory")
parser.add_argument(
"target_exe",
type=str,
help="Target executable within setup directory without coverage instrumentation",
)
parser.add_argument(
"target_coverage_exe",
type=str,
help="Target executable within setup directory with coverage instrumentation",
)
parser.add_argument("project", type=str, help="Name of project")
parser.add_argument("name", type=str, help="Name of target")
parser.add_argument("build", type=str, help="Target build version.")
parser.add_argument("pool_name", type=str, help="VM pool to use")
parser.add_argument("tools", type=str, help="tools directory")
parser.add_argument(
"--duration", type=int, default=24, help="Hours to run the fuzzing task"
)
parser.add_argument("--inputs", help="seeds to use")
args = parser.parse_args()
of = Onefuzz()
logging.basicConfig(level=logging.WARNING)
of.logger.setLevel(logging.INFO)
job = of.template.libfuzzer.basic(
args.project,
args.name,
args.build,
args.pool_name,
target_exe=args.target_exe,
setup_dir=args.setup_dir,
duration=args.duration,
inputs=args.inputs,
)
helper = JobHelper(
of,
of.logger,
args.project,
args.name,
args.build,
args.duration,
pool_name=args.pool_name,
target_exe=args.target_exe,
job=job,
)
helper.define_containers(
ContainerType.setup,
ContainerType.analysis,
ContainerType.inputs,
ContainerType.tools,
)
helper.create_containers()
of.containers.files.upload_file(
helper.containers[ContainerType.tools], f"{args.tools}/source-coverage.sh"
)
containers = [
(ContainerType.setup, helper.containers[ContainerType.setup]),
(ContainerType.analysis, helper.containers[ContainerType.analysis]),
(ContainerType.tools, helper.containers[ContainerType.tools]),
# note, analysis is typically for crashes, but this is analyzing inputs
(ContainerType.crashes, helper.containers[ContainerType.inputs]),
]
of.logger.info("Creating generic_analysis task")
of.tasks.create(
helper.job.job_id,
TaskType.generic_analysis,
helper.setup_relative_blob_name(args.target_coverage_exe, args.setup_dir),
containers,
pool_name=args.pool_name,
duration=args.duration,
analyzer_exe="{tools_dir}/source-coverage.sh",
analyzer_options=["{target_exe}", "{output_dir}", "{input}"],
)
print(f"job:{helper.job.json(indent=4)}")
if __name__ == "__main__":
main()

91
src/cli/examples/llvm-source-coverage/source-coverage.py Executable file
View File

@ -0,0 +1,91 @@
#!/usr/bin/env python
#
# Copyright (c) Microsoft Corporation.
# Licensed under the MIT License.
import argparse
import logging
from onefuzztypes.enums import ContainerType, TaskType
from onefuzz.api import Onefuzz
from onefuzz.templates import JobHelper
def main() -> None:
parser = argparse.ArgumentParser(
formatter_class=argparse.ArgumentDefaultsHelpFormatter
)
parser.add_argument("setup_dir", type=str, help="Target setup directory")
parser.add_argument(
"target_coverage_exe",
type=str,
help="Target executable within setup directory with coverage instrumentation",
)
parser.add_argument("project", type=str, help="Name of project")
parser.add_argument("name", type=str, help="Name of target")
parser.add_argument("build", type=str, help="Target build version.")
parser.add_argument("pool_name", type=str, help="VM pool to use")
parser.add_argument("tools", type=str, help="tools directory")
parser.add_argument(
"--duration", type=int, default=24, help="Hours to run the fuzzing task"
)
parser.add_argument("--inputs", help="seeds to use")
args = parser.parse_args()
of = Onefuzz()
logging.basicConfig(level=logging.WARNING)
of.logger.setLevel(logging.INFO)
helper = JobHelper(
of,
of.logger,
args.project,
args.name,
args.build,
args.duration,
pool_name=args.pool_name,
target_exe=args.target_coverage_exe,
)
helper.define_containers(
ContainerType.setup,
ContainerType.analysis,
ContainerType.inputs,
ContainerType.tools,
)
helper.create_containers()
helper.upload_setup(args.setup_dir, args.target_coverage_exe)
if args.inputs:
helper.upload_inputs(args.inputs)
of.containers.files.upload_file(
helper.containers[ContainerType.tools], f"{args.tools}/source-coverage.sh"
)
containers = [
(ContainerType.setup, helper.containers[ContainerType.setup]),
(ContainerType.analysis, helper.containers[ContainerType.analysis]),
(ContainerType.tools, helper.containers[ContainerType.tools]),
# note, analysis is typically for crashes, but this is analyzing inputs
(ContainerType.crashes, helper.containers[ContainerType.inputs]),
]
of.logger.info("Creating generic_analysis task")
of.tasks.create(
helper.job.job_id,
TaskType.generic_analysis,
helper.setup_relative_blob_name(args.target_coverage_exe, args.setup_dir),
containers,
pool_name=args.pool_name,
duration=args.duration,
analyzer_exe="{tools_dir}/source-coverage.sh",
analyzer_options=["{target_exe}", "{output_dir}", "{input}"],
)
print(f"job:{helper.job.json(indent=4)}")
if __name__ == "__main__":
main()

48
src/cli/examples/llvm-source-coverage/tools/source-coverage.sh Executable file
View File

@ -0,0 +1,48 @@
#!/bin/bash
#
# Copyright (c) Microsoft Corporation.
# Licensed under the MIT License.
set -ex
if [ $# -lt 3 ]; then
echo usage: FUZZER RESULT_DIR FILE [... FILE_N]
exit 1
fi
FUZZER=$1; shift
RESULTS=$1; shift
mkdir -p ${RESULTS}/inputs
MERGED=${RESULTS}/coverage.profdata
REPORT=${RESULTS}/coverage.report
LCOV=${RESULTS}/coverage.lcov
COV_TOOL=$(which llvm-cov || which llvm-cov-10)
PROF_TOOL=$(which llvm-profdata || which llvm-profdata-10)
for file in $@; do
SHA=$(sha256sum ${file} | cut -d ' ' -f 1)
RAW=${RESULTS}/inputs/${SHA}.profraw
if [ -f ${RAW} ]; then
continue
fi
LLVM_PROFILE_FILE=${RAW} ${FUZZER} ${file}
if [ ! -f ${RAW} ]; then
echo "no coverage file generated ${RAW}"
exit 1
fi
if [ -f ${MERGED} ]; then
${PROF_TOOL} merge -output ${MERGED}.tmp ${RAW} ${MERGED}
mv ${MERGED}.tmp ${MERGED}
else
${PROF_TOOL} merge -output ${MERGED} ${RAW}
fi
${COV_TOOL} export ${FUZZER} -instr-profile=${MERGED} > ${REPORT}
${COV_TOOL} export ${FUZZER} -instr-profile=${MERGED} --format lcov > ${LCOV}
done