ExternalVendorCode/onefuzz
1
0
Fork 0
mirror of https://github.com/microsoft/onefuzz.git synced 2025-06-10 17:21:34 +00:00
Code Issues Packages Projects Releases Wiki Activity
onefuzz/src/cli/examples/llvm-source-coverage/README.md
bmc-msft 198d765cb4
Add generic_analysis example that performs coverage analysis (#1072) 
This adds an example script and tool that enables LLVM source-based coverage using the `generic_analysis` task.

This provides:
1. sample python script that launches the template and then the analysis task
1. sample `analysis_exe` wrapper script that launches the LLVM coverage tools
1. sample libfuzzer target for the example
1. walk through submitting the jobs and inspecting the results
2021-07-21 17:12:24 +00:00

7.0 KiB
Raw Blame History

Collecting Source Coverage using Analysis Tasks

The generic_analysis task can be used to perform a user-defined analysis of a target executable for every test input from some storage container.

Running an application compiled with LLVM's source-based code coverage with each input can be used to generate source based coverage information.

This example demonstrates using generic_analysis and the LLVM source coverage tools to provide source-based coverage on every input for a job. For more information, see Custom Analysis Tasks

  • source-coverage-libfuzzer.py: A wrapper that will launch a standard libfuzzer basic job with a source-based coverage task. (used below)
  • source-coverage.py: A wrapper that will launch a new job comprised of a source-based coverage task
  • setup: a basic libFuzzer target that builds with and without source coverage enabled
  • tools/source-coverage.sh: a script that wraps llvm-profdata and llvm-cov to perform the source analysis

This example generates the following data in the analysis container:

  • inputs/SHA256_OF_INPUT.profraw: the "raw" coverage data for each input analyzed
  • coverage.profdata: The merged coverage data using llvm-profdata
  • coverage.report: The JSON report of the merged coverage data provided by llvm-cov export
  • coverage.lcov : The lcov report of the merged coverage data provided by llvm-cov export --format lcov
❯ # build our libfuzzer
❯ cd setup/ 
❯ ls
Makefile  simple.c
❯ make
clang -g3 -fsanitize=fuzzer -fsanitize=address simple.c -o fuzz.exe
clang -g3 -fsanitize=fuzzer -fprofile-instr-generate -fcoverage-mapping simple.c -o fuzz-coverage.exe 
❯ cd ..
❯ # submit our basic job with an additional analysis task
❯ ./source-coverage-libfuzzer.py setup/ setup/fuzz.exe ./setup/fuzz-coverage.exe coverage-example 1 1 linux-1 ./tools/
INFO:onefuzz:creating libfuzzer from template
INFO:onefuzz:creating job (runtime: 24 hours)
INFO:onefuzz:created job: 61bc5c7c-d24f-4ebc-9bac-bec8fe040ade
INFO:onefuzz:using container: oft-setup-d1100b49a03c5a9483f140cee0676b87
INFO:onefuzz:using container: oft-inputs-6f3b76e7e841532bb7714375f564d483
INFO:onefuzz:using container: oft-crashes-6f3b76e7e841532bb7714375f564d483
INFO:onefuzz:using container: oft-reports-6f3b76e7e841532bb7714375f564d483
INFO:onefuzz:using container: oft-unique-reports-6f3b76e7e841532bb7714375f564d483
INFO:onefuzz:using container: oft-unique-inputs-6f3b76e7e841532bb7714375f564d483
INFO:onefuzz:using container: oft-no-repro-6f3b76e7e841532bb7714375f564d483
INFO:onefuzz:using container: oft-coverage-d1100b49a03c5a9483f140cee0676b87
INFO:onefuzz:using container: oft-regression-reports-06bdcba10b5f5e45bdb38ed924856426
INFO:onefuzz:uploading setup dir `setup/`
INFO:onefuzz:creating libfuzzer_regression task
INFO:onefuzz:creating libfuzzer task
INFO:onefuzz:creating coverage task
INFO:onefuzz:creating libfuzzer_crash_report task
INFO:onefuzz:done creating tasks
INFO:onefuzz:using container: oft-setup-d1100b49a03c5a9483f140cee0676b87
INFO:onefuzz:using container: oft-analysis-6f3b76e7e841532bb7714375f564d483
INFO:onefuzz:using container: oft-inputs-6f3b76e7e841532bb7714375f564d483
INFO:onefuzz:using container: oft-tools-6f3b76e7e841532bb7714375f564d483
INFO:onefuzz:Creating generic_analysis task
job:{
    "timestamp": null,
    "job_id": "61bc5c7c-d24f-4ebc-9bac-bec8fe040ade",
    "state": "init",
    "config": {
        "project": "coverage-example",
        "name": "1",
        "build": "1",
        "duration": 24
    },
    "error": null,
    "end_time": null,
    "task_info": null,
    "user_info": {
        "application_id": "00000000-0000-0000-0000-000000000000",
        "object_id": "00000000-0000-0000-0000-000000000000",
        "upn": "example@contoso.com"
    }
}
❯ # a little while later, check on the status of our job
❯ onefuzz
job: 61bc5c7c-d24f-4ebc-9bac-bec8fe040ade
project:coverage-example name:1 build:1

tasks:
50e1b076 target:fuzz-coverage.exe state:running type:generic_analysis
63880445 target:fuzz.exe state:stopped type:libfuzzer_regression
77fb6177 target:fuzz.exe state:running type:coverage
a8e3338c target:fuzz.exe state:running type:libfuzzer_crash_report
aae7ba1b target:fuzz.exe state:running type:libfuzzer_fuzz

containers:
setup           count:4     name:oft-setup-d1100b49a03c5a9483f140cee0676b87
analysis        count:14    name:oft-analysis-6f3b76e7e841532bb7714375f564d483
tools           count:1     name:oft-tools-6f3b76e7e841532bb7714375f564d483
crashes         count:11    name:oft-inputs-6f3b76e7e841532bb7714375f564d483
crashes         count:4     name:oft-crashes-6f3b76e7e841532bb7714375f564d483
unique_reports  count:3     name:oft-unique-reports-6f3b76e7e841532bb7714375f564d483
regression_reports count:0     name:oft-regression-reports-06bdcba10b5f5e45bdb38ed924856426
coverage        count:1     name:oft-coverage-d1100b49a03c5a9483f140cee0676b87
readonly_inputs count:11    name:oft-inputs-6f3b76e7e841532bb7714375f564d483
reports         count:4     name:oft-reports-6f3b76e7e841532bb7714375f564d483
no_repro        count:0     name:oft-no-repro-6f3b76e7e841532bb7714375f564d483
inputs          count:11    name:oft-inputs-6f3b76e7e841532bb7714375f564d483
❯ # lets check on the results of the analysis thus far
❯ onefuzz containers files list oft-analysis-6f3b76e7e841532bb7714375f564d483
{
    "files": [
        "coverage.lcov",
        "coverage.profdata",
        "coverage.report",
        "inputs/06a7e66b4ddb9d43b9007e20f351c8076a2f5c5c13ec6d683e1307eeee472f7a.profraw",
        "inputs/075de2b906dbd7066da008cab735bee896370154603579a50122f9b88545bd45.profraw",
        "inputs/0fc4f9bfb1e6850b77e130904c0d5f8d0bfabe9a658efee7c4c41ad0015bff22.profraw",
        "inputs/15dab3cc1c78958bc8c6d959cf708c2062e8327d3db873c2629b243c7e1a1759.profraw",
        "inputs/3ebe1b59762a1c8020c1efe3747dd07f0e30617ed60b4e6a5bee16b6ea421dd0.profraw",
        "inputs/594e519ae499312b29433b7dd8a97ff068defcba9755b6d5d00e84c524d67b06.profraw",
        "inputs/75558b9c2275acb05f57066ce1199be864c7affffece0b952edac02e785bbc9f.profraw",
        "inputs/bc9b8634ef85180578a9b501c901ce394ccd9087096fa4f298e4fc3752e60804.profraw",
        "inputs/c6b27b6743b120d83d5cc1d37b0f51acddcb69ff544763e7552efb7b575bac38.profraw",
        "inputs/c8bc644c4ddaaeafdb76142b72577e1f923b6797d87d254025f2fdf2b8225540.profraw",
        "inputs/e5e1b99e66064d2e9414a37158465eb4fdc1a8120b9fa8e10e9301b5fc25bc98.profraw"
    ]
}
❯ # this parses the report and checks that it's an coverage json report as we expect
❯ 1f containers files get oft-analysis-6f3b76e7e841532bb7714375f564d483 coverage.report | jq .type
"llvm.coverage.json.export"
❯ # now let's inspect the merged lcov file
❯ 1f containers files get oft-analysis-6f3b76e7e841532bb7714375f564d483 coverage.lcov |head -n 10
SF:/home/USERNAME/onefuzz/src/cli/examples/llvm-source-coverage/setup/simple.c
FN:8,LLVMFuzzerTestOneInput
FNDA:6,LLVMFuzzerTestOneInput
FNF:1
FNH:1
DA:8,6
DA:9,6
DA:10,6
DA:11,6
DA:12,1
❯