mirror of
https://github.com/AFLplusplus/AFLplusplus.git
synced 2025-06-14 02:58:08 +00:00
fff7f1c558
* Pure Python (3.6) port of benchmark.sh as benchmark.py, no other changes * Test standard and persistent modes separately * Add support for multi-core benchmarking * Save the results to a json file * Allow config of all experiment params, average across runs * Add start_time_of_run and total_execs_per_sec, cleanup for PR * benchmark: cleanup, add results, add a data exploration notebook * benchmark: add a README, lower default runs from 5 to 3 * benchmark: notebook wording tweaks * copy 'detect_leaks=0' from ASAN to LSAN fix for issue #1733, set "detect_leaks=0" when ASAN_OPTIONS contains it and LSAN_OPTIONS are not set. * fix of fix: make sure ASAN_OPTIONS and LSAN_OPTIONS agree on leak detection * fix lsan fix * clang-format 16->17 * Add missing initialisation for havoc_queued during the custom mutator's stage. * fix dictionary and cmin * Use direct call to write to OpenBSD The linker on OpenBSD emits a warning when linking this file: warning: syscall() may go away, please rewrite code to use direct calls * Fix possible doc inconsistency for custom mutator's queue_get function. * update todos * benchmark: Add support for COMPARISON file * benchmark: show the number of cores used in COMPARISON * benchmark: lower minimum Python version to 3.8 * benchmark: use afl's execs/s; increase CPU model width * benchmark: disallow duplicate entries for the same CPU in COMPARISON * Update benchmark.py * fix inf in stats * Fix benchmark.py * missing closing parenthesis * Update benchmark.py * benchmark: remove self-calculation of execs/sec * benchmark: update COMPARISON * benchmark: Update Jupyter notebook and results file. * benchmark: rename afl_execs_per_sec to execs_per_sec * benchmark: update README * update * add benchmark * nits * add benchmarks * Update unicornafl ref * Pass correct Nyx ID when creating a Nyx runner * Fix typo in docker pull command, add exampe to mount current dir as volume (#1914) * mini fix * add custom_post_run.c * update afl-fuzz-run * update python module * format code * update * merge function * changes * code format * improve cmplog * nit * nit * fix * fix * Stop hardcoding the path /usr/local/lib/afl in afl-ld-lto.c and respect the configured PREFIX. * Add benchmark for Raspberry Pi 5 * ryzen 5950 benchmark * add missing raspery5 * comparison -> comparison.md * removing options "-Wl,-rpath" "LLVM_LIBDIR" when using gcc * fixing -Wl,-rpath=<LLVM_LIBDIR> * nits * fix * afl-cc fixes * nit * add n_fuzz to ignore_timeouts * fix * Fix #1927 * in-depth blog post * add AFL_FUZZER_LOOPCOUNT * AFL_FUZZER_LOOPCOUNT * fix 2 mutation bugs * v4.09c release * v4.10a init * switch to explore powerschedule as default * fix MUT_INSERTASCIINUM * fix MUT_STRATEGY_ARRAY_SIZE * fix bad fix for MUT_STRATEGY_ARRAY_SIZE * remove afl-network-client on uninstall * update nyx * Improve binary-only related docs * llvm 18 build fixes. * code format * Fix custom_send link Add a leading '/' to walk in the repo root instead of current dir. * Use ../ instead * initial simple injection detection support * inject docs * fix for issue #1916, iLLVM crash in split-floatingpoint-compares * LLVM 17 bug workaround * finish injection implementation * remove tmp todo * update changelog * forgot to add the injection pass * Output afl-clang-fast stuffs only if necessary (#1912) * afl-cc header * afl-cc common declarations - Add afl-cc-state.c - Strip includes, find_object, debug/be_quiet/have_*/callname setting from afl-cc.c - Use debugf_args in main - Modify execvp stuffs to fit new aflcc struct * afl-cc show usage * afl-cc mode selecting 1. compiler_mode by callname in argv[0] 2. compiler_mode by env "AFL_CC_COMPILER" 3. compiler_mode/instrument_mode by command line options "--afl-..." 4. instrument_mode/compiler_mode by various env vars including "AFL_LLVM_INSTRUMENT" 5. final checking steps 6. print "... - mode: %s-%s\n" 7. determine real argv[0] according to compiler_mode * afl-cc macro defs * afl-cc linking behaviors * afl-cc fsanitize behaviors * afl-cc misc * afl-cc body update * afl-cc all-in-one formated with custom-format.py * nits --------- Co-authored-by: vanhauser-thc <vh@thc.org> * changelog * update grammar mutator * lto llvm 12+ * docs(custom_mutators): fix missing ':' (#1953) * Fix broken LTO mode and response file support (#1948) * Strip `-Wl,-no-undefined` during compilation (#1952) Make the compiler wrapper stripping `-Wl,-no-undefined` in addition to `-Wl,--no-undefined`. Both versions of the flag are accepted by clang and, therefore, used by building systems in the wild (e.g., samba will not build without this fix). * Remove dead code in write_to_testcase (#1955) The custom_mutators_count check in if case is duplicate with if condition. The else case is custom_mutators_count == 0, neither custom_mutator_list iteration nor sent check needed. Signed-off-by: Xeonacid <h.dwwwwww@gmail.com> * update qemuafl * WIP: Add ability to generate drcov trace using QEMU backend (#1956) * Document new drcov QEMU plugin * Add link to lightkeeper for QEMU drcov file loading --------- Co-authored-by: Jean-Romain Garnier <jean-romain.garnier@airbus.com> * code format * changelog * sleep on uid != 0 afl-system-config * fix segv about skip_next, warn on unsupported cases of linking options (#1958) * todos * ensure afl-cc only allows available compiler modes * update grammar mutator * disable aslr on apple * fix for arm64 * help selective instrumentation * typos * macos * add compiler test script * apple fixes --------- Signed-off-by: Xeonacid <h.dwwwwww@gmail.com> Co-authored-by: Chris Ball <chris@printf.net> Co-authored-by: hexcoder <hexcoder-@users.noreply.github.com> Co-authored-by: hexcoder- <heiko@hexco.de> Co-authored-by: Manuel Carrasco <m.carrasco@imperial.ac.uk> Co-authored-by: Jasper Lievisse Adriaanse <j@jasper.la> Co-authored-by: ifyGecko <26214995+ifyGecko@users.noreply.github.com> Co-authored-by: Dominik Maier <domenukk@gmail.com> Co-authored-by: Christian Holler (:decoder) <choller@mozilla.com> Co-authored-by: Carlo Maragno <ste.maragno@gmail.com> Co-authored-by: yangzao <yangzaocn@outlook.com> Co-authored-by: Romain Geissler <romain.geissler@amadeus.com> Co-authored-by: Jakob Lell <jakob@jakoblell.com> Co-authored-by: vincenzo MEZZELA <vincenzo.mezzela@amadeus.com> Co-authored-by: Andrea Fioraldi <andreafioraldi@gmail.com> Co-authored-by: Bet4 <0xbet4@gmail.com> Co-authored-by: David Carlier <devnexen@gmail.com> Co-authored-by: Xeonacid <h.dwwwwww@gmail.com> Co-authored-by: Sonic <50692172+SonicStark@users.noreply.github.com> Co-authored-by: Nils Bars <nils.bars@rub.de> Co-authored-by: Jean-Romain Garnier <7504819+JRomainG@users.noreply.github.com> Co-authored-by: Jean-Romain Garnier <jean-romain.garnier@airbus.com>
1446 lines
125 KiB
Plaintext
1446 lines
125 KiB
Plaintext
{
|
||
"cells": [
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 142,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"# benchmark.ipynb\n",
|
||
"# Part of the aflplusplus project, requires an ipynb (Jupyter) editor or viewer.\n",
|
||
"# Author: Chris Ball <chris@printf.net>\n",
|
||
"import json\n",
|
||
"import pandas as pd\n",
|
||
"with open(\"benchmark-results.jsonl\") as f:\n",
|
||
" lines = f.read().splitlines()\n",
|
||
"json_lines = [json.loads(line) for line in lines]\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"### Translate the JSON Lines entries into a single pandas DataFrame\n",
|
||
"\n",
|
||
"We have JSON Lines in [benchmark-results.jsonl](benchmark-results.jsonl) that look like this:"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 143,
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"{\n",
|
||
" \"config\": {\n",
|
||
" \"afl_persistent_config\": true,\n",
|
||
" \"afl_system_config\": true,\n",
|
||
" \"afl_version\": \"++4.09a\",\n",
|
||
" \"comment\": \"i9-9900k, 16GB DDR4-3000, Arch Linux\",\n",
|
||
" \"compiler\": \"clang version 16.0.6\",\n",
|
||
" \"target_arch\": \"x86_64-pc-linux-gnu\"\n",
|
||
" },\n",
|
||
" \"hardware\": {\n",
|
||
" \"cpu_fastest_core_mhz\": 4788.77,\n",
|
||
" \"cpu_model\": \"Intel(R) Core(TM) i9-9900K CPU @ 3.60GHz\",\n",
|
||
" \"cpu_threads\": 16\n",
|
||
" },\n",
|
||
" \"targets\": {\n",
|
||
" \"test-instr\": {\n",
|
||
" \"singlecore\": {\n",
|
||
" \"execs_per_sec\": 9845.64,\n",
|
||
" \"execs_total\": 98545,\n",
|
||
" \"fuzzers_used\": 1\n",
|
||
" }\n",
|
||
" }\n",
|
||
" }\n",
|
||
"}\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"print(json.dumps(json.loads(lines[0]), indent=2))"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"The [pd.json_normalize()](https://pandas.pydata.org/docs/reference/api/pandas.json_normalize.html]) method translates this into a flat table that we can perform queries against:"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 144,
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"data": {
|
||
"text/html": [
|
||
"<div>\n",
|
||
"<style scoped>\n",
|
||
" .dataframe tbody tr th:only-of-type {\n",
|
||
" vertical-align: middle;\n",
|
||
" }\n",
|
||
"\n",
|
||
" .dataframe tbody tr th {\n",
|
||
" vertical-align: top;\n",
|
||
" }\n",
|
||
"\n",
|
||
" .dataframe thead th {\n",
|
||
" text-align: right;\n",
|
||
" }\n",
|
||
"</style>\n",
|
||
"<table border=\"1\" class=\"dataframe\">\n",
|
||
" <thead>\n",
|
||
" <tr style=\"text-align: right;\">\n",
|
||
" <th></th>\n",
|
||
" <th>config.afl_persistent_config</th>\n",
|
||
" <th>config.afl_system_config</th>\n",
|
||
" <th>config.afl_version</th>\n",
|
||
" <th>config.comment</th>\n",
|
||
" <th>config.compiler</th>\n",
|
||
" <th>config.target_arch</th>\n",
|
||
" <th>hardware.cpu_fastest_core_mhz</th>\n",
|
||
" <th>hardware.cpu_model</th>\n",
|
||
" <th>hardware.cpu_threads</th>\n",
|
||
" <th>targets.test-instr.singlecore.execs_per_sec</th>\n",
|
||
" <th>...</th>\n",
|
||
" <th>targets.test-instr.singlecore.fuzzers_used</th>\n",
|
||
" <th>targets.test-instr-persist-shmem.singlecore.execs_per_sec</th>\n",
|
||
" <th>targets.test-instr-persist-shmem.singlecore.execs_total</th>\n",
|
||
" <th>targets.test-instr-persist-shmem.singlecore.fuzzers_used</th>\n",
|
||
" <th>targets.test-instr-persist-shmem.multicore.execs_per_sec</th>\n",
|
||
" <th>targets.test-instr-persist-shmem.multicore.execs_total</th>\n",
|
||
" <th>targets.test-instr-persist-shmem.multicore.fuzzers_used</th>\n",
|
||
" <th>targets.test-instr.multicore.execs_per_sec</th>\n",
|
||
" <th>targets.test-instr.multicore.execs_total</th>\n",
|
||
" <th>targets.test-instr.multicore.fuzzers_used</th>\n",
|
||
" </tr>\n",
|
||
" </thead>\n",
|
||
" <tbody>\n",
|
||
" <tr>\n",
|
||
" <th>0</th>\n",
|
||
" <td>True</td>\n",
|
||
" <td>True</td>\n",
|
||
" <td>++4.09a</td>\n",
|
||
" <td>i9-9900k, 16GB DDR4-3000, Arch Linux</td>\n",
|
||
" <td>clang version 16.0.6</td>\n",
|
||
" <td>x86_64-pc-linux-gnu</td>\n",
|
||
" <td>4788.770</td>\n",
|
||
" <td>Intel(R) Core(TM) i9-9900K CPU @ 3.60GHz</td>\n",
|
||
" <td>16</td>\n",
|
||
" <td>9845.64</td>\n",
|
||
" <td>...</td>\n",
|
||
" <td>1.0</td>\n",
|
||
" <td>NaN</td>\n",
|
||
" <td>NaN</td>\n",
|
||
" <td>NaN</td>\n",
|
||
" <td>NaN</td>\n",
|
||
" <td>NaN</td>\n",
|
||
" <td>NaN</td>\n",
|
||
" <td>NaN</td>\n",
|
||
" <td>NaN</td>\n",
|
||
" <td>NaN</td>\n",
|
||
" </tr>\n",
|
||
" <tr>\n",
|
||
" <th>1</th>\n",
|
||
" <td>True</td>\n",
|
||
" <td>True</td>\n",
|
||
" <td>++4.09a</td>\n",
|
||
" <td>i9-9900k, 16GB DDR4-3000, Arch Linux</td>\n",
|
||
" <td>clang version 16.0.6</td>\n",
|
||
" <td>x86_64-pc-linux-gnu</td>\n",
|
||
" <td>4989.281</td>\n",
|
||
" <td>Intel(R) Core(TM) i9-9900K CPU @ 3.60GHz</td>\n",
|
||
" <td>16</td>\n",
|
||
" <td>NaN</td>\n",
|
||
" <td>...</td>\n",
|
||
" <td>NaN</td>\n",
|
||
" <td>125682.73</td>\n",
|
||
" <td>1257330.0</td>\n",
|
||
" <td>1.0</td>\n",
|
||
" <td>NaN</td>\n",
|
||
" <td>NaN</td>\n",
|
||
" <td>NaN</td>\n",
|
||
" <td>NaN</td>\n",
|
||
" <td>NaN</td>\n",
|
||
" <td>NaN</td>\n",
|
||
" </tr>\n",
|
||
" <tr>\n",
|
||
" <th>2</th>\n",
|
||
" <td>True</td>\n",
|
||
" <td>True</td>\n",
|
||
" <td>++4.09a</td>\n",
|
||
" <td>i9-9900k, 16GB DDR4-3000, Arch Linux</td>\n",
|
||
" <td>clang version 16.0.6</td>\n",
|
||
" <td>x86_64-pc-linux-gnu</td>\n",
|
||
" <td>4799.415</td>\n",
|
||
" <td>Intel(R) Core(TM) i9-9900K CPU @ 3.60GHz</td>\n",
|
||
" <td>16</td>\n",
|
||
" <td>NaN</td>\n",
|
||
" <td>...</td>\n",
|
||
" <td>NaN</td>\n",
|
||
" <td>NaN</td>\n",
|
||
" <td>NaN</td>\n",
|
||
" <td>NaN</td>\n",
|
||
" <td>120293.77</td>\n",
|
||
" <td>1203058.0</td>\n",
|
||
" <td>1.0</td>\n",
|
||
" <td>NaN</td>\n",
|
||
" <td>NaN</td>\n",
|
||
" <td>NaN</td>\n",
|
||
" </tr>\n",
|
||
" <tr>\n",
|
||
" <th>3</th>\n",
|
||
" <td>True</td>\n",
|
||
" <td>True</td>\n",
|
||
" <td>++4.09a</td>\n",
|
||
" <td>i9-9900k, 16GB DDR4-3000, Arch Linux</td>\n",
|
||
" <td>clang version 16.0.6</td>\n",
|
||
" <td>x86_64-pc-linux-gnu</td>\n",
|
||
" <td>4703.293</td>\n",
|
||
" <td>Intel(R) Core(TM) i9-9900K CPU @ 3.60GHz</td>\n",
|
||
" <td>16</td>\n",
|
||
" <td>NaN</td>\n",
|
||
" <td>...</td>\n",
|
||
" <td>NaN</td>\n",
|
||
" <td>NaN</td>\n",
|
||
" <td>NaN</td>\n",
|
||
" <td>NaN</td>\n",
|
||
" <td>231429.96</td>\n",
|
||
" <td>2314531.0</td>\n",
|
||
" <td>2.0</td>\n",
|
||
" <td>NaN</td>\n",
|
||
" <td>NaN</td>\n",
|
||
" <td>NaN</td>\n",
|
||
" </tr>\n",
|
||
" <tr>\n",
|
||
" <th>4</th>\n",
|
||
" <td>True</td>\n",
|
||
" <td>True</td>\n",
|
||
" <td>++4.09a</td>\n",
|
||
" <td>i9-9900k, 16GB DDR4-3000, Arch Linux</td>\n",
|
||
" <td>clang version 16.0.6</td>\n",
|
||
" <td>x86_64-pc-linux-gnu</td>\n",
|
||
" <td>4800.375</td>\n",
|
||
" <td>Intel(R) Core(TM) i9-9900K CPU @ 3.60GHz</td>\n",
|
||
" <td>16</td>\n",
|
||
" <td>NaN</td>\n",
|
||
" <td>...</td>\n",
|
||
" <td>NaN</td>\n",
|
||
" <td>NaN</td>\n",
|
||
" <td>NaN</td>\n",
|
||
" <td>NaN</td>\n",
|
||
" <td>346759.33</td>\n",
|
||
" <td>3468290.0</td>\n",
|
||
" <td>3.0</td>\n",
|
||
" <td>NaN</td>\n",
|
||
" <td>NaN</td>\n",
|
||
" <td>NaN</td>\n",
|
||
" </tr>\n",
|
||
" </tbody>\n",
|
||
"</table>\n",
|
||
"<p>5 rows × 21 columns</p>\n",
|
||
"</div>"
|
||
],
|
||
"text/plain": [
|
||
" config.afl_persistent_config config.afl_system_config config.afl_version \\\n",
|
||
"0 True True ++4.09a \n",
|
||
"1 True True ++4.09a \n",
|
||
"2 True True ++4.09a \n",
|
||
"3 True True ++4.09a \n",
|
||
"4 True True ++4.09a \n",
|
||
"\n",
|
||
" config.comment config.compiler \\\n",
|
||
"0 i9-9900k, 16GB DDR4-3000, Arch Linux clang version 16.0.6 \n",
|
||
"1 i9-9900k, 16GB DDR4-3000, Arch Linux clang version 16.0.6 \n",
|
||
"2 i9-9900k, 16GB DDR4-3000, Arch Linux clang version 16.0.6 \n",
|
||
"3 i9-9900k, 16GB DDR4-3000, Arch Linux clang version 16.0.6 \n",
|
||
"4 i9-9900k, 16GB DDR4-3000, Arch Linux clang version 16.0.6 \n",
|
||
"\n",
|
||
" config.target_arch hardware.cpu_fastest_core_mhz \\\n",
|
||
"0 x86_64-pc-linux-gnu 4788.770 \n",
|
||
"1 x86_64-pc-linux-gnu 4989.281 \n",
|
||
"2 x86_64-pc-linux-gnu 4799.415 \n",
|
||
"3 x86_64-pc-linux-gnu 4703.293 \n",
|
||
"4 x86_64-pc-linux-gnu 4800.375 \n",
|
||
"\n",
|
||
" hardware.cpu_model hardware.cpu_threads \\\n",
|
||
"0 Intel(R) Core(TM) i9-9900K CPU @ 3.60GHz 16 \n",
|
||
"1 Intel(R) Core(TM) i9-9900K CPU @ 3.60GHz 16 \n",
|
||
"2 Intel(R) Core(TM) i9-9900K CPU @ 3.60GHz 16 \n",
|
||
"3 Intel(R) Core(TM) i9-9900K CPU @ 3.60GHz 16 \n",
|
||
"4 Intel(R) Core(TM) i9-9900K CPU @ 3.60GHz 16 \n",
|
||
"\n",
|
||
" targets.test-instr.singlecore.execs_per_sec ... \\\n",
|
||
"0 9845.64 ... \n",
|
||
"1 NaN ... \n",
|
||
"2 NaN ... \n",
|
||
"3 NaN ... \n",
|
||
"4 NaN ... \n",
|
||
"\n",
|
||
" targets.test-instr.singlecore.fuzzers_used \\\n",
|
||
"0 1.0 \n",
|
||
"1 NaN \n",
|
||
"2 NaN \n",
|
||
"3 NaN \n",
|
||
"4 NaN \n",
|
||
"\n",
|
||
" targets.test-instr-persist-shmem.singlecore.execs_per_sec \\\n",
|
||
"0 NaN \n",
|
||
"1 125682.73 \n",
|
||
"2 NaN \n",
|
||
"3 NaN \n",
|
||
"4 NaN \n",
|
||
"\n",
|
||
" targets.test-instr-persist-shmem.singlecore.execs_total \\\n",
|
||
"0 NaN \n",
|
||
"1 1257330.0 \n",
|
||
"2 NaN \n",
|
||
"3 NaN \n",
|
||
"4 NaN \n",
|
||
"\n",
|
||
" targets.test-instr-persist-shmem.singlecore.fuzzers_used \\\n",
|
||
"0 NaN \n",
|
||
"1 1.0 \n",
|
||
"2 NaN \n",
|
||
"3 NaN \n",
|
||
"4 NaN \n",
|
||
"\n",
|
||
" targets.test-instr-persist-shmem.multicore.execs_per_sec \\\n",
|
||
"0 NaN \n",
|
||
"1 NaN \n",
|
||
"2 120293.77 \n",
|
||
"3 231429.96 \n",
|
||
"4 346759.33 \n",
|
||
"\n",
|
||
" targets.test-instr-persist-shmem.multicore.execs_total \\\n",
|
||
"0 NaN \n",
|
||
"1 NaN \n",
|
||
"2 1203058.0 \n",
|
||
"3 2314531.0 \n",
|
||
"4 3468290.0 \n",
|
||
"\n",
|
||
" targets.test-instr-persist-shmem.multicore.fuzzers_used \\\n",
|
||
"0 NaN \n",
|
||
"1 NaN \n",
|
||
"2 1.0 \n",
|
||
"3 2.0 \n",
|
||
"4 3.0 \n",
|
||
"\n",
|
||
" targets.test-instr.multicore.execs_per_sec \\\n",
|
||
"0 NaN \n",
|
||
"1 NaN \n",
|
||
"2 NaN \n",
|
||
"3 NaN \n",
|
||
"4 NaN \n",
|
||
"\n",
|
||
" targets.test-instr.multicore.execs_total \\\n",
|
||
"0 NaN \n",
|
||
"1 NaN \n",
|
||
"2 NaN \n",
|
||
"3 NaN \n",
|
||
"4 NaN \n",
|
||
"\n",
|
||
" targets.test-instr.multicore.fuzzers_used \n",
|
||
"0 NaN \n",
|
||
"1 NaN \n",
|
||
"2 NaN \n",
|
||
"3 NaN \n",
|
||
"4 NaN \n",
|
||
"\n",
|
||
"[5 rows x 21 columns]"
|
||
]
|
||
},
|
||
"execution_count": 144,
|
||
"metadata": {},
|
||
"output_type": "execute_result"
|
||
}
|
||
],
|
||
"source": [
|
||
"\n",
|
||
"df = pd.json_normalize(json_lines)\n",
|
||
"df.head()"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"### Graph prep\n",
|
||
"\n",
|
||
"We're looking for a line graph showing lines for each fuzz target, in both singlecore and multicore modes, in each config setting -- where the x-axis is number of cores, and the y-axis is execs_per_sec.\n",
|
||
"\n",
|
||
"First, a quick check that the number of rows matched what we'd intuitively expect:"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 145,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"i7 = df.query(\"`config.comment` == 'i9-9900k, 16GB DDR4-3000, Arch Linux'\")\n",
|
||
"assert len(i7) == 185"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 146,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"def build_graphdf_from_query(query: pd.DataFrame):\n",
|
||
" \"\"\"Build a table suitable for graphing from a subset of the dataframe.\"\"\"\n",
|
||
" graphdata = []\n",
|
||
" max_fuzzers = int(query[[\"targets.test-instr-persist-shmem.multicore.fuzzers_used\", \"targets.test-instr.multicore.fuzzers_used\"]].max(axis=1).max(axis=0))\n",
|
||
" for _, row in query.iterrows():\n",
|
||
" for target in [\"test-instr-persist-shmem\", \"test-instr\"]:\n",
|
||
" for mode in [\"multicore\", \"singlecore\"]:\n",
|
||
" label = \"\"\n",
|
||
" if not row[f\"targets.{target}.{mode}.execs_per_sec\"] > 0:\n",
|
||
" continue\n",
|
||
" execs_per_sec = row[f\"targets.{target}.{mode}.execs_per_sec\"]\n",
|
||
" parallel_fuzzers = row[f\"targets.{target}.{mode}.fuzzers_used\"]\n",
|
||
" afl_persistent_config = row[\"config.afl_persistent_config\"]\n",
|
||
" afl_system_config = row[\"config.afl_system_config\"]\n",
|
||
" if target == \"test-instr-persist-shmem\":\n",
|
||
" label += \"shmem\"\n",
|
||
" else:\n",
|
||
" label += \"base\"\n",
|
||
" if mode == \"multicore\":\n",
|
||
" label += \"-multicore\"\n",
|
||
" else:\n",
|
||
" label += \"-singlecore\"\n",
|
||
" if afl_persistent_config:\n",
|
||
" label += \"+persist-conf\"\n",
|
||
" if afl_system_config:\n",
|
||
" label += \"+system-conf\"\n",
|
||
" \n",
|
||
" if label == \"shmem-multicore+persist-conf+system-conf\":\n",
|
||
" graphdata.append({\"execs_per_sec\": execs_per_sec, \"parallel_fuzzers\": parallel_fuzzers, \"afl_persistent_config\": afl_persistent_config, \"afl_system_config\": afl_system_config, \"label\": \"Multicore: Persistent mode/shared memory + kernel config\"})\n",
|
||
" if label == \"shmem-multicore\":\n",
|
||
" graphdata.append({\"execs_per_sec\": execs_per_sec, \"parallel_fuzzers\": parallel_fuzzers, \"afl_persistent_config\": afl_persistent_config, \"afl_system_config\": afl_system_config, \"label\": \"Multicore: Persistent mode/shared memory without kernel config\"})\n",
|
||
" if label == \"base-multicore+persist-conf+system-conf\":\n",
|
||
" graphdata.append({\"execs_per_sec\": execs_per_sec, \"parallel_fuzzers\": parallel_fuzzers, \"afl_persistent_config\": afl_persistent_config, \"afl_system_config\": afl_system_config, \"label\": \"Multicore: Non-persistent mode + kernel config\"})\n",
|
||
" if label == \"shmem-singlecore+persist-conf+system-conf\":\n",
|
||
" for i in range(1, max_fuzzers + 1):\n",
|
||
" graphdata.append({\"execs_per_sec\": execs_per_sec, \"parallel_fuzzers\": float(i), \"afl_persistent_config\": afl_persistent_config, \"afl_system_config\": afl_system_config, \"label\": \"Singlecore: Persistent mode/shared memory + kernel config\"})\n",
|
||
" if label == \"base-singlecore+persist-conf+system-conf\":\n",
|
||
" for i in range(1, max_fuzzers + 1):\n",
|
||
" graphdata.append({\"execs_per_sec\": execs_per_sec, \"parallel_fuzzers\": float(i), \"afl_persistent_config\": afl_persistent_config, \"afl_system_config\": afl_system_config, \"label\": \"Singlecore: Non-persistent mode + kernel config\"})\n",
|
||
" return pd.DataFrame.from_records(graphdata).sort_values(\"label\", ascending=False)\n",
|
||
"\n",
|
||
"graphdf = build_graphdf_from_query(i7)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 147,
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"data": {
|
||
"image/svg+xml": [
|
||
"<svg class=\"main-svg\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"1200\" height=\"400\" style=\"\" viewBox=\"0 0 1200 400\"><rect x=\"0\" y=\"0\" width=\"1200\" height=\"400\" style=\"fill: rgb(255, 255, 255); fill-opacity: 1;\"/><defs id=\"defs-9d6f0e\"><g class=\"clips\"><clipPath id=\"clip9d6f0exyplot\" class=\"plotclip\"><rect width=\"707\" height=\"220\"/></clipPath><clipPath class=\"axesclip\" id=\"clip9d6f0ex\"><rect x=\"80\" y=\"0\" width=\"707\" height=\"400\"/></clipPath><clipPath class=\"axesclip\" id=\"clip9d6f0ey\"><rect x=\"0\" y=\"100\" width=\"1200\" height=\"220\"/></clipPath><clipPath class=\"axesclip\" id=\"clip9d6f0exy\"><rect x=\"80\" y=\"100\" width=\"707\" height=\"220\"/></clipPath></g><g class=\"gradients\"/><g class=\"patterns\"/></defs><g class=\"bglayer\"><rect class=\"bg\" x=\"80\" y=\"100\" width=\"707\" height=\"220\" style=\"fill: rgb(229, 236, 246); fill-opacity: 1; stroke-width: 0;\"/></g><g class=\"layer-below\"><g class=\"imagelayer\"/><g class=\"shapelayer\"/></g><g class=\"cartesianlayer\"><g class=\"subplot xy\"><g class=\"layer-subplot\"><g class=\"shapelayer\"/><g class=\"imagelayer\"/></g><g class=\"minor-gridlayer\"><g class=\"x\"/><g class=\"y\"/></g><g class=\"gridlayer\"><g class=\"x\"><path class=\"xgrid crisp\" transform=\"translate(100.2,0)\" d=\"M0,100v220\" style=\"stroke: rgb(255, 255, 255); stroke-opacity: 1; stroke-width: 1px;\"/><path class=\"xgrid crisp\" transform=\"translate(120.4,0)\" d=\"M0,100v220\" style=\"stroke: rgb(255, 255, 255); stroke-opacity: 1; stroke-width: 1px;\"/><path class=\"xgrid crisp\" transform=\"translate(140.6,0)\" d=\"M0,100v220\" style=\"stroke: rgb(255, 255, 255); stroke-opacity: 1; stroke-width: 1px;\"/><path class=\"xgrid crisp\" transform=\"translate(160.8,0)\" d=\"M0,100v220\" style=\"stroke: rgb(255, 255, 255); stroke-opacity: 1; stroke-width: 1px;\"/><path class=\"xgrid crisp\" transform=\"translate(181,0)\" d=\"M0,100v220\" style=\"stroke: rgb(255, 255, 255); stroke-opacity: 1; stroke-width: 1px;\"/><path class=\"xgrid crisp\" transform=\"translate(201.2,0)\" d=\"M0,100v220\" style=\"stroke: rgb(255, 255, 255); stroke-opacity: 1; stroke-width: 1px;\"/><path class=\"xgrid crisp\" transform=\"translate(221.4,0)\" d=\"M0,100v220\" style=\"stroke: rgb(255, 255, 255); stroke-opacity: 1; stroke-width: 1px;\"/><path class=\"xgrid crisp\" transform=\"translate(241.6,0)\" d=\"M0,100v220\" style=\"stroke: rgb(255, 255, 255); stroke-opacity: 1; stroke-width: 1px;\"/><path class=\"xgrid crisp\" transform=\"translate(261.8,0)\" d=\"M0,100v220\" style=\"stroke: rgb(255, 255, 255); stroke-opacity: 1; stroke-width: 1px;\"/><path class=\"xgrid crisp\" transform=\"translate(282,0)\" d=\"M0,100v220\" style=\"stroke: rgb(255, 255, 255); stroke-opacity: 1; stroke-width: 1px;\"/><path class=\"xgrid crisp\" transform=\"translate(302.2,0)\" d=\"M0,100v220\" style=\"stroke: rgb(255, 255, 255); stroke-opacity: 1; stroke-width: 1px;\"/><path class=\"xgrid crisp\" transform=\"translate(322.4,0)\" d=\"M0,100v220\" style=\"stroke: rgb(255, 255, 255); stroke-opacity: 1; stroke-width: 1px;\"/><path class=\"xgrid crisp\" transform=\"translate(342.6,0)\" d=\"M0,100v220\" style=\"stroke: rgb(255, 255, 255); stroke-opacity: 1; stroke-width: 1px;\"/><path class=\"xgrid crisp\" transform=\"translate(362.8,0)\" d=\"M0,100v220\" style=\"stroke: rgb(255, 255, 255); stroke-opacity: 1; stroke-width: 1px;\"/><path class=\"xgrid crisp\" transform=\"translate(383,0)\" d=\"M0,100v220\" style=\"stroke: rgb(255, 255, 255); stroke-opacity: 1; stroke-width: 1px;\"/><path class=\"xgrid crisp\" transform=\"translate(403.2,0)\" d=\"M0,100v220\" style=\"stroke: rgb(255, 255, 255); stroke-opacity: 1; stroke-width: 1px;\"/><path class=\"xgrid crisp\" transform=\"translate(423.4,0)\" d=\"M0,100v220\" style=\"stroke: rgb(255, 255, 255); stroke-opacity: 1; stroke-width: 1px;\"/><path class=\"xgrid crisp\" transform=\"translate(443.6,0)\" d=\"M0,100v220\" style=\"stroke: rgb(255, 255, 255); stroke-opacity: 1; stroke-width: 1px;\"/><path class=\"xgrid crisp\" transform=\"translate(463.8,0)\" d=\"M0,100v220\" style=\"stroke: rgb(255, 255, 255); stroke-opacity: 1; stroke-width: 1px;\"/><path class=\"xgrid crisp\" transform=\"translate(484,0)\" d=\"M0,100v220\" style=\"stroke: rgb(255, 255, 255); stroke-opacity: 1; stroke-width: 1px;\"/><path class=\"xgrid crisp\" transform=\"translate(504.2,0)\" d=\"M0,100v220\" style=\"stroke: rgb(255, 255, 255); stroke-opacity: 1; stroke-width: 1px;\"/><path class=\"xgrid crisp\" transform=\"translate(524.4,0)\" d=\"M0,100v220\" style=\"stroke: rgb(255, 255, 255); stroke-opacity: 1; stroke-width: 1px;\"/><path class=\"xgrid crisp\" transform=\"translate(544.6,0)\" d=\"M0,100v220\" style=\"stroke: rgb(255, 255, 255); stroke-opacity: 1; stroke-width: 1px;\"/><path class=\"xgrid crisp\" transform=\"translate(564.8,0)\" d=\"M0,100v220\" style=\"stroke: rgb(255, 255, 255); stroke-opacity: 1; stroke-width: 1px;\"/><path class=\"xgrid crisp\" transform=\"translate(585,0)\" d=\"M0,100v220\" style=\"stroke: rgb(255, 255, 255); stroke-opacity: 1; stroke-width: 1px;\"/><path class=\"xgrid crisp\" transform=\"translate(605.2,0)\" d=\"M0,100v220\" style=\"stroke: rgb(255, 255, 255); stroke-opacity: 1; stroke-width: 1px;\"/><path class=\"xgrid crisp\" transform=\"translate(625.4,0)\" d=\"M0,100v220\" style=\"stroke: rgb(255, 255, 255); stroke-opacity: 1; stroke-width: 1px;\"/><path class=\"xgrid crisp\" transform=\"translate(645.6,0)\" d=\"M0,100v220\" style=\"stroke: rgb(255, 255, 255); stroke-opacity: 1; stroke-width: 1px;\"/><path class=\"xgrid crisp\" transform=\"translate(665.8,0)\" d=\"M0,100v220\" style=\"stroke: rgb(255, 255, 255); stroke-opacity: 1; stroke-width: 1px;\"/><path class=\"xgrid crisp\" transform=\"translate(686,0)\" d=\"M0,100v220\" style=\"stroke: rgb(255, 255, 255); stroke-opacity: 1; stroke-width: 1px;\"/><path class=\"xgrid crisp\" transform=\"translate(706.2,0)\" d=\"M0,100v220\" style=\"stroke: rgb(255, 255, 255); stroke-opacity: 1; stroke-width: 1px;\"/><path class=\"xgrid crisp\" transform=\"translate(726.4,0)\" d=\"M0,100v220\" style=\"stroke: rgb(255, 255, 255); stroke-opacity: 1; stroke-width: 1px;\"/><path class=\"xgrid crisp\" transform=\"translate(746.6,0)\" d=\"M0,100v220\" style=\"stroke: rgb(255, 255, 255); stroke-opacity: 1; stroke-width: 1px;\"/><path class=\"xgrid crisp\" transform=\"translate(766.8,0)\" d=\"M0,100v220\" style=\"stroke: rgb(255, 255, 255); stroke-opacity: 1; stroke-width: 1px;\"/></g><g class=\"y\"><path class=\"ygrid crisp\" transform=\"translate(0,309)\" d=\"M80,0h707\" style=\"stroke: rgb(255, 255, 255); stroke-opacity: 1; stroke-width: 1px;\"/><path class=\"ygrid crisp\" transform=\"translate(0,269.4)\" d=\"M80,0h707\" style=\"stroke: rgb(255, 255, 255); stroke-opacity: 1; stroke-width: 1px;\"/><path class=\"ygrid crisp\" transform=\"translate(0,229.8)\" d=\"M80,0h707\" style=\"stroke: rgb(255, 255, 255); stroke-opacity: 1; stroke-width: 1px;\"/><path class=\"ygrid crisp\" transform=\"translate(0,190.2)\" d=\"M80,0h707\" style=\"stroke: rgb(255, 255, 255); stroke-opacity: 1; stroke-width: 1px;\"/><path class=\"ygrid crisp\" transform=\"translate(0,150.6)\" d=\"M80,0h707\" style=\"stroke: rgb(255, 255, 255); stroke-opacity: 1; stroke-width: 1px;\"/><path class=\"ygrid crisp\" transform=\"translate(0,111)\" d=\"M80,0h707\" style=\"stroke: rgb(255, 255, 255); stroke-opacity: 1; stroke-width: 1px;\"/></g></g><g class=\"zerolinelayer\"><path class=\"yzl zl crisp\" transform=\"translate(0,310.68)\" d=\"M80,0h707\" style=\"stroke: rgb(255, 255, 255); stroke-opacity: 1; stroke-width: 2px;\"/></g><path class=\"xlines-below\"/><path class=\"ylines-below\"/><g class=\"overlines-below\"/><g class=\"xaxislayer-below\"/><g class=\"yaxislayer-below\"/><g class=\"overaxes-below\"/><g class=\"plot\" transform=\"translate(80,100)\" clip-path=\"url(#clip9d6f0exyplot)\"><g class=\"scatterlayer mlayer\"><g class=\"trace scatter traceff0d16\" style=\"stroke-miterlimit: 2; opacity: 1;\"><g class=\"fills\"/><g class=\"errorbars\"/><g class=\"lines\"><path class=\"js-line\" d=\"M0,208.85L80.8,202.77L101,201.36L141.4,198.59L161.6,198L323.2,193.39L343.4,193.43L464.6,193.88L484.8,193.92L707,194.3\" style=\"vector-effect: none; fill: none; stroke: rgb(99, 110, 250); stroke-opacity: 1; stroke-width: 2px; opacity: 1;\"/></g><g class=\"points\"/><g class=\"text\"/></g><g class=\"trace scatter tracef305d6\" style=\"stroke-miterlimit: 2; opacity: 1;\"><g class=\"fills\"/><g class=\"errorbars\"/><g class=\"lines\"><path class=\"js-line\" d=\"M0,190.13L101,94.86L121.2,77L141.4,58.03L161.6,47.37L202,37L222.2,30.79L242.4,25.2L262.6,20.96L303,11L323.2,16.78L343.4,12.45L363.6,13.25L383.8,14.21L404,16.48L424.2,20.66L444.4,15.41L464.6,15.27L484.8,22.42L505,19.96L525.2,22.91L545.4,20.22L565.6,20.92L585.8,29.86L606,19.39L626.2,19.56L646.4,23.9L666.6,21.39L686.8,32.77L707,24.08\" style=\"vector-effect: none; fill: none; stroke: rgb(239, 85, 59); stroke-opacity: 1; stroke-width: 2px; opacity: 1;\"/></g><g class=\"points\"/><g class=\"text\"/></g><g class=\"trace scatter tracef64091\" style=\"stroke-miterlimit: 2; opacity: 1;\"><g class=\"fills\"/><g class=\"errorbars\"/><g class=\"lines\"><path class=\"js-line\" d=\"M0,195.2L40.4,165.81L60.6,152.75L141.4,97.63L161.6,93.7L303,66.53L323.2,66.65L383.8,65.9L404,66.5L444.4,66.49L464.6,67.67L505,67.82L525.2,69.28L565.6,69.56L585.8,70.99L626.2,71.86L646.4,73.11L666.6,71.31L686.8,71.91L707,71.07\" style=\"vector-effect: none; fill: none; stroke: rgb(0, 204, 150); stroke-opacity: 1; stroke-width: 2px; opacity: 1;\"/></g><g class=\"points\"/><g class=\"text\"/></g><g class=\"trace scatter trace642013\" style=\"stroke-miterlimit: 2; opacity: 1;\"><g class=\"fills\"/><g class=\"errorbars\"/><g class=\"lines\"><path class=\"js-line\" d=\"M0,209L707,209\" style=\"vector-effect: none; fill: none; stroke: rgb(171, 99, 250); stroke-opacity: 1; stroke-width: 2px; opacity: 1;\"/></g><g class=\"points\"/><g class=\"text\"/></g><g class=\"trace scatter traceff0076\" style=\"stroke-miterlimit: 2; opacity: 1;\"><g class=\"fills\"/><g class=\"errorbars\"/><g class=\"lines\"><path class=\"js-line\" d=\"M0,189.21L707,189.21\" style=\"vector-effect: none; fill: none; stroke: rgb(255, 161, 90); stroke-opacity: 1; stroke-width: 2px; opacity: 1;\"/></g><g class=\"points\"/><g class=\"text\"/></g></g></g><g class=\"overplot\"/><path class=\"xlines-above crisp\" d=\"M0,0\" style=\"fill: none;\"/><path class=\"ylines-above crisp\" d=\"M0,0\" style=\"fill: none;\"/><g class=\"overlines-above\"/><g class=\"xaxislayer-above\"><g class=\"xtick\"><text text-anchor=\"middle\" x=\"0\" y=\"333\" transform=\"translate(80,0)\" style=\"font-family: 'Open Sans', verdana, arial, sans-serif; font-size: 12px; fill: rgb(42, 63, 95); fill-opacity: 1; white-space: pre; opacity: 1;\">1</text></g><g class=\"xtick\"><text text-anchor=\"middle\" x=\"0\" y=\"333\" style=\"font-family: 'Open Sans', verdana, arial, sans-serif; font-size: 12px; fill: rgb(42, 63, 95); fill-opacity: 1; white-space: pre; opacity: 1;\" transform=\"translate(100.2,0)\">2</text></g><g class=\"xtick\"><text text-anchor=\"middle\" x=\"0\" y=\"333\" style=\"font-family: 'Open Sans', verdana, arial, sans-serif; font-size: 12px; fill: rgb(42, 63, 95); fill-opacity: 1; white-space: pre; opacity: 1;\" transform=\"translate(120.4,0)\">3</text></g><g class=\"xtick\"><text text-anchor=\"middle\" x=\"0\" y=\"333\" style=\"font-family: 'Open Sans', verdana, arial, sans-serif; font-size: 12px; fill: rgb(42, 63, 95); fill-opacity: 1; white-space: pre; opacity: 1;\" transform=\"translate(140.6,0)\">4</text></g><g class=\"xtick\"><text text-anchor=\"middle\" x=\"0\" y=\"333\" style=\"font-family: 'Open Sans', verdana, arial, sans-serif; font-size: 12px; fill: rgb(42, 63, 95); fill-opacity: 1; white-space: pre; opacity: 1;\" transform=\"translate(160.8,0)\">5</text></g><g class=\"xtick\"><text text-anchor=\"middle\" x=\"0\" y=\"333\" style=\"font-family: 'Open Sans', verdana, arial, sans-serif; font-size: 12px; fill: rgb(42, 63, 95); fill-opacity: 1; white-space: pre; opacity: 1;\" transform=\"translate(181,0)\">6</text></g><g class=\"xtick\"><text text-anchor=\"middle\" x=\"0\" y=\"333\" style=\"font-family: 'Open Sans', verdana, arial, sans-serif; font-size: 12px; fill: rgb(42, 63, 95); fill-opacity: 1; white-space: pre; opacity: 1;\" transform=\"translate(201.2,0)\">7</text></g><g class=\"xtick\"><text text-anchor=\"middle\" x=\"0\" y=\"333\" style=\"font-family: 'Open Sans', verdana, arial, sans-serif; font-size: 12px; fill: rgb(42, 63, 95); fill-opacity: 1; white-space: pre; opacity: 1;\" transform=\"translate(221.4,0)\">8</text></g><g class=\"xtick\"><text text-anchor=\"middle\" x=\"0\" y=\"333\" style=\"font-family: 'Open Sans', verdana, arial, sans-serif; font-size: 12px; fill: rgb(42, 63, 95); fill-opacity: 1; white-space: pre; opacity: 1;\" transform=\"translate(241.6,0)\">9</text></g><g class=\"xtick\"><text text-anchor=\"middle\" x=\"0\" y=\"333\" style=\"font-family: 'Open Sans', verdana, arial, sans-serif; font-size: 12px; fill: rgb(42, 63, 95); fill-opacity: 1; white-space: pre; opacity: 1;\" transform=\"translate(261.8,0)\">10</text></g><g class=\"xtick\"><text text-anchor=\"middle\" x=\"0\" y=\"333\" style=\"font-family: 'Open Sans', verdana, arial, sans-serif; font-size: 12px; fill: rgb(42, 63, 95); fill-opacity: 1; white-space: pre; opacity: 1;\" transform=\"translate(282,0)\">11</text></g><g class=\"xtick\"><text text-anchor=\"middle\" x=\"0\" y=\"333\" style=\"font-family: 'Open Sans', verdana, arial, sans-serif; font-size: 12px; fill: rgb(42, 63, 95); fill-opacity: 1; white-space: pre; opacity: 1;\" transform=\"translate(302.2,0)\">12</text></g><g class=\"xtick\"><text text-anchor=\"middle\" x=\"0\" y=\"333\" style=\"font-family: 'Open Sans', verdana, arial, sans-serif; font-size: 12px; fill: rgb(42, 63, 95); fill-opacity: 1; white-space: pre; opacity: 1;\" transform=\"translate(322.4,0)\">13</text></g><g class=\"xtick\"><text text-anchor=\"middle\" x=\"0\" y=\"333\" style=\"font-family: 'Open Sans', verdana, arial, sans-serif; font-size: 12px; fill: rgb(42, 63, 95); fill-opacity: 1; white-space: pre; opacity: 1;\" transform=\"translate(342.6,0)\">14</text></g><g class=\"xtick\"><text text-anchor=\"middle\" x=\"0\" y=\"333\" style=\"font-family: 'Open Sans', verdana, arial, sans-serif; font-size: 12px; fill: rgb(42, 63, 95); fill-opacity: 1; white-space: pre; opacity: 1;\" transform=\"translate(362.8,0)\">15</text></g><g class=\"xtick\"><text text-anchor=\"middle\" x=\"0\" y=\"333\" style=\"font-family: 'Open Sans', verdana, arial, sans-serif; font-size: 12px; fill: rgb(42, 63, 95); fill-opacity: 1; white-space: pre; opacity: 1;\" transform=\"translate(383,0)\">16</text></g><g class=\"xtick\"><text text-anchor=\"middle\" x=\"0\" y=\"333\" style=\"font-family: 'Open Sans', verdana, arial, sans-serif; font-size: 12px; fill: rgb(42, 63, 95); fill-opacity: 1; white-space: pre; opacity: 1;\" transform=\"translate(403.2,0)\">17</text></g><g class=\"xtick\"><text text-anchor=\"middle\" x=\"0\" y=\"333\" style=\"font-family: 'Open Sans', verdana, arial, sans-serif; font-size: 12px; fill: rgb(42, 63, 95); fill-opacity: 1; white-space: pre; opacity: 1;\" transform=\"translate(423.4,0)\">18</text></g><g class=\"xtick\"><text text-anchor=\"middle\" x=\"0\" y=\"333\" style=\"font-family: 'Open Sans', verdana, arial, sans-serif; font-size: 12px; fill: rgb(42, 63, 95); fill-opacity: 1; white-space: pre; opacity: 1;\" transform=\"translate(443.6,0)\">19</text></g><g class=\"xtick\"><text text-anchor=\"middle\" x=\"0\" y=\"333\" style=\"font-family: 'Open Sans', verdana, arial, sans-serif; font-size: 12px; fill: rgb(42, 63, 95); fill-opacity: 1; white-space: pre; opacity: 1;\" transform=\"translate(463.8,0)\">20</text></g><g class=\"xtick\"><text text-anchor=\"middle\" x=\"0\" y=\"333\" style=\"font-family: 'Open Sans', verdana, arial, sans-serif; font-size: 12px; fill: rgb(42, 63, 95); fill-opacity: 1; white-space: pre; opacity: 1;\" transform=\"translate(484,0)\">21</text></g><g class=\"xtick\"><text text-anchor=\"middle\" x=\"0\" y=\"333\" style=\"font-family: 'Open Sans', verdana, arial, sans-serif; font-size: 12px; fill: rgb(42, 63, 95); fill-opacity: 1; white-space: pre; opacity: 1;\" transform=\"translate(504.2,0)\">22</text></g><g class=\"xtick\"><text text-anchor=\"middle\" x=\"0\" y=\"333\" style=\"font-family: 'Open Sans', verdana, arial, sans-serif; font-size: 12px; fill: rgb(42, 63, 95); fill-opacity: 1; white-space: pre; opacity: 1;\" transform=\"translate(524.4,0)\">23</text></g><g class=\"xtick\"><text text-anchor=\"middle\" x=\"0\" y=\"333\" style=\"font-family: 'Open Sans', verdana, arial, sans-serif; font-size: 12px; fill: rgb(42, 63, 95); fill-opacity: 1; white-space: pre; opacity: 1;\" transform=\"translate(544.6,0)\">24</text></g><g class=\"xtick\"><text text-anchor=\"middle\" x=\"0\" y=\"333\" style=\"font-family: 'Open Sans', verdana, arial, sans-serif; font-size: 12px; fill: rgb(42, 63, 95); fill-opacity: 1; white-space: pre; opacity: 1;\" transform=\"translate(564.8,0)\">25</text></g><g class=\"xtick\"><text text-anchor=\"middle\" x=\"0\" y=\"333\" style=\"font-family: 'Open Sans', verdana, arial, sans-serif; font-size: 12px; fill: rgb(42, 63, 95); fill-opacity: 1; white-space: pre; opacity: 1;\" transform=\"translate(585,0)\">26</text></g><g class=\"xtick\"><text text-anchor=\"middle\" x=\"0\" y=\"333\" style=\"font-family: 'Open Sans', verdana, arial, sans-serif; font-size: 12px; fill: rgb(42, 63, 95); fill-opacity: 1; white-space: pre; opacity: 1;\" transform=\"translate(605.2,0)\">27</text></g><g class=\"xtick\"><text text-anchor=\"middle\" x=\"0\" y=\"333\" style=\"font-family: 'Open Sans', verdana, arial, sans-serif; font-size: 12px; fill: rgb(42, 63, 95); fill-opacity: 1; white-space: pre; opacity: 1;\" transform=\"translate(625.4,0)\">28</text></g><g class=\"xtick\"><text text-anchor=\"middle\" x=\"0\" y=\"333\" style=\"font-family: 'Open Sans', verdana, arial, sans-serif; font-size: 12px; fill: rgb(42, 63, 95); fill-opacity: 1; white-space: pre; opacity: 1;\" transform=\"translate(645.6,0)\">29</text></g><g class=\"xtick\"><text text-anchor=\"middle\" x=\"0\" y=\"333\" style=\"font-family: 'Open Sans', verdana, arial, sans-serif; font-size: 12px; fill: rgb(42, 63, 95); fill-opacity: 1; white-space: pre; opacity: 1;\" transform=\"translate(665.8,0)\">30</text></g><g class=\"xtick\"><text text-anchor=\"middle\" x=\"0\" y=\"333\" style=\"font-family: 'Open Sans', verdana, arial, sans-serif; font-size: 12px; fill: rgb(42, 63, 95); fill-opacity: 1; white-space: pre; opacity: 1;\" transform=\"translate(686,0)\">31</text></g><g class=\"xtick\"><text text-anchor=\"middle\" x=\"0\" y=\"333\" style=\"font-family: 'Open Sans', verdana, arial, sans-serif; font-size: 12px; fill: rgb(42, 63, 95); fill-opacity: 1; white-space: pre; opacity: 1;\" transform=\"translate(706.2,0)\">32</text></g><g class=\"xtick\"><text text-anchor=\"middle\" x=\"0\" y=\"333\" style=\"font-family: 'Open Sans', verdana, arial, sans-serif; font-size: 12px; fill: rgb(42, 63, 95); fill-opacity: 1; white-space: pre; opacity: 1;\" transform=\"translate(726.4,0)\">33</text></g><g class=\"xtick\"><text text-anchor=\"middle\" x=\"0\" y=\"333\" style=\"font-family: 'Open Sans', verdana, arial, sans-serif; font-size: 12px; fill: rgb(42, 63, 95); fill-opacity: 1; white-space: pre; opacity: 1;\" transform=\"translate(746.6,0)\">34</text></g><g class=\"xtick\"><text text-anchor=\"middle\" x=\"0\" y=\"333\" style=\"font-family: 'Open Sans', verdana, arial, sans-serif; font-size: 12px; fill: rgb(42, 63, 95); fill-opacity: 1; white-space: pre; opacity: 1;\" transform=\"translate(766.8,0)\">35</text></g><g class=\"xtick\"><text text-anchor=\"middle\" x=\"0\" y=\"333\" style=\"font-family: 'Open Sans', verdana, arial, sans-serif; font-size: 12px; fill: rgb(42, 63, 95); fill-opacity: 1; white-space: pre; opacity: 1;\" transform=\"translate(787,0)\">36</text></g></g><g class=\"yaxislayer-above\"><g class=\"ytick\"><text text-anchor=\"end\" x=\"79\" y=\"4.199999999999999\" transform=\"translate(0,309)\" style=\"font-family: 'Open Sans', verdana, arial, sans-serif; font-size: 12px; fill: rgb(42, 63, 95); fill-opacity: 1; white-space: pre; opacity: 1;\">1x</text></g><g class=\"ytick\"><text text-anchor=\"end\" x=\"79\" y=\"4.199999999999999\" style=\"font-family: 'Open Sans', verdana, arial, sans-serif; font-size: 12px; fill: rgb(42, 63, 95); fill-opacity: 1; white-space: pre; opacity: 1;\" transform=\"translate(0,269.4)\">25x</text></g><g class=\"ytick\"><text text-anchor=\"end\" x=\"79\" y=\"4.199999999999999\" style=\"font-family: 'Open Sans', verdana, arial, sans-serif; font-size: 12px; fill: rgb(42, 63, 95); fill-opacity: 1; white-space: pre; opacity: 1;\" transform=\"translate(0,229.8)\">48x</text></g><g class=\"ytick\"><text text-anchor=\"end\" x=\"79\" y=\"4.199999999999999\" style=\"font-family: 'Open Sans', verdana, arial, sans-serif; font-size: 12px; fill: rgb(42, 63, 95); fill-opacity: 1; white-space: pre; opacity: 1;\" transform=\"translate(0,190.2)\">72x</text></g><g class=\"ytick\"><text text-anchor=\"end\" x=\"79\" y=\"4.199999999999999\" style=\"font-family: 'Open Sans', verdana, arial, sans-serif; font-size: 12px; fill: rgb(42, 63, 95); fill-opacity: 1; white-space: pre; opacity: 1;\" transform=\"translate(0,150.6)\">95x</text></g><g class=\"ytick\"><text text-anchor=\"end\" x=\"79\" y=\"4.199999999999999\" style=\"font-family: 'Open Sans', verdana, arial, sans-serif; font-size: 12px; fill: rgb(42, 63, 95); fill-opacity: 1; white-space: pre; opacity: 1;\" transform=\"translate(0,111)\">119x</text></g></g><g class=\"overaxes-above\"/></g></g><g class=\"polarlayer\"/><g class=\"smithlayer\"/><g class=\"ternarylayer\"/><g class=\"geolayer\"/><g class=\"funnelarealayer\"/><g class=\"pielayer\"/><g class=\"iciclelayer\"/><g class=\"treemaplayer\"/><g class=\"sunburstlayer\"/><g class=\"glimages\"/><defs id=\"topdefs-9d6f0e\"><g class=\"clips\"/><clipPath id=\"legend9d6f0e\"><rect width=\"387\" height=\"124\" x=\"0\" y=\"0\"/></clipPath></defs><g class=\"layer-above\"><g class=\"imagelayer\"/><g class=\"shapelayer\"/></g><g class=\"infolayer\"><g class=\"legend\" pointer-events=\"all\" transform=\"translate(801.14,100)\"><rect class=\"bg\" shape-rendering=\"crispEdges\" width=\"387\" height=\"124\" x=\"0\" y=\"0\" style=\"stroke: rgb(68, 68, 68); stroke-opacity: 1; fill: rgb(255, 255, 255); fill-opacity: 1; stroke-width: 0px;\"/><g class=\"scrollbox\" transform=\"\" clip-path=\"url(#legend9d6f0e)\"><text class=\"legendtitletext\" text-anchor=\"start\" x=\"2\" y=\"18.2\" style=\"font-family: 'Open Sans', verdana, arial, sans-serif; font-size: 14px; fill: rgb(42, 63, 95); fill-opacity: 1; white-space: pre;\">Configuration</text><g class=\"groups\" transform=\"\"><g class=\"traces\" transform=\"translate(0,32.7)\" style=\"opacity: 1;\"><text class=\"legendtext\" text-anchor=\"start\" x=\"40\" y=\"4.680000000000001\" style=\"font-family: 'Open Sans', verdana, arial, sans-serif; font-size: 12px; fill: rgb(42, 63, 95); fill-opacity: 1; white-space: pre;\">Multicore: Non-persistent mode + kernel config</text><g class=\"layers\" style=\"opacity: 1;\"><g class=\"legendfill\"/><g class=\"legendlines\"><path class=\"js-line\" d=\"M5,0h30\" style=\"fill: none; stroke: rgb(99, 110, 250); stroke-opacity: 1; stroke-width: 2px;\"/></g><g class=\"legendsymbols\"><g class=\"legendpoints\"/></g></g><rect class=\"legendtoggle\" x=\"0\" y=\"-9.5\" width=\"381.484375\" height=\"19\" style=\"fill: rgb(0, 0, 0); fill-opacity: 0;\"/></g></g><g class=\"groups\" transform=\"\"><g class=\"traces\" transform=\"translate(0,51.7)\" style=\"opacity: 1;\"><text class=\"legendtext\" text-anchor=\"start\" x=\"40\" y=\"4.680000000000001\" style=\"font-family: 'Open Sans', verdana, arial, sans-serif; font-size: 12px; fill: rgb(42, 63, 95); fill-opacity: 1; white-space: pre;\">Multicore: Persistent mode/shared memory + kernel config</text><g class=\"layers\" style=\"opacity: 1;\"><g class=\"legendfill\"/><g class=\"legendlines\"><path class=\"js-line\" d=\"M5,0h30\" style=\"fill: none; stroke: rgb(239, 85, 59); stroke-opacity: 1; stroke-width: 2px;\"/></g><g class=\"legendsymbols\"><g class=\"legendpoints\"/></g></g><rect class=\"legendtoggle\" x=\"0\" y=\"-9.5\" width=\"381.484375\" height=\"19\" style=\"fill: rgb(0, 0, 0); fill-opacity: 0;\"/></g></g><g class=\"groups\" transform=\"\"><g class=\"traces\" transform=\"translate(0,70.7)\" style=\"opacity: 1;\"><text class=\"legendtext\" text-anchor=\"start\" x=\"40\" y=\"4.680000000000001\" style=\"font-family: 'Open Sans', verdana, arial, sans-serif; font-size: 12px; fill: rgb(42, 63, 95); fill-opacity: 1; white-space: pre;\">Multicore: Persistent mode/shared memory without kernel config</text><g class=\"layers\" style=\"opacity: 1;\"><g class=\"legendfill\"/><g class=\"legendlines\"><path class=\"js-line\" d=\"M5,0h30\" style=\"fill: none; stroke: rgb(0, 204, 150); stroke-opacity: 1; stroke-width: 2px;\"/></g><g class=\"legendsymbols\"><g class=\"legendpoints\"/></g></g><rect class=\"legendtoggle\" x=\"0\" y=\"-9.5\" width=\"381.484375\" height=\"19\" style=\"fill: rgb(0, 0, 0); fill-opacity: 0;\"/></g></g><g class=\"groups\" transform=\"\"><g class=\"traces\" transform=\"translate(0,89.7)\" style=\"opacity: 1;\"><text class=\"legendtext\" text-anchor=\"start\" x=\"40\" y=\"4.680000000000001\" style=\"font-family: 'Open Sans', verdana, arial, sans-serif; font-size: 12px; fill: rgb(42, 63, 95); fill-opacity: 1; white-space: pre;\">Singlecore: Non-persistent mode + kernel config</text><g class=\"layers\" style=\"opacity: 1;\"><g class=\"legendfill\"/><g class=\"legendlines\"><path class=\"js-line\" d=\"M5,0h30\" style=\"fill: none; stroke: rgb(171, 99, 250); stroke-opacity: 1; stroke-width: 2px;\"/></g><g class=\"legendsymbols\"><g class=\"legendpoints\"/></g></g><rect class=\"legendtoggle\" x=\"0\" y=\"-9.5\" width=\"381.484375\" height=\"19\" style=\"fill: rgb(0, 0, 0); fill-opacity: 0;\"/></g></g><g class=\"groups\" transform=\"\"><g class=\"traces\" transform=\"translate(0,108.7)\" style=\"opacity: 1;\"><text class=\"legendtext\" text-anchor=\"start\" x=\"40\" y=\"4.680000000000001\" style=\"font-family: 'Open Sans', verdana, arial, sans-serif; font-size: 12px; fill: rgb(42, 63, 95); fill-opacity: 1; white-space: pre;\">Singlecore: Persistent mode/shared memory + kernel config</text><g class=\"layers\" style=\"opacity: 1;\"><g class=\"legendfill\"/><g class=\"legendlines\"><path class=\"js-line\" d=\"M5,0h30\" style=\"fill: none; stroke: rgb(255, 161, 90); stroke-opacity: 1; stroke-width: 2px;\"/></g><g class=\"legendsymbols\"><g class=\"legendpoints\"/></g></g><rect class=\"legendtoggle\" x=\"0\" y=\"-9.5\" width=\"381.484375\" height=\"19\" style=\"fill: rgb(0, 0, 0); fill-opacity: 0;\"/></g></g></g><rect class=\"scrollbar\" rx=\"20\" ry=\"3\" width=\"0\" height=\"0\" x=\"0\" y=\"0\" style=\"fill: rgb(128, 139, 164); fill-opacity: 1;\"/></g><g class=\"g-gtitle\"><text class=\"gtitle\" x=\"60\" y=\"50\" text-anchor=\"start\" dy=\"0em\" style=\"font-family: 'Open Sans', verdana, arial, sans-serif; font-size: 17px; fill: rgb(42, 63, 95); opacity: 1; font-weight: normal; white-space: pre;\">Fuzzer performance</text></g><g class=\"g-xtitle\"><text class=\"xtitle\" x=\"433.5\" y=\"360.3\" text-anchor=\"middle\" style=\"font-family: 'Open Sans', verdana, arial, sans-serif; font-size: 14px; fill: rgb(42, 63, 95); opacity: 1; font-weight: normal; white-space: pre;\">Number of parallel fuzzers</text></g><g class=\"g-ytitle\"><text class=\"ytitle\" transform=\"rotate(-90,29.559375000000003,210)\" x=\"29.559375000000003\" y=\"210\" text-anchor=\"middle\" style=\"font-family: 'Open Sans', verdana, arial, sans-serif; font-size: 14px; fill: rgb(42, 63, 95); opacity: 1; font-weight: normal; white-space: pre;\">Fuzz target executions per second</text></g></g></svg>"
|
||
]
|
||
},
|
||
"metadata": {},
|
||
"output_type": "display_data"
|
||
}
|
||
],
|
||
"source": [
|
||
"import numpy as np\n",
|
||
"pd.options.plotting.backend = \"plotly\"\n",
|
||
"\n",
|
||
"# Right now our table has absolute values of execs per sec, but it's more useful\n",
|
||
"# to show relative perf (vs 1.0x baseline)\n",
|
||
"pivotdf = graphdf.pivot(index=\"parallel_fuzzers\", columns=\"label\", values=\"execs_per_sec\")\n",
|
||
"fig = pivotdf.plot(\n",
|
||
" title=\"Fuzzer performance\",\n",
|
||
" labels={\n",
|
||
" \"label\": \"Configuration\",\n",
|
||
" \"parallel_fuzzers\": \"Number of parallel fuzzers\",\n",
|
||
" \"value\": \"Fuzz target executions per second\"\n",
|
||
" }\n",
|
||
")\n",
|
||
"\n",
|
||
"# Compute tick values and their labels for the primary Y-axis\n",
|
||
"tickvals = np.linspace(graphdf['execs_per_sec'].min(), graphdf['execs_per_sec'].max(), 6)\n",
|
||
"ticktext = [f\"{val:.0f}x\" for val in tickvals / graphdf['execs_per_sec'].min()]\n",
|
||
"# Update the primary Y-axis with custom tick labels\n",
|
||
"fig.update_yaxes(tickvals=tickvals, ticktext=ticktext)\n",
|
||
"fig.update_xaxes(tickvals=list(range(1,36+1)))\n",
|
||
"fig.update_layout(width=1200, height=400)\n",
|
||
"fig.show(\"svg\")\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"Here's what the table that produced this graph looks like:"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 148,
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"data": {
|
||
"text/html": [
|
||
"<div>\n",
|
||
"<style scoped>\n",
|
||
" .dataframe tbody tr th:only-of-type {\n",
|
||
" vertical-align: middle;\n",
|
||
" }\n",
|
||
"\n",
|
||
" .dataframe tbody tr th {\n",
|
||
" vertical-align: top;\n",
|
||
" }\n",
|
||
"\n",
|
||
" .dataframe thead th {\n",
|
||
" text-align: right;\n",
|
||
" }\n",
|
||
"</style>\n",
|
||
"<table border=\"1\" class=\"dataframe\">\n",
|
||
" <thead>\n",
|
||
" <tr style=\"text-align: right;\">\n",
|
||
" <th>label</th>\n",
|
||
" <th>Multicore: Non-persistent mode + kernel config</th>\n",
|
||
" <th>Multicore: Persistent mode/shared memory + kernel config</th>\n",
|
||
" <th>Multicore: Persistent mode/shared memory without kernel config</th>\n",
|
||
" <th>Singlecore: Non-persistent mode + kernel config</th>\n",
|
||
" <th>Singlecore: Persistent mode/shared memory + kernel config</th>\n",
|
||
" </tr>\n",
|
||
" <tr>\n",
|
||
" <th>parallel_fuzzers</th>\n",
|
||
" <th></th>\n",
|
||
" <th></th>\n",
|
||
" <th></th>\n",
|
||
" <th></th>\n",
|
||
" <th></th>\n",
|
||
" </tr>\n",
|
||
" </thead>\n",
|
||
" <tbody>\n",
|
||
" <tr>\n",
|
||
" <th>1.0</th>\n",
|
||
" <td>10714.79</td>\n",
|
||
" <td>120293.77</td>\n",
|
||
" <td>90641.62</td>\n",
|
||
" <td>9845.64</td>\n",
|
||
" <td>125682.73</td>\n",
|
||
" </tr>\n",
|
||
" <tr>\n",
|
||
" <th>2.0</th>\n",
|
||
" <td>20493.07</td>\n",
|
||
" <td>231429.96</td>\n",
|
||
" <td>178184.19</td>\n",
|
||
" <td>9845.64</td>\n",
|
||
" <td>125682.73</td>\n",
|
||
" </tr>\n",
|
||
" <tr>\n",
|
||
" <th>3.0</th>\n",
|
||
" <td>29660.06</td>\n",
|
||
" <td>346759.33</td>\n",
|
||
" <td>262652.86</td>\n",
|
||
" <td>9845.64</td>\n",
|
||
" <td>125682.73</td>\n",
|
||
" </tr>\n",
|
||
" <tr>\n",
|
||
" <th>4.0</th>\n",
|
||
" <td>37875.57</td>\n",
|
||
" <td>455340.06</td>\n",
|
||
" <td>339119.32</td>\n",
|
||
" <td>9845.64</td>\n",
|
||
" <td>125682.73</td>\n",
|
||
" </tr>\n",
|
||
" <tr>\n",
|
||
" <th>5.0</th>\n",
|
||
" <td>46326.75</td>\n",
|
||
" <td>568405.15</td>\n",
|
||
" <td>420239.94</td>\n",
|
||
" <td>9845.64</td>\n",
|
||
" <td>125682.73</td>\n",
|
||
" </tr>\n",
|
||
" <tr>\n",
|
||
" <th>6.0</th>\n",
|
||
" <td>54595.48</td>\n",
|
||
" <td>678030.96</td>\n",
|
||
" <td>498062.02</td>\n",
|
||
" <td>9845.64</td>\n",
|
||
" <td>125682.73</td>\n",
|
||
" </tr>\n",
|
||
" <tr>\n",
|
||
" <th>7.0</th>\n",
|
||
" <td>62720.98</td>\n",
|
||
" <td>782585.04</td>\n",
|
||
" <td>578495.44</td>\n",
|
||
" <td>9845.64</td>\n",
|
||
" <td>125682.73</td>\n",
|
||
" </tr>\n",
|
||
" <tr>\n",
|
||
" <th>8.0</th>\n",
|
||
" <td>70777.99</td>\n",
|
||
" <td>893618.35</td>\n",
|
||
" <td>661836.22</td>\n",
|
||
" <td>9845.64</td>\n",
|
||
" <td>125682.73</td>\n",
|
||
" </tr>\n",
|
||
" <tr>\n",
|
||
" <th>9.0</th>\n",
|
||
" <td>74236.02</td>\n",
|
||
" <td>956026.15</td>\n",
|
||
" <td>684808.49</td>\n",
|
||
" <td>9845.64</td>\n",
|
||
" <td>125682.73</td>\n",
|
||
" </tr>\n",
|
||
" <tr>\n",
|
||
" <th>10.0</th>\n",
|
||
" <td>78134.94</td>\n",
|
||
" <td>984942.13</td>\n",
|
||
" <td>707094.65</td>\n",
|
||
" <td>9845.64</td>\n",
|
||
" <td>125682.73</td>\n",
|
||
" </tr>\n",
|
||
" <tr>\n",
|
||
" <th>11.0</th>\n",
|
||
" <td>81886.33</td>\n",
|
||
" <td>1016758.62</td>\n",
|
||
" <td>732106.17</td>\n",
|
||
" <td>9845.64</td>\n",
|
||
" <td>125682.73</td>\n",
|
||
" </tr>\n",
|
||
" <tr>\n",
|
||
" <th>12.0</th>\n",
|
||
" <td>85923.44</td>\n",
|
||
" <td>1053087.90</td>\n",
|
||
" <td>752910.17</td>\n",
|
||
" <td>9845.64</td>\n",
|
||
" <td>125682.73</td>\n",
|
||
" </tr>\n",
|
||
" <tr>\n",
|
||
" <th>13.0</th>\n",
|
||
" <td>89696.95</td>\n",
|
||
" <td>1085797.87</td>\n",
|
||
" <td>776179.85</td>\n",
|
||
" <td>9845.64</td>\n",
|
||
" <td>125682.73</td>\n",
|
||
" </tr>\n",
|
||
" <tr>\n",
|
||
" <th>14.0</th>\n",
|
||
" <td>93540.52</td>\n",
|
||
" <td>1110640.20</td>\n",
|
||
" <td>797520.58</td>\n",
|
||
" <td>9845.64</td>\n",
|
||
" <td>125682.73</td>\n",
|
||
" </tr>\n",
|
||
" <tr>\n",
|
||
" <th>15.0</th>\n",
|
||
" <td>97641.51</td>\n",
|
||
" <td>1138984.22</td>\n",
|
||
" <td>822235.41</td>\n",
|
||
" <td>9845.64</td>\n",
|
||
" <td>125682.73</td>\n",
|
||
" </tr>\n",
|
||
" <tr>\n",
|
||
" <th>16.0</th>\n",
|
||
" <td>101692.65</td>\n",
|
||
" <td>1168943.19</td>\n",
|
||
" <td>843897.51</td>\n",
|
||
" <td>9845.64</td>\n",
|
||
" <td>125682.73</td>\n",
|
||
" </tr>\n",
|
||
" <tr>\n",
|
||
" <th>17.0</th>\n",
|
||
" <td>101236.75</td>\n",
|
||
" <td>1135093.91</td>\n",
|
||
" <td>843177.15</td>\n",
|
||
" <td>9845.64</td>\n",
|
||
" <td>125682.73</td>\n",
|
||
" </tr>\n",
|
||
" <tr>\n",
|
||
" <th>18.0</th>\n",
|
||
" <td>101006.28</td>\n",
|
||
" <td>1160430.45</td>\n",
|
||
" <td>844779.09</td>\n",
|
||
" <td>9845.64</td>\n",
|
||
" <td>125682.73</td>\n",
|
||
" </tr>\n",
|
||
" <tr>\n",
|
||
" <th>19.0</th>\n",
|
||
" <td>99952.26</td>\n",
|
||
" <td>1155769.97</td>\n",
|
||
" <td>846060.74</td>\n",
|
||
" <td>9845.64</td>\n",
|
||
" <td>125682.73</td>\n",
|
||
" </tr>\n",
|
||
" <tr>\n",
|
||
" <th>20.0</th>\n",
|
||
" <td>99798.64</td>\n",
|
||
" <td>1150156.26</td>\n",
|
||
" <td>847556.23</td>\n",
|
||
" <td>9845.64</td>\n",
|
||
" <td>125682.73</td>\n",
|
||
" </tr>\n",
|
||
" <tr>\n",
|
||
" <th>21.0</th>\n",
|
||
" <td>99018.86</td>\n",
|
||
" <td>1136873.58</td>\n",
|
||
" <td>844022.97</td>\n",
|
||
" <td>9845.64</td>\n",
|
||
" <td>125682.73</td>\n",
|
||
" </tr>\n",
|
||
" <tr>\n",
|
||
" <th>22.0</th>\n",
|
||
" <td>98600.87</td>\n",
|
||
" <td>1112404.25</td>\n",
|
||
" <td>845818.70</td>\n",
|
||
" <td>9845.64</td>\n",
|
||
" <td>125682.73</td>\n",
|
||
" </tr>\n",
|
||
" <tr>\n",
|
||
" <th>23.0</th>\n",
|
||
" <td>98634.02</td>\n",
|
||
" <td>1143131.72</td>\n",
|
||
" <td>844118.27</td>\n",
|
||
" <td>9845.64</td>\n",
|
||
" <td>125682.73</td>\n",
|
||
" </tr>\n",
|
||
" <tr>\n",
|
||
" <th>24.0</th>\n",
|
||
" <td>98352.90</td>\n",
|
||
" <td>1143931.38</td>\n",
|
||
" <td>837189.02</td>\n",
|
||
" <td>9845.64</td>\n",
|
||
" <td>125682.73</td>\n",
|
||
" </tr>\n",
|
||
" <tr>\n",
|
||
" <th>25.0</th>\n",
|
||
" <td>98118.63</td>\n",
|
||
" <td>1102090.61</td>\n",
|
||
" <td>834712.31</td>\n",
|
||
" <td>9845.64</td>\n",
|
||
" <td>125682.73</td>\n",
|
||
" </tr>\n",
|
||
" <tr>\n",
|
||
" <th>26.0</th>\n",
|
||
" <td>97752.45</td>\n",
|
||
" <td>1116518.70</td>\n",
|
||
" <td>836344.12</td>\n",
|
||
" <td>9845.64</td>\n",
|
||
" <td>125682.73</td>\n",
|
||
" </tr>\n",
|
||
" <tr>\n",
|
||
" <th>27.0</th>\n",
|
||
" <td>97864.07</td>\n",
|
||
" <td>1099224.19</td>\n",
|
||
" <td>827784.91</td>\n",
|
||
" <td>9845.64</td>\n",
|
||
" <td>125682.73</td>\n",
|
||
" </tr>\n",
|
||
" <tr>\n",
|
||
" <th>28.0</th>\n",
|
||
" <td>97821.80</td>\n",
|
||
" <td>1114945.37</td>\n",
|
||
" <td>828641.27</td>\n",
|
||
" <td>9845.64</td>\n",
|
||
" <td>125682.73</td>\n",
|
||
" </tr>\n",
|
||
" <tr>\n",
|
||
" <th>29.0</th>\n",
|
||
" <td>97564.87</td>\n",
|
||
" <td>1110889.91</td>\n",
|
||
" <td>826123.67</td>\n",
|
||
" <td>9845.64</td>\n",
|
||
" <td>125682.73</td>\n",
|
||
" </tr>\n",
|
||
" <tr>\n",
|
||
" <th>30.0</th>\n",
|
||
" <td>98508.10</td>\n",
|
||
" <td>1058548.28</td>\n",
|
||
" <td>817765.77</td>\n",
|
||
" <td>9845.64</td>\n",
|
||
" <td>125682.73</td>\n",
|
||
" </tr>\n",
|
||
" <tr>\n",
|
||
" <th>31.0</th>\n",
|
||
" <td>98238.96</td>\n",
|
||
" <td>1119804.85</td>\n",
|
||
" <td>816556.66</td>\n",
|
||
" <td>9845.64</td>\n",
|
||
" <td>125682.73</td>\n",
|
||
" </tr>\n",
|
||
" <tr>\n",
|
||
" <th>32.0</th>\n",
|
||
" <td>98363.93</td>\n",
|
||
" <td>1118828.99</td>\n",
|
||
" <td>812661.77</td>\n",
|
||
" <td>9845.64</td>\n",
|
||
" <td>125682.73</td>\n",
|
||
" </tr>\n",
|
||
" <tr>\n",
|
||
" <th>33.0</th>\n",
|
||
" <td>96758.69</td>\n",
|
||
" <td>1093426.61</td>\n",
|
||
" <td>805352.16</td>\n",
|
||
" <td>9845.64</td>\n",
|
||
" <td>125682.73</td>\n",
|
||
" </tr>\n",
|
||
" <tr>\n",
|
||
" <th>34.0</th>\n",
|
||
" <td>96327.00</td>\n",
|
||
" <td>1108123.59</td>\n",
|
||
" <td>815888.26</td>\n",
|
||
" <td>9845.64</td>\n",
|
||
" <td>125682.73</td>\n",
|
||
" </tr>\n",
|
||
" <tr>\n",
|
||
" <th>35.0</th>\n",
|
||
" <td>95913.98</td>\n",
|
||
" <td>1041486.52</td>\n",
|
||
" <td>812348.56</td>\n",
|
||
" <td>9845.64</td>\n",
|
||
" <td>125682.73</td>\n",
|
||
" </tr>\n",
|
||
" <tr>\n",
|
||
" <th>36.0</th>\n",
|
||
" <td>95871.39</td>\n",
|
||
" <td>1092395.61</td>\n",
|
||
" <td>817278.03</td>\n",
|
||
" <td>9845.64</td>\n",
|
||
" <td>125682.73</td>\n",
|
||
" </tr>\n",
|
||
" </tbody>\n",
|
||
"</table>\n",
|
||
"</div>"
|
||
],
|
||
"text/plain": [
|
||
"label Multicore: Non-persistent mode + kernel config \\\n",
|
||
"parallel_fuzzers \n",
|
||
"1.0 10714.79 \n",
|
||
"2.0 20493.07 \n",
|
||
"3.0 29660.06 \n",
|
||
"4.0 37875.57 \n",
|
||
"5.0 46326.75 \n",
|
||
"6.0 54595.48 \n",
|
||
"7.0 62720.98 \n",
|
||
"8.0 70777.99 \n",
|
||
"9.0 74236.02 \n",
|
||
"10.0 78134.94 \n",
|
||
"11.0 81886.33 \n",
|
||
"12.0 85923.44 \n",
|
||
"13.0 89696.95 \n",
|
||
"14.0 93540.52 \n",
|
||
"15.0 97641.51 \n",
|
||
"16.0 101692.65 \n",
|
||
"17.0 101236.75 \n",
|
||
"18.0 101006.28 \n",
|
||
"19.0 99952.26 \n",
|
||
"20.0 99798.64 \n",
|
||
"21.0 99018.86 \n",
|
||
"22.0 98600.87 \n",
|
||
"23.0 98634.02 \n",
|
||
"24.0 98352.90 \n",
|
||
"25.0 98118.63 \n",
|
||
"26.0 97752.45 \n",
|
||
"27.0 97864.07 \n",
|
||
"28.0 97821.80 \n",
|
||
"29.0 97564.87 \n",
|
||
"30.0 98508.10 \n",
|
||
"31.0 98238.96 \n",
|
||
"32.0 98363.93 \n",
|
||
"33.0 96758.69 \n",
|
||
"34.0 96327.00 \n",
|
||
"35.0 95913.98 \n",
|
||
"36.0 95871.39 \n",
|
||
"\n",
|
||
"label Multicore: Persistent mode/shared memory + kernel config \\\n",
|
||
"parallel_fuzzers \n",
|
||
"1.0 120293.77 \n",
|
||
"2.0 231429.96 \n",
|
||
"3.0 346759.33 \n",
|
||
"4.0 455340.06 \n",
|
||
"5.0 568405.15 \n",
|
||
"6.0 678030.96 \n",
|
||
"7.0 782585.04 \n",
|
||
"8.0 893618.35 \n",
|
||
"9.0 956026.15 \n",
|
||
"10.0 984942.13 \n",
|
||
"11.0 1016758.62 \n",
|
||
"12.0 1053087.90 \n",
|
||
"13.0 1085797.87 \n",
|
||
"14.0 1110640.20 \n",
|
||
"15.0 1138984.22 \n",
|
||
"16.0 1168943.19 \n",
|
||
"17.0 1135093.91 \n",
|
||
"18.0 1160430.45 \n",
|
||
"19.0 1155769.97 \n",
|
||
"20.0 1150156.26 \n",
|
||
"21.0 1136873.58 \n",
|
||
"22.0 1112404.25 \n",
|
||
"23.0 1143131.72 \n",
|
||
"24.0 1143931.38 \n",
|
||
"25.0 1102090.61 \n",
|
||
"26.0 1116518.70 \n",
|
||
"27.0 1099224.19 \n",
|
||
"28.0 1114945.37 \n",
|
||
"29.0 1110889.91 \n",
|
||
"30.0 1058548.28 \n",
|
||
"31.0 1119804.85 \n",
|
||
"32.0 1118828.99 \n",
|
||
"33.0 1093426.61 \n",
|
||
"34.0 1108123.59 \n",
|
||
"35.0 1041486.52 \n",
|
||
"36.0 1092395.61 \n",
|
||
"\n",
|
||
"label Multicore: Persistent mode/shared memory without kernel config \\\n",
|
||
"parallel_fuzzers \n",
|
||
"1.0 90641.62 \n",
|
||
"2.0 178184.19 \n",
|
||
"3.0 262652.86 \n",
|
||
"4.0 339119.32 \n",
|
||
"5.0 420239.94 \n",
|
||
"6.0 498062.02 \n",
|
||
"7.0 578495.44 \n",
|
||
"8.0 661836.22 \n",
|
||
"9.0 684808.49 \n",
|
||
"10.0 707094.65 \n",
|
||
"11.0 732106.17 \n",
|
||
"12.0 752910.17 \n",
|
||
"13.0 776179.85 \n",
|
||
"14.0 797520.58 \n",
|
||
"15.0 822235.41 \n",
|
||
"16.0 843897.51 \n",
|
||
"17.0 843177.15 \n",
|
||
"18.0 844779.09 \n",
|
||
"19.0 846060.74 \n",
|
||
"20.0 847556.23 \n",
|
||
"21.0 844022.97 \n",
|
||
"22.0 845818.70 \n",
|
||
"23.0 844118.27 \n",
|
||
"24.0 837189.02 \n",
|
||
"25.0 834712.31 \n",
|
||
"26.0 836344.12 \n",
|
||
"27.0 827784.91 \n",
|
||
"28.0 828641.27 \n",
|
||
"29.0 826123.67 \n",
|
||
"30.0 817765.77 \n",
|
||
"31.0 816556.66 \n",
|
||
"32.0 812661.77 \n",
|
||
"33.0 805352.16 \n",
|
||
"34.0 815888.26 \n",
|
||
"35.0 812348.56 \n",
|
||
"36.0 817278.03 \n",
|
||
"\n",
|
||
"label Singlecore: Non-persistent mode + kernel config \\\n",
|
||
"parallel_fuzzers \n",
|
||
"1.0 9845.64 \n",
|
||
"2.0 9845.64 \n",
|
||
"3.0 9845.64 \n",
|
||
"4.0 9845.64 \n",
|
||
"5.0 9845.64 \n",
|
||
"6.0 9845.64 \n",
|
||
"7.0 9845.64 \n",
|
||
"8.0 9845.64 \n",
|
||
"9.0 9845.64 \n",
|
||
"10.0 9845.64 \n",
|
||
"11.0 9845.64 \n",
|
||
"12.0 9845.64 \n",
|
||
"13.0 9845.64 \n",
|
||
"14.0 9845.64 \n",
|
||
"15.0 9845.64 \n",
|
||
"16.0 9845.64 \n",
|
||
"17.0 9845.64 \n",
|
||
"18.0 9845.64 \n",
|
||
"19.0 9845.64 \n",
|
||
"20.0 9845.64 \n",
|
||
"21.0 9845.64 \n",
|
||
"22.0 9845.64 \n",
|
||
"23.0 9845.64 \n",
|
||
"24.0 9845.64 \n",
|
||
"25.0 9845.64 \n",
|
||
"26.0 9845.64 \n",
|
||
"27.0 9845.64 \n",
|
||
"28.0 9845.64 \n",
|
||
"29.0 9845.64 \n",
|
||
"30.0 9845.64 \n",
|
||
"31.0 9845.64 \n",
|
||
"32.0 9845.64 \n",
|
||
"33.0 9845.64 \n",
|
||
"34.0 9845.64 \n",
|
||
"35.0 9845.64 \n",
|
||
"36.0 9845.64 \n",
|
||
"\n",
|
||
"label Singlecore: Persistent mode/shared memory + kernel config \n",
|
||
"parallel_fuzzers \n",
|
||
"1.0 125682.73 \n",
|
||
"2.0 125682.73 \n",
|
||
"3.0 125682.73 \n",
|
||
"4.0 125682.73 \n",
|
||
"5.0 125682.73 \n",
|
||
"6.0 125682.73 \n",
|
||
"7.0 125682.73 \n",
|
||
"8.0 125682.73 \n",
|
||
"9.0 125682.73 \n",
|
||
"10.0 125682.73 \n",
|
||
"11.0 125682.73 \n",
|
||
"12.0 125682.73 \n",
|
||
"13.0 125682.73 \n",
|
||
"14.0 125682.73 \n",
|
||
"15.0 125682.73 \n",
|
||
"16.0 125682.73 \n",
|
||
"17.0 125682.73 \n",
|
||
"18.0 125682.73 \n",
|
||
"19.0 125682.73 \n",
|
||
"20.0 125682.73 \n",
|
||
"21.0 125682.73 \n",
|
||
"22.0 125682.73 \n",
|
||
"23.0 125682.73 \n",
|
||
"24.0 125682.73 \n",
|
||
"25.0 125682.73 \n",
|
||
"26.0 125682.73 \n",
|
||
"27.0 125682.73 \n",
|
||
"28.0 125682.73 \n",
|
||
"29.0 125682.73 \n",
|
||
"30.0 125682.73 \n",
|
||
"31.0 125682.73 \n",
|
||
"32.0 125682.73 \n",
|
||
"33.0 125682.73 \n",
|
||
"34.0 125682.73 \n",
|
||
"35.0 125682.73 \n",
|
||
"36.0 125682.73 "
|
||
]
|
||
},
|
||
"execution_count": 148,
|
||
"metadata": {},
|
||
"output_type": "execute_result"
|
||
}
|
||
],
|
||
"source": [
|
||
"pivotdf"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"You can totally ignore the code cell directly below (unless you're curious). It's just preparing Markdown for the block below it to render. Jupyter Notebooks aren't able to use code variables inside Markdown blocks, so I have to do this instead."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 149,
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"data": {
|
||
"text/markdown": [
|
||
"\n",
|
||
"### Line graph analysis\n",
|
||
"Here are a few things that jump out from the graph above. Let's start at the bottom of the graph.\n",
|
||
"\n",
|
||
"#### test-instr vs. test-instr-persist-shmem\n",
|
||
"\n",
|
||
"This graph is scaled so that the single-core, non-persistent-mode performance (9845 execs per second) is\n",
|
||
"represented as **1.0x**. If you build and run a fuzzer without creating a persistent mode harness for it, and without running fuzzers in parallel, this is the performance\n",
|
||
"you get on this machine.\n",
|
||
"\n",
|
||
"#### Multicore test-instr\n",
|
||
"\n",
|
||
"By running as many parallel fuzzers are there are CPU threads, we can reach 101692 execs per second, which is **10.3x** that base speed.\n",
|
||
"\n",
|
||
"#### Persistent mode + shared memory\n",
|
||
"\n",
|
||
"##### Singlecore\n",
|
||
"\n",
|
||
"By modifying the harness to use persistent mode with shared memory as described [here](https://github.com/AFLplusplus/AFLplusplus/blob/stable/instrumentation/README.persistent_mode.md#4-persistent-mode),\n",
|
||
"we end up with **12.8x** base speed. So -- perhaps counter-intuively -- if you have a choice between switching to using multiple cores or rewriting\n",
|
||
"the harness to use persistent mode on a single core, it is better (at least on this machine) to use persistent mode on a single core, than to use non-persistent mode on all cores.\n",
|
||
"\n",
|
||
"##### Multicore\n",
|
||
"\n",
|
||
"By scaling up that persistent mode with shared memory harness across cores, and with kernel mitigations still turned on (see next section), we get to\n",
|
||
"**86.1x** base speed.\n",
|
||
"\n",
|
||
"#### Kernel config\n",
|
||
"\n",
|
||
"By \"kernel config\", I'm referring to booting the Linux kernel with `mitigations=off`, which is a meta-parameter for disabling *all* hardware vulnerability meltdowns (such as Spectre,\n",
|
||
"Meltdown, Retbleed, etc) introduced in Linux v5.2. Disabling these results in a `execs_per_sec` increase of 321386 execs -- the difference between\n",
|
||
"118.7x (mitigations off) and 86.1x (mitigations on) base speed. Turning on mitigations\n",
|
||
"reduced the overall performance by 27%!\n",
|
||
"\n",
|
||
"One way to think about this is that the mitigations turn this 16-thread CPU into a 7-thread CPU, since the number of execs reached with 16 threads and mitigations on is around the same\n",
|
||
"number of execs reached with 7 threads and mitigations off.\n",
|
||
"\n",
|
||
"Or if we want to think in terms of cores, then the average number of execs gained per core in the initial eight is 110474 execs per sec, but the loss due to\n",
|
||
"mitigations is 321386 execs per sec, which is the averaged performance of 2.9 cores.\n",
|
||
"\n",
|
||
"With kernel mitigations turned off, we reach our highest available execs_per_sec speed on this machine, which is **118.7x** higher\n",
|
||
"than where we started from.\n",
|
||
"\n",
|
||
"#### How many parallel fuzzers should we use on this machine?\n",
|
||
"\n",
|
||
"* Using >16 is worse than using 16. Makes sense.\n",
|
||
"* So, we should use the number of CPUs in /proc/cpuinfo (threads) to get the best performance. But if we did halve the number of\n",
|
||
" fuzzers, we would surprisingly only lose 23%\n",
|
||
" of performance. This could be a good tradeoff in terms of cost.\n"
|
||
],
|
||
"text/plain": [
|
||
"<IPython.core.display.Markdown object>"
|
||
]
|
||
},
|
||
"execution_count": 149,
|
||
"metadata": {},
|
||
"output_type": "execute_result"
|
||
}
|
||
],
|
||
"source": [
|
||
"# (You can ignore reading this code cell.)\n",
|
||
"from IPython.display import Markdown as md\n",
|
||
"singlecore_base_execs = pivotdf.iloc[0][\"Singlecore: Non-persistent mode + kernel config\"]\n",
|
||
"singlecore_persist_execs = pivotdf.iloc[0][\"Singlecore: Persistent mode/shared memory + kernel config\"]\n",
|
||
"multicore_fuzzers_max_execs = int(pivotdf[\"Multicore: Persistent mode/shared memory + kernel config\"].idxmax())\n",
|
||
"multicore_base_max_execs = pivotdf[\"Multicore: Non-persistent mode + kernel config\"].max()\n",
|
||
"factor_for_execs = lambda execs: round(execs / singlecore_base_execs, 1)\n",
|
||
"\n",
|
||
"multicore_persistent_without_mitigations_label = \"Multicore: Persistent mode/shared memory + kernel config\"\n",
|
||
"multicore_max_execs_mitigations_off = pivotdf[multicore_persistent_without_mitigations_label].max()\n",
|
||
"multicore_max_execs_mitigations_off_only_cores = pivotdf.loc[multicore_fuzzers_max_execs / 2][multicore_persistent_without_mitigations_label]\n",
|
||
"multicore_max_execs_mitigations_on = pivotdf[\"Multicore: Persistent mode/shared memory without kernel config\"].max()\n",
|
||
"multicore_avg_gain_per_core = pivotdf.loc[pivotdf.index <= 8][\"Multicore: Persistent mode/shared memory + kernel config\"].diff().dropna().mean()\n",
|
||
"mitigations_off_increase = int(multicore_max_execs_mitigations_off - multicore_max_execs_mitigations_on)\n",
|
||
"\n",
|
||
"md(f\"\"\"\n",
|
||
"### Line graph analysis\n",
|
||
"Here are a few things that jump out from the graph above. Let's start at the bottom of the graph.\n",
|
||
"\n",
|
||
"#### test-instr vs. test-instr-persist-shmem\n",
|
||
"\n",
|
||
"This graph is scaled so that the single-core, non-persistent-mode performance ({int(singlecore_base_execs)} execs per second) is\n",
|
||
"represented as **1.0x**. If you build and run a fuzzer without creating a persistent mode harness for it, and without running fuzzers in parallel, this is the performance\n",
|
||
"you get on this machine.\n",
|
||
"\n",
|
||
"#### Multicore test-instr\n",
|
||
"\n",
|
||
"By running as many parallel fuzzers are there are CPU threads, we can reach {int(multicore_base_max_execs)} execs per second, which is **{factor_for_execs(multicore_base_max_execs)}x** that base speed.\n",
|
||
"\n",
|
||
"#### Persistent mode + shared memory\n",
|
||
"\n",
|
||
"##### Singlecore\n",
|
||
"\n",
|
||
"By modifying the harness to use persistent mode with shared memory as described [here](https://github.com/AFLplusplus/AFLplusplus/blob/stable/instrumentation/README.persistent_mode.md#4-persistent-mode),\n",
|
||
"we end up with **{factor_for_execs(singlecore_persist_execs)}x** base speed. So -- perhaps counter-intuively -- if you have a choice between switching to using multiple cores or rewriting\n",
|
||
"the harness to use persistent mode on a single core, it is better (at least on this machine) to use persistent mode on a single core, than to use non-persistent mode on all cores.\n",
|
||
"\n",
|
||
"##### Multicore\n",
|
||
"\n",
|
||
"By scaling up that persistent mode with shared memory harness across cores, and with kernel mitigations still turned on (see next section), we get to\n",
|
||
"**{factor_for_execs(multicore_max_execs_mitigations_on)}x** base speed.\n",
|
||
"\n",
|
||
"#### Kernel config\n",
|
||
"\n",
|
||
"By \"kernel config\", I'm referring to booting the Linux kernel with `mitigations=off`, which is a meta-parameter for disabling *all* hardware vulnerability meltdowns (such as Spectre,\n",
|
||
"Meltdown, Retbleed, etc) introduced in Linux v5.2. Disabling these results in a `execs_per_sec` increase of {mitigations_off_increase} execs -- the difference between\n",
|
||
"{factor_for_execs(multicore_max_execs_mitigations_off)}x (mitigations off) and {factor_for_execs(multicore_max_execs_mitigations_on)}x (mitigations on) base speed. Turning on mitigations\n",
|
||
"reduced the overall performance by {abs(round(((multicore_max_execs_mitigations_on - multicore_max_execs_mitigations_off) / multicore_max_execs_mitigations_off) * 100))}%!\n",
|
||
"\n",
|
||
"One way to think about this is that the mitigations turn this 16-thread CPU into a 7-thread CPU, since the number of execs reached with 16 threads and mitigations on is around the same\n",
|
||
"number of execs reached with 7 threads and mitigations off.\n",
|
||
"\n",
|
||
"Or if we want to think in terms of cores, then the average number of execs gained per core in the initial eight is {int(multicore_avg_gain_per_core)} execs per sec, but the loss due to\n",
|
||
"mitigations is {mitigations_off_increase} execs per sec, which is the averaged performance of {round(mitigations_off_increase / multicore_avg_gain_per_core, 1)} cores.\n",
|
||
"\n",
|
||
"With kernel mitigations turned off, we reach our highest available execs_per_sec speed on this machine, which is **{factor_for_execs(multicore_max_execs_mitigations_off)}x** higher\n",
|
||
"than where we started from.\n",
|
||
"\n",
|
||
"#### How many parallel fuzzers should we use on this machine?\n",
|
||
"\n",
|
||
"* Using >16 is worse than using 16. Makes sense.\n",
|
||
"* So, we should use the number of CPUs in /proc/cpuinfo (threads) to get the best performance. But if we did halve the number of\n",
|
||
" fuzzers, we would surprisingly only lose {abs(int(((multicore_max_execs_mitigations_off_only_cores - multicore_max_execs_mitigations_off) / multicore_max_execs_mitigations_off) * 100))}%\n",
|
||
" of performance. This could be a good tradeoff in terms of cost.\n",
|
||
"\"\"\")\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"### Example with more cores\n",
|
||
"\n",
|
||
"While there was some nuance here, the answer was pretty straightforward -- use the number of CPU threads you have access to. What if there were more threads? Here the experiment is repeated on an AWS EC2 \"r6a.48xlarge\" spot instance with 192 vCPUs:"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 150,
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"data": {
|
||
"text/html": [
|
||
"<div>\n",
|
||
"<style scoped>\n",
|
||
" .dataframe tbody tr th:only-of-type {\n",
|
||
" vertical-align: middle;\n",
|
||
" }\n",
|
||
"\n",
|
||
" .dataframe tbody tr th {\n",
|
||
" vertical-align: top;\n",
|
||
" }\n",
|
||
"\n",
|
||
" .dataframe thead th {\n",
|
||
" text-align: right;\n",
|
||
" }\n",
|
||
"</style>\n",
|
||
"<table border=\"1\" class=\"dataframe\">\n",
|
||
" <thead>\n",
|
||
" <tr style=\"text-align: right;\">\n",
|
||
" <th></th>\n",
|
||
" <th>config.afl_persistent_config</th>\n",
|
||
" <th>config.afl_system_config</th>\n",
|
||
" <th>config.afl_version</th>\n",
|
||
" <th>config.comment</th>\n",
|
||
" <th>config.compiler</th>\n",
|
||
" <th>config.target_arch</th>\n",
|
||
" <th>hardware.cpu_fastest_core_mhz</th>\n",
|
||
" <th>hardware.cpu_model</th>\n",
|
||
" <th>hardware.cpu_threads</th>\n",
|
||
" <th>targets.test-instr-persist-shmem.multicore.execs_per_sec</th>\n",
|
||
" <th>targets.test-instr-persist-shmem.multicore.execs_total</th>\n",
|
||
" <th>targets.test-instr-persist-shmem.multicore.fuzzers_used</th>\n",
|
||
" </tr>\n",
|
||
" </thead>\n",
|
||
" <tbody>\n",
|
||
" <tr>\n",
|
||
" <th>223</th>\n",
|
||
" <td>True</td>\n",
|
||
" <td>True</td>\n",
|
||
" <td>++4.09a</td>\n",
|
||
" <td>AWS EC2 r6a.48xlarge spot instance</td>\n",
|
||
" <td>clang version 15.0.7 (Amazon Linux 15.0.7-3.am...</td>\n",
|
||
" <td>x86_64-amazon-linux-gnu</td>\n",
|
||
" <td>3514.326</td>\n",
|
||
" <td>AMD EPYC 7R13 Processor</td>\n",
|
||
" <td>192</td>\n",
|
||
" <td>119469.35</td>\n",
|
||
" <td>1194813.0</td>\n",
|
||
" <td>1.0</td>\n",
|
||
" </tr>\n",
|
||
" <tr>\n",
|
||
" <th>224</th>\n",
|
||
" <td>True</td>\n",
|
||
" <td>True</td>\n",
|
||
" <td>++4.09a</td>\n",
|
||
" <td>AWS EC2 r6a.48xlarge spot instance</td>\n",
|
||
" <td>clang version 15.0.7 (Amazon Linux 15.0.7-3.am...</td>\n",
|
||
" <td>x86_64-amazon-linux-gnu</td>\n",
|
||
" <td>3599.748</td>\n",
|
||
" <td>AMD EPYC 7R13 Processor</td>\n",
|
||
" <td>192</td>\n",
|
||
" <td>237177.20</td>\n",
|
||
" <td>2372250.0</td>\n",
|
||
" <td>2.0</td>\n",
|
||
" </tr>\n",
|
||
" </tbody>\n",
|
||
"</table>\n",
|
||
"</div>"
|
||
],
|
||
"text/plain": [
|
||
" config.afl_persistent_config config.afl_system_config \\\n",
|
||
"223 True True \n",
|
||
"224 True True \n",
|
||
"\n",
|
||
" config.afl_version config.comment \\\n",
|
||
"223 ++4.09a AWS EC2 r6a.48xlarge spot instance \n",
|
||
"224 ++4.09a AWS EC2 r6a.48xlarge spot instance \n",
|
||
"\n",
|
||
" config.compiler \\\n",
|
||
"223 clang version 15.0.7 (Amazon Linux 15.0.7-3.am... \n",
|
||
"224 clang version 15.0.7 (Amazon Linux 15.0.7-3.am... \n",
|
||
"\n",
|
||
" config.target_arch hardware.cpu_fastest_core_mhz \\\n",
|
||
"223 x86_64-amazon-linux-gnu 3514.326 \n",
|
||
"224 x86_64-amazon-linux-gnu 3599.748 \n",
|
||
"\n",
|
||
" hardware.cpu_model hardware.cpu_threads \\\n",
|
||
"223 AMD EPYC 7R13 Processor 192 \n",
|
||
"224 AMD EPYC 7R13 Processor 192 \n",
|
||
"\n",
|
||
" targets.test-instr-persist-shmem.multicore.execs_per_sec \\\n",
|
||
"223 119469.35 \n",
|
||
"224 237177.20 \n",
|
||
"\n",
|
||
" targets.test-instr-persist-shmem.multicore.execs_total \\\n",
|
||
"223 1194813.0 \n",
|
||
"224 2372250.0 \n",
|
||
"\n",
|
||
" targets.test-instr-persist-shmem.multicore.fuzzers_used \n",
|
||
"223 1.0 \n",
|
||
"224 2.0 "
|
||
]
|
||
},
|
||
"execution_count": 150,
|
||
"metadata": {},
|
||
"output_type": "execute_result"
|
||
}
|
||
],
|
||
"source": [
|
||
"r6a = df.query(\"`config.comment` == 'AWS EC2 r6a.48xlarge spot instance'\")\n",
|
||
"r6a.head(2).dropna(axis=1)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 151,
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"data": {
|
||
"text/html": [
|
||
"<div>\n",
|
||
"<style scoped>\n",
|
||
" .dataframe tbody tr th:only-of-type {\n",
|
||
" vertical-align: middle;\n",
|
||
" }\n",
|
||
"\n",
|
||
" .dataframe tbody tr th {\n",
|
||
" vertical-align: top;\n",
|
||
" }\n",
|
||
"\n",
|
||
" .dataframe thead th {\n",
|
||
" text-align: right;\n",
|
||
" }\n",
|
||
"</style>\n",
|
||
"<table border=\"1\" class=\"dataframe\">\n",
|
||
" <thead>\n",
|
||
" <tr style=\"text-align: right;\">\n",
|
||
" <th></th>\n",
|
||
" <th>execs_per_sec</th>\n",
|
||
" <th>parallel_fuzzers</th>\n",
|
||
" <th>afl_persistent_config</th>\n",
|
||
" <th>afl_system_config</th>\n",
|
||
" <th>label</th>\n",
|
||
" </tr>\n",
|
||
" </thead>\n",
|
||
" <tbody>\n",
|
||
" <tr>\n",
|
||
" <th>0</th>\n",
|
||
" <td>119469.35</td>\n",
|
||
" <td>1.0</td>\n",
|
||
" <td>True</td>\n",
|
||
" <td>True</td>\n",
|
||
" <td>Multicore: Persistent mode/shared memory + ker...</td>\n",
|
||
" </tr>\n",
|
||
" <tr>\n",
|
||
" <th>1</th>\n",
|
||
" <td>237177.20</td>\n",
|
||
" <td>2.0</td>\n",
|
||
" <td>True</td>\n",
|
||
" <td>True</td>\n",
|
||
" <td>Multicore: Persistent mode/shared memory + ker...</td>\n",
|
||
" </tr>\n",
|
||
" </tbody>\n",
|
||
"</table>\n",
|
||
"</div>"
|
||
],
|
||
"text/plain": [
|
||
" execs_per_sec parallel_fuzzers afl_persistent_config afl_system_config \\\n",
|
||
"0 119469.35 1.0 True True \n",
|
||
"1 237177.20 2.0 True True \n",
|
||
"\n",
|
||
" label \n",
|
||
"0 Multicore: Persistent mode/shared memory + ker... \n",
|
||
"1 Multicore: Persistent mode/shared memory + ker... "
|
||
]
|
||
},
|
||
"execution_count": 151,
|
||
"metadata": {},
|
||
"output_type": "execute_result"
|
||
}
|
||
],
|
||
"source": [
|
||
"r6a_graphdf = build_graphdf_from_query(r6a)\n",
|
||
"r6a_graphdf.head(2)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 152,
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"data": {
|
||
"image/svg+xml": [
|
||
"<svg class=\"main-svg\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"1200\" height=\"400\" style=\"\" viewBox=\"0 0 1200 400\"><rect x=\"0\" y=\"0\" width=\"1200\" height=\"400\" style=\"fill: rgb(255, 255, 255); fill-opacity: 1;\"/><defs id=\"defs-cbb66b\"><g class=\"clips\"><clipPath id=\"clipcbb66bxyplot\" class=\"plotclip\"><rect width=\"737\" height=\"220\"/></clipPath><clipPath class=\"axesclip\" id=\"clipcbb66bx\"><rect x=\"80\" y=\"0\" width=\"737\" height=\"400\"/></clipPath><clipPath class=\"axesclip\" id=\"clipcbb66by\"><rect x=\"0\" y=\"100\" width=\"1200\" height=\"220\"/></clipPath><clipPath class=\"axesclip\" id=\"clipcbb66bxy\"><rect x=\"80\" y=\"100\" width=\"737\" height=\"220\"/></clipPath></g><g class=\"gradients\"/><g class=\"patterns\"/></defs><g class=\"bglayer\"><rect class=\"bg\" x=\"80\" y=\"100\" width=\"737\" height=\"220\" style=\"fill: rgb(229, 236, 246); fill-opacity: 1; stroke-width: 0;\"/></g><g class=\"layer-below\"><g class=\"imagelayer\"/><g class=\"shapelayer\"/></g><g class=\"cartesianlayer\"><g class=\"subplot xy\"><g class=\"layer-subplot\"><g class=\"shapelayer\"/><g class=\"imagelayer\"/></g><g class=\"minor-gridlayer\"><g class=\"x\"/><g class=\"y\"/></g><g class=\"gridlayer\"><g class=\"x\"><path class=\"xgrid crisp\" transform=\"translate(91.58,0)\" d=\"M0,100v220\" style=\"stroke: rgb(255, 255, 255); stroke-opacity: 1; stroke-width: 1px;\"/><path class=\"xgrid crisp\" transform=\"translate(107.01,0)\" d=\"M0,100v220\" style=\"stroke: rgb(255, 255, 255); stroke-opacity: 1; stroke-width: 1px;\"/><path class=\"xgrid crisp\" transform=\"translate(122.45,0)\" d=\"M0,100v220\" style=\"stroke: rgb(255, 255, 255); stroke-opacity: 1; stroke-width: 1px;\"/><path class=\"xgrid crisp\" transform=\"translate(137.88,0)\" d=\"M0,100v220\" style=\"stroke: rgb(255, 255, 255); stroke-opacity: 1; stroke-width: 1px;\"/><path class=\"xgrid crisp\" transform=\"translate(153.31,0)\" d=\"M0,100v220\" style=\"stroke: rgb(255, 255, 255); stroke-opacity: 1; stroke-width: 1px;\"/><path class=\"xgrid crisp\" transform=\"translate(168.75,0)\" d=\"M0,100v220\" style=\"stroke: rgb(255, 255, 255); stroke-opacity: 1; stroke-width: 1px;\"/><path class=\"xgrid crisp\" transform=\"translate(184.18,0)\" d=\"M0,100v220\" style=\"stroke: rgb(255, 255, 255); stroke-opacity: 1; stroke-width: 1px;\"/><path class=\"xgrid crisp\" transform=\"translate(199.62,0)\" d=\"M0,100v220\" style=\"stroke: rgb(255, 255, 255); stroke-opacity: 1; stroke-width: 1px;\"/><path class=\"xgrid crisp\" transform=\"translate(215.05,0)\" d=\"M0,100v220\" style=\"stroke: rgb(255, 255, 255); stroke-opacity: 1; stroke-width: 1px;\"/><path class=\"xgrid crisp\" transform=\"translate(230.49,0)\" d=\"M0,100v220\" style=\"stroke: rgb(255, 255, 255); stroke-opacity: 1; stroke-width: 1px;\"/><path class=\"xgrid crisp\" transform=\"translate(245.92,0)\" d=\"M0,100v220\" style=\"stroke: rgb(255, 255, 255); stroke-opacity: 1; stroke-width: 1px;\"/><path class=\"xgrid crisp\" transform=\"translate(261.36,0)\" d=\"M0,100v220\" style=\"stroke: rgb(255, 255, 255); stroke-opacity: 1; stroke-width: 1px;\"/><path class=\"xgrid crisp\" transform=\"translate(276.78999999999996,0)\" d=\"M0,100v220\" style=\"stroke: rgb(255, 255, 255); stroke-opacity: 1; stroke-width: 1px;\"/><path class=\"xgrid crisp\" transform=\"translate(292.23,0)\" d=\"M0,100v220\" style=\"stroke: rgb(255, 255, 255); stroke-opacity: 1; stroke-width: 1px;\"/><path class=\"xgrid crisp\" transform=\"translate(307.65999999999997,0)\" d=\"M0,100v220\" style=\"stroke: rgb(255, 255, 255); stroke-opacity: 1; stroke-width: 1px;\"/><path class=\"xgrid crisp\" transform=\"translate(323.09000000000003,0)\" d=\"M0,100v220\" style=\"stroke: rgb(255, 255, 255); stroke-opacity: 1; stroke-width: 1px;\"/><path class=\"xgrid crisp\" transform=\"translate(338.53,0)\" d=\"M0,100v220\" style=\"stroke: rgb(255, 255, 255); stroke-opacity: 1; stroke-width: 1px;\"/><path class=\"xgrid crisp\" transform=\"translate(353.96,0)\" d=\"M0,100v220\" style=\"stroke: rgb(255, 255, 255); stroke-opacity: 1; stroke-width: 1px;\"/><path class=\"xgrid crisp\" transform=\"translate(369.4,0)\" d=\"M0,100v220\" style=\"stroke: rgb(255, 255, 255); stroke-opacity: 1; stroke-width: 1px;\"/><path class=\"xgrid crisp\" transform=\"translate(384.83,0)\" d=\"M0,100v220\" style=\"stroke: rgb(255, 255, 255); stroke-opacity: 1; stroke-width: 1px;\"/><path class=\"xgrid crisp\" transform=\"translate(400.27,0)\" d=\"M0,100v220\" style=\"stroke: rgb(255, 255, 255); stroke-opacity: 1; stroke-width: 1px;\"/><path class=\"xgrid crisp\" transform=\"translate(415.7,0)\" d=\"M0,100v220\" style=\"stroke: rgb(255, 255, 255); stroke-opacity: 1; stroke-width: 1px;\"/><path class=\"xgrid crisp\" transform=\"translate(431.14,0)\" d=\"M0,100v220\" style=\"stroke: rgb(255, 255, 255); stroke-opacity: 1; stroke-width: 1px;\"/><path class=\"xgrid crisp\" transform=\"translate(446.57,0)\" d=\"M0,100v220\" style=\"stroke: rgb(255, 255, 255); stroke-opacity: 1; stroke-width: 1px;\"/><path class=\"xgrid crisp\" transform=\"translate(462.01,0)\" d=\"M0,100v220\" style=\"stroke: rgb(255, 255, 255); stroke-opacity: 1; stroke-width: 1px;\"/><path class=\"xgrid crisp\" transform=\"translate(477.44,0)\" d=\"M0,100v220\" style=\"stroke: rgb(255, 255, 255); stroke-opacity: 1; stroke-width: 1px;\"/><path class=\"xgrid crisp\" transform=\"translate(492.87,0)\" d=\"M0,100v220\" style=\"stroke: rgb(255, 255, 255); stroke-opacity: 1; stroke-width: 1px;\"/><path class=\"xgrid crisp\" transform=\"translate(508.31,0)\" d=\"M0,100v220\" style=\"stroke: rgb(255, 255, 255); stroke-opacity: 1; stroke-width: 1px;\"/><path class=\"xgrid crisp\" transform=\"translate(523.74,0)\" d=\"M0,100v220\" style=\"stroke: rgb(255, 255, 255); stroke-opacity: 1; stroke-width: 1px;\"/><path class=\"xgrid crisp\" transform=\"translate(539.1800000000001,0)\" d=\"M0,100v220\" style=\"stroke: rgb(255, 255, 255); stroke-opacity: 1; stroke-width: 1px;\"/><path class=\"xgrid crisp\" transform=\"translate(554.61,0)\" d=\"M0,100v220\" style=\"stroke: rgb(255, 255, 255); stroke-opacity: 1; stroke-width: 1px;\"/><path class=\"xgrid crisp\" transform=\"translate(570.05,0)\" d=\"M0,100v220\" style=\"stroke: rgb(255, 255, 255); stroke-opacity: 1; stroke-width: 1px;\"/><path class=\"xgrid crisp\" transform=\"translate(585.48,0)\" d=\"M0,100v220\" style=\"stroke: rgb(255, 255, 255); stroke-opacity: 1; stroke-width: 1px;\"/><path class=\"xgrid crisp\" transform=\"translate(600.92,0)\" d=\"M0,100v220\" style=\"stroke: rgb(255, 255, 255); stroke-opacity: 1; stroke-width: 1px;\"/><path class=\"xgrid crisp\" transform=\"translate(616.35,0)\" d=\"M0,100v220\" style=\"stroke: rgb(255, 255, 255); stroke-opacity: 1; stroke-width: 1px;\"/><path class=\"xgrid crisp\" transform=\"translate(631.79,0)\" d=\"M0,100v220\" style=\"stroke: rgb(255, 255, 255); stroke-opacity: 1; stroke-width: 1px;\"/><path class=\"xgrid crisp\" transform=\"translate(647.22,0)\" d=\"M0,100v220\" style=\"stroke: rgb(255, 255, 255); stroke-opacity: 1; stroke-width: 1px;\"/><path class=\"xgrid crisp\" transform=\"translate(662.65,0)\" d=\"M0,100v220\" style=\"stroke: rgb(255, 255, 255); stroke-opacity: 1; stroke-width: 1px;\"/><path class=\"xgrid crisp\" transform=\"translate(678.09,0)\" d=\"M0,100v220\" style=\"stroke: rgb(255, 255, 255); stroke-opacity: 1; stroke-width: 1px;\"/><path class=\"xgrid crisp\" transform=\"translate(693.52,0)\" d=\"M0,100v220\" style=\"stroke: rgb(255, 255, 255); stroke-opacity: 1; stroke-width: 1px;\"/><path class=\"xgrid crisp\" transform=\"translate(708.96,0)\" d=\"M0,100v220\" style=\"stroke: rgb(255, 255, 255); stroke-opacity: 1; stroke-width: 1px;\"/><path class=\"xgrid crisp\" transform=\"translate(724.39,0)\" d=\"M0,100v220\" style=\"stroke: rgb(255, 255, 255); stroke-opacity: 1; stroke-width: 1px;\"/><path class=\"xgrid crisp\" transform=\"translate(739.83,0)\" d=\"M0,100v220\" style=\"stroke: rgb(255, 255, 255); stroke-opacity: 1; stroke-width: 1px;\"/><path class=\"xgrid crisp\" transform=\"translate(755.26,0)\" d=\"M0,100v220\" style=\"stroke: rgb(255, 255, 255); stroke-opacity: 1; stroke-width: 1px;\"/><path class=\"xgrid crisp\" transform=\"translate(770.7,0)\" d=\"M0,100v220\" style=\"stroke: rgb(255, 255, 255); stroke-opacity: 1; stroke-width: 1px;\"/><path class=\"xgrid crisp\" transform=\"translate(786.13,0)\" d=\"M0,100v220\" style=\"stroke: rgb(255, 255, 255); stroke-opacity: 1; stroke-width: 1px;\"/><path class=\"xgrid crisp\" transform=\"translate(801.57,0)\" d=\"M0,100v220\" style=\"stroke: rgb(255, 255, 255); stroke-opacity: 1; stroke-width: 1px;\"/></g><g class=\"y\"><path class=\"ygrid crisp\" transform=\"translate(0,309)\" d=\"M80,0h737\" style=\"stroke: rgb(255, 255, 255); stroke-opacity: 1; stroke-width: 1px;\"/><path class=\"ygrid crisp\" transform=\"translate(0,269.4)\" d=\"M80,0h737\" style=\"stroke: rgb(255, 255, 255); stroke-opacity: 1; stroke-width: 1px;\"/><path class=\"ygrid crisp\" transform=\"translate(0,229.8)\" d=\"M80,0h737\" style=\"stroke: rgb(255, 255, 255); stroke-opacity: 1; stroke-width: 1px;\"/><path class=\"ygrid crisp\" transform=\"translate(0,190.2)\" d=\"M80,0h737\" style=\"stroke: rgb(255, 255, 255); stroke-opacity: 1; stroke-width: 1px;\"/><path class=\"ygrid crisp\" transform=\"translate(0,150.6)\" d=\"M80,0h737\" style=\"stroke: rgb(255, 255, 255); stroke-opacity: 1; stroke-width: 1px;\"/><path class=\"ygrid crisp\" transform=\"translate(0,111)\" d=\"M80,0h737\" style=\"stroke: rgb(255, 255, 255); stroke-opacity: 1; stroke-width: 1px;\"/></g></g><g class=\"zerolinelayer\"/><path class=\"xlines-below\"/><path class=\"ylines-below\"/><g class=\"overlines-below\"/><g class=\"xaxislayer-below\"/><g class=\"yaxislayer-below\"/><g class=\"overaxes-below\"/><g class=\"plot\" transform=\"translate(80,100)\" clip-path=\"url(#clipcbb66bxyplot)\"><g class=\"scatterlayer mlayer\"><g class=\"trace scatter trace9250e0\" style=\"stroke-miterlimit: 2; opacity: 1;\"><g class=\"fills\"/><g class=\"errorbars\"/><g class=\"lines\"><path class=\"js-line\" d=\"M0,209L27.01,102.25L30.87,92.62L38.59,70.35L42.45,56.73L54.02,21.04L57.88,11L61.74,26.13L65.6,32.45L77.17,55.69L81.03,63.3L88.75,77.57L92.61,91.06L100.32,99.97L104.18,101.5L119.62,110.06L123.48,115.54L131.19,117.24L135.05,116.27L138.91,118.19L142.77,119.19L150.49,122.55L154.35,126.03L158.2,124.89L162.06,125.39L165.92,124.75L169.78,125.59L173.64,126.69L177.5,128.95L181.36,130.38L185.21,138.51L196.79,138.25L200.65,139.44L204.51,138.5L208.37,138.36L212.23,138.9L216.08,143.74L219.94,144.38L223.8,143.28L235.38,142.43L239.24,143.32L243.09,142.87L246.95,145.91L250.81,146.52L254.67,145.26L266.25,144.24L270.1,145.13L273.96,144.33L277.82,147.9L281.68,147.87L285.54,146.98L300.97,145.4L304.83,146.29L308.69,149.5L312.55,149.05L316.41,147.52L320.27,147.61L324.13,146.65L327.98,147.64L335.7,146L339.56,149.34L343.42,149.32L347.28,148.33L354.99,148.62L358.85,146.68L366.57,146.81L370.43,148.02L378.15,149.76L382.01,149.62L389.72,151.19L393.58,151.08L405.16,153.04L409.02,152.44L412.87,153.36L416.73,152.85L439.88,154.48L443.74,153.92L451.46,154.74L455.32,154.56L459.18,154.9L463.04,157.41L470.75,159.8L474.61,160.48L482.33,161.72L486.19,163.29L501.62,165.38L505.48,164.47L532.49,167.38L536.35,166.72L540.21,167.26L544.07,166.74L551.79,166.87L555.64,168.82L563.36,168.82L567.22,168L582.65,167.75L586.51,168.83L594.23,168.31L598.09,167.88L601.95,168.3L605.81,167.68L613.52,166.74L617.38,167.9L636.68,166.35L640.53,166.66L648.25,166.11L652.11,166.24L667.54,164.76L671.4,164.41L675.26,164.4L679.12,165.51L682.98,164.93L686.84,165.64L690.7,164.08L694.55,163.93L698.41,163.27L702.27,163.55L706.13,162.53L709.99,162.91L713.85,163.78L717.71,162.38L733.14,162.64L737,161.91\" style=\"vector-effect: none; fill: none; stroke: rgb(99, 110, 250); stroke-opacity: 1; stroke-width: 2px; opacity: 1;\"/></g><g class=\"points\"/><g class=\"text\"/></g></g></g><g class=\"overplot\"/><path class=\"xlines-above crisp\" d=\"M0,0\" style=\"fill: none;\"/><path class=\"ylines-above crisp\" d=\"M0,0\" style=\"fill: none;\"/><g class=\"overlines-above\"/><g class=\"xaxislayer-above\"><g class=\"xtick\"><text text-anchor=\"start\" x=\"0\" y=\"333\" transform=\"translate(91.58,0) rotate(90,0,327)\" style=\"font-family: 'Open Sans', verdana, arial, sans-serif; font-size: 12px; fill: rgb(42, 63, 95); fill-opacity: 1; white-space: pre; opacity: 1;\">4</text></g><g class=\"xtick\"><text text-anchor=\"start\" x=\"0\" y=\"333\" style=\"font-family: 'Open Sans', verdana, arial, sans-serif; font-size: 12px; fill: rgb(42, 63, 95); fill-opacity: 1; white-space: pre; opacity: 1;\" transform=\"translate(107.01,0) rotate(90,0,327)\">8</text></g><g class=\"xtick\"><text text-anchor=\"start\" x=\"0\" y=\"333\" style=\"font-family: 'Open Sans', verdana, arial, sans-serif; font-size: 12px; fill: rgb(42, 63, 95); fill-opacity: 1; white-space: pre; opacity: 1;\" transform=\"translate(122.45,0) rotate(90,0,327)\">12</text></g><g class=\"xtick\"><text text-anchor=\"start\" x=\"0\" y=\"333\" style=\"font-family: 'Open Sans', verdana, arial, sans-serif; font-size: 12px; fill: rgb(42, 63, 95); fill-opacity: 1; white-space: pre; opacity: 1;\" transform=\"translate(137.88,0) rotate(90,0,327)\">16</text></g><g class=\"xtick\"><text text-anchor=\"start\" x=\"0\" y=\"333\" style=\"font-family: 'Open Sans', verdana, arial, sans-serif; font-size: 12px; fill: rgb(42, 63, 95); fill-opacity: 1; white-space: pre; opacity: 1;\" transform=\"translate(153.31,0) rotate(90,0,327)\">20</text></g><g class=\"xtick\"><text text-anchor=\"start\" x=\"0\" y=\"333\" style=\"font-family: 'Open Sans', verdana, arial, sans-serif; font-size: 12px; fill: rgb(42, 63, 95); fill-opacity: 1; white-space: pre; opacity: 1;\" transform=\"translate(168.75,0) rotate(90,0,327)\">24</text></g><g class=\"xtick\"><text text-anchor=\"start\" x=\"0\" y=\"333\" style=\"font-family: 'Open Sans', verdana, arial, sans-serif; font-size: 12px; fill: rgb(42, 63, 95); fill-opacity: 1; white-space: pre; opacity: 1;\" transform=\"translate(184.18,0) rotate(90,0,327)\">28</text></g><g class=\"xtick\"><text text-anchor=\"start\" x=\"0\" y=\"333\" style=\"font-family: 'Open Sans', verdana, arial, sans-serif; font-size: 12px; fill: rgb(42, 63, 95); fill-opacity: 1; white-space: pre; opacity: 1;\" transform=\"translate(199.62,0) rotate(90,0,327)\">32</text></g><g class=\"xtick\"><text text-anchor=\"start\" x=\"0\" y=\"333\" style=\"font-family: 'Open Sans', verdana, arial, sans-serif; font-size: 12px; fill: rgb(42, 63, 95); fill-opacity: 1; white-space: pre; opacity: 1;\" transform=\"translate(215.05,0) rotate(90,0,327)\">36</text></g><g class=\"xtick\"><text text-anchor=\"start\" x=\"0\" y=\"333\" style=\"font-family: 'Open Sans', verdana, arial, sans-serif; font-size: 12px; fill: rgb(42, 63, 95); fill-opacity: 1; white-space: pre; opacity: 1;\" transform=\"translate(230.49,0) rotate(90,0,327)\">40</text></g><g class=\"xtick\"><text text-anchor=\"start\" x=\"0\" y=\"333\" style=\"font-family: 'Open Sans', verdana, arial, sans-serif; font-size: 12px; fill: rgb(42, 63, 95); fill-opacity: 1; white-space: pre; opacity: 1;\" transform=\"translate(245.92,0) rotate(90,0,327)\">44</text></g><g class=\"xtick\"><text text-anchor=\"start\" x=\"0\" y=\"333\" style=\"font-family: 'Open Sans', verdana, arial, sans-serif; font-size: 12px; fill: rgb(42, 63, 95); fill-opacity: 1; white-space: pre; opacity: 1;\" transform=\"translate(261.36,0) rotate(90,0,327)\">48</text></g><g class=\"xtick\"><text text-anchor=\"start\" x=\"0\" y=\"333\" style=\"font-family: 'Open Sans', verdana, arial, sans-serif; font-size: 12px; fill: rgb(42, 63, 95); fill-opacity: 1; white-space: pre; opacity: 1;\" transform=\"translate(276.78999999999996,0) rotate(90,0,327)\">52</text></g><g class=\"xtick\"><text text-anchor=\"start\" x=\"0\" y=\"333\" style=\"font-family: 'Open Sans', verdana, arial, sans-serif; font-size: 12px; fill: rgb(42, 63, 95); fill-opacity: 1; white-space: pre; opacity: 1;\" transform=\"translate(292.23,0) rotate(90,0,327)\">56</text></g><g class=\"xtick\"><text text-anchor=\"start\" x=\"0\" y=\"333\" style=\"font-family: 'Open Sans', verdana, arial, sans-serif; font-size: 12px; fill: rgb(42, 63, 95); fill-opacity: 1; white-space: pre; opacity: 1;\" transform=\"translate(307.65999999999997,0) rotate(90,0,327)\">60</text></g><g class=\"xtick\"><text text-anchor=\"start\" x=\"0\" y=\"333\" style=\"font-family: 'Open Sans', verdana, arial, sans-serif; font-size: 12px; fill: rgb(42, 63, 95); fill-opacity: 1; white-space: pre; opacity: 1;\" transform=\"translate(323.09000000000003,0) rotate(90,0,327)\">64</text></g><g class=\"xtick\"><text text-anchor=\"start\" x=\"0\" y=\"333\" style=\"font-family: 'Open Sans', verdana, arial, sans-serif; font-size: 12px; fill: rgb(42, 63, 95); fill-opacity: 1; white-space: pre; opacity: 1;\" transform=\"translate(338.53,0) rotate(90,0,327)\">68</text></g><g class=\"xtick\"><text text-anchor=\"start\" x=\"0\" y=\"333\" style=\"font-family: 'Open Sans', verdana, arial, sans-serif; font-size: 12px; fill: rgb(42, 63, 95); fill-opacity: 1; white-space: pre; opacity: 1;\" transform=\"translate(353.96,0) rotate(90,0,327)\">72</text></g><g class=\"xtick\"><text text-anchor=\"start\" x=\"0\" y=\"333\" style=\"font-family: 'Open Sans', verdana, arial, sans-serif; font-size: 12px; fill: rgb(42, 63, 95); fill-opacity: 1; white-space: pre; opacity: 1;\" transform=\"translate(369.4,0) rotate(90,0,327)\">76</text></g><g class=\"xtick\"><text text-anchor=\"start\" x=\"0\" y=\"333\" style=\"font-family: 'Open Sans', verdana, arial, sans-serif; font-size: 12px; fill: rgb(42, 63, 95); fill-opacity: 1; white-space: pre; opacity: 1;\" transform=\"translate(384.83,0) rotate(90,0,327)\">80</text></g><g class=\"xtick\"><text text-anchor=\"start\" x=\"0\" y=\"333\" style=\"font-family: 'Open Sans', verdana, arial, sans-serif; font-size: 12px; fill: rgb(42, 63, 95); fill-opacity: 1; white-space: pre; opacity: 1;\" transform=\"translate(400.27,0) rotate(90,0,327)\">84</text></g><g class=\"xtick\"><text text-anchor=\"start\" x=\"0\" y=\"333\" style=\"font-family: 'Open Sans', verdana, arial, sans-serif; font-size: 12px; fill: rgb(42, 63, 95); fill-opacity: 1; white-space: pre; opacity: 1;\" transform=\"translate(415.7,0) rotate(90,0,327)\">88</text></g><g class=\"xtick\"><text text-anchor=\"start\" x=\"0\" y=\"333\" style=\"font-family: 'Open Sans', verdana, arial, sans-serif; font-size: 12px; fill: rgb(42, 63, 95); fill-opacity: 1; white-space: pre; opacity: 1;\" transform=\"translate(431.14,0) rotate(90,0,327)\">92</text></g><g class=\"xtick\"><text text-anchor=\"start\" x=\"0\" y=\"333\" style=\"font-family: 'Open Sans', verdana, arial, sans-serif; font-size: 12px; fill: rgb(42, 63, 95); fill-opacity: 1; white-space: pre; opacity: 1;\" transform=\"translate(446.57,0) rotate(90,0,327)\">96</text></g><g class=\"xtick\"><text text-anchor=\"start\" x=\"0\" y=\"333\" style=\"font-family: 'Open Sans', verdana, arial, sans-serif; font-size: 12px; fill: rgb(42, 63, 95); fill-opacity: 1; white-space: pre; opacity: 1;\" transform=\"translate(462.01,0) rotate(90,0,327)\">100</text></g><g class=\"xtick\"><text text-anchor=\"start\" x=\"0\" y=\"333\" style=\"font-family: 'Open Sans', verdana, arial, sans-serif; font-size: 12px; fill: rgb(42, 63, 95); fill-opacity: 1; white-space: pre; opacity: 1;\" transform=\"translate(477.44,0) rotate(90,0,327)\">104</text></g><g class=\"xtick\"><text text-anchor=\"start\" x=\"0\" y=\"333\" style=\"font-family: 'Open Sans', verdana, arial, sans-serif; font-size: 12px; fill: rgb(42, 63, 95); fill-opacity: 1; white-space: pre; opacity: 1;\" transform=\"translate(492.87,0) rotate(90,0,327)\">108</text></g><g class=\"xtick\"><text text-anchor=\"start\" x=\"0\" y=\"333\" style=\"font-family: 'Open Sans', verdana, arial, sans-serif; font-size: 12px; fill: rgb(42, 63, 95); fill-opacity: 1; white-space: pre; opacity: 1;\" transform=\"translate(508.31,0) rotate(90,0,327)\">112</text></g><g class=\"xtick\"><text text-anchor=\"start\" x=\"0\" y=\"333\" style=\"font-family: 'Open Sans', verdana, arial, sans-serif; font-size: 12px; fill: rgb(42, 63, 95); fill-opacity: 1; white-space: pre; opacity: 1;\" transform=\"translate(523.74,0) rotate(90,0,327)\">116</text></g><g class=\"xtick\"><text text-anchor=\"start\" x=\"0\" y=\"333\" style=\"font-family: 'Open Sans', verdana, arial, sans-serif; font-size: 12px; fill: rgb(42, 63, 95); fill-opacity: 1; white-space: pre; opacity: 1;\" transform=\"translate(539.1800000000001,0) rotate(90,0,327)\">120</text></g><g class=\"xtick\"><text text-anchor=\"start\" x=\"0\" y=\"333\" style=\"font-family: 'Open Sans', verdana, arial, sans-serif; font-size: 12px; fill: rgb(42, 63, 95); fill-opacity: 1; white-space: pre; opacity: 1;\" transform=\"translate(554.61,0) rotate(90,0,327)\">124</text></g><g class=\"xtick\"><text text-anchor=\"start\" x=\"0\" y=\"333\" style=\"font-family: 'Open Sans', verdana, arial, sans-serif; font-size: 12px; fill: rgb(42, 63, 95); fill-opacity: 1; white-space: pre; opacity: 1;\" transform=\"translate(570.05,0) rotate(90,0,327)\">128</text></g><g class=\"xtick\"><text text-anchor=\"start\" x=\"0\" y=\"333\" style=\"font-family: 'Open Sans', verdana, arial, sans-serif; font-size: 12px; fill: rgb(42, 63, 95); fill-opacity: 1; white-space: pre; opacity: 1;\" transform=\"translate(585.48,0) rotate(90,0,327)\">132</text></g><g class=\"xtick\"><text text-anchor=\"start\" x=\"0\" y=\"333\" style=\"font-family: 'Open Sans', verdana, arial, sans-serif; font-size: 12px; fill: rgb(42, 63, 95); fill-opacity: 1; white-space: pre; opacity: 1;\" transform=\"translate(600.92,0) rotate(90,0,327)\">136</text></g><g class=\"xtick\"><text text-anchor=\"start\" x=\"0\" y=\"333\" style=\"font-family: 'Open Sans', verdana, arial, sans-serif; font-size: 12px; fill: rgb(42, 63, 95); fill-opacity: 1; white-space: pre; opacity: 1;\" transform=\"translate(616.35,0) rotate(90,0,327)\">140</text></g><g class=\"xtick\"><text text-anchor=\"start\" x=\"0\" y=\"333\" style=\"font-family: 'Open Sans', verdana, arial, sans-serif; font-size: 12px; fill: rgb(42, 63, 95); fill-opacity: 1; white-space: pre; opacity: 1;\" transform=\"translate(631.79,0) rotate(90,0,327)\">144</text></g><g class=\"xtick\"><text text-anchor=\"start\" x=\"0\" y=\"333\" style=\"font-family: 'Open Sans', verdana, arial, sans-serif; font-size: 12px; fill: rgb(42, 63, 95); fill-opacity: 1; white-space: pre; opacity: 1;\" transform=\"translate(647.22,0) rotate(90,0,327)\">148</text></g><g class=\"xtick\"><text text-anchor=\"start\" x=\"0\" y=\"333\" style=\"font-family: 'Open Sans', verdana, arial, sans-serif; font-size: 12px; fill: rgb(42, 63, 95); fill-opacity: 1; white-space: pre; opacity: 1;\" transform=\"translate(662.65,0) rotate(90,0,327)\">152</text></g><g class=\"xtick\"><text text-anchor=\"start\" x=\"0\" y=\"333\" style=\"font-family: 'Open Sans', verdana, arial, sans-serif; font-size: 12px; fill: rgb(42, 63, 95); fill-opacity: 1; white-space: pre; opacity: 1;\" transform=\"translate(678.09,0) rotate(90,0,327)\">156</text></g><g class=\"xtick\"><text text-anchor=\"start\" x=\"0\" y=\"333\" style=\"font-family: 'Open Sans', verdana, arial, sans-serif; font-size: 12px; fill: rgb(42, 63, 95); fill-opacity: 1; white-space: pre; opacity: 1;\" transform=\"translate(693.52,0) rotate(90,0,327)\">160</text></g><g class=\"xtick\"><text text-anchor=\"start\" x=\"0\" y=\"333\" style=\"font-family: 'Open Sans', verdana, arial, sans-serif; font-size: 12px; fill: rgb(42, 63, 95); fill-opacity: 1; white-space: pre; opacity: 1;\" transform=\"translate(708.96,0) rotate(90,0,327)\">164</text></g><g class=\"xtick\"><text text-anchor=\"start\" x=\"0\" y=\"333\" style=\"font-family: 'Open Sans', verdana, arial, sans-serif; font-size: 12px; fill: rgb(42, 63, 95); fill-opacity: 1; white-space: pre; opacity: 1;\" transform=\"translate(724.39,0) rotate(90,0,327)\">168</text></g><g class=\"xtick\"><text text-anchor=\"start\" x=\"0\" y=\"333\" style=\"font-family: 'Open Sans', verdana, arial, sans-serif; font-size: 12px; fill: rgb(42, 63, 95); fill-opacity: 1; white-space: pre; opacity: 1;\" transform=\"translate(739.83,0) rotate(90,0,327)\">172</text></g><g class=\"xtick\"><text text-anchor=\"start\" x=\"0\" y=\"333\" style=\"font-family: 'Open Sans', verdana, arial, sans-serif; font-size: 12px; fill: rgb(42, 63, 95); fill-opacity: 1; white-space: pre; opacity: 1;\" transform=\"translate(755.26,0) rotate(90,0,327)\">176</text></g><g class=\"xtick\"><text text-anchor=\"start\" x=\"0\" y=\"333\" style=\"font-family: 'Open Sans', verdana, arial, sans-serif; font-size: 12px; fill: rgb(42, 63, 95); fill-opacity: 1; white-space: pre; opacity: 1;\" transform=\"translate(770.7,0) rotate(90,0,327)\">180</text></g><g class=\"xtick\"><text text-anchor=\"start\" x=\"0\" y=\"333\" style=\"font-family: 'Open Sans', verdana, arial, sans-serif; font-size: 12px; fill: rgb(42, 63, 95); fill-opacity: 1; white-space: pre; opacity: 1;\" transform=\"translate(786.13,0) rotate(90,0,327)\">184</text></g><g class=\"xtick\"><text text-anchor=\"start\" x=\"0\" y=\"333\" style=\"font-family: 'Open Sans', verdana, arial, sans-serif; font-size: 12px; fill: rgb(42, 63, 95); fill-opacity: 1; white-space: pre; opacity: 1;\" transform=\"translate(801.57,0) rotate(90,0,327)\">188</text></g><g class=\"xtick\"><text text-anchor=\"start\" x=\"0\" y=\"333\" style=\"font-family: 'Open Sans', verdana, arial, sans-serif; font-size: 12px; fill: rgb(42, 63, 95); fill-opacity: 1; white-space: pre; opacity: 1;\" transform=\"translate(817,0) rotate(90,0,327)\">192</text></g></g><g class=\"yaxislayer-above\"><g class=\"ytick\"><text text-anchor=\"end\" x=\"79\" y=\"4.199999999999999\" transform=\"translate(0,309)\" style=\"font-family: 'Open Sans', verdana, arial, sans-serif; font-size: 12px; fill: rgb(42, 63, 95); fill-opacity: 1; white-space: pre; opacity: 1;\">12x</text></g><g class=\"ytick\"><text text-anchor=\"end\" x=\"79\" y=\"4.199999999999999\" style=\"font-family: 'Open Sans', verdana, arial, sans-serif; font-size: 12px; fill: rgb(42, 63, 95); fill-opacity: 1; white-space: pre; opacity: 1;\" transform=\"translate(0,269.4)\">43x</text></g><g class=\"ytick\"><text text-anchor=\"end\" x=\"79\" y=\"4.199999999999999\" style=\"font-family: 'Open Sans', verdana, arial, sans-serif; font-size: 12px; fill: rgb(42, 63, 95); fill-opacity: 1; white-space: pre; opacity: 1;\" transform=\"translate(0,229.8)\">74x</text></g><g class=\"ytick\"><text text-anchor=\"end\" x=\"79\" y=\"4.199999999999999\" style=\"font-family: 'Open Sans', verdana, arial, sans-serif; font-size: 12px; fill: rgb(42, 63, 95); fill-opacity: 1; white-space: pre; opacity: 1;\" transform=\"translate(0,190.2)\">104x</text></g><g class=\"ytick\"><text text-anchor=\"end\" x=\"79\" y=\"4.199999999999999\" style=\"font-family: 'Open Sans', verdana, arial, sans-serif; font-size: 12px; fill: rgb(42, 63, 95); fill-opacity: 1; white-space: pre; opacity: 1;\" transform=\"translate(0,150.6)\">135x</text></g><g class=\"ytick\"><text text-anchor=\"end\" x=\"79\" y=\"4.199999999999999\" style=\"font-family: 'Open Sans', verdana, arial, sans-serif; font-size: 12px; fill: rgb(42, 63, 95); fill-opacity: 1; white-space: pre; opacity: 1;\" transform=\"translate(0,111)\">166x</text></g></g><g class=\"overaxes-above\"/></g></g><g class=\"polarlayer\"/><g class=\"smithlayer\"/><g class=\"ternarylayer\"/><g class=\"geolayer\"/><g class=\"funnelarealayer\"/><g class=\"pielayer\"/><g class=\"iciclelayer\"/><g class=\"treemaplayer\"/><g class=\"sunburstlayer\"/><g class=\"glimages\"/><defs id=\"topdefs-cbb66b\"><g class=\"clips\"/><clipPath id=\"legendcbb66b\"><rect width=\"356\" height=\"48\" x=\"0\" y=\"0\"/></clipPath></defs><g class=\"layer-above\"><g class=\"imagelayer\"/><g class=\"shapelayer\"/></g><g class=\"infolayer\"><g class=\"legend\" pointer-events=\"all\" transform=\"translate(831.74,100)\"><rect class=\"bg\" shape-rendering=\"crispEdges\" width=\"356\" height=\"48\" x=\"0\" y=\"0\" style=\"stroke: rgb(68, 68, 68); stroke-opacity: 1; fill: rgb(255, 255, 255); fill-opacity: 1; stroke-width: 0px;\"/><g class=\"scrollbox\" transform=\"\" clip-path=\"url(#legendcbb66b)\"><text class=\"legendtitletext\" text-anchor=\"start\" x=\"2\" y=\"18.2\" style=\"font-family: 'Open Sans', verdana, arial, sans-serif; font-size: 14px; fill: rgb(42, 63, 95); fill-opacity: 1; white-space: pre;\">Configuration</text><g class=\"groups\" transform=\"\"><g class=\"traces\" transform=\"translate(0,32.7)\" style=\"opacity: 1;\"><text class=\"legendtext\" text-anchor=\"start\" x=\"40\" y=\"4.680000000000001\" style=\"font-family: 'Open Sans', verdana, arial, sans-serif; font-size: 12px; fill: rgb(42, 63, 95); fill-opacity: 1; white-space: pre;\">Multicore: Persistent mode/shared memory + kernel config</text><g class=\"layers\" style=\"opacity: 1;\"><g class=\"legendfill\"/><g class=\"legendlines\"><path class=\"js-line\" d=\"M5,0h30\" style=\"fill: none; stroke: rgb(99, 110, 250); stroke-opacity: 1; stroke-width: 2px;\"/></g><g class=\"legendsymbols\"><g class=\"legendpoints\"/></g></g><rect class=\"legendtoggle\" x=\"0\" y=\"-9.5\" width=\"350.46875\" height=\"19\" style=\"fill: rgb(0, 0, 0); fill-opacity: 0;\"/></g></g></g><rect class=\"scrollbar\" rx=\"20\" ry=\"3\" width=\"0\" height=\"0\" x=\"0\" y=\"0\" style=\"fill: rgb(128, 139, 164); fill-opacity: 1;\"/></g><g class=\"g-gtitle\"><text class=\"gtitle\" x=\"60\" y=\"50\" text-anchor=\"start\" dy=\"0em\" style=\"font-family: 'Open Sans', verdana, arial, sans-serif; font-size: 17px; fill: rgb(42, 63, 95); opacity: 1; font-weight: normal; white-space: pre;\">Fuzzer performance</text></g><g class=\"g-xtitle\"><text class=\"xtitle\" x=\"448.5\" y=\"371.659375\" text-anchor=\"middle\" style=\"font-family: 'Open Sans', verdana, arial, sans-serif; font-size: 14px; fill: rgb(42, 63, 95); opacity: 1; font-weight: normal; white-space: pre;\">Number of parallel fuzzers</text></g><g class=\"g-ytitle\"><text class=\"ytitle\" transform=\"rotate(-90,28.668750000000003,210)\" x=\"28.668750000000003\" y=\"210\" text-anchor=\"middle\" style=\"font-family: 'Open Sans', verdana, arial, sans-serif; font-size: 14px; fill: rgb(42, 63, 95); opacity: 1; font-weight: normal; white-space: pre;\">Fuzz target executions per second</text></g></g></svg>"
|
||
]
|
||
},
|
||
"metadata": {},
|
||
"output_type": "display_data"
|
||
}
|
||
],
|
||
"source": [
|
||
"r6a_pivotdf = r6a_graphdf.pivot(index=\"parallel_fuzzers\", columns=\"label\", values=\"execs_per_sec\")\n",
|
||
"r6a_fig = r6a_pivotdf.plot(\n",
|
||
" title=\"Fuzzer performance\",\n",
|
||
" labels={\n",
|
||
" \"label\": \"Configuration\",\n",
|
||
" \"parallel_fuzzers\": \"Number of parallel fuzzers\",\n",
|
||
" \"value\": \"Fuzz target executions per second\"\n",
|
||
" }\n",
|
||
")\n",
|
||
"\n",
|
||
"# Compute tick values and their labels for the primary Y-axis\n",
|
||
"tickvals = np.linspace(r6a_graphdf['execs_per_sec'].min(), r6a_graphdf['execs_per_sec'].max(), 6)\n",
|
||
"ticktext = [f\"{val:.0f}x\" for val in tickvals / graphdf['execs_per_sec'].min()]\n",
|
||
"# Update the primary Y-axis with custom tick labels\n",
|
||
"r6a_fig.update_yaxes(tickvals=tickvals, ticktext=ticktext)\n",
|
||
"r6a_fig.update_xaxes(tickvals=list(range(0,200+1, 4)))\n",
|
||
"r6a_fig.update_layout(width=1200, height=400)\n",
|
||
"r6a_fig.show(\"svg\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"### Line graph analysis\n",
|
||
"\n",
|
||
"This is a shocking result for a 192 vCPU machine -- our optimal number of parallel fuzzers was 16! Using 32 parallel fuzzers gives less performance than using 8 fuzzers. Using 192 parallel fuzzers (the physical number of threads in this machine) gives the same performance as using 4 fuzzers.\n",
|
||
"\n",
|
||
"This is clearly a cautionary tale about measuring before simply using the number of hardware threads in your machine. But does this mean that AFL++ is a bad fuzzer, or that AWS tricked us and gave us a 16-thread machine instead of a 192-thread one?\n",
|
||
"\n",
|
||
"No, probably not -- the most likely cause here (other than a horrible bug) may be that we're already saturating the Linux kernel's ability to service system calls (although we're definitely hitting such a limit way earlier than I expected). A good way to test this theory would be to run more system-call-servicers (read: kernels!) at once on this machine; one way to do that is to use hardware virtualization with KVM. "
|
||
]
|
||
}
|
||
],
|
||
"metadata": {
|
||
"kernelspec": {
|
||
"display_name": "Python 3",
|
||
"language": "python",
|
||
"name": "python3"
|
||
},
|
||
"language_info": {
|
||
"codemirror_mode": {
|
||
"name": "ipython",
|
||
"version": 3
|
||
},
|
||
"file_extension": ".py",
|
||
"mimetype": "text/x-python",
|
||
"name": "python",
|
||
"nbconvert_exporter": "python",
|
||
"pygments_lexer": "ipython3",
|
||
"version": "3.11.5"
|
||
},
|
||
"orig_nbformat": 4
|
||
},
|
||
"nbformat": 4,
|
||
"nbformat_minor": 2
|
||
}
|