Merge "perf_tips.md" into "best_practices.md" and "fuzzing_in_depth.md"

docs/best_practices.md

@@ -64,11 +64,11 @@ which allows you to define network state with different type of data packets.
 
 1. Use [llvm_mode](../instrumentation/README.llvm.md): afl-clang-lto (llvm >= 11) or afl-clang-fast (llvm >= 9 recommended).
 2. Use [persistent mode](../instrumentation/README.persistent_mode.md) (x2-x20 speed increase).
-3. Use the [AFL++ snapshot module](https://github.com/AFLplusplus/AFL-Snapshot-LKM) (x2 speed increase).
+3. Instrument just what you are interested in, see [instrumentation/README.instrument_list.md](../instrumentation/README.instrument_list.md).
 4. If you do not use shmem persistent mode, use `AFL_TMPDIR` to put the input file directory on a tempfs location, see [env_variables.md](env_variables.md).
 5. Improve Linux kernel performance: modify `/etc/default/grub`, set `GRUB_CMDLINE_LINUX_DEFAULT="ibpb=off ibrs=off kpti=off l1tf=off mds=off mitigations=off no_stf_barrier noibpb noibrs nopcid nopti nospec_store_bypass_disable nospectre_v1 nospectre_v2 pcid=off pti=off spec_store_bypass_disable=off spectre_v2=off stf_barrier=off"`; then `update-grub` and `reboot` (warning: makes the system less secure).
 6. Running on an `ext2` filesystem with `noatime` mount option will be a bit faster than on any other journaling filesystem.
-7. Use your cores ([fuzzing_in_depth.md:b) Using multiple cores](fuzzing_in_depth.md#b-using-multiple-cores))!
+7. Use your cores ([fuzzing_in_depth.md:3c) Using multiple cores](fuzzing_in_depth.md#c-using-multiple-cores))!
 
 ### Improving stability
 

docs/fuzzing_in_depth.md

@@ -419,7 +419,7 @@ as test data in there.
 
 If you do not want anything special, the defaults are already usually best,
 hence all you need is to specify the seed input directory with the result of
-step [2a. Collect inputs](#a-collect-inputs):
+step [2a) Collect inputs](#a-collect-inputs):
 `afl-fuzz -i input -o output -- bin/target -d @@`
 Note that the directory specified with -o will be created if it does not exist.
 
@@ -438,11 +438,6 @@ If you need to stop and re-start the fuzzing, use the same command line options
 mode!) and switch the input directory with a dash (`-`):
 `afl-fuzz -i - -o output -- bin/target -d @@`
 
-Memory limits are not enforced by afl-fuzz by default and the system may run out
-of memory. You can decrease the memory with the `-m` option, the value is in MB.
-If this is too small for the target, you can usually see this by afl-fuzz
-bailing with the message that it could not connect to the forkserver.
-
 Adding a dictionary is helpful. See the directory
 [dictionaries/](../dictionaries/) if something is already included for your data
 format, and tell afl-fuzz to load that dictionary by adding `-x
@@ -472,7 +467,26 @@ is:
 
 All labels are explained in [status_screen.md](status_screen.md).
 
-#### b) Using multiple cores
+#### b) Keeping memory use and timeouts in check
+
+Memory limits are not enforced by afl-fuzz by default and the system may run out
+of memory. You can decrease the memory with the `-m` option, the value is in MB.
+If this is too small for the target, you can usually see this by afl-fuzz
+bailing with the message that it could not connect to the forkserver.
+
+Consider setting low values for `-m` and `-t`.
+
+For programs that are nominally very fast, but get sluggish for some inputs, you
+can also try setting `-t` values that are more punishing than what `afl-fuzz`
+dares to use on its own. On fast and idle machines, going down to `-t 5` may be
+a viable plan.
+
+The `-m` parameter is worth looking at, too. Some programs can end up spending a
+fair amount of time allocating and initializing megabytes of memory when
+presented with pathological inputs. Low `-m` values can make them give up sooner
+and not waste CPU time.
+
+#### c) Using multiple cores
 
 If you want to seriously fuzz then use as many cores/threads as possible to fuzz
 your target.
@@ -537,7 +551,7 @@ directory of a different fuzzer is, e.g. `-F /src/target/honggfuzz`. Using
 honggfuzz (with `-n 1` or `-n 2`) and libfuzzer in parallel is highly
 recommended!
 
-#### c) Using multiple machines for fuzzing
+#### d) Using multiple machines for fuzzing
 
 Maybe you have more than one machine you want to fuzz the same target on.
 Simply start the `afl-fuzz` (and perhaps libfuzzer, honggfuzz, ...)
@@ -575,7 +589,7 @@ done
 You can run this manually, per cron job - as you need it. There is a more
 complex and configurable script in `utils/distributed_fuzzing`.
 
-#### d) The status of the fuzz campaign
+#### e) The status of the fuzz campaign
 
 AFL++ comes with the `afl-whatsup` script to show the status of the fuzzing
 campaign.
@@ -593,7 +607,7 @@ afl-plot, which generates an index.html file and a graphs that show how the
 fuzzing instance is performing. The syntax is `afl-plot instance_dir web_dir`,
 e.g., `afl-plot out/default /srv/www/htdocs/plot`.
 
-#### e) Stopping fuzzing, restarting fuzzing, adding new seeds
+#### f) Stopping fuzzing, restarting fuzzing, adding new seeds
 
 To stop an afl-fuzz run, simply press Control-C.
 
@@ -608,7 +622,7 @@ are in `newseeds/` directory:
 AFL_BENCH_JUST_ONE=1 AFL_FAST_CAL=1 afl-fuzz -i newseeds -o out -S newseeds -- ./target
 ```
 
-#### f) Checking the coverage of the fuzzing
+#### g) Checking the coverage of the fuzzing
 
 The `paths found` value is a bad indicator for checking how good the coverage
 is.
@@ -648,7 +662,7 @@ individual fuzzing campaigns each with one of these options set. E.g., if you
 fuzz a library to convert image formats and your target is the png to tiff API
 then you will not touch any of the other library APIs and features.
 
-#### g) How long to fuzz a target?
+#### h) How long to fuzz a target?
 
 This is a difficult question. Basically if no new path is found for a long time
 (e.g. for a day or a week) then you can expect that your fuzzing won't be
@@ -660,7 +674,7 @@ Keep the queue/ directory (for future fuzzings of the same or similar targets)
 and use them to seed other good fuzzers like libfuzzer with the -entropic switch
 or honggfuzz.
 
-#### h) Improve the speed!
+#### i) Improve the speed!
 
 * Use [persistent mode](../instrumentation/README.persistent_mode.md) (x2-x20
   speed increase)
@@ -675,11 +689,11 @@ or honggfuzz.
   also just run `sudo afl-persistent-config`
 * Linux: Running on an `ext2` filesystem with `noatime` mount option will be a
   bit faster than on any other journaling filesystem
-* Use your cores! [b) Using multiple cores](#b-using-multiple-cores)
+* Use your cores! [3c) Using multiple cores](#c-using-multiple-cores)
 * Run `sudo afl-system-config` before starting the first afl-fuzz instance after
   a reboot
 
-#### i) Going beyond crashes
+#### j) Going beyond crashes
 
 Fuzzing is a wonderful and underutilized technique for discovering non-crashing
 design and implementation errors, too. Quite a few interesting bugs have been
@@ -703,7 +717,7 @@ conditional with `#ifdef FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION` (a flag also
 shared with libfuzzer and honggfuzz) or `#ifdef __AFL_COMPILER` (this one is
 just for AFL++).
 
-#### j) Known limitations & areas for improvement
+#### k) Known limitations & areas for improvement
 
 Here are some of the most important caveats for AFL++:
 

docs/perf_tips.md

@@ -1,209 +0,0 @@
-## Tips for performance optimization
-
-  This file provides tips for troubleshooting slow or wasteful fuzzing jobs.
-  See README.md for the general instruction manual.
-
-## 1. Keep your test cases small
-
-This is probably the single most important step to take! Large test cases do
-not merely take more time and memory to be parsed by the tested binary, but
-also make the fuzzing process dramatically less efficient in several other
-ways.
-
-To illustrate, let's say that you're randomly flipping bits in a file, one bit
-at a time. Let's assume that if you flip bit #47, you will hit a security bug;
-flipping any other bit just results in an invalid document.
-
-Now, if your starting test case is 100 bytes long, you will have a 71% chance of
-triggering the bug within the first 1,000 execs - not bad! But if the test case
-is 1 kB long, the probability that we will randomly hit the right pattern in
-the same timeframe goes down to 11%. And if it has 10 kB of non-essential
-cruft, the odds plunge to 1%.
-
-On top of that, with larger inputs, the binary may be now running 5-10x times
-slower than before - so the overall drop in fuzzing efficiency may be easily
-as high as 500x or so.
-
-In practice, this means that you shouldn't fuzz image parsers with your
-vacation photos. Generate a tiny 16x16 picture instead, and run it through
-`jpegtran` or `pngcrunch` for good measure. The same goes for most other types
-of documents.
-
-There's plenty of small starting test cases in ../testcases/ - try them out
-or submit new ones!
-
-If you want to start with a larger, third-party corpus, run `afl-cmin` with an
-aggressive timeout on that data set first.
-
-## 2. Use a simpler target
-
-Consider using a simpler target binary in your fuzzing work. For example, for
-image formats, bundled utilities such as `djpeg`, `readpng`, or `gifhisto` are
-considerably (10-20x) faster than the convert tool from ImageMagick - all while exercising roughly the same library-level image parsing code.
-
-Even if you don't have a lightweight harness for a particular target, remember
-that you can always use another, related library to generate a corpus that will
-be then manually fed to a more resource-hungry program later on.
-
-Also note that reading the fuzzing input via stdin is faster than reading from
-a file.
-
-## 3. Use LLVM persistent instrumentation
-
-The LLVM mode offers a "persistent", in-process fuzzing mode that can
-work well for certain types of self-contained libraries, and for fast targets,
-can offer performance gains up to 5-10x; and a "deferred fork server" mode
-that can offer huge benefits for programs with high startup overhead. Both
-modes require you to edit the source code of the fuzzed program, but the
-changes often amount to just strategically placing a single line or two.
-
-If there are important data comparisons performed (e.g. `strcmp(ptr, MAGIC_HDR)`)
-then using laf-intel (see instrumentation/README.laf-intel.md) will help `afl-fuzz` a lot
-to get to the important parts in the code.
-
-If you are only interested in specific parts of the code being fuzzed, you can
-instrument_files the files that are actually relevant. This improves the speed and
-accuracy of afl. See instrumentation/README.instrument_list.md
-
-## 4. Profile and optimize the binary
-
-Check for any parameters or settings that obviously improve performance. For
-example, the djpeg utility that comes with IJG jpeg and libjpeg-turbo can be
-called with:
-
-```bash
-  -dct fast -nosmooth -onepass -dither none -scale 1/4
-```
-
-...and that will speed things up. There is a corresponding drop in the quality
-of decoded images, but it's probably not something you care about.
-
-In some programs, it is possible to disable output altogether, or at least use
-an output format that is computationally inexpensive. For example, with image
-transcoding tools, converting to a BMP file will be a lot faster than to PNG.
-
-With some laid-back parsers, enabling "strict" mode (i.e., bailing out after
-first error) may result in smaller files and improved run time without
-sacrificing coverage; for example, for sqlite, you may want to specify -bail.
-
-If the program is still too slow, you can use `strace -tt` or an equivalent
-profiling tool to see if the targeted binary is doing anything silly.
-Sometimes, you can speed things up simply by specifying `/dev/null` as the
-config file, or disabling some compile-time features that aren't really needed
-for the job (try `./configure --help`). One of the notoriously resource-consuming
-things would be calling other utilities via `exec*()`, `popen()`, `system()`, or
-equivalent calls; for example, tar can invoke external decompression tools
-when it decides that the input file is a compressed archive.
-
-Some programs may also intentionally call `sleep()`, `usleep()`, or `nanosleep()`;
-vim is a good example of that. Other programs may attempt `fsync()` and so on.
-There are third-party libraries that make it easy to get rid of such code,
-e.g.:
-
-  https://launchpad.net/libeatmydata
-
-In programs that are slow due to unavoidable initialization overhead, you may
-want to try the LLVM deferred forkserver mode (see README.llvm.md),
-which can give you speed gains up to 10x, as mentioned above.
-
-Last but not least, if you are using ASAN and the performance is unacceptable,
-consider turning it off for now, and manually examining the generated corpus
-with an ASAN-enabled binary later on.
-
-## 5. Instrument just what you need
-
-Instrument just the libraries you actually want to stress-test right now, one
-at a time. Let the program use system-wide, non-instrumented libraries for
-any functionality you don't actually want to fuzz. For example, in most
-cases, it doesn't make to instrument `libgmp` just because you're testing a
-crypto app that relies on it for bignum math.
-
-Beware of programs that come with oddball third-party libraries bundled with
-their source code (Spidermonkey is a good example of this). Check `./configure`
-options to use non-instrumented system-wide copies instead.
-
-## 6. Parallelize your fuzzers
-
-The fuzzer is designed to need ~1 core per job. This means that on a, say,
-4-core system, you can easily run four parallel fuzzing jobs with relatively
-little performance hit. For tips on how to do that, see parallel_fuzzing.md.
-
-The `afl-gotcpu` utility can help you understand if you still have idle CPU
-capacity on your system. (It won't tell you about memory bandwidth, cache
-misses, or similar factors, but they are less likely to be a concern.)
-
-## 7. Keep memory use and timeouts in check
-
-Consider setting low values for `-m` and `-t`.
-
-For programs that are nominally very fast, but get sluggish for some inputs,
-you can also try setting `-t` values that are more punishing than what `afl-fuzz`
-dares to use on its own. On fast and idle machines, going down to `-t 5` may be
-a viable plan.
-
-The `-m` parameter is worth looking at, too. Some programs can end up spending
-a fair amount of time allocating and initializing megabytes of memory when
-presented with pathological inputs. Low `-m` values can make them give up sooner
-and not waste CPU time.
-
-## 8. Check OS configuration
-
-There are several OS-level factors that may affect fuzzing speed:
-
-  - If you have no risk of power loss then run your fuzzing on a tmpfs
-    partition. This increases the performance noticably.
-    Alternatively you can use `AFL_TMPDIR` to point to a tmpfs location to
-    just write the input file to a tmpfs.
-  - High system load. Use idle machines where possible. Kill any non-essential
-    CPU hogs (idle browser windows, media players, complex screensavers, etc).
-  - Network filesystems, either used for fuzzer input / output, or accessed by
-    the fuzzed binary to read configuration files (pay special attention to the
-    home directory - many programs search it for dot-files).
-  - Disable all the spectre, meltdown etc. security countermeasures in the
-    kernel if your machine is properly separated:
-
-```
-ibpb=off ibrs=off kpti=off l1tf=off mds=off mitigations=off
-no_stf_barrier noibpb noibrs nopcid nopti nospec_store_bypass_disable
-nospectre_v1 nospectre_v2 pcid=off pti=off spec_store_bypass_disable=off
-spectre_v2=off stf_barrier=off
-```
-    In most Linux distributions you can put this into a `/etc/default/grub`
-    variable.
-    You can use `sudo afl-persistent-config` to set these options for you.
-
-The following list of changes are made when executing `afl-system-config`:
- 
-  - On-demand CPU scaling. The Linux `ondemand` governor performs its analysis
-    on a particular schedule and is known to underestimate the needs of
-    short-lived processes spawned by `afl-fuzz` (or any other fuzzer). On Linux,
-    this can be fixed with:
-
-``` bash
-    cd /sys/devices/system/cpu
-    echo performance | tee cpu*/cpufreq/scaling_governor
-```
-
-    On other systems, the impact of CPU scaling will be different; when fuzzing,
-    use OS-specific tools to find out if all cores are running at full speed.
-  - Transparent huge pages. Some allocators, such as `jemalloc`, can incur a
-    heavy fuzzing penalty when transparent huge pages (THP) are enabled in the
-    kernel. You can disable this via:
-
-```bash
-    echo never > /sys/kernel/mm/transparent_hugepage/enabled
-```
-
-  - Suboptimal scheduling strategies. The significance of this will vary from
-    one target to another, but on Linux, you may want to make sure that the
-    following options are set:
-
-```bash
-    echo 1 >/proc/sys/kernel/sched_child_runs_first
-    echo 1 >/proc/sys/kernel/sched_autogroup_enabled
-```
-
-    Setting a different scheduling policy for the fuzzer process - say
-    `SCHED_RR` - can usually speed things up, too, but needs to be done with
-    care.
-