mirror of
https://github.com/AFLplusplus/AFLplusplus.git
synced 2025-06-11 01:31:37 +00:00
Merge "perf_tips.md" into "best_practices.md" and "fuzzing_in_depth.md"
This commit is contained in:
parent
5b480f9451
commit
fce93647cc
@ -64,11 +64,11 @@ which allows you to define network state with different type of data packets.
|
||||
|
||||
1. Use [llvm_mode](../instrumentation/README.llvm.md): afl-clang-lto (llvm >= 11) or afl-clang-fast (llvm >= 9 recommended).
|
||||
2. Use [persistent mode](../instrumentation/README.persistent_mode.md) (x2-x20 speed increase).
|
||||
3. Use the [AFL++ snapshot module](https://github.com/AFLplusplus/AFL-Snapshot-LKM) (x2 speed increase).
|
||||
3. Instrument just what you are interested in, see [instrumentation/README.instrument_list.md](../instrumentation/README.instrument_list.md).
|
||||
4. If you do not use shmem persistent mode, use `AFL_TMPDIR` to put the input file directory on a tempfs location, see [env_variables.md](env_variables.md).
|
||||
5. Improve Linux kernel performance: modify `/etc/default/grub`, set `GRUB_CMDLINE_LINUX_DEFAULT="ibpb=off ibrs=off kpti=off l1tf=off mds=off mitigations=off no_stf_barrier noibpb noibrs nopcid nopti nospec_store_bypass_disable nospectre_v1 nospectre_v2 pcid=off pti=off spec_store_bypass_disable=off spectre_v2=off stf_barrier=off"`; then `update-grub` and `reboot` (warning: makes the system less secure).
|
||||
6. Running on an `ext2` filesystem with `noatime` mount option will be a bit faster than on any other journaling filesystem.
|
||||
7. Use your cores ([fuzzing_in_depth.md:b) Using multiple cores](fuzzing_in_depth.md#b-using-multiple-cores))!
|
||||
7. Use your cores ([fuzzing_in_depth.md:3c) Using multiple cores](fuzzing_in_depth.md#c-using-multiple-cores))!
|
||||
|
||||
### Improving stability
|
||||
|
||||
|
@ -419,7 +419,7 @@ as test data in there.
|
||||
|
||||
If you do not want anything special, the defaults are already usually best,
|
||||
hence all you need is to specify the seed input directory with the result of
|
||||
step [2a. Collect inputs](#a-collect-inputs):
|
||||
step [2a) Collect inputs](#a-collect-inputs):
|
||||
`afl-fuzz -i input -o output -- bin/target -d @@`
|
||||
Note that the directory specified with -o will be created if it does not exist.
|
||||
|
||||
@ -438,11 +438,6 @@ If you need to stop and re-start the fuzzing, use the same command line options
|
||||
mode!) and switch the input directory with a dash (`-`):
|
||||
`afl-fuzz -i - -o output -- bin/target -d @@`
|
||||
|
||||
Memory limits are not enforced by afl-fuzz by default and the system may run out
|
||||
of memory. You can decrease the memory with the `-m` option, the value is in MB.
|
||||
If this is too small for the target, you can usually see this by afl-fuzz
|
||||
bailing with the message that it could not connect to the forkserver.
|
||||
|
||||
Adding a dictionary is helpful. See the directory
|
||||
[dictionaries/](../dictionaries/) if something is already included for your data
|
||||
format, and tell afl-fuzz to load that dictionary by adding `-x
|
||||
@ -472,7 +467,26 @@ is:
|
||||
|
||||
All labels are explained in [status_screen.md](status_screen.md).
|
||||
|
||||
#### b) Using multiple cores
|
||||
#### b) Keeping memory use and timeouts in check
|
||||
|
||||
Memory limits are not enforced by afl-fuzz by default and the system may run out
|
||||
of memory. You can decrease the memory with the `-m` option, the value is in MB.
|
||||
If this is too small for the target, you can usually see this by afl-fuzz
|
||||
bailing with the message that it could not connect to the forkserver.
|
||||
|
||||
Consider setting low values for `-m` and `-t`.
|
||||
|
||||
For programs that are nominally very fast, but get sluggish for some inputs, you
|
||||
can also try setting `-t` values that are more punishing than what `afl-fuzz`
|
||||
dares to use on its own. On fast and idle machines, going down to `-t 5` may be
|
||||
a viable plan.
|
||||
|
||||
The `-m` parameter is worth looking at, too. Some programs can end up spending a
|
||||
fair amount of time allocating and initializing megabytes of memory when
|
||||
presented with pathological inputs. Low `-m` values can make them give up sooner
|
||||
and not waste CPU time.
|
||||
|
||||
#### c) Using multiple cores
|
||||
|
||||
If you want to seriously fuzz then use as many cores/threads as possible to fuzz
|
||||
your target.
|
||||
@ -537,7 +551,7 @@ directory of a different fuzzer is, e.g. `-F /src/target/honggfuzz`. Using
|
||||
honggfuzz (with `-n 1` or `-n 2`) and libfuzzer in parallel is highly
|
||||
recommended!
|
||||
|
||||
#### c) Using multiple machines for fuzzing
|
||||
#### d) Using multiple machines for fuzzing
|
||||
|
||||
Maybe you have more than one machine you want to fuzz the same target on.
|
||||
Simply start the `afl-fuzz` (and perhaps libfuzzer, honggfuzz, ...)
|
||||
@ -575,7 +589,7 @@ done
|
||||
You can run this manually, per cron job - as you need it. There is a more
|
||||
complex and configurable script in `utils/distributed_fuzzing`.
|
||||
|
||||
#### d) The status of the fuzz campaign
|
||||
#### e) The status of the fuzz campaign
|
||||
|
||||
AFL++ comes with the `afl-whatsup` script to show the status of the fuzzing
|
||||
campaign.
|
||||
@ -593,7 +607,7 @@ afl-plot, which generates an index.html file and a graphs that show how the
|
||||
fuzzing instance is performing. The syntax is `afl-plot instance_dir web_dir`,
|
||||
e.g., `afl-plot out/default /srv/www/htdocs/plot`.
|
||||
|
||||
#### e) Stopping fuzzing, restarting fuzzing, adding new seeds
|
||||
#### f) Stopping fuzzing, restarting fuzzing, adding new seeds
|
||||
|
||||
To stop an afl-fuzz run, simply press Control-C.
|
||||
|
||||
@ -608,7 +622,7 @@ are in `newseeds/` directory:
|
||||
AFL_BENCH_JUST_ONE=1 AFL_FAST_CAL=1 afl-fuzz -i newseeds -o out -S newseeds -- ./target
|
||||
```
|
||||
|
||||
#### f) Checking the coverage of the fuzzing
|
||||
#### g) Checking the coverage of the fuzzing
|
||||
|
||||
The `paths found` value is a bad indicator for checking how good the coverage
|
||||
is.
|
||||
@ -648,7 +662,7 @@ individual fuzzing campaigns each with one of these options set. E.g., if you
|
||||
fuzz a library to convert image formats and your target is the png to tiff API
|
||||
then you will not touch any of the other library APIs and features.
|
||||
|
||||
#### g) How long to fuzz a target?
|
||||
#### h) How long to fuzz a target?
|
||||
|
||||
This is a difficult question. Basically if no new path is found for a long time
|
||||
(e.g. for a day or a week) then you can expect that your fuzzing won't be
|
||||
@ -660,7 +674,7 @@ Keep the queue/ directory (for future fuzzings of the same or similar targets)
|
||||
and use them to seed other good fuzzers like libfuzzer with the -entropic switch
|
||||
or honggfuzz.
|
||||
|
||||
#### h) Improve the speed!
|
||||
#### i) Improve the speed!
|
||||
|
||||
* Use [persistent mode](../instrumentation/README.persistent_mode.md) (x2-x20
|
||||
speed increase)
|
||||
@ -675,11 +689,11 @@ or honggfuzz.
|
||||
also just run `sudo afl-persistent-config`
|
||||
* Linux: Running on an `ext2` filesystem with `noatime` mount option will be a
|
||||
bit faster than on any other journaling filesystem
|
||||
* Use your cores! [b) Using multiple cores](#b-using-multiple-cores)
|
||||
* Use your cores! [3c) Using multiple cores](#c-using-multiple-cores)
|
||||
* Run `sudo afl-system-config` before starting the first afl-fuzz instance after
|
||||
a reboot
|
||||
|
||||
#### i) Going beyond crashes
|
||||
#### j) Going beyond crashes
|
||||
|
||||
Fuzzing is a wonderful and underutilized technique for discovering non-crashing
|
||||
design and implementation errors, too. Quite a few interesting bugs have been
|
||||
@ -703,7 +717,7 @@ conditional with `#ifdef FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION` (a flag also
|
||||
shared with libfuzzer and honggfuzz) or `#ifdef __AFL_COMPILER` (this one is
|
||||
just for AFL++).
|
||||
|
||||
#### j) Known limitations & areas for improvement
|
||||
#### k) Known limitations & areas for improvement
|
||||
|
||||
Here are some of the most important caveats for AFL++:
|
||||
|
||||
|
@ -1,209 +0,0 @@
|
||||
## Tips for performance optimization
|
||||
|
||||
This file provides tips for troubleshooting slow or wasteful fuzzing jobs.
|
||||
See README.md for the general instruction manual.
|
||||
|
||||
## 1. Keep your test cases small
|
||||
|
||||
This is probably the single most important step to take! Large test cases do
|
||||
not merely take more time and memory to be parsed by the tested binary, but
|
||||
also make the fuzzing process dramatically less efficient in several other
|
||||
ways.
|
||||
|
||||
To illustrate, let's say that you're randomly flipping bits in a file, one bit
|
||||
at a time. Let's assume that if you flip bit #47, you will hit a security bug;
|
||||
flipping any other bit just results in an invalid document.
|
||||
|
||||
Now, if your starting test case is 100 bytes long, you will have a 71% chance of
|
||||
triggering the bug within the first 1,000 execs - not bad! But if the test case
|
||||
is 1 kB long, the probability that we will randomly hit the right pattern in
|
||||
the same timeframe goes down to 11%. And if it has 10 kB of non-essential
|
||||
cruft, the odds plunge to 1%.
|
||||
|
||||
On top of that, with larger inputs, the binary may be now running 5-10x times
|
||||
slower than before - so the overall drop in fuzzing efficiency may be easily
|
||||
as high as 500x or so.
|
||||
|
||||
In practice, this means that you shouldn't fuzz image parsers with your
|
||||
vacation photos. Generate a tiny 16x16 picture instead, and run it through
|
||||
`jpegtran` or `pngcrunch` for good measure. The same goes for most other types
|
||||
of documents.
|
||||
|
||||
There's plenty of small starting test cases in ../testcases/ - try them out
|
||||
or submit new ones!
|
||||
|
||||
If you want to start with a larger, third-party corpus, run `afl-cmin` with an
|
||||
aggressive timeout on that data set first.
|
||||
|
||||
## 2. Use a simpler target
|
||||
|
||||
Consider using a simpler target binary in your fuzzing work. For example, for
|
||||
image formats, bundled utilities such as `djpeg`, `readpng`, or `gifhisto` are
|
||||
considerably (10-20x) faster than the convert tool from ImageMagick - all while exercising roughly the same library-level image parsing code.
|
||||
|
||||
Even if you don't have a lightweight harness for a particular target, remember
|
||||
that you can always use another, related library to generate a corpus that will
|
||||
be then manually fed to a more resource-hungry program later on.
|
||||
|
||||
Also note that reading the fuzzing input via stdin is faster than reading from
|
||||
a file.
|
||||
|
||||
## 3. Use LLVM persistent instrumentation
|
||||
|
||||
The LLVM mode offers a "persistent", in-process fuzzing mode that can
|
||||
work well for certain types of self-contained libraries, and for fast targets,
|
||||
can offer performance gains up to 5-10x; and a "deferred fork server" mode
|
||||
that can offer huge benefits for programs with high startup overhead. Both
|
||||
modes require you to edit the source code of the fuzzed program, but the
|
||||
changes often amount to just strategically placing a single line or two.
|
||||
|
||||
If there are important data comparisons performed (e.g. `strcmp(ptr, MAGIC_HDR)`)
|
||||
then using laf-intel (see instrumentation/README.laf-intel.md) will help `afl-fuzz` a lot
|
||||
to get to the important parts in the code.
|
||||
|
||||
If you are only interested in specific parts of the code being fuzzed, you can
|
||||
instrument_files the files that are actually relevant. This improves the speed and
|
||||
accuracy of afl. See instrumentation/README.instrument_list.md
|
||||
|
||||
## 4. Profile and optimize the binary
|
||||
|
||||
Check for any parameters or settings that obviously improve performance. For
|
||||
example, the djpeg utility that comes with IJG jpeg and libjpeg-turbo can be
|
||||
called with:
|
||||
|
||||
```bash
|
||||
-dct fast -nosmooth -onepass -dither none -scale 1/4
|
||||
```
|
||||
|
||||
...and that will speed things up. There is a corresponding drop in the quality
|
||||
of decoded images, but it's probably not something you care about.
|
||||
|
||||
In some programs, it is possible to disable output altogether, or at least use
|
||||
an output format that is computationally inexpensive. For example, with image
|
||||
transcoding tools, converting to a BMP file will be a lot faster than to PNG.
|
||||
|
||||
With some laid-back parsers, enabling "strict" mode (i.e., bailing out after
|
||||
first error) may result in smaller files and improved run time without
|
||||
sacrificing coverage; for example, for sqlite, you may want to specify -bail.
|
||||
|
||||
If the program is still too slow, you can use `strace -tt` or an equivalent
|
||||
profiling tool to see if the targeted binary is doing anything silly.
|
||||
Sometimes, you can speed things up simply by specifying `/dev/null` as the
|
||||
config file, or disabling some compile-time features that aren't really needed
|
||||
for the job (try `./configure --help`). One of the notoriously resource-consuming
|
||||
things would be calling other utilities via `exec*()`, `popen()`, `system()`, or
|
||||
equivalent calls; for example, tar can invoke external decompression tools
|
||||
when it decides that the input file is a compressed archive.
|
||||
|
||||
Some programs may also intentionally call `sleep()`, `usleep()`, or `nanosleep()`;
|
||||
vim is a good example of that. Other programs may attempt `fsync()` and so on.
|
||||
There are third-party libraries that make it easy to get rid of such code,
|
||||
e.g.:
|
||||
|
||||
https://launchpad.net/libeatmydata
|
||||
|
||||
In programs that are slow due to unavoidable initialization overhead, you may
|
||||
want to try the LLVM deferred forkserver mode (see README.llvm.md),
|
||||
which can give you speed gains up to 10x, as mentioned above.
|
||||
|
||||
Last but not least, if you are using ASAN and the performance is unacceptable,
|
||||
consider turning it off for now, and manually examining the generated corpus
|
||||
with an ASAN-enabled binary later on.
|
||||
|
||||
## 5. Instrument just what you need
|
||||
|
||||
Instrument just the libraries you actually want to stress-test right now, one
|
||||
at a time. Let the program use system-wide, non-instrumented libraries for
|
||||
any functionality you don't actually want to fuzz. For example, in most
|
||||
cases, it doesn't make to instrument `libgmp` just because you're testing a
|
||||
crypto app that relies on it for bignum math.
|
||||
|
||||
Beware of programs that come with oddball third-party libraries bundled with
|
||||
their source code (Spidermonkey is a good example of this). Check `./configure`
|
||||
options to use non-instrumented system-wide copies instead.
|
||||
|
||||
## 6. Parallelize your fuzzers
|
||||
|
||||
The fuzzer is designed to need ~1 core per job. This means that on a, say,
|
||||
4-core system, you can easily run four parallel fuzzing jobs with relatively
|
||||
little performance hit. For tips on how to do that, see parallel_fuzzing.md.
|
||||
|
||||
The `afl-gotcpu` utility can help you understand if you still have idle CPU
|
||||
capacity on your system. (It won't tell you about memory bandwidth, cache
|
||||
misses, or similar factors, but they are less likely to be a concern.)
|
||||
|
||||
## 7. Keep memory use and timeouts in check
|
||||
|
||||
Consider setting low values for `-m` and `-t`.
|
||||
|
||||
For programs that are nominally very fast, but get sluggish for some inputs,
|
||||
you can also try setting `-t` values that are more punishing than what `afl-fuzz`
|
||||
dares to use on its own. On fast and idle machines, going down to `-t 5` may be
|
||||
a viable plan.
|
||||
|
||||
The `-m` parameter is worth looking at, too. Some programs can end up spending
|
||||
a fair amount of time allocating and initializing megabytes of memory when
|
||||
presented with pathological inputs. Low `-m` values can make them give up sooner
|
||||
and not waste CPU time.
|
||||
|
||||
## 8. Check OS configuration
|
||||
|
||||
There are several OS-level factors that may affect fuzzing speed:
|
||||
|
||||
- If you have no risk of power loss then run your fuzzing on a tmpfs
|
||||
partition. This increases the performance noticably.
|
||||
Alternatively you can use `AFL_TMPDIR` to point to a tmpfs location to
|
||||
just write the input file to a tmpfs.
|
||||
- High system load. Use idle machines where possible. Kill any non-essential
|
||||
CPU hogs (idle browser windows, media players, complex screensavers, etc).
|
||||
- Network filesystems, either used for fuzzer input / output, or accessed by
|
||||
the fuzzed binary to read configuration files (pay special attention to the
|
||||
home directory - many programs search it for dot-files).
|
||||
- Disable all the spectre, meltdown etc. security countermeasures in the
|
||||
kernel if your machine is properly separated:
|
||||
|
||||
```
|
||||
ibpb=off ibrs=off kpti=off l1tf=off mds=off mitigations=off
|
||||
no_stf_barrier noibpb noibrs nopcid nopti nospec_store_bypass_disable
|
||||
nospectre_v1 nospectre_v2 pcid=off pti=off spec_store_bypass_disable=off
|
||||
spectre_v2=off stf_barrier=off
|
||||
```
|
||||
In most Linux distributions you can put this into a `/etc/default/grub`
|
||||
variable.
|
||||
You can use `sudo afl-persistent-config` to set these options for you.
|
||||
|
||||
The following list of changes are made when executing `afl-system-config`:
|
||||
|
||||
- On-demand CPU scaling. The Linux `ondemand` governor performs its analysis
|
||||
on a particular schedule and is known to underestimate the needs of
|
||||
short-lived processes spawned by `afl-fuzz` (or any other fuzzer). On Linux,
|
||||
this can be fixed with:
|
||||
|
||||
``` bash
|
||||
cd /sys/devices/system/cpu
|
||||
echo performance | tee cpu*/cpufreq/scaling_governor
|
||||
```
|
||||
|
||||
On other systems, the impact of CPU scaling will be different; when fuzzing,
|
||||
use OS-specific tools to find out if all cores are running at full speed.
|
||||
- Transparent huge pages. Some allocators, such as `jemalloc`, can incur a
|
||||
heavy fuzzing penalty when transparent huge pages (THP) are enabled in the
|
||||
kernel. You can disable this via:
|
||||
|
||||
```bash
|
||||
echo never > /sys/kernel/mm/transparent_hugepage/enabled
|
||||
```
|
||||
|
||||
- Suboptimal scheduling strategies. The significance of this will vary from
|
||||
one target to another, but on Linux, you may want to make sure that the
|
||||
following options are set:
|
||||
|
||||
```bash
|
||||
echo 1 >/proc/sys/kernel/sched_child_runs_first
|
||||
echo 1 >/proc/sys/kernel/sched_autogroup_enabled
|
||||
```
|
||||
|
||||
Setting a different scheduling policy for the fuzzer process - say
|
||||
`SCHED_RR` - can usually speed things up, too, but needs to be done with
|
||||
care.
|
||||
|
Loading…
x
Reference in New Issue
Block a user