I have observed two problems:
1. A sudden "stability cliff" where stability drops precipitously.
2. A sudden jump to a 100% saturated "density map".
Both issues are due to attempted "recalibration" of a case at the
beginning of fuzz_one_original() or mopt_common_fuzzing(). See the
comments "CALIBRATION (only if failed earlier on)" in those functions
and the subsequent call to calibrate_case().
At those calls to calibrate_case(), afl->fsrv.trace_bits holds
trace_bits for a run of the SUT on a prior queue entry. However,
calibrate_case() may use the trace_bits as if they apply to the
current queue entry (afl->queue_cur).
Most often this bug causes the "stability cliff". Trace bits are
compared for runs on distinct inputs, which can be very different.
The result is a sudden drop in stability.
Sometimes it leads to the "saturated map" problem. A saturated
density map arises if the trace bits on the previous entry were
"simplified" by simplify_trace(). Simplified traces only contain the
values 1 and 128. They are meant to be compared against
virgin_crashes and virgin_tmouts.
However, this bug causes the (stale) simplified trace to be compared
against virgin_bits during a call to has_new_bits(), which causes
every byte in vigin_bits to be something other than 255. The overall
map density is determined by the percentage of bytes not 255, which
will be 100%. Worse, AFL++ will be unable to detect novel occurrences
of edge counts 1 and 128 going forward.
This patch avoids the above issues by clearing q->exec_cksum when
calibration fails. Recalibrations are forced to start with a fresh
trace on the queue entry.
Thanks to @andreafioraldi for suggesting the current, improved patch.
