Fix saturated maps & stability cliff in recalibration

I have observed two problems:

  1. A sudden "stability cliff" where stability drops precipitously.

  2. A sudden jump to a 100% saturated "density map".

Both issues are due to attempted "recalibration" of a case at the
beginning of fuzz_one_original() or mopt_common_fuzzing().  See the
comments "CALIBRATION (only if failed earlier on)" in those functions
and the subsequent call to calibrate_case().

At those calls to calibrate_case(), afl->fsrv.trace_bits holds
trace_bits for a run of the SUT on a prior queue entry.  However,
calibrate_case() may use the trace_bits as if they apply to the
current queue entry (afl->queue_cur).

Most often this bug causes the "stability cliff".  Trace bits are
compared for runs on distinct inputs, which can be very different.
The result is a sudden drop in stability.

Sometimes it leads to the "saturated map" problem.  A saturated
density map arises if the trace bits on the previous entry were
"simplified" by simplify_trace().  Simplified traces only contain the
values 1 and 128.  They are meant to be compared against
virgin_crashes and virgin_tmouts.

However, this bug causes the (stale) simplified trace to be compared
against virgin_bits during a call to has_new_bits(), which causes
every byte in vigin_bits to be something other than 255.  The overall
map density is determined by the percentage of bytes not 255, which
will be 100%.  Worse, AFL++ will be unable to detect novel occurrences
of edge counts 1 and 128 going forward.

This patch avoids the above issues by clearing q->exec_cksum when
calibration fails.  Recalibrations are forced to start with a fresh
trace on the queue entry.

Thanks to @andreafioraldi for suggesting the current, improved patch.

src/afl-fuzz-run.c

@@ -374,6 +374,8 @@ u8 calibrate_case(afl_state_t *afl, struct queue_entry *q, u8 *use_mem,
 
 abort_calibration:
 
+  if (q->cal_failed) { q->exec_cksum = 0; }
+
   if (new_bits == 2 && !q->has_new_cov) {
 
     q->has_new_cov = 1;