While looking at the source code of the havoc mutations I realized that there seems to be a typo / copy+paste error with the SUBBYTE_ mutation. It is currently incrementing, instead of decrementing the value.
Alternative Fix: Change the documentation to "/* Decrease byte by minus 1. */" to make it work as documented :-P
* Revert "fixed potential double free in custom trim (#881)"
This reverts commit e9d2f72382.
* Revert "fix custom trim for increasing data"
This reverts commit 86a8ef168d.
* Fix memory errors when trim causes testcase growth
Modify trim_case_custom to avoid writing into in_buf because
some custom mutators can cause the testcase to grow rather than
shrink.
Instead of modifying in_buf directly, we write the update out
to the disk when trimming is complete, and then the caller is
responsible for refreshing the in-memory buffer from the file.
This is still a bit sketchy because it does need to modify q->len in
order to notify the upper layers that something changed, and it could
end up telling upper layer code that the q->len is *bigger* than
the buffer (q->testcase_buf) that contains it, which is asking
for trouble down the line somewhere...
* Fix an unlikely situation
Put back some `unlikely()` calls that were in
the e9d2f72382 commit that was
reverted.
