When UBSan was enabled and 'make test' was executed, the following
runtime error was observed:
libradamsa.c:26055:29: runtime error: left shift of 16777215 by 8 places cannot be represented in type 'int'
This issue was caused by a left shift operation on a signed integer.
The fix involved changing the integer literal to an unsigned integer by
adding a 'U' suffix, ensuring the left shift operation produces the
correct value without overflow.
Fixes: 8178f4df ("remove radamsa, add radamsa custom mutator")
After commit 0c9d8e59 ("Fix undefined behavior by casting to uint64_t
before left shift"), the -Wno-shift-count-overflow flag is no longer
necessary to suppress compilation warnings. This commit removes the
flag, allowing the compiler to provide appropriate warnings for any
potential future cases of shifting too many bits.
Additionally, after commit eaf4a299 ("make redqueen hashmap not
default"), the hashmap is no longer in use. Therefore, this commit also
removes the hashmap from the GNUmakefile.
The memory allocation check in afl_custom_fuzz function was incorrect.
The condition was erroneously checking if ptr was non-null, whereas it
should return 0 when ptr is null. Correct the condition to properly
handle memory allocation failures.
Fixes: 32ffa266 ("max_len support")
According to the C standard, left-shifting a value by an amount greater
than or equal to the width of its promoted type results in undefined
behavior. To prevent potential unexpected results, explicitly cast the
uint8_t variable type to uint64_t before performing the left shift
operation by 56 bits. This ensures the operation is well-defined and
adheres to the standard.
Fixes: 40df85d1 ("adjust cmplog header")
