wget/fuzz
sulfastor 4ce2f93600 Hi, Thank you again Darshit for your response. The RejectHeaderField rule rejects ANY header
of the header field while RejectHeader rejects ONLY the specified full header.
	Since we wanted to be sure a header field is not sent to the server we wrote this rule.

	* doc/wget.texi: Added --disable-header documentation.
        * fuzz/wget_options_fuzzer.dict: Update with --disable-header inputs.
        * src/http.c (disabled_header): Checks for disabled headers
	(request_set_header): Doesn't let header to be set if disabled
	(gethttp): frees disabled header to let overriding
        * src/init.c (cmd_dis_header), (check_user_disabled_header) added new option disabled_headers.
        * src/main.c: added new option --disable-header, added help description
        * src/options.h: added new option --disable-header
	* src/utils.h (vec_remove_header)
	* src/utils.c (vec_remove_header) removes all header instances from vector
        * testenv/Makefile.am: Added new test files
        * testenv/server/http/http_server.py: Added new rule RejectHeaderField
        * testenv/conf/reject_header_field.py: Added new rule RejectHeaderField
        * testenv/README: Added help description for new rule
        * testenv/Test-disable-default-headers.py: Test without using --header
        * testenv/Test-disable-headers-after.py: Test using --header before --disable-header
        * testenv/Test-disable-headers-before.py: Test using --header after --disable-header

Signed-off-by: sulfastor <torresmoisesa@gmail.com>, adham elkarn <adhamelkarn@hotmail.com>
2019-05-30 10:01:24 +02:00
..
wget_cookie_fuzzer.in * fuzz/*.in/*: Update corpora from OSS-fuzz 2019-04-26 14:47:17 +02:00
wget_css_fuzzer.in * fuzz/*.in/*: Update corpora from OSS-fuzz 2019-04-26 14:47:17 +02:00
wget_css_fuzzer.repro Fix buffer overflow in CSS parser 2018-04-26 22:40:28 +02:00
wget_ftpls_fuzzer.in * fuzz/*.in/*: Update corpora from OSS-fuzz 2019-04-26 14:47:17 +02:00
wget_html_fuzzer.in * fuzz/*.in/*: Update corpora from OSS-fuzz 2019-04-26 14:47:17 +02:00
wget_netrc_fuzzer.in * fuzz/*.in/*: Update corpora from OSS-fuzz 2019-04-26 14:47:17 +02:00
wget_options_fuzzer.in * fuzz/*.in/*: Update corpora from OSS-fuzz 2019-04-26 14:47:17 +02:00
wget_read_hunk_fuzzer.in * fuzz/*.in/*: Update corpora from OSS-fuzz 2019-04-26 14:47:17 +02:00
wget_robots_fuzzer.in * fuzz/*.in/*: Update corpora from OSS-fuzz 2019-04-26 14:47:17 +02:00
wget_url_fuzzer.in * fuzz/*.in/*: Update corpora from OSS-fuzz 2019-04-26 14:47:17 +02:00
fuzzer.h Fix STDERR closing/restoring in fuzzers 2019-02-19 15:50:34 +01:00
get_all_corpora Add OSS-Fuzz infrastruture 2018-04-16 09:58:51 +02:00
get_ossfuzz_corpora * fuzz/get_ossfuzz_corpora: Remove -f from unzip, mkdir *.in/ 2019-04-26 14:14:12 +02:00
glob_crash.c Update copyright statements 2019-02-10 11:50:17 +01:00
main.c * fuzz/main.c: Fix paths for WIN32 2019-05-07 17:05:32 +02:00
Makefile.am Add new fuzzer wget_read_hunk_fuzzer.c 2019-03-25 14:57:07 +01:00
README.md * fuzz/README.md: Add CFLAGS for undefined sanitizer 2018-04-24 11:30:06 +02:00
run-afl.sh Update copyright statements 2019-02-10 11:50:17 +01:00
run-clang.sh Update copyright statements 2019-02-10 11:50:17 +01:00
view-coverage.sh Add OSS-Fuzz infrastruture 2018-04-16 09:58:51 +02:00
wget_cookie_fuzzer.c Fix fuzz/ tests for OpenBSD 2019-02-19 17:18:49 +01:00
wget_cookie_fuzzer.dict Add new fuzzer for the Set-Cookie parser 2018-04-27 12:56:25 +02:00
wget_css_fuzzer.c Fix fuzz/ tests for OpenBSD 2019-02-19 17:18:49 +01:00
wget_ftpls_fuzzer.c * fuzz/wget_ftpls_fuzzer.c: Fix fuzzer 2019-03-03 17:05:44 +01:00
wget_ftpls_fuzzer.dict Add new fuzzer for the FTP listing parsers 2018-04-21 19:24:25 +02:00
wget_html_fuzzer.c Fix fuzz/ tests for OpenBSD 2019-02-19 17:18:49 +01:00
wget_html_fuzzer.dict Add new HTML parser fuzzer 2018-04-20 22:33:58 +02:00
wget_netrc_fuzzer.c * fuzz/wget_netrc_fuzzer.c: Fix fuzzer 2019-03-03 17:05:58 +01:00
wget_netrc_fuzzer.dict Add new fuzzer for the .netrc parser 2018-04-28 20:49:57 +02:00
wget_options_fuzzer.c Fix fuzz/ tests for OpenBSD 2019-02-19 17:18:49 +01:00
wget_options_fuzzer.dict Hi, Thank you again Darshit for your response. The RejectHeaderField rule rejects ANY header 2019-05-30 10:01:24 +02:00
wget_read_hunk_fuzzer.c Fix corner case in processing server response 2019-03-25 16:05:56 +01:00
wget_robots_fuzzer.c Fix fuzz/ tests for OpenBSD 2019-02-19 17:18:49 +01:00
wget_url_fuzzer.c Fix fuzz/ tests for OpenBSD 2019-02-19 17:18:49 +01:00

Fuzzers

These are fuzzers designed for use with libFuzzer or afl. They can be used to run on Google's OSS-Fuzz (https://github.com/google/oss-fuzz/).

The convention used here is that the initial values for each parser fuzzer are taken from the $NAME.in directory.

Crash reproducers from OSS-Fuzz are put into $NAME.repro directory for regression testing with top dir 'make check' or 'make check-valgrind'.

Running a fuzzer using clang

Use the following commands on top dir:

export CC=clang-6.0
export CXX=clang++-6.0
# address sanitizer:
export CFLAGS="-O1 -g -fno-omit-frame-pointer -gline-tables-only -DFUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION -fsanitize=undefined,integer,nullability -fsanitize=address -fsanitize-address-use-after-scope -fsanitize-coverage=trace-pc-guard,trace-cmp"
# undefined sanitizer;
export CFLAGS="-O1 -fno-omit-frame-pointer -gline-tables-only -DFUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION -fsanitize=bool,array-bounds,float-divide-by-zero,function,integer-divide-by-zero,return,shift,signed-integer-overflow,vla-bound,vptr -fno-sanitize-recover=bool,array-bounds,float-divide-by-zero,function,integer-divide-by-zero,return,shift,signed-integer-overflow,vla-bound,vptr -fsanitize=fuzzer-no-link"
export CXXFLAGS="$CFLAGS -stdlib=libc++"
export LIB_FUZZING_ENGINE="-lFuzzer -lstdc++"
./configure --enable-fuzzing --without-metalink --without-zlib --disable-pcre --without-libuuid
make clean
make -j$(nproc)
cd fuzz

# run wget_options_fuzzer
UBSAN_OPTIONS=print_stacktrace=1 ASAN_SYMBOLIZER_PATH=/usr/lib/llvm-6.0/bin/llvm-symbolizer \
  ./run-clang.sh wget_options_fuzzer

If you see a crash, then a crash corpora is written that can be used for further investigation. E.g.

==2410==ERROR: AddressSanitizer: heap-use-after-free on address 0x602000004e90 at pc 0x00000049cf9c bp 0x7fffb5543f70 sp 0x7fffb5543720
...
Test unit written to ./crash-adc83b19e793491b1c6ea0fd8b46cd9f32e592fc

To reproduce the crash:

./wget_options_fuzzer < ./crash-adc83b19e793491b1c6ea0fd8b46cd9f32e592fc

You can also copy/move that file into wget_options_fuzzer.repro/ and re-build the project without fuzzing for a valgrind run, if you like that better. Just a ./configure and a make check-valgrind should reproduce it.

Running a fuzzer using AFL

Use the following commands on top dir:

$ export LIB_FUZZING_ENGINE=""
$ CC=afl-clang-fast ./configure --enable-fuzzing
$ make -j$(nproc) clean all
$ cd fuzz
$ ./run-afl.sh wget_options_fuzzer

Fuzz code coverage using the corpus directories *.in/

Code coverage reports currently work best with gcc+lcov+genhtml.

In the top directory:

CC=gcc CFLAGS="-O0 -g" ./configure
make fuzz-coverage
xdg-open lcov/index.html

To work on corpora for better coverage, cd fuzz and use e.g. ./view-coverage.sh wget_options_fuzzer.

Creating wget_options_fuzzer.dict

for i in `../src/wget --help|tr ' ' '\n'|grep ^--|cut -c 3-|sort`;do echo \"$i\"; done >wget_options_fuzzer.dict