1
0
mirror of https://github.com/mirror/wget.git synced 2025-01-26 04:10:42 +08:00
Commit Graph

2552 Commits

Author SHA1 Message Date
Tim Rühsen
b0f802c46c * src/ftp-ls.c (ftp_parse_winnt_ls): Fix heap-buffer-overflow
Fixes OSS-Fuzz issue .
This is a long standing bug affecting all versions <= 1.19.4.
2018-04-22 12:45:51 +02:00
Tim Rühsen
96c64a859d * src/ftp-ls.c (ftp_parse_winnt_ls): Fix heap-buffer-overflow
Fixes OSS-Fuzz issue .
This is a long standing bug affecting all versions <= 1.19.4.
2018-04-22 11:33:35 +02:00
Tim Rühsen
7d3da08537 * src/ftp-ls.c (eat_carets): Fix heap-buffer-overflow 2018-04-21 23:48:01 +02:00
Tim Rühsen
2b61c46183 * src/ftp-ls.c (ftp_parse_winnt_ls): Fix memleak 2018-04-21 22:52:01 +02:00
Tim Rühsen
f0d715b264 * src/ftp-ls.c (ftp_parse_vms_ls): Fix heap-buffer-overflow 2018-04-21 22:47:17 +02:00
Tim Rühsen
b3ff8ce3d5 * src/ftp-ls.c (ftp_parse_vms_ls): Fix heap-buffer-overflow 2018-04-21 22:45:03 +02:00
Tim Rühsen
c7014fbaea * src/ftp-ls.c (ftp_parse_vms_ls): Fix memleak 2018-04-21 22:42:30 +02:00
Tim Rühsen
407cd5f23b Add new fuzzer for the FTP listing parsers
* fuzz/Makefile.am: Add wget_ftpls_fuzzer
* fuzz/wget_ftpls_fuzzer.c: New fuzzer
* fuzz/wget_ftpls_fuzzer.dict: Fuzzer dictionary
* fuzz/wget_ftpls_fuzzer.in/starter: Starting corpus
* src/ftp-ls.c: Parsing function take FILE * as argument,
  new function ftp_parse_ls_fp()
* src/ftp.c: Remove static from freefileinfo()
* src/ftp.h: Add ftp_parse_ls_fp() and freefileinfo()
2018-04-21 19:24:25 +02:00
Tim Rühsen
7ecfe3ef70 * src/main.c (main): Fix memleak for fuzzing/testing 2018-04-21 18:21:52 +02:00
Tim Rühsen
7e635d173e * src/init.c: Fix fuzzing in case ~/.wgetrc doesn't exist 2018-04-21 16:33:45 +02:00
Tim Rühsen
23b0275feb Add new HTML parser fuzzer
* fuzz/Makefile.am: Add wget_html_fuzzer
* fuzz/wget_html_fuzzer.c: New fuzzer
* fuzz/wget_html_fuzzer.dict: HTML dictionary for fuzzing
* fuzz/wget_html_fuzzer.in: Initial corpora
* src/html-url.c: Add new function get_urls_html_fm()
* src/html-url.h: Add ne function get_urls_html_fm()
* src/wget.h: Fix define for fopen_wgetrc()
2018-04-20 22:33:58 +02:00
Tim Rühsen
c9a091ae45 * src/css-url.c (get_uri_string): Fix buffer overflow (read) 2018-04-20 11:37:52 +02:00
Tim Rühsen
7a5db30b01 * src/iri.h: Fix C++ compile error 2018-04-20 10:17:55 +02:00
Tim Rühsen
9d899d7bb7 * src/http.c: Download and scan CSS files in spider mode 2018-04-19 23:05:06 +02:00
Tim Rühsen
d25d036fba * src/css-url.c (get_urls_css): Call yylex_destroy() to reset CSS scanner 2018-04-19 23:05:06 +02:00
Tim Rühsen
ff3c7733b7 * src/html-url.h: Include needed header files 2018-04-18 20:41:08 +02:00
Tim Rühsen
3ae58dae13 Fix oss-fuzz issue with exit()
* src/wget.h: Define exit() as exit_wget()
* fuzz/wget_options_fuzzer.c: Implement exit_wget() and cleanup
2018-04-18 13:26:10 +02:00
Tim Rühsen
66b416b6cd Fix fopen/stdin issues with fuzzing
* fuzz/wget_options_fuzzer.c: Add fopen_wget() and fopen_wgetrc()
* src/utils.c: Use fopen_wgetrc() for config files,
  don't read from stdin when fuzzing
* src/wget.h: Define fopen as fopen_wget when fuzzing,
  define fopen_wgetrc as fopen when not fuzzing
2018-04-17 23:02:04 +02:00
Tim Rühsen
3c4a6506a5 * src/log.c: Don't check_redirect_output() when fuzzing 2018-04-17 12:40:47 +02:00
Tim Rühsen
fbb4cd231e * src/main.c (promt_for_password): Avoid getpass() when fuzzing 2018-04-17 12:15:18 +02:00
Tim Rühsen
3ceb6e5630 Fix double fclose() with -d while fuzzing
* src/ftp.c (ftp_loop_internal): Set warc_tmp to NULL after ffclose()
* src/init.c (cleanup): Set output_stream to NULL after fclose()
* src/log.c (log_close): Set global stream vars to NULL after closing
* src/recur.c (retrieve_tree): Set rejectedlog to NULL after closing
* src/warc.c (warc_close): Set stream vars to NULL after closing
2018-04-17 11:59:54 +02:00
Tim Rühsen
eaf167aaaa * src/main.c (main): Don't background if TESTING 2018-04-17 11:50:36 +02:00
Tim Rühsen
7d5de64fc9 * src/init.c (initialize): Return error, don't exit() 2018-04-17 11:42:43 +02:00
Tim Rühsen
70042265be * src/init.c (cmd_use_askpass): Return false on error 2018-04-16 23:04:53 +02:00
Tim Rühsen
64758655c4 * src/utils.c (compile_posix_regex): Hard-code string to regcomp
regcomp() may be too cpu + memory intensive for fuzzing.
See https://sourceware.org/glibc/wiki/Security%20Exceptions
2018-04-16 22:04:54 +02:00
Tim Rühsen
e737c4b10e Fix 2 more memleaks
* src/init.c (initialize): Use global var for wgetrc filename
* src/iri.c (find_locale): Return strdup'ed locale string
* src/options.h (struct options): Add wgetrcfile
2018-04-16 22:02:11 +02:00
Tim Rühsen
05a8c064e9 * src/init.c (cleanup): Set output_stream to NULL after closing 2018-04-16 13:22:29 +02:00
Tim Rühsen
01002a168a Fix homedir memory leaks
* src/hsts.c: Use opt.homedir
* src/init.c: Likewise
* src/main.c: Likewise
* src/netrc.c: Likewise
* src/options.h (struct options): Add homedir
2018-04-16 13:19:03 +02:00
Tim Rühsen
73fd57585c * src/main.c (main): Free opt.encoding_remote properly 2018-04-16 12:21:52 +02:00
Tim Rühsen
7963260e76 * src/host.c (wait_ares): Free ptimer 2018-04-16 11:58:18 +02:00
Tim Rühsen
99a7039def * src/init.c (cleanup): Free regex objects properly 2018-04-16 11:57:39 +02:00
Tim Rühsen
d7e3acb2cc * src/init.c (cleanup): Never call cleanup() twice 2018-04-16 09:58:51 +02:00
Tim Rühsen
e0860dd1ff * src/init.c (cmd_bytes_sum): Fix integer over- and underflow 2018-04-16 09:58:51 +02:00
Tim Rühsen
15ef79f808 * src/main.c (save_hsts): Free hsts_store after closing 2018-04-16 09:58:51 +02:00
Tim Rühsen
79385a29fd Use strtol() instead of selfmade function
* src/init.c (cmd_number): Use strtol() instead of selfmade function
* bootstrap.conf: Add strtol gnulib module
2018-04-16 09:58:51 +02:00
Tim Rühsen
55da9f71f0 * src/hsts.c (hsts_hash_func): Allow integer overflow 2018-04-16 09:58:51 +02:00
Tim Rühsen
bec9816f40 * init.c (cmd_spec_mirror): Fix uninitialzed stack variable 2018-04-16 09:58:51 +02:00
Tim Rühsen
b86294e1c9 * src/init.c (cleanup): Free more variables 2018-04-16 09:58:51 +02:00
Tim Rühsen
328438e69b * src/utils.c (fopen_stat): Early return to allow fuzzing/fmemopen 2018-04-16 09:58:51 +02:00
Tim Rühsen
36f029d2f0 * src/init.c (initialize): Free mem before exit() 2018-04-16 09:58:51 +02:00
Tim Rühsen
a4402120ad Add OSS-Fuzz infrastruture
* Makefile.am: Add fuzz/ to SUBDIRS
* cfg.mk: Fix 'make syntax-check'
* configure.ac: Add --enable-fuzzing
* fuzz/Makefile.am: New file
* fuzz/README.md: New file
* fuzz/fuzzer.h: New file
* fuzz/get_all_corpora: New file
* fuzz/get_ossfuzz_corpora: New file
* fuzz/glob_crash.c: New file
* fuzz/main.c: New file
* fuzz/run-afl.sh: New file
* fuzz/run-clang.sh: New file
* fuzz/view-coverage.sh: New file
* fuzz/wget_options_fuzzer.c: New file
* fuzz/wget_options_fuzzer.dict: New file
* src/init.c (cleanup): Free more resources
* src/main.c (init_switches): Initialize only once,
  (print_usage): Don't print if TESTING is defined
* src/utils.h: Include wget.h
2018-04-16 09:58:51 +02:00
Tim Rühsen
de54c970b2 Move unit-test code to tests/
* src/Makefile.am: Remove test.c and test.h
* src/test.c: Rename to tests/unit-tests.c
* src/test.h: Rename to tests/unit-tests.h
* tests/Makefile.am: Add unit-tests.c and unit-tests.h
* src/hsts.c: Amend #include
* src/http.c: Likewise
* src/init.c: Likewise
* src/metalink.c: Likewise
* src/res.c: Likewise
* src/url.c: Likewise
* src/utils.c: Likewise
2018-04-05 15:06:47 +02:00
Tim Rühsen
3e84963e84 * src/main.c: Rename main() -> main_wget() for unit tests 2018-04-05 15:06:47 +02:00
Tim Rühsen
f56f970bc2 Fix some issues found by 'infer' 2018-03-14 14:43:35 +01:00
Tim Rühsen
0b54043d17 * src/openssl.c: Fix build for OpenSSL 1.1.0 without TLS1_3_VERSION 2018-03-08 16:17:14 +01:00
Loganaden Velvindron
fde8cefd13 Add TLS1.3 support for OpenSSL build
* src/init.c: Add 'tlsv1_3 for --secure-protocol
* src/openssl.c (ssl_init): Enable TLS1.3 if possible
* src/options.h: Add secure_protocol_tlsv1_3
* doc/wget.texi: Add description of TLSv1_3

Copyright-paperwork-exempt: Yes
2018-03-08 15:30:14 +01:00
Tim Rühsen
ba2b0654b4 * src/main.c: Add help text for --retry-on-http-error
Reported-by: Giovanni Tirloni
2018-03-07 10:32:08 +01:00
Tim Rühsen
375bfa98dc * src/url.c (convert_fname): Fix invalid free on iconv_open() failure
Reported-by: Volkmar Klatt
2018-03-01 16:03:29 +01:00
Tim Rühsen
bea54e0da4 * src/mswindows.c: Fix prototype of fork_to_background()
Reported-by: Gisle Vanem
2018-02-21 19:05:15 +01:00
Tim Rühsen
9887b870d1 Use gnulib's utime()
* bootstrap.conf: Add modules utime and utime-h
* src/utils.c (touch): Remove own code for gnulib's utime()
2018-02-09 10:21:43 +01:00
Tim Rühsen
c722973212 Fix logging in background mode
* ../src/main.c: Re-init logfile if changed for background mode
* ../src/utils.c: fork_to_background() returns whether logfile changed
* ../src/utils.h: Set return type bool for fork_to_background()

Fixes: 
Reported-by: Noël Köthe
2018-02-09 10:21:43 +01:00
Tim Rühsen
bb7fa977a1 * src/http.c: Fix two typos in comments 2018-02-09 10:18:35 +01:00
Tim Rühsen
d27032c446 Mention list and bugtracker for --help and in man page
* doc/wget.texi: Mention list and bugtracker in man page
* src/main.c: Mention list and bugtracker for --help
2018-01-22 10:39:49 +01:00
Darshit Shah
d0a5d9f131 Switch off compression by default
Gzip compression has a number of bugs which need to be ironed out before
we can support it by default. Some of these stem from a misunderstanding
of the HTTP spec, but a lot of them are also due to many web servers not
being compliant with RFC 7231.

With this commit, I am marking GZip compression support as experimental
in GNU Wget pending further investigation and the addition of tests.

* src/init.c (defaults): Switch of compression support by default
* docs/wget.texi: State that compression is experimental
2018-01-21 10:51:11 +01:00
Darshit Shah
0d0a95a01b Revert "* src/init.c (defaults): Set compression_none as the default compression"
This reverts commit 8283ac0846.
2018-01-21 10:50:44 +01:00
Darshit Shah
8283ac0846 * src/init.c (defaults): Set compression_none as the default compression 2018-01-21 10:17:39 +01:00
Reiji
a7cc4e2b37 * src/http.c (gethttp): Fix bug that prevented all files from being decompressed
Signed-off-by: Darshit Shah <darnir@gnu.org>
2018-01-20 14:04:28 +01:00
Tim Rühsen
55d25fc20c * src/host.c (sufmatch): Fix to domain matching 2018-01-19 19:32:01 +01:00
Gisle Vanem
513cc1c0c8 * src/netrc.c: Fix Standalone compilation of netrc file 2018-01-17 14:44:52 +01:00
Darshit Shah
183fccdaad Update Copyright years 2018-01-14 11:24:43 +01:00
Darshit Shah
d26c6c0028 * src/netrc.c: Search for the correct netrc file on Windows 2018-01-14 10:55:03 +01:00
Tim Rühsen
047746eb76 * src/http.c: Exclude *.gz and *.tgz from decompression 2018-01-10 15:46:13 +01:00
Tim Rühsen
d8df356d4b * src/utils.c (wg_pin_peer_pubkey): Fix format warning 2017-12-31 13:03:25 +01:00
Peter Wu
220c24ecb5 Avoid redirecting output to file when tcgetpgrp fails
* src/log.c (check_redirect_output): tcgetpgrp can return -1 (ENOTTY),
be sure to check whether a valid controlling terminal exists before
redirecting.

Fixes: 
2017-12-31 12:59:15 +01:00
Darshit Shah
693cee0109 Don't assume a 416 response has no body
* http.c(gethttp): In case of a 416 response, try to drain the socket of
any bytes before reusing the connection

Reported-By: Iru Cai <mytbk920423@gmail.com>
2017-12-08 18:44:17 +01:00
Tim Rühsen
6aa6b669ef Support building with OpenSSL 1.1 w/o deprecated features
* src/openssl.c (ssl_init): Fix code for the subject's issue

Reported-by: Matthew Thode
2017-11-26 18:59:47 +01:00
Tim Rühsen
8551ceccfe Avoid link conversion after 304 Not Modified
* src/http.c (gethttp): Handle 304 before setting document content type

Fixes: 
Reported-by: Ben Fuchs
2017-11-25 19:33:03 +01:00
YX Hao
19060db44f Fix printing mutibyte chars as unprintable chars on Windows
* src/log.c (get_warc_log_fp): Fix return value to stderr
* src/main.c (main): Init logging as soon as possible,
  fix locale/charset on Windows
2017-11-16 12:23:20 +01:00
YX Hao
a9a953feee Convert remote path to local encoding
* src/url.c (url_file_name): Convert remote path to local encoding
2017-11-15 19:58:53 +01:00
Tim Rühsen
267cd51fff Do not use must-revalidate in Cache-Control header
As the bug report states, 'must-revalidate' is a request directive.

Fixes 
2017-11-10 10:57:46 +01:00
Darshit Shah
973c26ed7d Fix Segfault due to derefencing null ptr
* src/http.c(gethttp): When Encoding is gzip, ensure that the
Content-Type Header was actually seen. Without this, the "type" variable
is null causing a Segfault.

Reported-By: Noël Köthe <noel@debian.org>
2017-11-06 10:09:03 +01:00
Tim Rühsen
16d066f89c * src/http.c: Fix H_REDIRECTED 2017-11-03 22:23:04 +01:00
Tim Rühsen
a2477d487c * src/http.c: Add support for HTTP status code 308 2017-11-03 22:12:11 +01:00
Tim Rühsen
ba6b44f674 Fix heap overflow in HTTP protocol handling (CVE-2017-13090)
* src/retr.c (fd_read_body): Stop processing on negative chunk size

Reported-by: Antti Levomäki, Christian Jalio, Joonas Pihlaja from Forcepoint
Reported-by: Juhani Eronen from Finnish National Cyber Security Centre
2017-10-26 17:29:38 +02:00
Tim Rühsen
d892291fb8 Fix stack overflow in HTTP protocol handling (CVE-2017-13089)
* src/http.c (skip_short_body): Return error on negative chunk size

Reported-by: Antti Levomäki, Christian Jalio, Joonas Pihlaja from Forcepoint
Reported-by: Juhani Eronen from Finnish National Cyber Security Centre
2017-10-26 17:29:38 +02:00
YX Hao
27d78d944f Avoid unnecessary UTF-8 encoded fallback (trivial change)
* src/retr.c (retrieve_url): Check for changed URL on redirect
2017-10-25 14:26:36 +02:00
Tim Rühsen
60f033426f Add GNU extensions to .netrc parsing
src/netrc.c (parse_netrc): Add 'port' and 'force' extensions

Reported-by: September 20Tim Landscheidt
2017-09-27 12:42:06 +02:00
Josef Moellers
6f3b995993 Bail out on unexpected 416 server errors
* src/http.c (gethttp): Stop on 416 if file is incomplete
2017-09-18 16:45:49 +02:00
Tim Schlueter
c451eec155 Add gzip Content-Encoding decompression
* src/http.c (struct http_stat): Add remote_encoding field.
(read_response_body): Enable gzip decompression.
(initialize_request): Send gzip Accept-Encoding header.
(gethttp): Decompress files with gzip Content-Encoding.
* src/retr.c: include zlib.h.
(zalloc): New function.
(zfree): New function.
(fd_read_body): Decompress gzip data.
* src/retr.h (fd_read_body enum): Add rb_compressed_gzip flag.
2017-08-04 14:34:53 +02:00
Tim Schlueter
b543dfe783 Add --compression option
* doc/wget.texi: Add --compression documentation.
* src/init.c (cmd_spec_compression): New function.
(commands[]): Add opt.compression.
(defaults): Set default opt.compression value.
* src/main.c (option_data[]): Add struct for --compression.
(print_help, help[]): Add description for --compression.
(main): Add incompatibility checks for --compression.
* src/options.h (struct options): Add compression enum and field.
2017-08-04 14:34:53 +02:00
Tim Schlueter
08ed2a5530 Adjust Extension based on Content-Encoding
* doc/wget.texi (--adjust-extension, adjust_extension): Updated documentation.
* src/http.c (encoding_t): New enum.
(struct http_stat): Add local_encoding field.
(gethttp): --adjust-extension based on Content-Encoding.
2017-08-04 14:34:53 +02:00
Tim Rühsen
3ad3b3e36c * src/url.c (url_scheme): Use ASCII version of strncasecmp 2017-07-28 17:11:26 +02:00
Tim Rühsen
5fb6b6bd68 Fix misuse of strncasecmp
* src/http.c (set_content_type): Use c_strcasecmp instead of strncasecmp

See issue bug 
2017-07-28 16:56:27 +02:00
Tim Rühsen
21154bdc36 Check for 304 response before applying --adjust-extension
* src/http.c (gethttp): Move 304 code before --adjust-extension code

This fixes applying --adjust-extension in combination with 304
HTTP responses. It could lead to .html extensions to arbitrary
files.

Reported-by: anfractuosity
2017-06-13 11:25:20 +02:00
Tim Rühsen
ae293c945a Fix buffer overflow in Public Key Pinning
* src/utils.c (wget_base64_decode): Add param for destination size,
  (wg_pubkey_pem_to_der): Amend call to wget_base64_decode(),
  (wg_pin_peer_pubkey): Likewise and fix code style.
* src/utils.h: Add param to wget_base64_decode()
* src/http-ntlm.c (ntlm_input): Amend call to wget_base64_decode()
* src/http.c (skip_content_type): Likewise

Fixes 
2017-06-13 10:23:04 +02:00
Tomas Hozza
876def8ebe Add command line option to disable use of .netrc
Although internally code uses option for (not) reading .netrc for
credentials, it was not possible to turn this behavior off on command
line. Note that it was possible to turn it off using wgetrc.

Idea for this change came from Bruce Jerrick (bmj001@gmail.com).
Reference: https://bugzilla.redhat.com/show_bug.cgi?id=1425097

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2017-05-15 16:06:50 +02:00
Tomas Hozza
f8c3df1f40 Fixed getting of credentials from .netrc
There seemed to be a copy&paste error in http.c code, which decides
whether to get credentials from .netrc. In ftp.c "user" and "pass"
variables are char*, while in http.c, these are char**. For this reason
they should be dereferenced when determining if password and user login
is set to some value.

Also since both variables are dereferenced on lines above the changed
code, it does not really make sense to check if they are NULL.

This patch is based on fix from Bruce Jerrick <bmj001@gmail.com>.
Fedora bug: https://bugzilla.redhat.com/show_bug.cgi?id=1425097

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2017-05-15 16:06:50 +02:00
Tim Rühsen
936efc3564 * src/iri.c (idn_encode): Better IDNA 2003 compatibility 2017-05-11 11:56:26 +02:00
Tomas Hozza
0b41c7543a Mention TLSv1_1 and TLSv1_2 as secure-protocol values in help
* src/main.c: The --secure-protocol option accepts also values TLSv1_1
and TLSv1_2, as mentioned in the man page. However the help message
doesn't mention these two values. This patch adds TLSv1_1 and TLSv1_2 as
possible values to the help message.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2017-05-04 14:51:54 +02:00
Tim Rühsen
c4a2b2e77e * src/http.c (gethttp): Support Wayback Machine's X-Archive-Orig-last-modified 2017-05-03 16:37:11 +02:00
Tim Rühsen
56c78c4b09 * src/utils.c: Remove non-portable __builtin_unreachable() 2017-04-18 13:22:25 +02:00
Tim Rühsen
0ec46cb109 Skip iconv() usage if HAVE_ICONV is not defined
This helps on broken iconv implementations, e.g. Solaris.

Reported-by: Mojca Miklavec
2017-04-18 13:17:19 +02:00
Tim Rühsen
92bfe2a2e4 Fix charset transcoding issue for non-reversible codepoints
* src/url.c: Check iconv() against 0, not -1

On some libiconv implementations, unknown codepoints become
encoded as ?, e.g. when converting a non-ascii codepoint to ASCII.
This results in ambigious file names which also fails our tests.
2017-04-16 19:55:14 +02:00
Tim Rühsen
fc2f4233ed * src/iri.c: Fix WIN32 idn2_free, forgotten code 2017-04-16 19:50:10 +02:00
Darshit Shah
b2c38d33e1 * src/init.c: Set flstats correctly when using WGETRC env var 2017-04-14 01:16:49 +02:00
Tim Rühsen
6ef493b19e Fix use of idn2_free()
* src/connect.c (connect_to_ip): Use xfree() instead of idn2_free()
* src/host.c (lookup_host): Use xfree() instead of idn2_free()
* src/iri.h: Do not include idn2.h
* src/url.c (url_free): Use xfree() instead of idn2_free()
* src/url.h (struct url): Remove 'idn_allocated' from struct

Reported-by: Gisle Vanem
2017-04-08 11:05:55 +02:00
klemens
f381831d88 Fix typos in comments 2017-04-01 19:38:09 +02:00
Tim Rühsen
02d40a4676 * src/metalink.c (retrieve_from_metalink): Fix len in memset() 2017-03-31 13:15:27 +02:00
Vijo Cherian
400b8eba6c Safeguards against TOCTTOU
* src/utils.h: Add struct file_stat_s declaration,
  change prototypes of file_exists_p(),
  add prototypes for fopen_stat() and open_stat().
* src/utils.c: Extend file_exists_p(),
  new function fopen_stat() and open_stat(),
  add new param for file_exists_p().
* src/init.h: Add param file_stats_t to run_wgetrc().
* src/ftp.c: Amend calls to extended functions.
* src/hsts.c: Likewise.
* src/http.c: Likewise.
* src/init.c: Likewise.
* src/main.c: Likewise.
* src/metalink.c: Likewise.
* src/retr.c: Likewise.
* src/url.c: Likewise.

Added fopen_stat() and open_stat() that checks to makes sure the file didn't
change underneath us.
Return error from file_exists_p().
Added a way to return error from this file without major surgery to the
callers.

Fixes: 
2017-03-24 09:39:09 +01:00
Christof Horschitz
1d71645c06 * src/warc.c (warc_write_cdx_record): Escape URLs 2017-03-22 15:01:04 +01:00
Mike Frysinger
e249844143 Include libunistring headers only when used
* src/iri.c: Check for libidn2 < 0.14 to include libunistring headers

The unistring functions are used only when an older version of libidn2
is used, so don't include its headers either w/newer libdin2 versions.
2017-03-20 09:39:20 +01:00
Tim Rühsen
84a93f4127 Fix links to www.robotstxt.org
* NEWS: Fix links
* doc/wget.texi: Likewise
* src/res.c: Likewise

Reported-by: Noël Köthe
2017-03-18 19:05:38 +01:00
Tim Rühsen
90b487369a Include <arpa/inet.h> for Windows
Reported-by: Gisle Vanem
2017-03-08 13:00:54 +01:00
Tim Rühsen
57d748117f Fix updating HSTS entries
* src/hsts.c (hsts_store_entry): Always update 'created' field

Fixes: 
Reported-by: Deian Stefan, Atyansh Jaiswal, Jonathan Luck
2017-03-08 10:56:12 +01:00
Tim Rühsen
4d729e322f Fix CRLF injection in Wget host part
* src/url.c (url_parse): Reject control characters in host part of URL

Reported-by: Orange Tsai
2017-03-06 10:04:22 +01:00
Benjamin Esham
63c2aea255 * src/warc.c: Use warc_write_header_uri for all WARC-Target-URI fields
The WARC spec requires that all URIs be enclosed in angle brackets. This
was being done in most cases, but not for "WARC-Target-URI" fields in
WARC blocks of type "response", "resource", "revisit", and "metadata".
2017-03-04 12:51:39 +01:00
Tim Rühsen
ac4fed3220 Fix 504 status handling
* src/http.c (gethttp): Move 504 handling to correct place.
  (http_loop): Fix memeory leak.
* testenv/server/http/http_server.py: Add Content-Length header on non-2xx
  status codes with a body

Reported-by: Adam Sampson
2017-02-16 15:53:56 +01:00
YX Hao
cf5df5593d * src/url.c (url_file_name): Do not charset convert local directory
In a non-ASCII environment, the local path may contain non-ASCII
characters. The server responded file name must be converted before
it is concatenated to the local path. Conversion after concatenation
may result in 'iconv' errors.
2017-02-16 12:52:16 +01:00
Tim Rühsen
ac9be9b756 * src/main.c: Remove double 'verbose' option
Fixes: 
2017-02-12 21:18:23 +01:00
Tim Rühsen
aebd49d9d4 isrc/http.c (check_retry_on_http_error): Fix gcc warning 2017-02-11 11:54:21 +01:00
Tom Szilagyi
d6d00006a0 Add support for --retry-on-http-error
* doc/wget.text: Add documentation
* src/http.c: Add function check_retry_on_http_error ()
* src/init.c: Add opt.retry_on_http_error
* src/main.c: Add struct for retry-on-http-error to option_data[]
* src/options.h: Add retry_on_http_error to struct options
2017-02-11 11:50:24 +01:00
Tim Rühsen
05acf5d3f6 Revert "Add support for --retry-on-http-error"
This reverts commit 977276374d.
2017-02-11 11:45:11 +01:00
Tim Rühsen
80c62c238e Change libtool library deps to non-libtool deps
Reported-by: Yousong Zhou
Fixes: 
2017-02-10 17:20:42 +01:00
Tom Szilagyi
977276374d Add support for --retry-on-http-error
* doc/wget.texi: Add description for --retry-on-http-error
* src/http.c (gethttp):
Consider given HTTP response codes as non-fatal, transient errors.
Supply a comma-separated list of 3-digit HTTP response codes as
argument. Useful to work around special circumstances where retries
are required, but the server responds with an error code normally not
retried by Wget. Such errors might be 503 (Service Unavailable) and
429 (Too Many Requests). Retries enabled by this option are performed
subject to the normal retry timing and retry count limitations of
Wget.

Using this option is intended to support special use cases only and is
generally not recommended, as it can force retries even in cases where
the server is actually trying to decrease its load. Please use it
wisely and only if you know what you are doing.

Example use and a starting point for manual testing:
  wget --retry-on-http-error=429,503 http://httpbin.org/status/503
2017-02-09 21:17:20 +01:00
Tim Rühsen
d061e553a1 * src/http.c (initialize_request): Fix regression in .netrc auth
Reported-by: Axel Reinhold
2017-02-06 21:44:18 +01:00
Tim Rühsen
2ddd2b69e4 * src/iri.c (idn_encode): Fix memory leak 2017-02-06 21:39:44 +01:00
Tim Rühsen
31ac36e170 Fix include/define clash with gnulib's unlink module
* src/options.h: Rename options.unlink to options.unlink_requested
* src/init.c: Replace options unlink member by unlink_requested
* src/http.c: Likewise
* src/ftp.c: Likewise
2017-02-04 18:02:54 +01:00
Tim Rühsen
f2c4289557 * src/xattr.h: Fix #define fsetxattr for MacOS and FreeBSD
Reported-by: Zhiming Wang
2017-02-04 15:29:44 +01:00
Tim Rühsen
366d82f349 * src/utils.c: Move macro FMT_MAX_LENGTH into scope 2017-02-03 12:35:49 +01:00
Tim Rühsen
f2574e90b7 * src/utils.c: Fix -Wformat= warnings 2017-02-03 12:33:38 +01:00
Tim Rühsen
81b3aaf75c * src/gnutls.c: Fix -Wformat= warnings 2017-02-03 12:31:51 +01:00
Tim Rühsen
17d2f42a3d * src/iri.c: Remove unused macro IDNA_FLAGS 2017-02-03 12:28:37 +01:00
Tim Rühsen
638df40476 * src/iri.c: Remove use of __func__ macros 2017-02-03 12:28:05 +01:00
Tim Rühsen
00bafe72f1 * src/http.c: Fix -Wformat= warnings 2017-02-03 12:24:41 +01:00
Tim Rühsen
e777c01f43 * src/progress.c: Remove unused macro move_to_end 2017-02-03 12:18:14 +01:00
Tim Rühsen
3ba112ea57 * src/html-parse.c: Remove unused macro SKIP_NON_WS 2017-02-03 12:15:24 +01:00
Tim Rühsen
485fcfcc20 * src/hsts.c: Remove unused macro CHECK_EXPLICIT_PORT 2017-02-03 12:09:18 +01:00
Tim Rühsen
a5094731cd * src/hsts.c: Fix -Wformat= warnings 2017-02-03 12:08:08 +01:00
Tim Rühsen
3186eb2976 * src/hash.c: Explicitly convert float to int 2017-02-03 12:03:50 +01:00
Tim Rühsen
9947663af8 * src/ftp-ls.c: Fix -Wformat= warnings 2017-02-03 11:59:33 +01:00
Tim Rühsen
cfae085665 * src/ftp.c (ftp_retrieve_list): Add default to switch 2017-02-03 11:57:02 +01:00
Tim Rühsen
e69808256b * src/css-url.h: Remove redundant declaration 2017-02-03 11:53:28 +01:00
Tim Rühsen
11989ef669 * src/ftp.c: Fix -Wformat= warning 2017-02-03 11:52:08 +01:00
Tim Rühsen
5fceab6cb9 * src/http.c (test_parse_range_header): Fix constants 2017-02-03 10:32:42 +01:00
Tim Rühsen
d0e02a54ae * src/url.c (mkalldirs): Add newline to log message 2017-02-02 11:11:50 +01:00
Tim Rühsen
2e70409844 * src/cookies.c (check_domain_match): Add newline to DEBUG lines 2017-02-02 11:11:07 +01:00
Tim Rühsen
6de24fe3c0 * src/iri.c: Use TR46 non-transitional for toASCII conversion 2017-01-13 15:53:03 +01:00
Tim Rühsen
0ab3d92c85 * src/main.c: Fix _Noreturn compiler warnings 2017-01-13 15:50:19 +01:00
Tim Rühsen
4cf8af84e0 * src/utils.c: Fix _Noreturn compiler warning 2017-01-13 15:49:05 +01:00
Tim Rühsen
42b8761cbc * src/init.c (setval_internal): Fix sign compare warning 2017-01-13 15:47:02 +01:00
Tim Rühsen
fd0f759597 Replace home-grown portability code by gnulib modules
* bootstrap.conf: Add intprops, inttypes, limits-h, signal-h,
  stat, sys_types
* src/ftp.c: Replace 'struct_stat' by 'struct stat'
* src/hsts.c: Likewise
* src/http.c: Likewise
* src/main.c: Likewise
* src/netrc.c: Likewise
* src/retr.c: Likewise
* src/url.c: Likewise
* src/utils.c: Likewise
* src/sysdep.h: Remove old portability code

Further portability issues should be addressed by gnulib.
2017-01-13 15:38:15 +01:00
Tim Rühsen
a384f5e2e9 Replace WGET_* m4 macros by gnulib modules
* bootstrap.conf: Add hostent, inet_ntop, nanosleep, utimens
* configure.ac: Remove WGET_STRUCT_UTIMBUF, WGET_FNMATCH,
  WGET_NANOSLEEP, WGET_POSIX_CLOCK, WGET_NSL_SOCKET
* m4/wget.m4: Likewise
* src/Makefile.am: Add $(LIB_NANOSLEEP) $(LIB_POSIX_SPAWN) to LDADD
* tests/Makefile.am: Likewise
* src/host.c (print_address): Use inet_ntop also for IPV4
2017-01-13 12:54:35 +01:00
Tim Rühsen
5ae1f37902 Remove libidn vulnerability work-around
* src/iri.c (_utf8_is_valid): Removed

Since we are using libidn2 for IDNs, we no longer need
this work-around.
2017-01-13 12:03:47 +01:00
Tim Rühsen
0eb4a21b6c * src/iri.c (idn_encode): Use TR46 transitional if available 2017-01-13 11:45:18 +01:00
Tim Rühsen
1a01a6b2d0 Fix previous commit 2427ca4ac0 2017-01-07 15:59:11 +01:00
vijeth-aradhya
2427ca4ac0 Fix http.c and ftp.c passwd logic error
* src/ftp.c (getftp): Fix password/user selection
* src/http.c (initialize_request): Likewise

Before, netrc password won over interactive
--ask-password but now --ask-password wins
after change of program logic

Fixes Issue 
2017-01-06 16:18:40 +01:00
Giuseppe Scrivano
42c2ce71bc * src/main.c (main): Add missing \n in error message 2016-12-31 17:11:01 +01:00
Giuseppe Scrivano
def133f26f Check that fd_set has not fds bigger than FD_SETSIZE
* src/connect.c: check that the fd is not bigger than FD_SETSIZE
  before using FD_SET.  An fd_set cannot hold fds bigger than
  FD_SETSIZE, causing out-of-bounds write to a buffer on the stack.

Reported by: Jann Horn <jannh@google.com>
2016-12-28 12:24:19 +01:00
Nikos Mavrogiannopoulos
b9ed06afd8 Avoid calling the gnutls priority functions multiple times
* src/gnutls.c (ssl_connect_wget): Call gnutls_set_default_priority()
  for --secure-protocol=auto (default).

The patch fixes a behavior that may have unintended side-effects in
certain gnutls versions. Instead use the default priorities when no
options are given.

Signed-off-by: Nikos Mavrogiannopoulos <nmav@gnutls.org>
2016-12-20 14:48:31 +01:00
Tim Rühsen
1bdc20d774 Print debug message when skipping certain recursive downloads
* src/recur.c (retrieve_tree): Print debug message instead silently
  skipping recursive downloads.
2016-12-19 12:19:52 +01:00