* src/http.c (skip_short_body): Return error on negative chunk size
Reported-by: Antti Levomäki, Christian Jalio, Joonas Pihlaja from Forcepoint
Reported-by: Juhani Eronen from Finnish National Cyber Security Centre
* src/http.c (gethttp): Move 304 code before --adjust-extension code
This fixes applying --adjust-extension in combination with 304
HTTP responses. It could lead to .html extensions to arbitrary
files.
Reported-by: anfractuosity
Although internally code uses option for (not) reading .netrc for
credentials, it was not possible to turn this behavior off on command
line. Note that it was possible to turn it off using wgetrc.
Idea for this change came from Bruce Jerrick (bmj001@gmail.com).
Reference: https://bugzilla.redhat.com/show_bug.cgi?id=1425097
Signed-off-by: Tomas Hozza <thozza@redhat.com>
There seemed to be a copy&paste error in http.c code, which decides
whether to get credentials from .netrc. In ftp.c "user" and "pass"
variables are char*, while in http.c, these are char**. For this reason
they should be dereferenced when determining if password and user login
is set to some value.
Also since both variables are dereferenced on lines above the changed
code, it does not really make sense to check if they are NULL.
This patch is based on fix from Bruce Jerrick <bmj001@gmail.com>.
Fedora bug: https://bugzilla.redhat.com/show_bug.cgi?id=1425097
Signed-off-by: Tomas Hozza <thozza@redhat.com>
* src/main.c: The --secure-protocol option accepts also values TLSv1_1
and TLSv1_2, as mentioned in the man page. However the help message
doesn't mention these two values. This patch adds TLSv1_1 and TLSv1_2 as
possible values to the help message.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
* src/url.c: Check iconv() against 0, not -1
On some libiconv implementations, unknown codepoints become
encoded as ?, e.g. when converting a non-ascii codepoint to ASCII.
This results in ambigious file names which also fails our tests.
* src/connect.c (connect_to_ip): Use xfree() instead of idn2_free()
* src/host.c (lookup_host): Use xfree() instead of idn2_free()
* src/iri.h: Do not include idn2.h
* src/url.c (url_free): Use xfree() instead of idn2_free()
* src/url.h (struct url): Remove 'idn_allocated' from struct
Reported-by: Gisle Vanem
* src/utils.h: Add struct file_stat_s declaration,
change prototypes of file_exists_p(),
add prototypes for fopen_stat() and open_stat().
* src/utils.c: Extend file_exists_p(),
new function fopen_stat() and open_stat(),
add new param for file_exists_p().
* src/init.h: Add param file_stats_t to run_wgetrc().
* src/ftp.c: Amend calls to extended functions.
* src/hsts.c: Likewise.
* src/http.c: Likewise.
* src/init.c: Likewise.
* src/main.c: Likewise.
* src/metalink.c: Likewise.
* src/retr.c: Likewise.
* src/url.c: Likewise.
Added fopen_stat() and open_stat() that checks to makes sure the file didn't
change underneath us.
Return error from file_exists_p().
Added a way to return error from this file without major surgery to the
callers.
Fixes: #20369
* src/iri.c: Check for libidn2 < 0.14 to include libunistring headers
The unistring functions are used only when an older version of libidn2
is used, so don't include its headers either w/newer libdin2 versions.
The WARC spec requires that all URIs be enclosed in angle brackets. This
was being done in most cases, but not for "WARC-Target-URI" fields in
WARC blocks of type "response", "resource", "revisit", and "metadata".
* src/http.c (gethttp): Move 504 handling to correct place.
(http_loop): Fix memeory leak.
* testenv/server/http/http_server.py: Add Content-Length header on non-2xx
status codes with a body
Reported-by: Adam Sampson
In a non-ASCII environment, the local path may contain non-ASCII
characters. The server responded file name must be converted before
it is concatenated to the local path. Conversion after concatenation
may result in 'iconv' errors.
* doc/wget.texi: Add description for --retry-on-http-error
* src/http.c (gethttp):
Consider given HTTP response codes as non-fatal, transient errors.
Supply a comma-separated list of 3-digit HTTP response codes as
argument. Useful to work around special circumstances where retries
are required, but the server responds with an error code normally not
retried by Wget. Such errors might be 503 (Service Unavailable) and
429 (Too Many Requests). Retries enabled by this option are performed
subject to the normal retry timing and retry count limitations of
Wget.
Using this option is intended to support special use cases only and is
generally not recommended, as it can force retries even in cases where
the server is actually trying to decrease its load. Please use it
wisely and only if you know what you are doing.
Example use and a starting point for manual testing:
wget --retry-on-http-error=429,503 http://httpbin.org/status/503
