Commit Graph

3811 Commits

Author SHA1 Message Date
Darshit Shah
953bcdaba8 Sync the bootstrap script from gnulib 2018-01-14 11:28:51 +01:00
Darshit Shah
183fccdaad Update Copyright years 2018-01-14 11:24:43 +01:00
Darshit Shah
8240439e21 Replace HTTP urls with HTTPS where valid 2018-01-14 11:15:32 +01:00
Darshit Shah
a03716a0fe Update gnulib 2018-01-14 11:15:16 +01:00
Darshit Shah
d26c6c0028 * src/netrc.c: Search for the correct netrc file on Windows 2018-01-14 10:55:03 +01:00
Tim Rühsen
047746eb76 * src/http.c: Exclude *.gz and *.tgz from decompression 2018-01-10 15:46:13 +01:00
Tim Rühsen
d8df356d4b * src/utils.c (wg_pin_peer_pubkey): Fix format warning 2017-12-31 13:03:25 +01:00
Peter Wu
220c24ecb5 Avoid redirecting output to file when tcgetpgrp fails
* src/log.c (check_redirect_output): tcgetpgrp can return -1 (ENOTTY),
be sure to check whether a valid controlling terminal exists before
redirecting.

Fixes: #51181
2017-12-31 12:59:15 +01:00
Darshit Shah
3d2b2231cd Add new test for 416 responses
* testenv/server/http/http_server.py: If there are multiple requests in
which the requested range is unsatisfiable, then send a body in the in
the 2nd response onwards
* testenv/Test-416.py: New test to check how Wget handles 416 responses
2017-12-11 14:48:01 +01:00
Darshit Shah
693cee0109 Don't assume a 416 response has no body
* http.c(gethttp): In case of a 416 response, try to drain the socket of
any bytes before reusing the connection

Reported-By: Iru Cai <mytbk920423@gmail.com>
2017-12-08 18:44:17 +01:00
Tim Rühsen
6aa6b669ef Support building with OpenSSL 1.1 w/o deprecated features
* src/openssl.c (ssl_init): Fix code for the subject's issue

Reported-by: Matthew Thode
2017-11-26 18:59:47 +01:00
Tim Rühsen
8551ceccfe Avoid link conversion after 304 Not Modified
* src/http.c (gethttp): Handle 304 before setting document content type

Fixes: #52404
Reported-by: Ben Fuchs
2017-11-25 19:33:03 +01:00
YX Hao
19060db44f Fix printing mutibyte chars as unprintable chars on Windows
* src/log.c (get_warc_log_fp): Fix return value to stderr
* src/main.c (main): Init logging as soon as possible,
  fix locale/charset on Windows
2017-11-16 12:23:20 +01:00
Tim Rühsen
a30a5a9059 * tests/Test-https-weboftrust.px: Fix/add sslport 2017-11-16 12:12:16 +01:00
YX Hao
a9a953feee Convert remote path to local encoding
* src/url.c (url_file_name): Convert remote path to local encoding
2017-11-15 19:58:53 +01:00
Tim Rühsen
267cd51fff Do not use must-revalidate in Cache-Control header
As the bug report states, 'must-revalidate' is a request directive.

Fixes #52379
2017-11-10 10:57:46 +01:00
Darshit Shah
973c26ed7d Fix Segfault due to derefencing null ptr
* src/http.c(gethttp): When Encoding is gzip, ensure that the
Content-Type Header was actually seen. Without this, the "type" variable
is null causing a Segfault.

Reported-By: Noël Köthe <noel@debian.org>
2017-11-06 10:09:03 +01:00
Tim Rühsen
16d066f89c * src/http.c: Fix H_REDIRECTED 2017-11-03 22:23:04 +01:00
Tim Rühsen
a2477d487c * src/http.c: Add support for HTTP status code 308 2017-11-03 22:12:11 +01:00
Tim Rühsen
7ac72e063c Fix uninitialized value messages in tests
* tests/Test-https-*.px: Use correct variable in error message

Reported-by: Jeffrey Walton
2017-10-30 21:29:14 +01:00
Tim Rühsen
95891cbba3 Ignore dirmngr.conf, gpg.conf in test directories
* testenv/conf/expected_files.py: Ignore dirmngr.conf, gpg.conf

Libgpgme creates certain files if $HOME doesn't contain the .gnupg directory.
These files disturb some metalink tests if we don't ignore them.

Reported-by: Arkadiusz Miśkiewicz <arekm@maven.pl>
2017-10-30 13:04:09 +01:00
Tim Rühsen
2486320a2a * contrib/release: Cleanup, use lzip instead of xz 2017-10-27 11:26:37 +02:00
Tim Rühsen
ba6b44f674 Fix heap overflow in HTTP protocol handling (CVE-2017-13090)
* src/retr.c (fd_read_body): Stop processing on negative chunk size

Reported-by: Antti Levomäki, Christian Jalio, Joonas Pihlaja from Forcepoint
Reported-by: Juhani Eronen from Finnish National Cyber Security Centre
2017-10-26 17:29:38 +02:00
Tim Rühsen
d892291fb8 Fix stack overflow in HTTP protocol handling (CVE-2017-13089)
* src/http.c (skip_short_body): Return error on negative chunk size

Reported-by: Antti Levomäki, Christian Jalio, Joonas Pihlaja from Forcepoint
Reported-by: Juhani Eronen from Finnish National Cyber Security Centre
2017-10-26 17:29:38 +02:00
Tim Rühsen
bec4c215a8 Update for release 1.19.2
* NEWS: Add news for 1.19.2
* doc/wget.texi: Fix 1.20 to 1.19.2
2017-10-26 16:31:57 +02:00
YX Hao
27d78d944f Avoid unnecessary UTF-8 encoded fallback (trivial change)
* src/retr.c (retrieve_url): Check for changed URL on redirect
2017-10-25 14:26:36 +02:00
Tim Rühsen
60f033426f Add GNU extensions to .netrc parsing
src/netrc.c (parse_netrc): Add 'port' and 'force' extensions

Reported-by: September 20Tim Landscheidt
2017-09-27 12:42:06 +02:00
Josef Moellers
6f3b995993 Bail out on unexpected 416 server errors
* src/http.c (gethttp): Stop on 416 if file is incomplete
2017-09-18 16:45:49 +02:00
Tim Schlueter
c451eec155 Add gzip Content-Encoding decompression
* src/http.c (struct http_stat): Add remote_encoding field.
(read_response_body): Enable gzip decompression.
(initialize_request): Send gzip Accept-Encoding header.
(gethttp): Decompress files with gzip Content-Encoding.
* src/retr.c: include zlib.h.
(zalloc): New function.
(zfree): New function.
(fd_read_body): Decompress gzip data.
* src/retr.h (fd_read_body enum): Add rb_compressed_gzip flag.
2017-08-04 14:34:53 +02:00
Tim Schlueter
b543dfe783 Add --compression option
* doc/wget.texi: Add --compression documentation.
* src/init.c (cmd_spec_compression): New function.
(commands[]): Add opt.compression.
(defaults): Set default opt.compression value.
* src/main.c (option_data[]): Add struct for --compression.
(print_help, help[]): Add description for --compression.
(main): Add incompatibility checks for --compression.
* src/options.h (struct options): Add compression enum and field.
2017-08-04 14:34:53 +02:00
Tim Schlueter
08ed2a5530 Adjust Extension based on Content-Encoding
* doc/wget.texi (--adjust-extension, adjust_extension): Updated documentation.
* src/http.c (encoding_t): New enum.
(struct http_stat): Add local_encoding field.
(gethttp): --adjust-extension based on Content-Encoding.
2017-08-04 14:34:53 +02:00
Darshit Shah
951d3e4cdd Document gperf as a requirement 2017-07-31 14:56:32 +02:00
Tim Rühsen
3ad3b3e36c * src/url.c (url_scheme): Use ASCII version of strncasecmp 2017-07-28 17:11:26 +02:00
Tim Rühsen
5fb6b6bd68 Fix misuse of strncasecmp
* src/http.c (set_content_type): Use c_strcasecmp instead of strncasecmp

See issue bug #51576
2017-07-28 16:56:27 +02:00
Tim Rühsen
f42229b1fd Fix python test suite for GnuTLS 3.5.12+
* testenv/Test-*.py: Replace 127.0.0.1 by localhost
* testenv/certs/server-template.cfg: Likewise
* testenv/certs/server-cert.pem: Regenerate
* testenv/certs/server-crl.pem: Likewise
* testenv/test/base_test.py: Hardcode 'localhost' as server domain

Reported-by: Ludovic Courtès <ludo@gnu.org>
2017-07-09 11:39:05 +02:00
Tim Rühsen
21154bdc36 Check for 304 response before applying --adjust-extension
* src/http.c (gethttp): Move 304 code before --adjust-extension code

This fixes applying --adjust-extension in combination with 304
HTTP responses. It could lead to .html extensions to arbitrary
files.

Reported-by: anfractuosity
2017-06-13 11:25:20 +02:00
Tim Rühsen
ae293c945a Fix buffer overflow in Public Key Pinning
* src/utils.c (wget_base64_decode): Add param for destination size,
  (wg_pubkey_pem_to_der): Amend call to wget_base64_decode(),
  (wg_pin_peer_pubkey): Likewise and fix code style.
* src/utils.h: Add param to wget_base64_decode()
* src/http-ntlm.c (ntlm_input): Amend call to wget_base64_decode()
* src/http.c (skip_content_type): Likewise

Fixes #51227
2017-06-13 10:23:04 +02:00
Tim Rühsen
407c1f990a * doc/wget.texi: Mention --no-config 2017-06-02 10:13:22 +02:00
Tim Rühsen
86b46a34a5 * testenv/Test-recursive-basic.py: Check crawled files 2017-05-16 11:20:57 +02:00
Tim Rühsen
5d4ada1b7b Fix two Metalink tests if $HOME is changed
* conf/expected_files.py (gen_local_fs_snapshot): Skip processing
  of 'pubring.kbx'
2017-05-16 10:24:52 +02:00
Tomas Hozza
876def8ebe Add command line option to disable use of .netrc
Although internally code uses option for (not) reading .netrc for
credentials, it was not possible to turn this behavior off on command
line. Note that it was possible to turn it off using wgetrc.

Idea for this change came from Bruce Jerrick (bmj001@gmail.com).
Reference: https://bugzilla.redhat.com/show_bug.cgi?id=1425097

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2017-05-15 16:06:50 +02:00
Tomas Hozza
f8c3df1f40 Fixed getting of credentials from .netrc
There seemed to be a copy&paste error in http.c code, which decides
whether to get credentials from .netrc. In ftp.c "user" and "pass"
variables are char*, while in http.c, these are char**. For this reason
they should be dereferenced when determining if password and user login
is set to some value.

Also since both variables are dereferenced on lines above the changed
code, it does not really make sense to check if they are NULL.

This patch is based on fix from Bruce Jerrick <bmj001@gmail.com>.
Fedora bug: https://bugzilla.redhat.com/show_bug.cgi?id=1425097

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2017-05-15 16:06:50 +02:00
Tomas Hozza
17960b57d5 Added tests for HTTP authentication using credentials from .netrc
Getting credentials from .netrc has been broken from time to time, thus
adding a test coverage to prevent regressions.

Also added setting of "HOME" environment variable when executing wget,
to make sure LocalFiles like .netrc, which are created just for the
test, are actually used.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2017-05-15 16:06:50 +02:00
Tim Rühsen
40c0d30f5c Fix Test-https-badcerts.px
* util/createcerts.sh: Set date of invalid cert to +20 years
* tests/certs/invalid.crt: Re-generated
* tests/certs/invalid.key: Re-generated
2017-05-15 15:46:11 +02:00
Tim Rühsen
e2c702571f * util/createcerts.sh: Fix double equal (syntax-check) 2017-05-14 21:35:34 +02:00
Tim Rühsen
269fb6201d Fix Test-https-badcerts.px to work with GnuTLS
* tests/Test-https-badcerts.px: Remove OpenSSL requirement
* tests/certs/expired.crt: Re-created with valid dates
* tests/certs/expired.key: Likewise
* tests/certs/invalid.crt: Likewise
* tests/certs/invalid.key: Likewise
2017-05-14 11:10:15 +02:00
Vijo Cherian
c08778aeb3 New shell script to create the certs and keys required for TLS tests
* util/createcerts.sh: New file
2017-05-14 11:07:43 +02:00
Tim Rühsen
297c1e2ac3 Fix HTTPS testing for stricter OpenSSL
* testenv/certs/README: Amend cert creation extensions
* testenv/certs/ca-cert.pem: Created without OCSP signing purpose

Having the OCSP signing purpose set made newer versions of OpenSSL
fail due to stricter checking. Test version of OpenSSL was 1.1.0e.
2017-05-11 12:38:19 +02:00
Tim Rühsen
936efc3564 * src/iri.c (idn_encode): Better IDNA 2003 compatibility 2017-05-11 11:56:26 +02:00
Tim Rühsen
11d3de74ca * .gitlab-ci.yml: Also test OpenSSL build 2017-05-10 12:12:42 +02:00