Commit Graph

3835 Commits

Author SHA1 Message Date
Tim Rühsen
60f033426f Add GNU extensions to .netrc parsing
src/netrc.c (parse_netrc): Add 'port' and 'force' extensions

Reported-by: September 20Tim Landscheidt
2017-09-27 12:42:06 +02:00
Josef Moellers
6f3b995993 Bail out on unexpected 416 server errors
* src/http.c (gethttp): Stop on 416 if file is incomplete
2017-09-18 16:45:49 +02:00
Tim Schlueter
c451eec155 Add gzip Content-Encoding decompression
* src/http.c (struct http_stat): Add remote_encoding field.
(read_response_body): Enable gzip decompression.
(initialize_request): Send gzip Accept-Encoding header.
(gethttp): Decompress files with gzip Content-Encoding.
* src/retr.c: include zlib.h.
(zalloc): New function.
(zfree): New function.
(fd_read_body): Decompress gzip data.
* src/retr.h (fd_read_body enum): Add rb_compressed_gzip flag.
2017-08-04 14:34:53 +02:00
Tim Schlueter
b543dfe783 Add --compression option
* doc/wget.texi: Add --compression documentation.
* src/init.c (cmd_spec_compression): New function.
(commands[]): Add opt.compression.
(defaults): Set default opt.compression value.
* src/main.c (option_data[]): Add struct for --compression.
(print_help, help[]): Add description for --compression.
(main): Add incompatibility checks for --compression.
* src/options.h (struct options): Add compression enum and field.
2017-08-04 14:34:53 +02:00
Tim Schlueter
08ed2a5530 Adjust Extension based on Content-Encoding
* doc/wget.texi (--adjust-extension, adjust_extension): Updated documentation.
* src/http.c (encoding_t): New enum.
(struct http_stat): Add local_encoding field.
(gethttp): --adjust-extension based on Content-Encoding.
2017-08-04 14:34:53 +02:00
Darshit Shah
951d3e4cdd Document gperf as a requirement 2017-07-31 14:56:32 +02:00
Tim Rühsen
3ad3b3e36c * src/url.c (url_scheme): Use ASCII version of strncasecmp 2017-07-28 17:11:26 +02:00
Tim Rühsen
5fb6b6bd68 Fix misuse of strncasecmp
* src/http.c (set_content_type): Use c_strcasecmp instead of strncasecmp

See issue bug #51576
2017-07-28 16:56:27 +02:00
Tim Rühsen
f42229b1fd Fix python test suite for GnuTLS 3.5.12+
* testenv/Test-*.py: Replace 127.0.0.1 by localhost
* testenv/certs/server-template.cfg: Likewise
* testenv/certs/server-cert.pem: Regenerate
* testenv/certs/server-crl.pem: Likewise
* testenv/test/base_test.py: Hardcode 'localhost' as server domain

Reported-by: Ludovic Courtès <ludo@gnu.org>
2017-07-09 11:39:05 +02:00
Tim Rühsen
21154bdc36 Check for 304 response before applying --adjust-extension
* src/http.c (gethttp): Move 304 code before --adjust-extension code

This fixes applying --adjust-extension in combination with 304
HTTP responses. It could lead to .html extensions to arbitrary
files.

Reported-by: anfractuosity
2017-06-13 11:25:20 +02:00
Tim Rühsen
ae293c945a Fix buffer overflow in Public Key Pinning
* src/utils.c (wget_base64_decode): Add param for destination size,
  (wg_pubkey_pem_to_der): Amend call to wget_base64_decode(),
  (wg_pin_peer_pubkey): Likewise and fix code style.
* src/utils.h: Add param to wget_base64_decode()
* src/http-ntlm.c (ntlm_input): Amend call to wget_base64_decode()
* src/http.c (skip_content_type): Likewise

Fixes #51227
2017-06-13 10:23:04 +02:00
Tim Rühsen
407c1f990a * doc/wget.texi: Mention --no-config 2017-06-02 10:13:22 +02:00
Tim Rühsen
86b46a34a5 * testenv/Test-recursive-basic.py: Check crawled files 2017-05-16 11:20:57 +02:00
Tim Rühsen
5d4ada1b7b Fix two Metalink tests if $HOME is changed
* conf/expected_files.py (gen_local_fs_snapshot): Skip processing
  of 'pubring.kbx'
2017-05-16 10:24:52 +02:00
Tomas Hozza
876def8ebe Add command line option to disable use of .netrc
Although internally code uses option for (not) reading .netrc for
credentials, it was not possible to turn this behavior off on command
line. Note that it was possible to turn it off using wgetrc.

Idea for this change came from Bruce Jerrick (bmj001@gmail.com).
Reference: https://bugzilla.redhat.com/show_bug.cgi?id=1425097

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2017-05-15 16:06:50 +02:00
Tomas Hozza
f8c3df1f40 Fixed getting of credentials from .netrc
There seemed to be a copy&paste error in http.c code, which decides
whether to get credentials from .netrc. In ftp.c "user" and "pass"
variables are char*, while in http.c, these are char**. For this reason
they should be dereferenced when determining if password and user login
is set to some value.

Also since both variables are dereferenced on lines above the changed
code, it does not really make sense to check if they are NULL.

This patch is based on fix from Bruce Jerrick <bmj001@gmail.com>.
Fedora bug: https://bugzilla.redhat.com/show_bug.cgi?id=1425097

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2017-05-15 16:06:50 +02:00
Tomas Hozza
17960b57d5 Added tests for HTTP authentication using credentials from .netrc
Getting credentials from .netrc has been broken from time to time, thus
adding a test coverage to prevent regressions.

Also added setting of "HOME" environment variable when executing wget,
to make sure LocalFiles like .netrc, which are created just for the
test, are actually used.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2017-05-15 16:06:50 +02:00
Tim Rühsen
40c0d30f5c Fix Test-https-badcerts.px
* util/createcerts.sh: Set date of invalid cert to +20 years
* tests/certs/invalid.crt: Re-generated
* tests/certs/invalid.key: Re-generated
2017-05-15 15:46:11 +02:00
Tim Rühsen
e2c702571f * util/createcerts.sh: Fix double equal (syntax-check) 2017-05-14 21:35:34 +02:00
Tim Rühsen
269fb6201d Fix Test-https-badcerts.px to work with GnuTLS
* tests/Test-https-badcerts.px: Remove OpenSSL requirement
* tests/certs/expired.crt: Re-created with valid dates
* tests/certs/expired.key: Likewise
* tests/certs/invalid.crt: Likewise
* tests/certs/invalid.key: Likewise
2017-05-14 11:10:15 +02:00
Vijo Cherian
c08778aeb3 New shell script to create the certs and keys required for TLS tests
* util/createcerts.sh: New file
2017-05-14 11:07:43 +02:00
Tim Rühsen
297c1e2ac3 Fix HTTPS testing for stricter OpenSSL
* testenv/certs/README: Amend cert creation extensions
* testenv/certs/ca-cert.pem: Created without OCSP signing purpose

Having the OCSP signing purpose set made newer versions of OpenSSL
fail due to stricter checking. Test version of OpenSSL was 1.1.0e.
2017-05-11 12:38:19 +02:00
Tim Rühsen
936efc3564 * src/iri.c (idn_encode): Better IDNA 2003 compatibility 2017-05-11 11:56:26 +02:00
Tim Rühsen
11d3de74ca * .gitlab-ci.yml: Also test OpenSSL build 2017-05-10 12:12:42 +02:00
Tim Rühsen
35d5b67cc1 * .gitlab-ci.yml: Remove installation of texlive 2017-05-10 10:41:42 +02:00
Tim Rühsen
da50a1ecc3 Add certs/wotca.pem to avoid temp. file creation
* tests/certs/wotca.pem: New file
* tests/Test-https-weboftrust.px: Remove creation of wotca.pem

This change let Test-https-weboftrust.px survive VPATH builds.
2017-05-10 10:34:42 +02:00
Tim Rühsen
1068ca07d3 * cfg.mk: Exclude ^tests/certs/.* from syntax-check 2017-05-10 10:05:18 +02:00
Tim Rühsen
0666e0f375 * po/POTFILES.in: Remove spider.c (syntax-check) 2017-05-10 10:04:34 +02:00
Tim Rühsen
0d9f8280e0 * tests/Test-https-selfsigned.px: Add newline at EOF (syntax-check) 2017-05-10 10:04:30 +02:00
Vijo Cherian
17f0e16ecc Removed all uses of temp files.
Added needed files to GIT
2017-05-09 16:45:13 -07:00
Vijo Cherian
cc74f8b7e2 Made CRL related files in the repo, instead of trying to generate them 2017-05-09 15:37:28 -07:00
Tim Rühsen
2894a37189 Cleanup on exit in Test-https-*.px
* tests/Test-https-*.px: Cleanup on exit
2017-05-09 16:58:33 +02:00
Tim Rühsen
4669ba5e0c Auto-generate interca.conf and rootca.conf
* configure.ac: Add interca.conf.in and rootca.conf.in to AC_CONFIG_FILES
* tests/certs/interca.conf: Removed
* tests/certs/rootca.conf: Removed
* tests/certs/interca.conf.in: New file
* tests/certs/rootca.conf.in: New file
2017-05-09 16:04:48 +02:00
Tim Rühsen
b5c2d083ea Fix path and VPATH issues of new https/TLS tests
* tests/SSLTest.pm: Use $srcdir in read-only files,
  (_setup_server): Remove unneeded path fixation code
* tests/Test-https-*.px: Use $srcdir for read-only files,
  use $cdir for writable files
* tests/certs/interca.conf: Adjust paths
* tests/certs/rootca.conf: Adjust paths
2017-05-09 15:21:23 +02:00
Tim Rühsen
466afc62b0 Check for test server name resolution in tests
* tests/Test-https-*.px: Skip if test server name resolution fails
2017-05-09 11:16:30 +02:00
Tim Rühsen
f68d001626 * tests/Makefile.am: Enable Test-https-badcerts again 2017-05-09 11:16:30 +02:00
Tim Rühsen
ffe75d0867 Fix WgetFeature.pm to allow multiple required features
* tests/WgetFeature.cfg: Remove file
* tests/WgetFeature.pm: Extend to multiple features, cleanup
2017-05-09 11:16:30 +02:00
Tim Rühsen
9aa894853f * .gitlab-ci.yml: Add wgettestingserver to /etc/hosts 2017-05-09 11:08:32 +02:00
Tim Rühsen
5337b94ce5 * tests/SSLServer.pm: Check for IO::Socket::SSL 2017-05-09 11:08:32 +02:00
Tim Rühsen
a26e6f3527 * tests/Test-https-*: Change server port to <= 32767 2017-05-09 11:08:32 +02:00
Tim Rühsen
b9fb74ddfa Move https test server ports from >32767 to <= 32767
* Test-https-badcerts.px: Change port
* Test-https-crl.px: Likewise
* Test-https-weboftrust.px: Likewise
2017-05-09 11:08:32 +02:00
Tim Rühsen
3132049ae4 * tests/Makefile.am: Add SSLTest.pm and SSLServer.pm to EXTRA_DIST 2017-05-09 11:08:32 +02:00
Tim Rühsen
c0c42da653 * tests//Makefile.am: Disable Test-https-badcerts.px 2017-05-09 11:08:32 +02:00
Tim Rühsen
888cc82c9d Add Gitlab CI (Debian)
* .gitlab-ci.yml: New file
2017-05-09 11:08:32 +02:00
Tim Rühsen
fce1b689e8 * .travis.yml: Use trusty for libidn2-dev 2017-05-06 13:03:28 +02:00
Tim Rühsen
580067d1e6 * tests/certs/test-ca-key.pem: Add newline at EOF 2017-05-04 16:51:49 +02:00
Tim Rühsen
5c4cc011fe Add static HOSTSALIAS file
* tests/certs/wgethosts: New file
* tests/Test-https-*.px: Remove creation of wgethosts file
2017-05-04 16:51:49 +02:00
Vijo Cherian
2a96249469 Added new tests for SSL
* tests/Test-https-badcerts.px : New file
* tests/Test-https-clientcert.px : New file
* tests/Test-https-crl.px : New file
* tests/Test-https-weboftrust.px : New file
* tests/certs/interca.conf : New file
* tests/certs/rootca.conf : New file
* tests/certs/test-ca-key.pem : New file

Added all new SSL / HTTPS tests to make check
Added Test for SSL Web of Trust, accept only if CA chain of trust is intact.
Added a test script for client certificate
Added Test for crlfile option of wget
Added test to make sure that wget doesn't accept expired or invalid certs

Some clean up : 1, Removed cause of warnings from perl & other cosmetic changes
                2, Fix  make -j 4 check such that it passes all tests
2017-05-04 16:51:49 +02:00
Tomas Hozza
0b41c7543a Mention TLSv1_1 and TLSv1_2 as secure-protocol values in help
* src/main.c: The --secure-protocol option accepts also values TLSv1_1
and TLSv1_2, as mentioned in the man page. However the help message
doesn't mention these two values. This patch adds TLSv1_1 and TLSv1_2 as
possible values to the help message.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2017-05-04 14:51:54 +02:00
Tim Rühsen
c4a2b2e77e * src/http.c (gethttp): Support Wayback Machine's X-Archive-Orig-last-modified 2017-05-03 16:37:11 +02:00