Commit Graph

3747 Commits

Author SHA1 Message Date
Tim Rühsen
5337b94ce5 * tests/SSLServer.pm: Check for IO::Socket::SSL 2017-05-09 11:08:32 +02:00
Tim Rühsen
a26e6f3527 * tests/Test-https-*: Change server port to <= 32767 2017-05-09 11:08:32 +02:00
Tim Rühsen
b9fb74ddfa Move https test server ports from >32767 to <= 32767
* Test-https-badcerts.px: Change port
* Test-https-crl.px: Likewise
* Test-https-weboftrust.px: Likewise
2017-05-09 11:08:32 +02:00
Tim Rühsen
3132049ae4 * tests/Makefile.am: Add SSLTest.pm and SSLServer.pm to EXTRA_DIST 2017-05-09 11:08:32 +02:00
Tim Rühsen
c0c42da653 * tests//Makefile.am: Disable Test-https-badcerts.px 2017-05-09 11:08:32 +02:00
Tim Rühsen
888cc82c9d Add Gitlab CI (Debian)
* .gitlab-ci.yml: New file
2017-05-09 11:08:32 +02:00
Tim Rühsen
fce1b689e8 * .travis.yml: Use trusty for libidn2-dev 2017-05-06 13:03:28 +02:00
Tim Rühsen
580067d1e6 * tests/certs/test-ca-key.pem: Add newline at EOF 2017-05-04 16:51:49 +02:00
Tim Rühsen
5c4cc011fe Add static HOSTSALIAS file
* tests/certs/wgethosts: New file
* tests/Test-https-*.px: Remove creation of wgethosts file
2017-05-04 16:51:49 +02:00
Vijo Cherian
2a96249469 Added new tests for SSL
* tests/Test-https-badcerts.px : New file
* tests/Test-https-clientcert.px : New file
* tests/Test-https-crl.px : New file
* tests/Test-https-weboftrust.px : New file
* tests/certs/interca.conf : New file
* tests/certs/rootca.conf : New file
* tests/certs/test-ca-key.pem : New file

Added all new SSL / HTTPS tests to make check
Added Test for SSL Web of Trust, accept only if CA chain of trust is intact.
Added a test script for client certificate
Added Test for crlfile option of wget
Added test to make sure that wget doesn't accept expired or invalid certs

Some clean up : 1, Removed cause of warnings from perl & other cosmetic changes
                2, Fix  make -j 4 check such that it passes all tests
2017-05-04 16:51:49 +02:00
Tomas Hozza
0b41c7543a Mention TLSv1_1 and TLSv1_2 as secure-protocol values in help
* src/main.c: The --secure-protocol option accepts also values TLSv1_1
and TLSv1_2, as mentioned in the man page. However the help message
doesn't mention these two values. This patch adds TLSv1_1 and TLSv1_2 as
possible values to the help message.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2017-05-04 14:51:54 +02:00
Tim Rühsen
c4a2b2e77e * src/http.c (gethttp): Support Wayback Machine's X-Archive-Orig-last-modified 2017-05-03 16:37:11 +02:00
Vijo Cherian
f6376ac0dc Added new tests for SSL
* tests/Test-https-badcerts.px : New file
* tests/Test-https-clientcert.px : New file
* tests/Test-https-crl.px : New file
* tests/Test-https-weboftrust.px : New file
* tests/certs/interca.conf : New file
* tests/certs/rootca.conf : New file
* tests/certs/test-ca-key.pem : New file

Added all new SSL / HTTPS tests to make check
Added Test for SSL Web of Trust, accept only if CA chain of trust is intact.
Added a test script for client certificate
Added Test for crlfile option of wget
Added test to make sure that wget doesn't accept expired or invalid certs

Some clean up : Removed cause of warnings from perl & other cosmetic changes
2017-04-28 12:22:54 +02:00
Vijo Cherian
ac519c041e Added new tests for SSL
* tests/SSLServer.pm: New file
* tests/SSLTest.pm: New file
* tests/Test-https-pfs.px: New file
* tests/Test-https-selfsigned.px: New file
* tests/Test-https-tlsv1.px: New file
* tests/Test-https-tlsv1x.px: New file
* tests/certs/server.crt: New file
* tests/certs/server.key: New file
* tests/certs/test-ca-cert.pem: New file

Added 4 new test scripts all for SSL.
Added base pm for SSL testing.
Added SSL tests for TLSv1, TLSv1_1 and PFS.
Added test for self signed cert : check that it fails without
--no-check-certificate and passes with that flag.
2017-04-19 12:59:27 +02:00
Tim Rühsen
56c78c4b09 * src/utils.c: Remove non-portable __builtin_unreachable() 2017-04-18 13:22:25 +02:00
Tim Rühsen
0ec46cb109 Skip iconv() usage if HAVE_ICONV is not defined
This helps on broken iconv implementations, e.g. Solaris.

Reported-by: Mojca Miklavec
2017-04-18 13:17:19 +02:00
Tim Rühsen
67cb37ce5a Mention 'bash' for executing ./bootstrap
Reported-by: Mojca Miklavec
2017-04-18 10:47:15 +02:00
Tim Rühsen
0004d3ec22 * bootstrap.conf: Make 'sed' options more portable
'sed -i' works differently on FreeBSD.

Reported-by: Mojca Miklavec
2017-04-18 10:44:06 +02:00
Tim Rühsen
92bfe2a2e4 Fix charset transcoding issue for non-reversible codepoints
* src/url.c: Check iconv() against 0, not -1

On some libiconv implementations, unknown codepoints become
encoded as ?, e.g. when converting a non-ascii codepoint to ASCII.
This results in ambigious file names which also fails our tests.
2017-04-16 19:55:14 +02:00
Tim Rühsen
fc2f4233ed * src/iri.c: Fix WIN32 idn2_free, forgotten code 2017-04-16 19:50:10 +02:00
Darshit Shah
b2c38d33e1 * src/init.c: Set flstats correctly when using WGETRC env var 2017-04-14 01:16:49 +02:00
Tim Rühsen
6ef493b19e Fix use of idn2_free()
* src/connect.c (connect_to_ip): Use xfree() instead of idn2_free()
* src/host.c (lookup_host): Use xfree() instead of idn2_free()
* src/iri.h: Do not include idn2.h
* src/url.c (url_free): Use xfree() instead of idn2_free()
* src/url.h (struct url): Remove 'idn_allocated' from struct

Reported-by: Gisle Vanem
2017-04-08 11:05:55 +02:00
Anton Yuzhaninov
7ffe93cabb Fix perl warnings in tests
* tests/FTPServer.pm: Escape '{' in RE to fix warnings
* tests/FTPTest.pm: Likewise
* tests/HTTPServer.pm: Likewise
* tests/HTTPTest.pm: Likewise
* tests/Test-proxied-https-auth-keepalive.px: Likewise
* tests/Test-proxied-https-auth.px: Likewise
Escape '{' in RE to fix warnings:
Unescaped left brace in regex is deprecated, passed through in regex;
marked by <-- HERE in m/{{ <-- HERE port}}/
2017-04-05 20:04:12 +02:00
klemens
f381831d88 Fix typos in comments 2017-04-01 19:38:09 +02:00
Tim Rühsen
02d40a4676 * src/metalink.c (retrieve_from_metalink): Fix len in memset() 2017-03-31 13:15:27 +02:00
Tim Rühsen
e89267fbbc Add gnulib module group-member 2017-03-24 11:36:01 +01:00
Vijo Cherian
400b8eba6c Safeguards against TOCTTOU
* src/utils.h: Add struct file_stat_s declaration,
  change prototypes of file_exists_p(),
  add prototypes for fopen_stat() and open_stat().
* src/utils.c: Extend file_exists_p(),
  new function fopen_stat() and open_stat(),
  add new param for file_exists_p().
* src/init.h: Add param file_stats_t to run_wgetrc().
* src/ftp.c: Amend calls to extended functions.
* src/hsts.c: Likewise.
* src/http.c: Likewise.
* src/init.c: Likewise.
* src/main.c: Likewise.
* src/metalink.c: Likewise.
* src/retr.c: Likewise.
* src/url.c: Likewise.

Added fopen_stat() and open_stat() that checks to makes sure the file didn't
change underneath us.
Return error from file_exists_p().
Added a way to return error from this file without major surgery to the
callers.

Fixes: #20369
2017-03-24 09:39:09 +01:00
Tim Rühsen
90a0a7499c Update gnulib 2017-03-24 09:19:52 +01:00
Christof Horschitz
1d71645c06 * src/warc.c (warc_write_cdx_record): Escape URLs 2017-03-22 15:01:04 +01:00
Mike Frysinger
e249844143 Include libunistring headers only when used
* src/iri.c: Check for libidn2 < 0.14 to include libunistring headers

The unistring functions are used only when an older version of libidn2
is used, so don't include its headers either w/newer libdin2 versions.
2017-03-20 09:39:20 +01:00
Tim Rühsen
84a93f4127 Fix links to www.robotstxt.org
* NEWS: Fix links
* doc/wget.texi: Likewise
* src/res.c: Likewise

Reported-by: Noël Köthe
2017-03-18 19:05:38 +01:00
Tim Rühsen
f31b93424b * tests/WgetTests.pm: Add -d to Wget test options 2017-03-13 21:31:09 +01:00
Tim Rühsen
90b487369a Include <arpa/inet.h> for Windows
Reported-by: Gisle Vanem
2017-03-08 13:00:54 +01:00
Tim Rühsen
57d748117f Fix updating HSTS entries
* src/hsts.c (hsts_store_entry): Always update 'created' field

Fixes: #50490
Reported-by: Deian Stefan, Atyansh Jaiswal, Jonathan Luck
2017-03-08 10:56:12 +01:00
Tim Rühsen
4d729e322f Fix CRLF injection in Wget host part
* src/url.c (url_parse): Reject control characters in host part of URL

Reported-by: Orange Tsai
2017-03-06 10:04:22 +01:00
Benjamin Esham
63c2aea255 * src/warc.c: Use warc_write_header_uri for all WARC-Target-URI fields
The WARC spec requires that all URIs be enclosed in angle brackets. This
was being done in most cases, but not for "WARC-Target-URI" fields in
WARC blocks of type "response", "resource", "revisit", and "metadata".
2017-03-04 12:51:39 +01:00
Tim Rühsen
ac4fed3220 Fix 504 status handling
* src/http.c (gethttp): Move 504 handling to correct place.
  (http_loop): Fix memeory leak.
* testenv/server/http/http_server.py: Add Content-Length header on non-2xx
  status codes with a body

Reported-by: Adam Sampson
2017-02-16 15:53:56 +01:00
YX Hao
cf5df5593d * src/url.c (url_file_name): Do not charset convert local directory
In a non-ASCII environment, the local path may contain non-ASCII
characters. The server responded file name must be converted before
it is concatenated to the local path. Conversion after concatenation
may result in 'iconv' errors.
2017-02-16 12:52:16 +01:00
Tim Rühsen
2215ee8d10 * configure.ac: Remove manually resetting of LIBICONV variable
Fixes: #48193
Reported-by: ilovezfs
Reported-by: Charles
2017-02-15 20:03:58 +01:00
Tim Rühsen
32e26dc199 * bootstrap.conf: Call gperf to create lib/unicase/special-casing-table.h 2017-02-15 14:45:23 +01:00
Tim Rühsen
21ac4ae356 * bootstrap.conf: Fix latest gnulib to work with gperf < 3.1 2017-02-15 13:03:10 +01:00
Darshit Shah
838c185ca9 Pull GNULib to latest.
* gnulib: Pull to latest commit. This fixes an issue with Gperf 3.1
2017-02-14 15:08:50 +01:00
Tim Rühsen
ac9be9b756 * src/main.c: Remove double 'verbose' option
Fixes: #50290
2017-02-12 21:18:23 +01:00
Tim Rühsen
6620c5156f * NEWS: update 2017-02-11 12:03:54 +01:00
Tim Rühsen
aebd49d9d4 isrc/http.c (check_retry_on_http_error): Fix gcc warning 2017-02-11 11:54:21 +01:00
Tom Szilagyi
d6d00006a0 Add support for --retry-on-http-error
* doc/wget.text: Add documentation
* src/http.c: Add function check_retry_on_http_error ()
* src/init.c: Add opt.retry_on_http_error
* src/main.c: Add struct for retry-on-http-error to option_data[]
* src/options.h: Add retry_on_http_error to struct options
2017-02-11 11:50:24 +01:00
Tim Rühsen
05acf5d3f6 Revert "Add support for --retry-on-http-error"
This reverts commit 977276374d.
2017-02-11 11:45:11 +01:00
Tim Rühsen
80c62c238e Change libtool library deps to non-libtool deps
Reported-by: Yousong Zhou
Fixes: #50260
2017-02-10 17:20:42 +01:00
Tom Szilagyi
977276374d Add support for --retry-on-http-error
* doc/wget.texi: Add description for --retry-on-http-error
* src/http.c (gethttp):
Consider given HTTP response codes as non-fatal, transient errors.
Supply a comma-separated list of 3-digit HTTP response codes as
argument. Useful to work around special circumstances where retries
are required, but the server responds with an error code normally not
retried by Wget. Such errors might be 503 (Service Unavailable) and
429 (Too Many Requests). Retries enabled by this option are performed
subject to the normal retry timing and retry count limitations of
Wget.

Using this option is intended to support special use cases only and is
generally not recommended, as it can force retries even in cases where
the server is actually trying to decrease its load. Please use it
wisely and only if you know what you are doing.

Example use and a starting point for manual testing:
  wget --retry-on-http-error=429,503 http://httpbin.org/status/503
2017-02-09 21:17:20 +01:00
Tim Rühsen
785c4b92dd * tests/WgetTests.pm: Add --no-config to wget invocation
Without --no-config, wgetrc commands may disturb the tests.

Reported-by: Dagobert Michelsen
2017-02-09 16:34:27 +01:00