* src/recur.c (descend_redirect): Ignore WG_RR_LIST and WG_RR_REGEX
for redirections.
* testenv/Makefile.am: Add Test-recursive-redirect.py
* testenv/Test-recursive-redirect.py: New test
Test-recursive-redirect.py written by Dale R. Worley.
Reported-by: "Dale R. Worley" <worley@ariadne.com>
* testenv/misc/metalinkv3_xml.py: New Metalink/XML v3 python class
* testenv/Test-metalink-xml-abspath-trust.py: Update test to use the
new Metalink/XML v3 python class
* testenv/Test-metalink-xml-abspath.py: Update test to use the new
Metalink/XML v3 python class
* testenv/Test-metalink-xml-absprefix-trust.py: Update test to use the
new Metalink/XML v3 python class
* testenv/Test-metalink-xml-absprefix.py: Update test to use the new
Metalink/XML v3 python class
* testenv/Test-metalink-xml-continue.py: Update test to use the new
Metalink/XML v3 python class
* testenv/Test-metalink-xml-emptyprefix-trust.py: Update test to use
the new Metalink/XML v3 python class
* testenv/Test-metalink-xml-homepath-trust.py: Update test to use the
new Metalink/XML v3 python class
* testenv/Test-metalink-xml-homepath.py: Update test to use the new
Metalink/XML v3 python class
* testenv/Test-metalink-xml-homeprefix-trust.py: Update test to use
the new Metalink/XML v3 python class
* testenv/Test-metalink-xml-homeprefix.py: Update test to use the new
Metalink/XML v3 python class
* testenv/Test-metalink-xml-nohash.py: Update test to use the new
Metalink/XML v3 python class
* testenv/Test-metalink-xml-nourls.py: Update test to use the new
Metalink/XML v3 python class
* testenv/Test-metalink-xml-prefix-trust.py: Update test to use the
new Metalink/XML v3 python class
* testenv/Test-metalink-xml-prefix.py: Update test to use the new
Metalink/XML v3 python class
* testenv/Test-metalink-xml-relpath-trust.py: Update test to use the
new Metalink/XML v3 python class
* testenv/Test-metalink-xml-relpath.py: Update test to use the new
Metalink/XML v3 python class
* testenv/Test-metalink-xml-relprefix-trust.py: Update test to use the
new Metalink/XML v3 python class
* testenv/Test-metalink-xml-relprefix.py: Update test to use the new
Metalink/XML v3 python class
* testenv/Test-metalink-xml-size.py: Update test to use the new
Metalink/XML v3 python class
* testenv/Test-metalink-xml-trust.py: Update test to use the new
Metalink/XML v3 python class
* testenv/Test-metalink-xml.py: Update test to use the new
Metalink/XML v3 python class
* src/http.c (metalink_from_http): Process the Content-Type header.
Add an application/metalink4+xml URL as metalink metaurl. If the
option opt.content_disposition is true, the Content-Disposition's
filename is the metaurl's name
* doc/wget.texi: Update --content-disposition and --metalink-over-http
* doc/metalink-standard.txt: Update doc. Content-Type/Disposition
processing through --metalink-over-http. Update download naming
system about --trust-server-names and --content-disposition
* testenv/Makefile.am: Add new files
* testenv/Test-metalink-http-xml-type.py: New file. Metalink/HTTP
Content-Type/Disposition header automated Metalink/XML tests
* testenv/Test-metalink-http-xml-type-trust.py: New file. Metalink/HTTP
Content-Type/Disposition header with --trust-server-names automated
Metalink/XML tests
* testenv/Test-metalink-http-xml-type-content.py: New file. Metalink/HTTP
Content-Type/Disposition header with --content-disposition automated
Metalink/XML tests
* testenv/Test-metalink-http-xml-type-trust-content.py: New file.
Metalink/HTTP Content-Type/Disposition header with --trust-server-names
and --content-disposition automated Metalink/XML tests
Process the Content-Type header, identify an application/metalink4+xml
file. The Content-Disposition could provide an alternate name through
the "filename" field for the metalink xml file. Respectively, the cli
options --metalink-over-http and --content-disposition are required.
When Metalink/XML auto-processing, to use the Content-Disposition's
filename, the cli option --trust-server-names is also required.
* src/metalink.c (retrieve_from_metalink): If opt.trustservernames is
true, use the basename of the metaurl's name to save the xml file
* doc/metalink-standard.txt: Update doc. With --trust-server-names any
Metalink/HTTP Link application/metalink4+xml file is saved using the
basename of the "name" field, if any. Update Metalink/HTTP examples
* testenv/Makefile.am: Add new file
* testenv/Test-metalink-http-xml-trust-name.py: New file. Metalink/HTTP
automated Metalink/XML, save xml files using the "name" field tests
* src/metalink.c (retrieve_from_metalink): Reject any metalink:file
without hashes. Prompt the error and switch to the next file
* testenv/Makefile.am: Add new file
* testenv/Test-metalink-xml-nohash.py: New file. Metalink/XML with no
hashes tests
Prevent SIGSEGV.
* src/http.c (metalink_from_http): Fix hash_bin_len type. Use ssize_t
instead than size_t. Reject -1 as base64_decode() return value
* testenv/Makefile.am: Add new file
* testenv/Test-metalink-http-baddigest.py: New file. Metalink/HTTP
malformed base64 Digest header tests
On malformed base64 input, ssize_t base64_decode() returns -1. Such
value is too big for a size_t variable, and used as xmalloc() value
will exaust all the memory.
* NEWS: Mention the effect of --metalink-index over Metalink
* src/init.c: Add new option metalinkindex (opt.metalink_index),
initialize to -1
* src/main.c: Add new option metalink-index (--metalink-index=NUMBER)
* src/options.h: Add new option metalink_index (int)
* src/metalink.h: Add declaration of functions fetch_metalink_file(),
replace_metalink_basename()
* src/metalink.c: Add functions fetch_metalink_file() simple file
fetch, replace_metalink_basename() replace file basename
* src/metalink.c (retrieve_from_metalink): New. Process Metalink
application/metalink4+xml of opt.metalink_index ordinal number
* doc/wget.texi: Add new option metalink-index (--metalink-index)
documentation
* doc/metalink-standard.txt: Updated doc. Add documentation about
Metalink application/metalink4+xml metaurls download naming system
* doc/metalink-standard.txt: Update Metalink/XML and HTTP examples
* testenv/Makefile.am: Add new files
* testenv/Test-metalink-http-xml.py: New file. Metalink/HTTP automated
Metalink/XML "application/metalink4+xml" --metalink-index tests
* testenv/Test-metalink-http-xml-trust.py: New file. Metalink/HTTP
automated Metalink/XML "application/metalink4+xml" --metalink-index
retrieval with --trust-server-names tests
WARNING: Do not use lib/dirname.c (dir_name) to get the directory
name, it may append a dot '.' character to the directory name.
* testenv/Makefile.am: Add new file
* testenv/Test-metalink-xml-emptyprefix-trust.py: New file.
Metalink/XML empty directory prefix (--directory-prefix '') tests
Detect a '/' character wrongfully concatenated to an empty directory
prefix '' (not NULL), resulting in an absolute path as '/dir/file',
instead than 'dir/file'.
* src/metalink.h: Add declaration of function dequote_metalink_string()
* src/metalink.c: Add function dequote_metalink_string() remove
surrounding quotes from string, \' or \"
* src/metalink.c (find_key_value, find_key_values): Call dequote_metalink_string()
to remove the surrounding quotes from the parsed value
* src/metalink.c (test_find_key_value, test_find_key_values): Add
quoted key's values for unit-tests
* testenv/Makefile.am: Add new file
* testenv/Test-metalink-http-quoted.py: New file. Metalink/HTTP quoted
values tests
Some Metalink/HTTP keys, like "type" [2], may have a quoted value [1]:
Link: <http://example.com/example.ext.meta4>; rel=describedby;
type="application/metalink4+xml"
Wget was expecting a dequoted value from the Metalink module. This
patch addresses this problem.
References:
[1] Metalink/HTTP: Mirrors and Hashes
1.1. Example Metalink Server Response
https://tools.ietf.org/html/rfc6249#section-1.1
[2] Additional Link Relations
6. "type"
https://tools.ietf.org/html/rfc6903#section-6
* src/metalink.h: Add declaration of function clean_metalink_string()
* src/metalink.c: Add directive #include "xmemdup0.h"
* src/metalink.c: Add function clean_metalink_string() remove leading
and trailing white spaces and CRLF from string
* src/metalink.c (retrieve_from_metalink): Remove leading and trailing
white spaces and CRLF from url resource mres->url
* testenv/Makefile.am: Add new file
* testenv/Test-metalink-xml-urlbreak.py: New test. Metalink/XML white
spaces and CRLF in url resources tests
White spaces and CRLF are not automatically removed by libmetalink
from url strings. The Wget's Metalink module was unable to process
such url strings. This patch implements the processing of such url
strings cleaning off leading and trailing white spaces and CRLF.
If a parsed Metalink/XML url string contains strings separated by
CRLF, only the first of the series is accepted.
* testenv/Makefile.am: Add new file
* testenv/Test-metalink-xml-nourls.py: New file. Metalink/XML unknown
urls tests
Test if when there are no good Metalink url resources there is any
segmentation fault.
* src/wget.h (uerr_t): Add error code METALINK_SIZE_ERROR to enum
* src/metalink.c (retrieve_from_metalink): Use boolean variable
size_ok, when false set retr_err to METALINK_SIZE_ERROR
* testenv/Makefile.am: Add new file
* testenv/Test-metalink-xml-size.py: New file. Metalink/XML file size
tests (<size></size>)
Before this patch, no appropriate error code was returned to inform a
file size mismatch.
This patch introduces the error code METALINK_SIZE_ERROR to inform a
file size mismatch.
* NEWS: Mention the effect of --trust-server-names over Metalink
* src/metalink.h: Add declaration of function append_suffix_number()
* src/metalink.c: Add function append_suffix_number() append number to
string
* src/metalink.c (retrieve_from_metalink): Safer Metalink/XML and
Metalink/HTTP download naming system, opt.trustservernames based
* doc/metalink-standard.txt: Update doc. Explain new Metalink/XML and
Metalin/HTTP download naming system and --trust-server-names role
* testenv/Makefile.am: Add new files
* testenv/Test-metalink-xml-continue.py: Update test. Metalink/XML
continue/keep existing files (HTTP 416) with --continue tests
* testenv/Test-metalink-xml.py: Update test. Metalink/XML naming tests
* testenv/Test-metalink-xml-trust.py: New file. Metalink/XML naming
tests with --trust-server-names
* testenv/Test-metalink-xml-abspath.py: Update test. Metalink/XML
absolute path tests
* testenv/Test-metalink-xml-abspath-trust.py: New file. Metalink/XML
absolute path tests with --trust-server-names
* testenv/Test-metalink-xml-relpath.py: Update test. Metalink/XML
relative path tests
* testenv/Test-metalink-xml-relpath-trust.py: New file. Metalink/XML
relative path tests with --trust-server-names
* testenv/Test-metalink-xml-homepath.py: Update test. Metalink/XML
home path and ~ (tilde) tests
* testenv/Test-metalink-xml-homepath-trust.py: New file. Metalink/XML
home path and ~ (tilde) tests with --trust-server-names
* testenv/Test-metalink-xml-prefix.py: New file. Metalink/XML naming
tests with --directory-prefix
* testenv/Test-metalink-xml-prefix-trust.py: New file. Metalink/XML
naming tests with --directory-prefix and --trust-server-names
* testenv/Test-metalink-xml-absprefix.py: New file. Metalink/XML
absolute --directory-prefix tests
* testenv/Test-metalink-xml-absprefix-trust.py: New file. Metalink/XML
absolute --directory-prefix tests with --trust-server-names
* testenv/Test-metalink-xml-relprefix.py: New file. Metalink/XML
relative --directory-prefix tests
* testenv/Test-metalink-xml-relprefix-trust.py: New file. Metalink/XML
relative --directory-prefix tests with --trust-server-names
* testenv/Test-metalink-xml-homeprefix.py: New file. Metalink/XML home
--directory-prefix tests
* testenv/Test-metalink-xml-homeprefix-trust.py: New file. Metalink/XML
home --directory-prefix tests with --trust-server-names
The option --trust-server-names allows to use the file names parsed
from a Metalink/XML file. Without --trust-server-names, the safety
mechanism provides secure and predictable file names.
* testenv/Test-metalink-xml-relpath.py: Update test
* testenv/Test-metalink-xml-homepath.py: New file. Reject home paths
* testenv/Makefile.am: Add new file
When --input-metalink=<file> is used, each metalink:file name is
verified by libmetalink's metalink_check_safe_path(). By design,
absolute, relative and home paths are rejected.
At the moment of writing, when --metalink-over-http is used, absolute,
relative, and home paths aren't a concern. The destination file name
is a combination of URL's file name and cli's "Directory Options"
handled by src/url.c (url_file_name).
* NEWS: Mention the effects of --continue over Metalink
* src/metalink.c (retrieve_from_metalink): On download error, resume
output_stream with the next mres->url. Keep fully downloaded files
started with --continue, otherwise rename/remove the file
* testenv/Makefile.am: Add new file
* testenv/Test-metalink-xml-continue.py: New file. Metalink/XML
continue/keep existing files (HTTP 416) with --continue tests
Before this patch, with --continue, existing and/or fully retrieved
files which fail the sanity tests were renamed (--keep-badhash), or
removed.
This patch ensures that --continue doesn't rename/remove existing
and/or fully retrieved files (HTTP 416) which fail the sanity tests.
* testenv/Test-metalink-http.py: Use python .replace
* testenv/Test-metalink-xml.py: Use python .replace
* testenv/Test-metalink-xml-abspath.py: Use python .replace
* testenv/Test-metalink-xml-relpath.py: Use python .replace
Use python .replace instead than re.sub, remove 'import re'.
* testenv/Test-metalink-xml-abspath.py: Change Metalink/XML v3 file
name from test.meta4 into test.metalink
* testenv/Test-metalink-xml-relpath.py: Change Metalink/XML v3 file
name from test.meta4 into test.metalink
* testenv/Test-metalink-xml.py: Change Metalink/XML v3 file name from
test.meta4 into test.metalink
* testenv/Test-recursive-basic.py: New file. Test basic recursion
* testenv/Test-recursive-include.py: New File. Recursion test with
include directories
* testenv/Test-redirect.py: New File. Basic redirection tests
* testenv/Makefile.am: Add new tests to makefile
* Test-pinnedpubkey-hash-https.py: Read hashed pubkey from file
* Test-pinnedpubkey-hash-no-check-fail-https.py: Use invalid hash
* certs/make_ca.sh: New script to generate test certs non-interactive
* certs/ca-template.cfg: New file (template for CA cert)
* certs/server-template.cfg: New file (template for server cert)
* certs/server-pubkey-sha256.base64: New file (pubkey sha256 hash)
* testenv/Makefile.am: Add Test-missing-scheme-retval.py
* testenv: New file Test-missing-scheme-retval.py
Idea and Perl implementation by Zdenek Dohnal <zdohnal@redhat.com>
* testenv/Makefile.am: Add new tests
* testenv/Test-pinnedpubkey-der-https.py: New test
* testenv/Test-pinnedpubkey-der-no-check-https.py: New Test
* testenv/Test-pinnedpubkey-hash-https.py: New test
* testenv/Test-pinnedpubkey-hash-no-check-fail-https.py: New test
* testenv/Test-pinnedpubkey-pem-fail-https.py: New test
* testenv/Test-pinnedpubkey-pem-https.py: New test
* testenv/certs/README: How to generate public keys with openssl tool
* testenv/certs/server-pubkey.der: New key file (DER format)
* testenv/certs/server-pubkey.pem: New key file (PEM format)
* testenv/test/base_test.py (__init__): Use test file name for name,
remove 'name' parameter
* testenv/test/http_test.py (__init__): Remove 'name' parameter
* testenv/*.py: Remove TEST_NAME
Using a fixed string (TEST_NAME) to build the working directory
for testing caused random failures (or successes) when tests
share the same TEST_NAME value. Not easy to find without digging
into the python test suite code.
We now use the test file name, which is unique within the test
environment.
* testenv/Makefile.am: Set SSL_TESTS env variable
* testenv/Test--https-crl.py, testenv/Test--https.py,
testenv/Test-hsts.py: Return 77 (SKIP) if SSL/TLS is not configured
* testenv/test-auth-both.py: Add qop parameter for digest auth
* testenv/test-auth-digest.py: Same
* testenv/conf/authentication.py: Support additional parameters for
authentication
* testenv/servers/http/http_server.py: Same
* testenv/Makefile.am: added new test 'Test-hsts.py'.
* testenv/Test-hsts.py: new test for HSTS.
* testenv/conf/domains.py: new hook to override domain list.
* testenv/test/base_test.py: (__init__): new optional parameter
for tests 'req_protocols'.
(get_domain_addr): set the instance variables 'addr' and 'port'.
Return address as an array (domain, port) instead of string.
(gen_cmd_line): take into account domain and port.
* testenv/test/http_test.py (__init__): new optional parameter
'req_protocols'.
(setup): new function. Call to server_setup() decoupled from
begin() and moved here.
(begin): call to superclass to maintain backward compatibility.
Removed call to server_setup().
This patch adds a new parameter to the test suite called 'req_protocols',
and a new function called 'setup'. The ability for tests to be able to set some
extra parameters such as the actual requested protocols (with 'req_protocols')
became obvious when support for HSTS was added to Wget, where the requested URI
and the actual executed URI do not have to be the same. This new parameter is optional
and if not specified, the test suite behaves as before. Also, the new function 'setup'
is provided as a means to start the test HTTP server, but not launch the test yet
(this is done when calling 'begin', as usual), in case we want to query the address
and port in which the test server listens. If 'setup' is not called, it is automatically
invoked when calling 'begin'. With these measures, we preserve backward-compatibility with
existing tests.
* testenv/README: Describe how to use repeated header name.
* testenv/server/http/http_server.py (finish_headers): Send all
values from list if the header value is a Python list.
* configure.ac: Export WITH_SSL for use in Makefile.am
* testenv/Makefile.am: Add HTTPS tests to XFAIL_TESTS when !WITH_SSL
Reported-by: Ander Juaristi <ajuaristi@gmx.es>
* testenv/test/base_test.py: Use Valgrind SSL suppressions file for
tests
* testenv/valgrind-suppression-ssl, tests/valgrind-suppression-ssl:
Add new suppression files to suppress OpenSSL errors in valgrind
* tests/test-proxied-https-auth.px: Use the valgrind SSL
suppressions file for the test
* tests/test-proxied-https-auth-keepalive.px: Same
* src/exc/server_error.py: Add exception for GET to HEAD fallback.
* src/server/http/http_server.py: Do not send body if 304 return
code requested for a file.
