* testenv/README: Document $WGET_PATH
* testenv/test/base_test.py: Use $WGET_PATH instead ../src/wget if set
Previously tests in testenv/ directory were run only on wget binary
which was built from sources in src/ directory. However as a
wget maintainer in a Linux distribution, I would like to be able to run
upstream tests on the wget binary distributed with the distribution.
This change enables one to define WGET_PATH environment variable
to a path to wget binary which should be used by tests.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
of the header field while RejectHeader rejects ONLY the specified full header.
Since we wanted to be sure a header field is not sent to the server we wrote this rule.
* doc/wget.texi: Added --disable-header documentation.
* fuzz/wget_options_fuzzer.dict: Update with --disable-header inputs.
* src/http.c (disabled_header): Checks for disabled headers
(request_set_header): Doesn't let header to be set if disabled
(gethttp): frees disabled header to let overriding
* src/init.c (cmd_dis_header), (check_user_disabled_header) added new option disabled_headers.
* src/main.c: added new option --disable-header, added help description
* src/options.h: added new option --disable-header
* src/utils.h (vec_remove_header)
* src/utils.c (vec_remove_header) removes all header instances from vector
* testenv/Makefile.am: Added new test files
* testenv/server/http/http_server.py: Added new rule RejectHeaderField
* testenv/conf/reject_header_field.py: Added new rule RejectHeaderField
* testenv/README: Added help description for new rule
* testenv/Test-disable-default-headers.py: Test without using --header
* testenv/Test-disable-headers-after.py: Test using --header before --disable-header
* testenv/Test-disable-headers-before.py: Test using --header after --disable-header
Signed-off-by: sulfastor <torresmoisesa@gmail.com>, adham elkarn <adhamelkarn@hotmail.com>
In Fedora, we are implementing crypto policies, in order to enhance the
security of user systems. This is done on the system level by global
configuration. It may happen that due to the active policy, only
TLSv1.2 or higher will be available in crypto libraries. While wget as
a client will by default determine the minimal TLS version supported by
both client and server, the HTTPS server implementation in testenv/
hardcodes use of TLSv1. As a result all HTTPS related tests fail in
case a more hardened crypto policy is set on the Fedora system.
This change removes the explicit TLS version setting and leaves the
determination of the minimal supported TLS version on the server and
client.
More information about Fedora change can be found here:
https://fedoraproject.org/wiki/Changes/StrongCryptoSettings
Signed-off-by: Tomas Hozza <thozza@redhat.com>
* testenv/server/http/http_server.py: If there are multiple requests in
which the requested range is unsatisfiable, then send a body in the in
the 2nd response onwards
* testenv/Test-416.py: New test to check how Wget handles 416 responses
* testenv/conf/expected_files.py: Ignore dirmngr.conf, gpg.conf
Libgpgme creates certain files if $HOME doesn't contain the .gnupg directory.
These files disturb some metalink tests if we don't ignore them.
Reported-by: Arkadiusz Miśkiewicz <arekm@maven.pl>
Although internally code uses option for (not) reading .netrc for
credentials, it was not possible to turn this behavior off on command
line. Note that it was possible to turn it off using wgetrc.
Idea for this change came from Bruce Jerrick (bmj001@gmail.com).
Reference: https://bugzilla.redhat.com/show_bug.cgi?id=1425097
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Getting credentials from .netrc has been broken from time to time, thus
adding a test coverage to prevent regressions.
Also added setting of "HOME" environment variable when executing wget,
to make sure LocalFiles like .netrc, which are created just for the
test, are actually used.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
* testenv/certs/README: Amend cert creation extensions
* testenv/certs/ca-cert.pem: Created without OCSP signing purpose
Having the OCSP signing purpose set made newer versions of OpenSSL
fail due to stricter checking. Test version of OpenSSL was 1.1.0e.
* src/http.c (gethttp): Move 504 handling to correct place.
(http_loop): Fix memeory leak.
* testenv/server/http/http_server.py: Add Content-Length header on non-2xx
status codes with a body
Reported-by: Adam Sampson
* src/recur.c (descend_redirect): Ignore WG_RR_LIST and WG_RR_REGEX
for redirections.
* testenv/Makefile.am: Add Test-recursive-redirect.py
* testenv/Test-recursive-redirect.py: New test
Test-recursive-redirect.py written by Dale R. Worley.
Reported-by: "Dale R. Worley" <worley@ariadne.com>
* testenv/misc/metalinkv3_xml.py: New Metalink/XML v3 python class
* testenv/Test-metalink-xml-abspath-trust.py: Update test to use the
new Metalink/XML v3 python class
* testenv/Test-metalink-xml-abspath.py: Update test to use the new
Metalink/XML v3 python class
* testenv/Test-metalink-xml-absprefix-trust.py: Update test to use the
new Metalink/XML v3 python class
* testenv/Test-metalink-xml-absprefix.py: Update test to use the new
Metalink/XML v3 python class
* testenv/Test-metalink-xml-continue.py: Update test to use the new
Metalink/XML v3 python class
* testenv/Test-metalink-xml-emptyprefix-trust.py: Update test to use
the new Metalink/XML v3 python class
* testenv/Test-metalink-xml-homepath-trust.py: Update test to use the
new Metalink/XML v3 python class
* testenv/Test-metalink-xml-homepath.py: Update test to use the new
Metalink/XML v3 python class
* testenv/Test-metalink-xml-homeprefix-trust.py: Update test to use
the new Metalink/XML v3 python class
* testenv/Test-metalink-xml-homeprefix.py: Update test to use the new
Metalink/XML v3 python class
* testenv/Test-metalink-xml-nohash.py: Update test to use the new
Metalink/XML v3 python class
* testenv/Test-metalink-xml-nourls.py: Update test to use the new
Metalink/XML v3 python class
* testenv/Test-metalink-xml-prefix-trust.py: Update test to use the
new Metalink/XML v3 python class
* testenv/Test-metalink-xml-prefix.py: Update test to use the new
Metalink/XML v3 python class
* testenv/Test-metalink-xml-relpath-trust.py: Update test to use the
new Metalink/XML v3 python class
* testenv/Test-metalink-xml-relpath.py: Update test to use the new
Metalink/XML v3 python class
* testenv/Test-metalink-xml-relprefix-trust.py: Update test to use the
new Metalink/XML v3 python class
* testenv/Test-metalink-xml-relprefix.py: Update test to use the new
Metalink/XML v3 python class
* testenv/Test-metalink-xml-size.py: Update test to use the new
Metalink/XML v3 python class
* testenv/Test-metalink-xml-trust.py: Update test to use the new
Metalink/XML v3 python class
* testenv/Test-metalink-xml.py: Update test to use the new
Metalink/XML v3 python class
* src/http.c (metalink_from_http): Process the Content-Type header.
Add an application/metalink4+xml URL as metalink metaurl. If the
option opt.content_disposition is true, the Content-Disposition's
filename is the metaurl's name
* doc/wget.texi: Update --content-disposition and --metalink-over-http
* doc/metalink-standard.txt: Update doc. Content-Type/Disposition
processing through --metalink-over-http. Update download naming
system about --trust-server-names and --content-disposition
* testenv/Makefile.am: Add new files
* testenv/Test-metalink-http-xml-type.py: New file. Metalink/HTTP
Content-Type/Disposition header automated Metalink/XML tests
* testenv/Test-metalink-http-xml-type-trust.py: New file. Metalink/HTTP
Content-Type/Disposition header with --trust-server-names automated
Metalink/XML tests
* testenv/Test-metalink-http-xml-type-content.py: New file. Metalink/HTTP
Content-Type/Disposition header with --content-disposition automated
Metalink/XML tests
* testenv/Test-metalink-http-xml-type-trust-content.py: New file.
Metalink/HTTP Content-Type/Disposition header with --trust-server-names
and --content-disposition automated Metalink/XML tests
Process the Content-Type header, identify an application/metalink4+xml
file. The Content-Disposition could provide an alternate name through
the "filename" field for the metalink xml file. Respectively, the cli
options --metalink-over-http and --content-disposition are required.
When Metalink/XML auto-processing, to use the Content-Disposition's
filename, the cli option --trust-server-names is also required.
* src/metalink.c (retrieve_from_metalink): If opt.trustservernames is
true, use the basename of the metaurl's name to save the xml file
* doc/metalink-standard.txt: Update doc. With --trust-server-names any
Metalink/HTTP Link application/metalink4+xml file is saved using the
basename of the "name" field, if any. Update Metalink/HTTP examples
* testenv/Makefile.am: Add new file
* testenv/Test-metalink-http-xml-trust-name.py: New file. Metalink/HTTP
automated Metalink/XML, save xml files using the "name" field tests
* src/metalink.c (retrieve_from_metalink): Reject any metalink:file
without hashes. Prompt the error and switch to the next file
* testenv/Makefile.am: Add new file
* testenv/Test-metalink-xml-nohash.py: New file. Metalink/XML with no
hashes tests
Prevent SIGSEGV.
* src/http.c (metalink_from_http): Fix hash_bin_len type. Use ssize_t
instead than size_t. Reject -1 as base64_decode() return value
* testenv/Makefile.am: Add new file
* testenv/Test-metalink-http-baddigest.py: New file. Metalink/HTTP
malformed base64 Digest header tests
On malformed base64 input, ssize_t base64_decode() returns -1. Such
value is too big for a size_t variable, and used as xmalloc() value
will exaust all the memory.
* NEWS: Mention the effect of --metalink-index over Metalink
* src/init.c: Add new option metalinkindex (opt.metalink_index),
initialize to -1
* src/main.c: Add new option metalink-index (--metalink-index=NUMBER)
* src/options.h: Add new option metalink_index (int)
* src/metalink.h: Add declaration of functions fetch_metalink_file(),
replace_metalink_basename()
* src/metalink.c: Add functions fetch_metalink_file() simple file
fetch, replace_metalink_basename() replace file basename
* src/metalink.c (retrieve_from_metalink): New. Process Metalink
application/metalink4+xml of opt.metalink_index ordinal number
* doc/wget.texi: Add new option metalink-index (--metalink-index)
documentation
* doc/metalink-standard.txt: Updated doc. Add documentation about
Metalink application/metalink4+xml metaurls download naming system
* doc/metalink-standard.txt: Update Metalink/XML and HTTP examples
* testenv/Makefile.am: Add new files
* testenv/Test-metalink-http-xml.py: New file. Metalink/HTTP automated
Metalink/XML "application/metalink4+xml" --metalink-index tests
* testenv/Test-metalink-http-xml-trust.py: New file. Metalink/HTTP
automated Metalink/XML "application/metalink4+xml" --metalink-index
retrieval with --trust-server-names tests
WARNING: Do not use lib/dirname.c (dir_name) to get the directory
name, it may append a dot '.' character to the directory name.
* testenv/Makefile.am: Add new file
* testenv/Test-metalink-xml-emptyprefix-trust.py: New file.
Metalink/XML empty directory prefix (--directory-prefix '') tests
Detect a '/' character wrongfully concatenated to an empty directory
prefix '' (not NULL), resulting in an absolute path as '/dir/file',
instead than 'dir/file'.
* src/metalink.h: Add declaration of function dequote_metalink_string()
* src/metalink.c: Add function dequote_metalink_string() remove
surrounding quotes from string, \' or \"
* src/metalink.c (find_key_value, find_key_values): Call dequote_metalink_string()
to remove the surrounding quotes from the parsed value
* src/metalink.c (test_find_key_value, test_find_key_values): Add
quoted key's values for unit-tests
* testenv/Makefile.am: Add new file
* testenv/Test-metalink-http-quoted.py: New file. Metalink/HTTP quoted
values tests
Some Metalink/HTTP keys, like "type" [2], may have a quoted value [1]:
Link: <http://example.com/example.ext.meta4>; rel=describedby;
type="application/metalink4+xml"
Wget was expecting a dequoted value from the Metalink module. This
patch addresses this problem.
References:
[1] Metalink/HTTP: Mirrors and Hashes
1.1. Example Metalink Server Response
https://tools.ietf.org/html/rfc6249#section-1.1
[2] Additional Link Relations
6. "type"
https://tools.ietf.org/html/rfc6903#section-6
* src/metalink.h: Add declaration of function clean_metalink_string()
* src/metalink.c: Add directive #include "xmemdup0.h"
* src/metalink.c: Add function clean_metalink_string() remove leading
and trailing white spaces and CRLF from string
* src/metalink.c (retrieve_from_metalink): Remove leading and trailing
white spaces and CRLF from url resource mres->url
* testenv/Makefile.am: Add new file
* testenv/Test-metalink-xml-urlbreak.py: New test. Metalink/XML white
spaces and CRLF in url resources tests
White spaces and CRLF are not automatically removed by libmetalink
from url strings. The Wget's Metalink module was unable to process
such url strings. This patch implements the processing of such url
strings cleaning off leading and trailing white spaces and CRLF.
If a parsed Metalink/XML url string contains strings separated by
CRLF, only the first of the series is accepted.
* testenv/Makefile.am: Add new file
* testenv/Test-metalink-xml-nourls.py: New file. Metalink/XML unknown
urls tests
Test if when there are no good Metalink url resources there is any
segmentation fault.
* src/wget.h (uerr_t): Add error code METALINK_SIZE_ERROR to enum
* src/metalink.c (retrieve_from_metalink): Use boolean variable
size_ok, when false set retr_err to METALINK_SIZE_ERROR
* testenv/Makefile.am: Add new file
* testenv/Test-metalink-xml-size.py: New file. Metalink/XML file size
tests (<size></size>)
Before this patch, no appropriate error code was returned to inform a
file size mismatch.
This patch introduces the error code METALINK_SIZE_ERROR to inform a
file size mismatch.
* NEWS: Mention the effect of --trust-server-names over Metalink
* src/metalink.h: Add declaration of function append_suffix_number()
* src/metalink.c: Add function append_suffix_number() append number to
string
* src/metalink.c (retrieve_from_metalink): Safer Metalink/XML and
Metalink/HTTP download naming system, opt.trustservernames based
* doc/metalink-standard.txt: Update doc. Explain new Metalink/XML and
Metalin/HTTP download naming system and --trust-server-names role
* testenv/Makefile.am: Add new files
* testenv/Test-metalink-xml-continue.py: Update test. Metalink/XML
continue/keep existing files (HTTP 416) with --continue tests
* testenv/Test-metalink-xml.py: Update test. Metalink/XML naming tests
* testenv/Test-metalink-xml-trust.py: New file. Metalink/XML naming
tests with --trust-server-names
* testenv/Test-metalink-xml-abspath.py: Update test. Metalink/XML
absolute path tests
* testenv/Test-metalink-xml-abspath-trust.py: New file. Metalink/XML
absolute path tests with --trust-server-names
* testenv/Test-metalink-xml-relpath.py: Update test. Metalink/XML
relative path tests
* testenv/Test-metalink-xml-relpath-trust.py: New file. Metalink/XML
relative path tests with --trust-server-names
* testenv/Test-metalink-xml-homepath.py: Update test. Metalink/XML
home path and ~ (tilde) tests
* testenv/Test-metalink-xml-homepath-trust.py: New file. Metalink/XML
home path and ~ (tilde) tests with --trust-server-names
* testenv/Test-metalink-xml-prefix.py: New file. Metalink/XML naming
tests with --directory-prefix
* testenv/Test-metalink-xml-prefix-trust.py: New file. Metalink/XML
naming tests with --directory-prefix and --trust-server-names
* testenv/Test-metalink-xml-absprefix.py: New file. Metalink/XML
absolute --directory-prefix tests
* testenv/Test-metalink-xml-absprefix-trust.py: New file. Metalink/XML
absolute --directory-prefix tests with --trust-server-names
* testenv/Test-metalink-xml-relprefix.py: New file. Metalink/XML
relative --directory-prefix tests
* testenv/Test-metalink-xml-relprefix-trust.py: New file. Metalink/XML
relative --directory-prefix tests with --trust-server-names
* testenv/Test-metalink-xml-homeprefix.py: New file. Metalink/XML home
--directory-prefix tests
* testenv/Test-metalink-xml-homeprefix-trust.py: New file. Metalink/XML
home --directory-prefix tests with --trust-server-names
The option --trust-server-names allows to use the file names parsed
from a Metalink/XML file. Without --trust-server-names, the safety
mechanism provides secure and predictable file names.
* testenv/Test-metalink-xml-relpath.py: Update test
* testenv/Test-metalink-xml-homepath.py: New file. Reject home paths
* testenv/Makefile.am: Add new file
When --input-metalink=<file> is used, each metalink:file name is
verified by libmetalink's metalink_check_safe_path(). By design,
absolute, relative and home paths are rejected.
At the moment of writing, when --metalink-over-http is used, absolute,
relative, and home paths aren't a concern. The destination file name
is a combination of URL's file name and cli's "Directory Options"
handled by src/url.c (url_file_name).
* NEWS: Mention the effects of --continue over Metalink
* src/metalink.c (retrieve_from_metalink): On download error, resume
output_stream with the next mres->url. Keep fully downloaded files
started with --continue, otherwise rename/remove the file
* testenv/Makefile.am: Add new file
* testenv/Test-metalink-xml-continue.py: New file. Metalink/XML
continue/keep existing files (HTTP 416) with --continue tests
Before this patch, with --continue, existing and/or fully retrieved
files which fail the sanity tests were renamed (--keep-badhash), or
removed.
This patch ensures that --continue doesn't rename/remove existing
and/or fully retrieved files (HTTP 416) which fail the sanity tests.
* testenv/Test-metalink-http.py: Use python .replace
* testenv/Test-metalink-xml.py: Use python .replace
* testenv/Test-metalink-xml-abspath.py: Use python .replace
* testenv/Test-metalink-xml-relpath.py: Use python .replace
Use python .replace instead than re.sub, remove 'import re'.
* testenv/Test-metalink-xml-abspath.py: Change Metalink/XML v3 file
name from test.meta4 into test.metalink
* testenv/Test-metalink-xml-relpath.py: Change Metalink/XML v3 file
name from test.meta4 into test.metalink
* testenv/Test-metalink-xml.py: Change Metalink/XML v3 file name from
test.meta4 into test.metalink
* testenv/Test-recursive-basic.py: New file. Test basic recursion
* testenv/Test-recursive-include.py: New File. Recursion test with
include directories
* testenv/Test-redirect.py: New File. Basic redirection tests
* testenv/Makefile.am: Add new tests to makefile
* Test-pinnedpubkey-hash-https.py: Read hashed pubkey from file
* Test-pinnedpubkey-hash-no-check-fail-https.py: Use invalid hash
* certs/make_ca.sh: New script to generate test certs non-interactive
* certs/ca-template.cfg: New file (template for CA cert)
* certs/server-template.cfg: New file (template for server cert)
* certs/server-pubkey-sha256.base64: New file (pubkey sha256 hash)
* testenv/Makefile.am: Add Test-missing-scheme-retval.py
* testenv: New file Test-missing-scheme-retval.py
Idea and Perl implementation by Zdenek Dohnal <zdohnal@redhat.com>
* testenv/Makefile.am: Add new tests
* testenv/Test-pinnedpubkey-der-https.py: New test
* testenv/Test-pinnedpubkey-der-no-check-https.py: New Test
* testenv/Test-pinnedpubkey-hash-https.py: New test
* testenv/Test-pinnedpubkey-hash-no-check-fail-https.py: New test
* testenv/Test-pinnedpubkey-pem-fail-https.py: New test
* testenv/Test-pinnedpubkey-pem-https.py: New test
* testenv/certs/README: How to generate public keys with openssl tool
* testenv/certs/server-pubkey.der: New key file (DER format)
* testenv/certs/server-pubkey.pem: New key file (PEM format)
* testenv/test/base_test.py (__init__): Use test file name for name,
remove 'name' parameter
* testenv/test/http_test.py (__init__): Remove 'name' parameter
* testenv/*.py: Remove TEST_NAME
Using a fixed string (TEST_NAME) to build the working directory
for testing caused random failures (or successes) when tests
share the same TEST_NAME value. Not easy to find without digging
into the python test suite code.
We now use the test file name, which is unique within the test
environment.
