* testenv/Test-recursive-basic.py: New file. Test basic recursion
* testenv/Test-recursive-include.py: New File. Recursion test with
include directories
* testenv/Test-redirect.py: New File. Basic redirection tests
* testenv/Makefile.am: Add new tests to makefile
* configure.ac: Add --with-psl-file to set a PSL file
* src/cookies.c (check_domain_match): Load PSL_FILE with
fallback to built-in data.
This change allows package maintainers to make Wget use the latest
PSL (DAFSA or plain text), without updating libpsl itself.
E.g. Debian now comes with a DAFSA binary within the 'publicsuffix'
package which allows very fast loading (no parsing or processing needed).
* src/cookies.c (cookie_header): Use heap instead of stack.
* src/http.c (request_send): Likewise.
If wget has to handle an insanely large amount of cookies (~700,000 on
32 bit systems or ~530,000 on 64 bit systems), the stack is not large
enough to hold these pointers, leading to undefined behaviour according
to POSIX; expect a segmentation fault in real life. ;)
Signed-off-by: Tobias Stoeckmann <tobias@stoeckmann.org>
The signal handler for SIGALRM calls longjmp, but the handler is
installed before the jump target has been initialized. If another
process sends SIGALRM right between handler installation and target
initialization, the jump leads to undefined behavior.
This can easily be fixed by moving the signal handler installation
into the "SETJMP == 0" conditional block, which means that the target
has just been initialized.
* src/utils.c: call signal after SETJMP.
Signed-off-by: Tobias Stoeckmann <tobias@stoeckmann.org>
* src/init.c: Remove hyphens from command names
* src/main.c: Likewise
Options with hyphens (or underscores) in their command name cannot be
set in a wgetrc file.
Signed-off-by: Jeffery To <jeffery.to@gmail.com>
* src/metalink.c (retrieve_from_metalink): Continue file download if
opt.always_rest is true
Without --continue, download as a new file with an unique name (this
conforms to the old behaviour).
* bootstrap.conf: Add crypto/md2, and crypto/md4
* src/metalink.c (retrieve_from_metalink): Add md2, and md4 support
This patch adds support for the deprecated (insecure) md2, and md4
Message-Digest algorithms to the Metalink module.
* bootstrap.conf: Add crypto/sha512
* src/metalink.c (retrieve_from_metalink): Add md5, sha1, sha224,
sha384, and sha512 support
Metalink's checksum verification was limited to sha256. This patch
adds support for md5, sha1, sha224, sha384, and sha512.
* configure.ac: Check for xattr availability
* src/Makefile.am: Add xattr.c
* src/ftp.c: Include xattr.h.
(getftp): Set attributes if enabled.
* src/http.c: Include xattr.h.
(gethttp): Add parameter 'original_url',
set attributes if enabled.
(http_loop): Add 'original_url' to call of gethttp().
* src/init.c: Add new option --xattr.
* src/main.c: Add new option --xattr, add description to help text.
* src/options.h: Add new config member 'enable_xattr'.
* src/xatrr.c: New file.
* src/xattr.h: New file.
These attributes provide a lightweight method of later determining
where a file was downloaded from.
This patch changes:
* autoconf detects whether extended attributes are available and
enables the code if they are.
* The new flags --xattr and --no-xattr control whether xattr is enabled.
* The new command "xattr = (on|off)" can be used in ~/.wgetrc or /etc/wgetrc
* The original and redirected URLs are recorded as shown below.
* This works for both single fetches and recursive mode.
The attributes that are set are:
user.xdg.origin.url: The URL that the content was fetched from.
user.xdg.referrer.url: The URL that was originally requested.
Here is an example, where http://archive.org redirects to https://archive.org:
$ wget --xattr http://archive.org
...
$ getfattr -d index.html
user.xdg.origin.url="https://archive.org/"
user.xdg.referrer.url="http://archive.org/"
These attributes were chosen based on those stored by Google Chrome
https://bugs.chromium.org/p/chromium/issues/detail?id=45903
and curl https://github.com/curl/curl/blob/master/src/tool_xattr.c
* src/openssl.c (ssl_init): Use SSL_is_init_finished() instead of
SSL_state(), conditionally skip SSLeay function calls
The python test suite makes SSL_peek() hang, consuming 100% CPU time.
This does not happen on real world TLS connections, though, but needs
investigations.
* Test-pinnedpubkey-hash-https.py: Read hashed pubkey from file
* Test-pinnedpubkey-hash-no-check-fail-https.py: Use invalid hash
* certs/make_ca.sh: New script to generate test certs non-interactive
* certs/ca-template.cfg: New file (template for CA cert)
* certs/server-template.cfg: New file (template for server cert)
* certs/server-pubkey-sha256.base64: New file (pubkey sha256 hash)
* src/hsts.c (hsts_file_access_valid): we should check for "world-writable"
files only on Unix-based systems. It's difficult to mimic the same behavior
on Windows, so it's better to just not do it.
Reported-by: Gisle Vanem <gvanem@yahoo.no>
Reported-by: Eli Zaretskii <eliz@gnu.org>
If not --trust-server-names is used, FTP will also get the destination
file name from the original url specified by the user instead of the
redirected url. Closes CVE-2016-4971.
* src/ftp.c (ftp_get_listing): Add argument original_url.
(getftp): Likewise.
(ftp_loop_internal): Likewise. Use original_url to generate the
file name if --trust-server-names is not provided.
(ftp_retrieve_glob): Likewise.
(ftp_loop): Likewise.
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
* testenv/Makefile.am: Add Test-missing-scheme-retval.py
* testenv: New file Test-missing-scheme-retval.py
Idea and Perl implementation by Zdenek Dohnal <zdohnal@redhat.com>
* src/main.c (save_hsts): save the in-memory HSTS database to a file
only if something changed.
* src/hsts.c (struct hsts_store): new field 'changed'.
(hsts_match): update field 'changed' accordingly.
(hsts_store_entry): update field 'changed' accordingly.
(hsts_store_has_changed): new function.
* src/hsts.h (hsts_store_has_changed): new function.
* hsts.c (hsts_file_access_valid): check that the file is a regular
file, and that it's not world-writable.
(hsts_store_open): if the HSTS database file does not meet the
above requirements, disable HSTS at all.
* src/hsts.c (hsts_store_entry): strictly comply with RFC 6797.
RFC 6797 states in section 8.1 that the UA's cached information should
only be updated if:
"either or both of the max-age and includeSubDomains header field
value tokens are conveying information different than that already
maintained by the UA."
