Commit Graph

3581 Commits

Author SHA1 Message Date
Tobias Stoeckmann
a9d49e5b15 Fix signal race condition
The signal handler for SIGALRM calls longjmp, but the handler is
installed before the jump target has been initialized. If another
process sends SIGALRM right between handler installation and target
initialization, the jump leads to undefined behavior.

This can easily be fixed by moving the signal handler installation
into the "SETJMP == 0" conditional block, which means that the target
has just been initialized.

* src/utils.c: call signal after SETJMP.

Signed-off-by: Tobias Stoeckmann <tobias@stoeckmann.org>
2016-08-09 17:38:29 +02:00
Jeffery To
0fe79eeacb Remove hyphens from command names
* src/init.c: Remove hyphens from command names
* src/main.c: Likewise

Options with hyphens (or underscores) in their command name cannot be
set in a wgetrc file.

Signed-off-by: Jeffery To <jeffery.to@gmail.com>
2016-08-05 09:45:09 +02:00
Tim Rühsen
e3fb4c3859 * src/metalink.c (badhash_suffix): Fix quoting 2016-08-04 13:09:28 +02:00
Matthew White
943a6d585f Add new option --keep-badhash to keep Metalink's files with a bad hash
* src/init.c: Add keepbadhash
* src/main.c: Add keep-badhash
* src/options.h: Add keep_badhash
* doc/wget.texi: Add docs for --keep-badhash
* src/metalink.h: Add prototypes badhash_suffix(), badhash_or_remove()
* src/metalink.c: New functions badhash_suffix(), badhash_or_remove().
  (retrieve_from_metalink): Call badhash_or_remove() on download error

With --keep-badhash, append .badhash to Metalink's files with checksum
mismatch. (retrieve_from_metalink): unique_create() may append another
suffix to avoid overwriting existing files.

Without --keep-badhash, remove downloaded files with checksum mismatch
(this conforms to the old behaviour).
2016-08-04 12:03:49 +02:00
Tim Rühsen
7fad76db4c * src/metalink.c: Remove C++ style comments 2016-08-03 13:48:07 +02:00
Matthew White
3e7e29f358 Add gnulib modules 'link', 'unlink' and 'symlink'
* bootstrap.conf: Add 'link', 'unlink' and 'symlink'
2016-08-03 13:44:34 +02:00
Matthew White
e0b60fd073 New: --continue continues partially downloaded Metalink's files
* src/metalink.c (retrieve_from_metalink): Continue file download if
  opt.always_rest is true

Without --continue, download as a new file with an unique name (this
conforms to the old behaviour).
2016-08-03 13:37:27 +02:00
Matthew White
9db02a0c46 Add support for Metalink's md2, and md4 hashes
* bootstrap.conf: Add crypto/md2, and crypto/md4
* src/metalink.c (retrieve_from_metalink): Add md2, and md4 support

This patch adds support for the deprecated (insecure) md2, and md4
Message-Digest algorithms to the Metalink module.
2016-08-03 12:58:43 +02:00
Matthew White
edad3c1df3 Add support for Metalink's md5, sha1, sha224, sha384, and sha512 hashes
* bootstrap.conf: Add crypto/sha512
* src/metalink.c (retrieve_from_metalink): Add md5, sha1, sha224,
  sha384, and sha512 support

Metalink's checksum verification was limited to sha256. This patch
adds support for md5, sha1, sha224, sha384, and sha512.
2016-08-03 12:49:26 +02:00
Sean Burford
20cac2c5ab Style fixes and DEBUG on setxattr failure.
* src/ftp.c: Fix style.
* src/http.c: Likewise.
* src/xattr.h: Likewise.
* src/xattr.c: Likewise,
  (write_xattr_metadata): Print debug msg on error.
2016-07-27 17:05:57 +02:00
Sean Burford
a933bdd31e Keep fetched URLs in POSIX extended attributes
* configure.ac: Check for xattr availability
* src/Makefile.am: Add xattr.c
* src/ftp.c: Include xattr.h.
  (getftp): Set attributes if enabled.
* src/http.c: Include xattr.h.
  (gethttp): Add parameter 'original_url',
  set attributes if enabled.
  (http_loop): Add 'original_url' to call of gethttp().
* src/init.c: Add new option --xattr.
* src/main.c: Add new option --xattr, add description to help text.
* src/options.h: Add new config member 'enable_xattr'.
* src/xatrr.c: New file.
* src/xattr.h: New file.

These attributes provide a lightweight method of later determining
where a file was downloaded from.

This patch changes:
*   autoconf detects whether extended attributes are available and
    enables the code if they are.
*   The new flags --xattr and --no-xattr control whether xattr is enabled.
*   The new command "xattr = (on|off)" can be used in ~/.wgetrc or /etc/wgetrc
*   The original and redirected URLs are recorded as shown below.
*   This works for both single fetches and recursive mode.

The attributes that are set are:
user.xdg.origin.url: The URL that the content was fetched from.
user.xdg.referrer.url: The URL that was originally requested.

Here is an example, where http://archive.org redirects to https://archive.org:
$ wget --xattr http://archive.org
...
$ getfattr -d index.html
user.xdg.origin.url="https://archive.org/"
user.xdg.referrer.url="http://archive.org/"

These attributes were chosen based on those stored by Google Chrome
https://bugs.chromium.org/p/chromium/issues/detail?id=45903
and curl https://github.com/curl/curl/blob/master/src/tool_xattr.c
2016-07-22 13:42:23 +02:00
Noël Köthe
ef372a4f27 Fix typos
* ChangeLog-2014-12-10: invokation -> invocation
* doc/wget.texi: invokation -> invocation
* src/main.c: seperated -> separated
* src/options.h: seperated -> separated
* testenv/README: invokation -> invocation
* testenv/conf/wget_commands.py: invokation -> invocation
2016-07-02 19:01:24 +02:00
Tim Rühsen
0b151f51eb Fix creating docs when make uses 'sh -e'
* doc/Makefile.am: Save fallback for pod2man --utf8

Reported-by: Jérémie Courrèges-Anglas <jca@wxcvbn.org>
2016-06-30 15:19:26 +02:00
Tim Rühsen
309e72c74f Fix compilation for OpenSSL 1.1.0
* src/openssl.c (ssl_init): Use SSL_is_init_finished() instead of
  SSL_state(), conditionally skip SSLeay function calls

The python test suite makes SSL_peek() hang, consuming 100% CPU time.
This does not happen on real world TLS connections, though, but needs
investigations.
2016-06-30 13:24:33 +02:00
Tim Rühsen
2318c309d4 Add script to generate test certs non-interactive
* Test-pinnedpubkey-hash-https.py: Read hashed pubkey from file
* Test-pinnedpubkey-hash-no-check-fail-https.py: Use invalid hash
* certs/make_ca.sh: New script to generate test certs non-interactive
* certs/ca-template.cfg: New file (template for CA cert)
* certs/server-template.cfg: New file (template for server cert)
* certs/server-pubkey-sha256.base64: New file (pubkey sha256 hash)
2016-06-29 12:54:06 +02:00
Ander Juaristi
cdc3e28d8e Bypass world-writable checks on Windows
* src/hsts.c (hsts_file_access_valid): we should check for "world-writable"
   files only on Unix-based systems. It's difficult to mimic the same behavior
   on Windows, so it's better to just not do it.

Reported-by: Gisle Vanem <gvanem@yahoo.no>
Reported-by: Eli Zaretskii <eliz@gnu.org>
2016-06-27 09:54:32 +02:00
Tim Rühsen
43359f47c4 Update gnulib and bootstrap
* gnulib: Sync gnulib submodule with upstream
* bootstrap: Update to latest version from gnulib/build-aux/
2016-06-14 09:27:58 +02:00
Tim Rühsen
e1e7afb210 Use ICONV_CONST to avoid type warning for iconv()
* src/iri.c (do_conversion): Cast 2. param of iconv() to
 'ICONV_CONST char **'
* src/url.c (convert_fname): Likewise
2016-06-12 21:51:34 +02:00
Tim Rühsen
7e585fe23d Remove check for HAVE_ICONV in src/url.c
* src/url.c: Remove check for HAVE_ICONV
2016-06-12 21:49:23 +02:00
Tim Rühsen
526e98f7c4 Add gnulib module 'langinfo'
* bootstrap.conf: Add 'langinfo'
2016-06-12 20:55:03 +02:00
Tim Rühsen
d75f43f083 Include gnulib fcntl.h instead of sys/fcntl.h
* src/gnutls.c: Include gnulib fcntl.h
2016-06-12 17:06:31 +02:00
Tim Rühsen
d4f97dc9af Add libraries to LDADD for wget
* src/Makefile.am: Add $(GETADDRINFO_LIB) $(HOSTENT_LIB) $(INET_NTOP_LIB)
 $(LIBSOCKET) $(LIB_CLOCK_GETTIME) $(LIB_CRYPTO) $(LIB_SELECT)
 $(LTLIBICONV) $(LTLIBINTL) $(LTLIBTHREAD) $(SERVENT_LIB) to LDADD
2016-06-12 17:02:12 +02:00
Tim Rühsen
ade62eb400 Explicitely mention Python3 in README.checkout
* README.checkout: Python test suite needs Python3
2016-06-10 15:14:43 +02:00
Giuseppe Scrivano
bd7f36705b NEWS: update 2016-06-09 17:31:57 +02:00
Giuseppe Scrivano
e996e322ff ftp: understand --trust-server-names on a HTTP->FTP redirect
If not --trust-server-names is used, FTP will also get the destination
file name from the original url specified by the user instead of the
redirected url.  Closes CVE-2016-4971.

* src/ftp.c (ftp_get_listing): Add argument original_url.
(getftp): Likewise.
(ftp_loop_internal): Likewise.  Use original_url to generate the
file name if --trust-server-names is not provided.
(ftp_retrieve_glob): Likewise.
(ftp_loop): Likewise.

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2016-06-09 15:02:49 +02:00
Tim Rühsen
2bdfc4f521 Fix warnings for --disable-iri configure flag
* src/iri.h: Fix #define for parse_charset
* src/html-url.c: Surround some IRI code parts by #ifdef ENABLE_IRI
* src/http.c: Likewise
* src/iri.h: Likewise
* src/recur.c: Likewise
* src/retr.c: Likewise
2016-06-07 12:52:59 +02:00
Tim Rühsen
2c736abb4c Fix warning about redefinition of MAP_FAILED
* src/sysdep.h: Removed definition of MAP_FAILED
* src/utils.c: Check and define MAP_FAILED after including sys/mmap.h
2016-06-07 09:56:01 +02:00
Tim Rühsen
7c0752c4cb Add new Test for missing scheme behavior
* testenv/Makefile.am: Add Test-missing-scheme-retval.py
* testenv: New file Test-missing-scheme-retval.py

Idea and Perl implementation by Zdenek Dohnal <zdohnal@redhat.com>
2016-06-03 10:16:40 +02:00
Tim Rühsen
302aa79abb Fallback to pod2man without utf-8 on error
* doc/Makefile.am: Fallback to pod2man without utf-8 on error
2016-05-27 16:44:50 +02:00
Ander Juaristi
5224d752a5 Correct HSTS debug message
* src/main.c (save_hsts): save the in-memory HSTS database to a file
   only if something changed.
 * src/hsts.c (struct hsts_store): new field 'changed'.
   (hsts_match): update field 'changed' accordingly.
   (hsts_store_entry): update field 'changed' accordingly.
   (hsts_store_has_changed): new function.
 * src/hsts.h (hsts_store_has_changed): new function.
2016-05-26 16:37:51 +02:00
Ander Juaristi
2aaf12990c Check the HSTS file is not world-writable
* hsts.c (hsts_file_access_valid): check that the file is a regular
   file, and that it's not world-writable.
   (hsts_store_open): if the HSTS database file does not meet the
   above requirements, disable HSTS at all.
2016-05-26 16:29:29 +02:00
Tim Rühsen
2e6f629061 Improve description of Perl libraries needed for test suite
* README.checkout: Improve text about Perl libraries
2016-05-24 09:38:15 +02:00
Tim Rühsen
a952f81f3e Remove special handling for Emacs in progress bar code
* src/progress.c: Remove special 'emacs' code

Fixes #47989
2016-05-23 21:46:29 +02:00
Jernej Simončič
42cc84b6b6 Fix xsleep() for Windows (trivial change)
* src/mswindows.c (xsleep): Fix check for number of seconds
2016-04-25 15:50:23 +02:00
Sergio Gelato
96ab9cad88 More accurate log message from do_conversion()
* src/iri.c (do_conversion): More accurate log message
2016-04-17 15:28:48 +02:00
Tim Rühsen
268163444d Include sys/select.h if HAVE_LIBCARES
* src/hosts.c: Include sys/select.h if HAVE_LIBCARES

Reported-by: Gisle Vanem <gvanem@yahoo.no>
2016-04-17 14:18:55 +02:00
Gisle Vanem
53800415a9 Fix Windows gnulib/c-ares incompatibility of select()
* src/host.c: Undef 'select' on Windows
2016-04-17 14:15:51 +02:00
Tim Rühsen
528770ca8b Set X flags for python tests
* testenv/*.py: Set eXecute flags

Regression from commit 926e42d467
2016-04-15 21:36:51 +02:00
Ander Juaristi
2f1c6a05c8 Strictly comply with RFC 6797
* src/hsts.c (hsts_store_entry): strictly comply with RFC 6797.

RFC 6797 states in section 8.1 that the UA's cached information should
only be updated if:

    "either or both of the max-age and includeSubDomains header field
    value tokens are conveying information different than that already
    maintained by the UA."
2016-04-11 16:44:47 +02:00
Ander Juaristi
33d860e1ef Correct HSTS database file description
* src/hsts.c (hsts_store_dump): s/[:port]/<port>/
2016-04-11 16:44:41 +02:00
moparisthebest
99fc712f21 Implement tests for new pinnedpubkey option
* testenv/Makefile.am: Add new tests
* testenv/Test-pinnedpubkey-der-https.py: New test
* testenv/Test-pinnedpubkey-der-no-check-https.py: New Test
* testenv/Test-pinnedpubkey-hash-https.py: New test
* testenv/Test-pinnedpubkey-hash-no-check-fail-https.py: New test
* testenv/Test-pinnedpubkey-pem-fail-https.py: New test
* testenv/Test-pinnedpubkey-pem-https.py: New test
* testenv/certs/README: How to generate public keys with openssl tool
* testenv/certs/server-pubkey.der: New key file (DER format)
* testenv/certs/server-pubkey.pem: New key file (PEM format)
2016-04-11 16:25:09 +02:00
moparisthebest
54746578e9 Implement --pinnedpubkey option to pin public keys
* doc/wget.texi: Add description for --pinnedpubkey
* src/gnutls.c: New function pkp_pin_peer_pubkey(),
  (ssl_check_certificate): Check pinned cert via pkp_pin_peer_pubkey()
* src/init.c: Add option --pinnedpubkey
* src/main.c: Add option --pinnedpubkey
* src/openssl.c: New function pkp_pin_peer_pubkey(),
  (ssl_check_certificate): Check pinned cert via pkp_pin_peer_pubkey()
* src/options.h: Add new option variable 'pinnedpubkey'
* src/utils.c: New functions wg_pubkey_pem_to_der(), wg_pin_peer_pubkey()
* src/utils.h: Add prototype for wg_pin_peer_pubkey()
2016-04-11 16:18:05 +02:00
Tim Rühsen
926e42d467 Use test file name for temp working directory name
* testenv/test/base_test.py (__init__): Use test file name for name,
  remove 'name' parameter
* testenv/test/http_test.py (__init__): Remove 'name' parameter
* testenv/*.py: Remove TEST_NAME

Using a fixed string (TEST_NAME) to build the working directory
for testing caused random failures (or successes) when tests
share the same TEST_NAME value. Not easy to find without digging
into the python test suite code.
We now use the test file name, which is unique within the test
environment.
2016-04-11 15:35:05 +02:00
Tim Rühsen
d286d5408b Fix testenv/Test--rejected-log.py
* testenv/Test--rejected-log.py: Add missing tabs in expected output
2016-04-11 15:24:04 +02:00
Tim Rühsen
9883f9fb10 Fail python tests when post_hook errors
* testenv/test/base_test.py (__exit__): Return self.tests_passed
  (__test_cleanup): Set self.tests_passed to False on exception
2016-04-11 15:23:38 +02:00
Darshit Shah
d26377053d Print the fingerprint instead of the raw pointer in debugging message
* src/metalink.c (retrieve_from_metalink): Fix debug message to print the
fingerprint instead of a pointer.

* Do not delete the ChangeLog file since it is required by the Makefile
and breaks compilation
2016-03-29 15:09:04 +02:00
Darshit Shah
722675553c Revert "Print the fingerprint instead of the raw pointer in debugging message"
This reverts commit b916595168.
2016-03-29 15:07:29 +02:00
Tim Rühsen
281ad7dfb9 Fixed URLs and references in wget.texi
* wget.texi: Replace server.com by example.com,
  replace ftp://wuarchive.wustl.edu by https://example.com,
  use HTTPS instead of HTTP where possible,
  fix list archive reference,
  remove reference to wget-notify@addictivecode.org,
  change bugtracker URL to bugtracker on Savannah,
  replace yoyodyne.com by example.com,
  fix URL to VMS port
2016-03-28 18:33:33 +02:00
Giuseppe Scrivano
f3e63f0071 * metalink.c (retrieve_from_metalink): Fix typo 2016-03-25 16:46:39 +01:00
Giuseppe Scrivano
b916595168 Print the fingerprint instead of the raw pointer in debugging message
* src/metalink.c (retrieve_from_metalink): Fix debug message to print the
fingerprint instead of a pointer.
2016-03-25 16:23:19 +01:00