Commit Graph

171 Commits

Author SHA1 Message Date
Sean Burford
a933bdd31e Keep fetched URLs in POSIX extended attributes
* configure.ac: Check for xattr availability
* src/Makefile.am: Add xattr.c
* src/ftp.c: Include xattr.h.
  (getftp): Set attributes if enabled.
* src/http.c: Include xattr.h.
  (gethttp): Add parameter 'original_url',
  set attributes if enabled.
  (http_loop): Add 'original_url' to call of gethttp().
* src/init.c: Add new option --xattr.
* src/main.c: Add new option --xattr, add description to help text.
* src/options.h: Add new config member 'enable_xattr'.
* src/xatrr.c: New file.
* src/xattr.h: New file.

These attributes provide a lightweight method of later determining
where a file was downloaded from.

This patch changes:
*   autoconf detects whether extended attributes are available and
    enables the code if they are.
*   The new flags --xattr and --no-xattr control whether xattr is enabled.
*   The new command "xattr = (on|off)" can be used in ~/.wgetrc or /etc/wgetrc
*   The original and redirected URLs are recorded as shown below.
*   This works for both single fetches and recursive mode.

The attributes that are set are:
user.xdg.origin.url: The URL that the content was fetched from.
user.xdg.referrer.url: The URL that was originally requested.

Here is an example, where http://archive.org redirects to https://archive.org:
$ wget --xattr http://archive.org
...
$ getfattr -d index.html
user.xdg.origin.url="https://archive.org/"
user.xdg.referrer.url="http://archive.org/"

These attributes were chosen based on those stored by Google Chrome
https://bugs.chromium.org/p/chromium/issues/detail?id=45903
and curl https://github.com/curl/curl/blob/master/src/tool_xattr.c
2016-07-22 13:42:23 +02:00
Tim Rühsen
76ef65b23c Add options --bind-dns-address and --dns-servers
* README.checkout: Add description for libares
* configure.ac: Add check for libares
* doc/wget.texi: Add docs for the new options
* src/build_info.c.in: Add +/-cares for --version output
* src/host.c:
  (merge_address_lists): New static function
  (address_list_from_hostent): New static function
  (wait_ares): New static function
  (callback): New static function
  (lookup_host): Add libares resolver code
* src/init.c: Add new options,
  (cleanup): Add cleanup code
* src/main.c: Add global libares channel variable
  (cmdline_option option_data): Add new options
  (print_help): Add short descriptions
  (main): Add libares init code
* src/options.h (struct options): Add option members

The new options allow to specify alternative DNS servers and
an alternate packet route for the resolver packets.
Wget has to built with libares, enabled at configure time by
./configure --with-cares.
2016-03-23 09:26:22 +01:00
Tim Rühsen
c190677cab Set AM_SILENT_RULES to yes by default
* configure.ac: Set AM_SILENT_RULES to yes by default
2016-02-01 21:10:33 +01:00
Giuseppe Scrivano
b1c7777037 * configure.ac: change gettext version to 0.18.1 2015-11-20 11:58:42 +01:00
Ikey Doherty
8c12260d4b configure.ac: Use correct gettext version
Signed-off-by: Ikey Doherty <michael.i.doherty@intel.com>
2015-11-19 23:14:07 +01:00
Tim Rühsen
84c405e690 Let bootstrap/autoreconf work without GPGME installed
* configure.ac: Check for existance of AM_PATH_GPGME
2015-08-14 10:17:52 +02:00
Darshit Shah
595f219a17 Fix configure options for metalink
* configure.ac: Ensure metalink support can be properly disabled
2015-07-24 23:42:20 +05:30
Hubert Tarasiuk
37b58e3976 Metalink support.
* bootstrap.conf: Add crypto/sha256
* configure.ac: Look for libmetalink and GPGME
* doc/wget.texi: Add --input-metalink and --metalink-over-http
options description.
* po/POTFILES.in: Add metalink.c
* src/Makefile.am: Add new translation unit (metalink.c)
* src/http.c (http_stat): Add metalink field.
(free_stat): Free metalink field.
(find_key_value): Find value of given key in header string.
(has_key): Check if token exists in header string.
(find_key_values): Find all key=value pairs in header string.
(metalink_from_http): Obtain Metalink metadata from HTTP response.
(gethttp): Call metalink_from_http if requested.
(http_loop): Request Metalink metadata from HTTP response if should be.
Fall back to regular download if no Metalink metadata found.
* src/init.c: Add --input-metalink and --metalink-over-http options
* src/main.c (option_data): Handle --input-metalink and
--metalink-over-http cmd arguments.
(print_help): Print --input-metalink option description.
(main): Retrieve files from Metalink file
* src/metalink.c (retrieve_from_metalink): Download files described by
metalink.
(metalink_res_cmp): Comparator for resources priority-sorting.
* src/metalink.h: Create header for metalink.c
(RES_TYPE_SUPPORTED): Define supported resources media.
(DEFAULT_PRI): Default mirror priority for Metalink over HTTP.
(VALID_PRI_RANGE): Valid priority range.
* src/options.h (options): Add input_metalink option and metalink_over_http
options.
* src/utils.c (hex_to_string): Convert binary data to ASCII-hex.
* src/utils.h (hex_to_string): Add prototype.
* src/wget.h: Add metalink-related error enums
Add METALINK_METADATA flag for document type.
2015-07-20 15:30:39 +02:00
Tim Rühsen
bd0ffcf8bc Let HTTPS tests XFAIL when no TLS support configured
* configure.ac: Export WITH_SSL for use in Makefile.am
* testenv/Makefile.am: Add HTTPS tests to XFAIL_TESTS when !WITH_SSL

Reported-by: Ander Juaristi <ajuaristi@gmx.es>
2015-07-14 07:54:03 +02:00
Tim Ruehsen
6f62bc5cd8 configure.ac: Skip pkg-config for opensl and gnutls when prefix is given
Make --with-libssl-prefix and --with-libgnutls-prefix do the right thing,
no matter if pkg-config is installed or not.

Reported-by: Charles Diza <chdiza@gmail.com>
2014-12-25 15:21:44 +01:00
Tim Rühsen
05d73a926a configure.ac: Fix libuuid and uuid_create detection
--with-libuuid now explictly asks for libuuid.
--without-libuuid ignores libuuid and tries to use libc builtin functions.
Else try builtin functions first, libuuid second and fallback to Wget's own
function.
2014-12-18 10:42:26 +01:00
Tim Rühsen
0d9615292e configure.ac: Check for RAND_egd() in pkg-config success code 2014-12-17 12:39:00 +01:00
Jérémie Courrèges-Anglas
b5778699f0 openssl: Detect the availability of RAND_egd (tiny change)
Alternatives like LibreSSL don't provide RAND_egd() anymore.
Fixes compilation on OpenBSD.
2014-12-17 11:47:15 +01:00
Jérémie Courrèges-Anglas
b8c567a3ef Fix use of uuid libc functions (tiny change)
* Always attempt to detect uuid.h and uuid_create().
* Split libuuid and uuid.h implementations of warc_uuid_str(), since
  those APIs vary significantly.
* Correctly use the uuid.h functions
2014-12-17 09:24:31 +01:00
Tim Rühsen
71425f1bb3 Check for pcre.h in configure.ac 2014-12-10 10:22:39 +01:00
Tim Rühsen
a80986a8c9 Fix message in configure.ac for when GnuTLS has not been found 2014-12-05 12:18:30 +01:00
Darshit Shah
841efb0108 Assertions are now disabled by default 2014-11-21 17:45:19 +05:30
Tim Ruehsen
b19d8032c2 Fix libpsl<0.6.0 detection in configure.ac 2014-11-20 21:08:21 +01:00
Darshit Shah
acc76b1e02 Use pkg-config for libpsl checking 2014-11-20 16:57:45 +05:30
Darshit Shah
e4be5e3b9a Autotoolize configure.ac 2014-11-20 16:49:28 +05:30
Darshit Shah
bfbc2866c5 Add the -Wextra flag by default
After the Spring Cleaning on warnings in early 2014, GCC and Clang emit
only about two warnings when compiling Wget with -Wall -Wextra.

Enable both these flags by default so that new warnings in these areas
are more prominent.
2014-11-20 16:45:34 +05:30
Tim Rühsen
7a7a241e5b Use random() and srandom() if available.
Reported-by: Coverity scanner
2014-11-19 16:06:19 +01:00
Darshit Shah
ff6fc1f8dd Add a configure option to disable assertions
Adds a --disable-assert configure option. We **highly recommend** the
use of this option by distro maintainers so as to ensure a smooth
experience with Wget. Sometimes an assertion may fail even though Wget
continues to work perfectly. In such scenarios, we do not wish for
client installations of Wget to crash. Hence, for a stable experience,
please use the --disable-assert configure flag.
2014-11-15 13:39:25 +05:30
Tim Ruehsen
831b9d1880 Skip python tests if python3 is not available
Reported-by: kdevel@vogtner.de
2014-11-06 11:07:33 +01:00
Darshit Shah
fdd8b8c997 Fix libpsl configure code 2014-11-01 23:49:55 +05:30
Mike Frysinger
f48be4e0c6 use pkg-config to check gnutls deps if available
Newer versions of these packages ship with pkg-config files, so if we can
detect it via those, do so.  If that fails, fall back to the old methods.
2014-11-01 17:09:24 +01:00
Mike Frysinger
5ae5d2eaeb use pkg-config to check openssl deps if available
Newer versions of these packages ship with pkg-config files, so if we can
detect it via those, do so.  If that fails, fall back to the old methods.
2014-11-01 17:09:10 +01:00
Mike Frysinger
47d0d4c331 use pkg-config to check pcre deps if available
Newer versions of these packages ship with pkg-config files, so if we can
detect it via those, do so.  If that fails, fall back to the old methods.

Also add a configure flag to explicitly control its usage.
2014-11-01 17:08:52 +01:00
Mike Frysinger
d272339fda use pkg-config to check libuuid deps if available
Newer versions of these packages ship with pkg-config files, so if we can
detect it via those, do so.  If that fails, fall back to the old methods.

Also add a configure flag to explicitly control its usage.
2014-11-01 17:07:08 +01:00
Mike Frysinger
297d040ce1 use pkg-config to check zlib deps if available
Newer versions of these packages ship with pkg-config files, so if we can
detect it via those, do so.  If that fails, fall back to the old methods.
2014-11-01 17:06:33 +01:00
Tim Ruehsen
4152e98bb0 content for commit 605d9053e5 2014-10-28 20:21:33 +01:00
Tim Rühsen
9c34d4ed09 add ./configure valgrind support to test suites 2014-10-28 12:32:12 +01:00
Tim Rühsen
b64fa2eb84 Switched to parallel test harness 2014-10-01 14:41:32 +05:30
Darshit Shah
7cbe8141d1 Introducing Python based Test Environment
Squashed Commit, of the following commits:

7743384 Update documentation to reflect changes in code
b703633 Add feature that allows to ensure that Wget correctly crawls the website in recursive mode
0758f47 Add new test for recursive spider mode
43bb61b Smartly guess content type header
d4d0e63 Support substring replace in File Contents too
f578500 Compatibility fix with multiple servers
8b1a9b6 Extend Functionality to support spawning multiple servers
e84192a Use the provided calls to shutdown server instead of rewriting it
99659f3 Improve Documentation
cb94e52 Slight code cleanup. Remove unused code
886ac1a Shift to new Threading Model from Multiprocessing model
e74c2ec Add new test for POST Requests
48644f1 Print diff when file contents don't match
b6f9efe Add tests for Cookie support
4c9e6b4 Document pending work
e13bc90 Add new test to ensure Content Disposition and Auth work together
60d1f4d Add new Test for Continue command
738b299 Add test, Test-Head
9b9d16b Edit non-unique TEST_NAME variable
ae958db Minor optimizations to the way Server Rules are executed
50b4f0c The rules need not be a defaultdict.
dccc154 Introducing Python based Test Environment
2014-07-24 16:51:58 +05:30
Darshit Shah
cae32d41c1 Fix broken check for libpsl 2014-07-22 22:11:32 +05:30
Darshit Shah
a44841cbe2 Fix potential memory leak and libpsl configure 2014-07-21 13:25:54 +05:30
Giuseppe Scrivano
601401da71 warc: Fix UUID generation on FreeBSD 2014-06-24 15:05:19 +02:00
Giuseppe Scrivano
ffccc7d3be Do not depend on always defined macros 2014-06-12 18:49:15 +02:00
Giuseppe Scrivano
a22cd7394b Remove trailing whitespaces 2014-06-12 18:49:14 +02:00
Darshit Shah
854ebbf4dd Support libpsl for cookie domain checking 2014-06-05 16:20:34 +05:30
Giuseppe Scrivano
59080465d1 configure.ac: update copyright years 2014-02-06 18:34:25 +01:00
Tim Ruehsen
7e85e9509e fix bug #39844 2013-09-16 10:25:11 +02:00
Tim Ruehsen
ffb94036f2 openssl: fix build. 2013-07-23 15:45:30 +02:00
Tim Ruehsen
c19d76c024 ntlm: support libnettle. 2013-07-22 14:17:03 +02:00
Giuseppe Scrivano
22bd8011e8 Fallback to PATH_MAX when pathconf is not available 2012-10-07 13:09:56 +02:00
Giuseppe Scrivano
f4122c5094 Use git-version-gen instead of bzr-version-gen. 2012-05-13 16:46:15 +02:00
Gijs van Tulder
f5a1097871 Add support for -accept-regex and --reject-regex. 2012-05-09 21:18:23 +02:00
Ray Satiro
44ea82bc67 Fix build under mingw when OpenSSL is used. 2012-03-25 13:47:53 +02:00
Gijs van Tulder
0a8a898fbe Fix a linker error if zlib is not found. 2012-01-11 15:27:06 +01:00
Gijs van Tulder
e3820953b2 Add support for WARC files. 2011-11-04 22:25:00 +01:00
Alan Hourihane
85797041a6 configure: check for libz when GNU TLS is used. 2011-09-04 14:19:12 +02:00
Giuseppe Scrivano
baed30f057 Under mingw don't check for different OpenSSL when one is found. 2011-08-26 12:35:12 +02:00
Giuseppe Scrivano
5e3c9b55f2 Prefers utime over futimens when available. 2011-08-25 11:41:51 +02:00
Giuseppe Scrivano
6330e4f8b8 configure: do not hardcode libraries 2011-08-11 12:20:25 +02:00
Giuseppe Scrivano
f3c2686ef7 allow --with-libgnutls-prefix and --with-libssl-prefix. 2011-08-11 12:07:32 +02:00
Giuseppe Scrivano
13028b1bae configure: print usage string for --with-ssl. 2011-08-10 22:23:53 +02:00
Giuseppe Scrivano
228299b4f7 Fix a linker error on systems with an older gnutls version. 2011-08-10 21:35:59 +02:00
Ray Satiro
df0f9c0ce7 configure.ac: fix indentation. 2011-04-19 11:14:53 +02:00
Giuseppe Scrivano
1ad97cc7ca Do not check the host_os twice and if windres is available. 2011-04-17 12:11:33 +02:00
Ray Satiro
918be06e06 Detect dynamically linked OpenSSL libraries. 2011-04-16 18:49:07 +02:00
Giuseppe Scrivano
1cba1a7659 Fix openssl detection under Windows. 2011-04-04 00:13:53 +02:00
Giuseppe Scrivano
b1acde223f Fix build under Windows. 2011-04-03 20:23:31 +02:00
Giuseppe Scrivano
930075f467 configure: Fix gnutls detection. 2011-03-26 19:54:47 +01:00
Giuseppe Scrivano
2f6aa1d741 mass change: update copyright years. 2011-01-01 13:19:37 +01:00
Giuseppe Scrivano
8405493e1d gnutls: fix build under mingw32. 2010-10-23 13:50:31 +02:00
Giuseppe Scrivano
04e4bd614d Quote AC_CHECK_SIZEOF argument. 2010-07-11 03:01:32 +02:00
Giuseppe Scrivano
4c73a8fc26 By default use GNU TLS not OpenSSL. 2010-06-22 22:33:38 +02:00
Giuseppe Scrivano
f9993ec155 Remove `windows' subdirectory. 2010-06-17 23:08:23 +02:00
Giuseppe Scrivano
f9c765d8e6 Fix IPv6 detection under Windows. 2010-06-15 16:35:29 +02:00
Giuseppe Scrivano
23ebf5c52a Remove dist-bzip2 dist-lzma from automake options. 2010-06-10 10:08:17 +02:00
Giuseppe Scrivano
64ebab7160 Under Windows, do not link to wsock32. 2010-06-03 17:00:44 +02:00
Giuseppe Scrivano
c702d99f7b Use the gnulib snprintf module. 2010-05-27 13:59:46 +02:00
Giuseppe Scrivano
58a9721edf Use the gnulib md5 module. 2010-05-16 01:34:17 +02:00
Giuseppe Scrivano
e36538b53a Use a script to generate the version string. 2010-05-10 00:47:17 +02:00
Giuseppe Scrivano
293008f682 Mass update copyright years. 2010-05-08 21:56:15 +02:00
Giuseppe Scrivano
b2e34eb6a9 Add -lws2_32 under mingw32. 2010-05-07 12:51:39 +02:00
Giuseppe Scrivano
a8de649f28 Do not use macro WGET_SOCKLEN_T. 2010-05-07 10:11:15 +02:00
Giuseppe Scrivano
0cf88f48e6 Do not define ALL_LINGUAS in configure.ac. 2010-05-07 10:00:41 +02:00
Giuseppe Scrivano
29e3aa3a20 Fix a sanity check by the AC_CONFIG_SRCDIR macro. 2010-05-03 20:40:23 +02:00
Micah Cowan
663b1b3367 imported patch windows 2009-09-22 18:43:02 -07:00
Micah Cowan
a5739c1991 Bump devel version. 2009-09-22 10:25:33 -07:00
Micah Cowan
ba2a0b871a Backout release-specific changes. 2009-09-22 10:24:07 -07:00
Micah Cowan
780d1e674b imported patch one-dot-twelve 2009-09-22 09:39:49 -07:00
Micah Cowan
4a08094db8 [mq]: cfg-mk 2009-09-21 20:39:44 -07:00
Micah Cowan
e968919131 No .zip - save that for the binary distros. 2009-09-20 14:15:43 -07:00
Micah Cowan
2f1bb05129 Additional dist types. 2009-09-09 20:34:00 -07:00
Micah Cowan
cb555a94fa Check for idna.h in /usr/include/idn. 2009-09-05 13:33:52 -07:00
Steven Schubiger
d294668715 Silence autoconf warnings. 2009-09-04 20:27:58 +02:00
Micah Cowan
c9bf94ef25 Use a separate build-aux directory; just too many files in the top directory. 2009-09-04 01:43:36 -07:00
Micah Cowan
b9e9ad65cc Ran update-copyright. 2009-09-04 00:13:47 -07:00
Micah Cowan
227daa7415 Move AM_GNU_GETTEXT below AC_AIX. 2009-09-03 14:35:07 -07:00
Micah Cowan
3411bd9488 Use stamp-po mechanism, and update to gettext-0.17. 2009-09-03 09:40:59 -07:00
Micah Cowan
efd514c825 Fixes for when IRIs are disabled. 2009-07-03 22:36:14 -07:00
Micah Cowan
4f3dd68173 Merge with mainline. 2009-06-25 01:14:11 -07:00
Micah Cowan
e265ef9242 Declare h_errno if no declaration provided (HP-UX). 2009-06-12 01:32:44 -07:00
Saint Xavier
b30a0dd817 Automated merge. 2008-07-20 21:45:09 +02:00
Micah Cowan
2e2ac6ad2f Merge current tip with CSS stuff. 2008-07-11 16:30:51 -07:00
Saint Xavier
4c9adcd1e4 Check for libiconv needed for IRIs support 2008-06-26 19:14:14 +02:00
Saint Xavier
ed558a83f6 The prefered way is to avoid #ifdef flooding, so take it that way. Introduce iri.c and iri.h for achieving it 2008-06-19 22:07:03 +02:00
Saint Xavier
d66a45f828 Add autoconf code for IDN/IRIs support 2008-05-24 23:57:56 +02:00