Commit Graph

3675 Commits

Author SHA1 Message Date
Tim Rühsen
9947663af8 * src/ftp-ls.c: Fix -Wformat= warnings 2017-02-03 11:59:33 +01:00
Tim Rühsen
cfae085665 * src/ftp.c (ftp_retrieve_list): Add default to switch 2017-02-03 11:57:02 +01:00
Tim Rühsen
e69808256b * src/css-url.h: Remove redundant declaration 2017-02-03 11:53:28 +01:00
Tim Rühsen
11989ef669 * src/ftp.c: Fix -Wformat= warning 2017-02-03 11:52:08 +01:00
Tim Rühsen
5fceab6cb9 * src/http.c (test_parse_range_header): Fix constants 2017-02-03 10:32:42 +01:00
Dale R. Worley
d6eead1794 Improve documentation of --trust-server-names. 2017-02-02 12:10:43 +01:00
Tim Rühsen
d0e02a54ae * src/url.c (mkalldirs): Add newline to log message 2017-02-02 11:11:50 +01:00
Tim Rühsen
2e70409844 * src/cookies.c (check_domain_match): Add newline to DEBUG lines 2017-02-02 11:11:07 +01:00
Tim Rühsen
4734e8d4d7 * NEWS: update 2017-01-17 15:16:40 +01:00
Tim Rühsen
0fe8a5c40e * README.checkout: Link HTTPS where possible, update flex homepage 2017-01-17 14:25:15 +01:00
Tim Rühsen
1b9374ee59 * README.checkout: Fix libidn to libidn2 2017-01-17 14:07:16 +01:00
Tim Rühsen
6de24fe3c0 * src/iri.c: Use TR46 non-transitional for toASCII conversion 2017-01-13 15:53:03 +01:00
Tim Rühsen
0ab3d92c85 * src/main.c: Fix _Noreturn compiler warnings 2017-01-13 15:50:19 +01:00
Tim Rühsen
4cf8af84e0 * src/utils.c: Fix _Noreturn compiler warning 2017-01-13 15:49:05 +01:00
Tim Rühsen
42b8761cbc * src/init.c (setval_internal): Fix sign compare warning 2017-01-13 15:47:02 +01:00
Tim Rühsen
fd0f759597 Replace home-grown portability code by gnulib modules
* bootstrap.conf: Add intprops, inttypes, limits-h, signal-h,
  stat, sys_types
* src/ftp.c: Replace 'struct_stat' by 'struct stat'
* src/hsts.c: Likewise
* src/http.c: Likewise
* src/main.c: Likewise
* src/netrc.c: Likewise
* src/retr.c: Likewise
* src/url.c: Likewise
* src/utils.c: Likewise
* src/sysdep.h: Remove old portability code

Further portability issues should be addressed by gnulib.
2017-01-13 15:38:15 +01:00
Tim Rühsen
a384f5e2e9 Replace WGET_* m4 macros by gnulib modules
* bootstrap.conf: Add hostent, inet_ntop, nanosleep, utimens
* configure.ac: Remove WGET_STRUCT_UTIMBUF, WGET_FNMATCH,
  WGET_NANOSLEEP, WGET_POSIX_CLOCK, WGET_NSL_SOCKET
* m4/wget.m4: Likewise
* src/Makefile.am: Add $(LIB_NANOSLEEP) $(LIB_POSIX_SPAWN) to LDADD
* tests/Makefile.am: Likewise
* src/host.c (print_address): Use inet_ntop also for IPV4
2017-01-13 12:54:35 +01:00
Tim Rühsen
5ae1f37902 Remove libidn vulnerability work-around
* src/iri.c (_utf8_is_valid): Removed

Since we are using libidn2 for IDNs, we no longer need
this work-around.
2017-01-13 12:03:47 +01:00
Tim Rühsen
0eb4a21b6c * src/iri.c (idn_encode): Use TR46 transitional if available 2017-01-13 11:45:18 +01:00
Tim Rühsen
60dc3c21b2 * gnulibx: Update 2017-01-13 10:59:25 +01:00
Tim Rühsen
1a01a6b2d0 Fix previous commit 2427ca4ac0 2017-01-07 15:59:11 +01:00
vijeth-aradhya
2427ca4ac0 Fix http.c and ftp.c passwd logic error
* src/ftp.c (getftp): Fix password/user selection
* src/http.c (initialize_request): Likewise

Before, netrc password won over interactive
--ask-password but now --ask-password wins
after change of program logic

Fixes Issue #48811
2017-01-06 16:18:40 +01:00
Giuseppe Scrivano
42c2ce71bc * src/main.c (main): Add missing \n in error message 2016-12-31 17:11:01 +01:00
Giuseppe Scrivano
def133f26f Check that fd_set has not fds bigger than FD_SETSIZE
* src/connect.c: check that the fd is not bigger than FD_SETSIZE
  before using FD_SET.  An fd_set cannot hold fds bigger than
  FD_SETSIZE, causing out-of-bounds write to a buffer on the stack.

Reported by: Jann Horn <jannh@google.com>
2016-12-28 12:24:19 +01:00
Nikos Mavrogiannopoulos
b9ed06afd8 Avoid calling the gnutls priority functions multiple times
* src/gnutls.c (ssl_connect_wget): Call gnutls_set_default_priority()
  for --secure-protocol=auto (default).

The patch fixes a behavior that may have unintended side-effects in
certain gnutls versions. Instead use the default priorities when no
options are given.

Signed-off-by: Nikos Mavrogiannopoulos <nmav@gnutls.org>
2016-12-20 14:48:31 +01:00
Tim Rühsen
1bdc20d774 Print debug message when skipping certain recursive downloads
* src/recur.c (retrieve_tree): Print debug message instead silently
  skipping recursive downloads.
2016-12-19 12:19:52 +01:00
Rahul Bedarkar
e4e9d3c1c8 Rename base64_{encode,decode} (trivial patch)
* src/http-ntlm.c: Rename base64_{encode,decode}
* src/http.c: Likewise
* src/utils.c: Likewise
* src/utils.h: Likewise

When statically linking with gnutls, we get definition clash error for
base64_encode which is also defined by gnutls.

To prevent definition clash, rename base64_{encode,decode}

Signed-off-by: Rahul Bedarkar <rahul.bedarkar@imgtec.com>
2016-12-14 15:52:52 +01:00
Tim Rühsen
dcdd618b18 Add support for psl_latest()
* configure.ac: Add check for psl_latest(),
  remove --with-psl-file
* src/cookies.c (check_domain_match): Use psl_latest() if available
2016-12-11 21:04:40 +01:00
Piotr Wajda
3c796b9a85 Respect -o parameter again
* log.c: don't choose log output dynamically when opt.lfilename is set

 Regression introduced by dd5c549f6a
 Reported-by: Dale R. Worley
2016-11-09 13:32:14 +01:00
Tim Rühsen
00ae9b4ee2 Move Wget from IDN2003 (libidn) to IDN2008 (libidn2)
* .travis.yml: Install libidn2-dev instead libidn11-dev.
* bootstrap.conf: Add modules libunistring-optional, unistr/base,
  unicase/tolower.
* configure.ac: Check for libidn2.
* src/Makefile.am: Add $(LTLIBUNISTRING) to LDADD.
* tests/Makefile.am: Set LDADD similar to LDADD in src/Makefile.am
* src/connect.c: Use libidn2 code instead of libidn.
* src/host.c: Likewise.
* src/iri.c: Likewise.
* src/iri.h: Likewise.
* src/options.h: Likewise.
* src/url.c: Likewise.
* src/url.h: Likewise.
* src/log.c: Fix C99 comment.

IDN2003 should not be used any more due to security concerns.
We use libunistring (resp. the unicode code from gnulib) for
lowercasing UTF-8 before we give data to libidn2.
TR#46 is missing, no support in libidn2 nor in libunistring.
2016-11-07 11:03:42 +01:00
Tim Rühsen
2242d5aee4 Use -O0 in contrib scripts
* contrib/check-hard: Use -O0 instead of -O2 to speed up script.
* contrib/travis-ci: Likewise.
2016-11-03 15:47:41 +01:00
Tim Rühsen
77a546348c Update gnulib 2016-11-03 13:28:17 +01:00
Tim Rühsen
be5517f98f * src/metalink.c: Fix typo 'suceeded' -> 'succeeded'
Reported-by: Göran Uddeborg <goeran@uddeborg.se>,
             Anders Jonsson <anders.jonsson@norsjovallen.se>
2016-10-22 22:21:39 +02:00
losgrandes
dd5c549f6a Fixes #45790: wget prints it's progress even when background
* src/log.c: Use tcgetpgrp(STDIN_FILENO) != getpgrp() to determine when to print to STD* or logfile.
  Deprecate log_request_redirect_output function.
  Use different file handles for STD* and logfile, to easily switch between them when changing fg/bg.
* src/log.h: Make redirect_output function externally linked.
* src/main.c: Don't use deprecated log_request_redirect_output function. Use redirect_output instead.
* src/mswindows.c: Don't use deprecated log_request_redirect_output function. Use redirect_output instead.
2016-10-21 19:33:29 +02:00
losgrandes
78e0ec5f03 Fixes #46584: wget --spider always returns zero exit status
* src/ftp.c: Return error as exit value if even one file doesn't exist
2016-10-21 10:24:28 +02:00
Tim Rühsen
807d1c7d94 * src/http.c (gethttp): Accept 206 for request w/o Range header
Fixes: #49319
2016-10-12 14:59:36 +02:00
Tim Rühsen
3d1d5b317b * tests/Makefile.am: Let WGETRC work on *nix and Windows 2016-10-07 13:27:07 +02:00
Tim Rühsen
517d799b6f Properly include iconv.h
* src/iri.c: Check HAVE_ICONV to include iconv.h
* src/url.c: Same
2016-10-07 13:24:15 +02:00
Tim Rühsen
f79bb20f1b * bootstrap.conf: Add pipe-posix for pipe() 2016-10-07 13:23:27 +02:00
Tim Rühsen
e5164a8260 Amend redirection behavior
* src/recur.c (descend_redirect): Ignore WG_RR_LIST and WG_RR_REGEX
  for redirections.
* testenv/Makefile.am: Add Test-recursive-redirect.py
* testenv/Test-recursive-redirect.py: New test

Test-recursive-redirect.py written by Dale R. Worley.

Reported-by: "Dale R. Worley" <worley@ariadne.com>
2016-10-07 11:49:07 +02:00
Matthew White
a35f3dd4f6 New: Metalink/XML v3 python class, update tests to use this class
* testenv/misc/metalinkv3_xml.py: New Metalink/XML v3 python class
* testenv/Test-metalink-xml-abspath-trust.py: Update test to use the
  new Metalink/XML v3 python class
* testenv/Test-metalink-xml-abspath.py: Update test to use the new
  Metalink/XML v3 python class
* testenv/Test-metalink-xml-absprefix-trust.py: Update test to use the
  new Metalink/XML v3 python class
* testenv/Test-metalink-xml-absprefix.py: Update test to use the new
  Metalink/XML v3 python class
* testenv/Test-metalink-xml-continue.py: Update test to use the new
  Metalink/XML v3 python class
* testenv/Test-metalink-xml-emptyprefix-trust.py: Update test to use
  the new Metalink/XML v3 python class
* testenv/Test-metalink-xml-homepath-trust.py: Update test to use the
  new Metalink/XML v3 python class
* testenv/Test-metalink-xml-homepath.py: Update test to use the new
  Metalink/XML v3 python class
* testenv/Test-metalink-xml-homeprefix-trust.py: Update test to use
  the new Metalink/XML v3 python class
* testenv/Test-metalink-xml-homeprefix.py: Update test to use the new
  Metalink/XML v3 python class
* testenv/Test-metalink-xml-nohash.py: Update test to use the new
  Metalink/XML v3 python class
* testenv/Test-metalink-xml-nourls.py: Update test to use the new
  Metalink/XML v3 python class
* testenv/Test-metalink-xml-prefix-trust.py: Update test to use the
  new Metalink/XML v3 python class
* testenv/Test-metalink-xml-prefix.py: Update test to use the new
  Metalink/XML v3 python class
* testenv/Test-metalink-xml-relpath-trust.py: Update test to use the
  new Metalink/XML v3 python class
* testenv/Test-metalink-xml-relpath.py: Update test to use the new
  Metalink/XML v3 python class
* testenv/Test-metalink-xml-relprefix-trust.py: Update test to use the
  new Metalink/XML v3 python class
* testenv/Test-metalink-xml-relprefix.py: Update test to use the new
  Metalink/XML v3 python class
* testenv/Test-metalink-xml-size.py: Update test to use the new
  Metalink/XML v3 python class
* testenv/Test-metalink-xml-trust.py: Update test to use the new
  Metalink/XML v3 python class
* testenv/Test-metalink-xml.py: Update test to use the new
  Metalink/XML v3 python class
2016-09-30 19:44:06 +02:00
Matthew White
c403e67935 New: --metalink-over-http Content-Type/Disposition Metalink/XML processing
* src/http.c (metalink_from_http): Process the Content-Type header.
  Add an application/metalink4+xml URL as metalink metaurl.  If the
  option opt.content_disposition is true, the Content-Disposition's
  filename is the metaurl's name
* doc/wget.texi: Update --content-disposition and --metalink-over-http
* doc/metalink-standard.txt: Update doc. Content-Type/Disposition
  processing through --metalink-over-http. Update download naming
  system about --trust-server-names and --content-disposition
* testenv/Makefile.am: Add new files
* testenv/Test-metalink-http-xml-type.py: New file. Metalink/HTTP
  Content-Type/Disposition header automated Metalink/XML tests
* testenv/Test-metalink-http-xml-type-trust.py: New file. Metalink/HTTP
  Content-Type/Disposition header with --trust-server-names automated
  Metalink/XML tests
* testenv/Test-metalink-http-xml-type-content.py: New file. Metalink/HTTP
  Content-Type/Disposition header with --content-disposition automated
  Metalink/XML tests
* testenv/Test-metalink-http-xml-type-trust-content.py: New file.
  Metalink/HTTP Content-Type/Disposition header with --trust-server-names
  and --content-disposition automated Metalink/XML tests

Process the Content-Type header, identify an application/metalink4+xml
file.  The Content-Disposition could provide an alternate name through
the "filename" field for the metalink xml file.  Respectively, the cli
options --metalink-over-http and --content-disposition are required.

When Metalink/XML auto-processing, to use the Content-Disposition's
filename, the cli option --trust-server-names is also required.
2016-09-30 19:44:06 +02:00
Matthew White
3021466817 Bugfix: Set NULL variable due to --content-disposition to Metalink origin
* src/http.c (http_loop): Prevent SIGSEGV when hstat.local_file is
  NULL, opt.content_disposition has a role in leaving the value unset
* src/http.c (gethttp): If hs->local_file is NULL (aka http_loop()'s
  hstat.local_file), set it to the value of hs->metalink->origin
2016-09-30 19:44:06 +02:00
Matthew White
c89767d8d1 New: --trust-server-names saves Metalink/HTTP xml files using the "name" field
* src/metalink.c (retrieve_from_metalink): If opt.trustservernames is
  true, use the basename of the metaurl's name to save the xml file
* doc/metalink-standard.txt: Update doc. With --trust-server-names any
  Metalink/HTTP Link application/metalink4+xml file is saved using the
  basename of the "name" field, if any. Update Metalink/HTTP examples
* testenv/Makefile.am: Add new file
* testenv/Test-metalink-http-xml-trust-name.py: New file. Metalink/HTTP
  automated Metalink/XML, save xml files using the "name" field tests
2016-09-30 19:44:06 +02:00
Matthew White
f030cdf8e2 Bugfix: Detect when a metalink:file doesn't have any hash
* src/metalink.c (retrieve_from_metalink): Reject any metalink:file
  without hashes. Prompt the error and switch to the next file
* testenv/Makefile.am: Add new file
* testenv/Test-metalink-xml-nohash.py: New file. Metalink/XML with no
  hashes tests

Prevent SIGSEGV.
2016-09-30 19:44:06 +02:00
Matthew White
5dccb2a9ce Bugfix: Detect malformed base64 Metalink/HTTP Digest header
* src/http.c (metalink_from_http): Fix hash_bin_len type. Use ssize_t
  instead than size_t. Reject -1 as base64_decode() return value
* testenv/Makefile.am: Add new file
* testenv/Test-metalink-http-baddigest.py: New file. Metalink/HTTP
  malformed base64 Digest header tests

On malformed base64 input, ssize_t base64_decode() returns -1. Such
value is too big for a size_t variable, and used as xmalloc() value
will exaust all the memory.
2016-09-30 19:44:06 +02:00
Matthew White
0538e791fb New option --metalink-index to process Metalink application/metalink4+xml
* NEWS: Mention the effect of --metalink-index over Metalink
* src/init.c: Add new option metalinkindex (opt.metalink_index),
  initialize to -1
* src/main.c: Add new option metalink-index (--metalink-index=NUMBER)
* src/options.h: Add new option metalink_index (int)
* src/metalink.h: Add declaration of functions fetch_metalink_file(),
  replace_metalink_basename()
* src/metalink.c: Add functions fetch_metalink_file() simple file
  fetch, replace_metalink_basename() replace file basename
* src/metalink.c (retrieve_from_metalink): New. Process Metalink
  application/metalink4+xml of opt.metalink_index ordinal number
* doc/wget.texi: Add new option metalink-index (--metalink-index)
  documentation
* doc/metalink-standard.txt: Updated doc. Add documentation about
  Metalink application/metalink4+xml metaurls download naming system
* doc/metalink-standard.txt: Update Metalink/XML and HTTP examples
* testenv/Makefile.am: Add new files
* testenv/Test-metalink-http-xml.py: New file. Metalink/HTTP automated
  Metalink/XML "application/metalink4+xml" --metalink-index tests
* testenv/Test-metalink-http-xml-trust.py: New file. Metalink/HTTP
  automated Metalink/XML "application/metalink4+xml" --metalink-index
  retrieval with --trust-server-names tests

WARNING: Do not use lib/dirname.c (dir_name) to get the directory
name, it may append a dot '.' character to the directory name.
2016-09-30 19:44:06 +02:00
Matthew White
acb1d1a668 Bugfix: Prevent sorting when there are less than two elements
* src/utils.c (stable_sort): Add condition nmemb > 1, sort only when
  there is more than one element

Prevent SIGSEGV.
2016-09-30 19:44:06 +02:00
Matthew White
628fb565c7 New: Parse Metalink/HTTP header for application/metalink4+xml
* src/http.c (metalink_from_http): Parse Metalink/HTTP header for
  metaurls application/metalink4+xml media types
* src/metalink.h: Add function declaration metalink_meta_cmp()
* src/metalink.c: Add function metalink_meta_cmp() compare metalink
  metaurls priorities

Add Metalink/HTTP application/metalink4+xml media types as metaurls to
the metalink variable that will be used to download the files.
2016-09-30 19:44:05 +02:00
Matthew White
64ab90ee66 New test: Metalink shall not concatenate '/' to an empty directory prefix
* testenv/Makefile.am: Add new file
* testenv/Test-metalink-xml-emptyprefix-trust.py: New file.
  Metalink/XML empty directory prefix (--directory-prefix '') tests

Detect a '/' character wrongfully concatenated to an empty directory
prefix '' (not NULL), resulting in an absolute path as '/dir/file',
instead than 'dir/file'.
2016-09-30 19:44:05 +02:00