Commit Graph

571 Commits

Author SHA1 Message Date
Tim Rühsen
77929eda1b Simplify url_error function.
* src/url.c (url_error): simplify, remove url arg, return const char *.
* src/url.h (url_error): remove url arg, return const char *.
* src/html-url.c (get_urls_file): Simplify call to url_error(),
  remove call to free().
* src/http.c (metalink_from_http): Likewise.
* src/main.c (main): Likewise.
* src/metalink.c (retrieve_from_metalink, fetch_metalink_file): Likewise.
* src/recur.c (retrieve_tree): Likewise.
* src/res.c (res_retrieve_file): Likewise.
* src/retr.c (retrieve_url, retrieve_from_file): Likewise.
2023-02-18 17:25:56 +01:00
Tim Rühsen
4223996930 Update copyright years 2023-02-17 13:46:58 +01:00
Tim Rühsen
cb114fbbf7 Fix HSTS portability by using int64_t instead of time_t.
* src/hsts.c: Use int64_t instead of time_t.
* src/http.c: Use int64_t for parsing Strict-Transport-Security.
2022-03-20 20:57:10 +01:00
Tim Rühsen
1cda2bb5d5 src/http.c (time_to_rfc1123): Fix -Wformat-nonliteral 2022-03-20 20:53:10 +01:00
Darshit Shah
be936bda56 * Update Copyright years 2022-02-26 15:49:52 +01:00
Aarni Koskela
446afdca21 * src/http.c (parse_strict_transport_security): Fix typo in string
Copyright-paperwork-exempt: Yes
2022-01-31 18:01:51 +01:00
Tim Rühsen
c7a37d82ee * src/http.c (http_loop): Fix memleak 2021-12-22 13:06:34 +01:00
Darshit Shah
f75fcf2985 * src/http.c (http_loop): Hide password when printing status with -nv
Reported-By: Per Lundberg <perlun@gmail.com>
Closes: #61492
2021-12-01 23:38:52 +01:00
Tim Rühsen
c778ac20b4 * src/http.c (gethttp): Add cheap extra check to help static analyzers 2021-06-06 15:34:06 +02:00
Tim Rühsen
027d294114 * src/http.c (initialize_request): Send Host HTTP header first
This solves an issue where the server expects the Host: header
as first one. This seems plausible (ahem) as the Host: header is the
only one that is required.
2021-05-03 17:49:58 +02:00
Darshit Shah
5a7f2f7e87 Run make update-copyright 2021-01-01 12:31:01 +01:00
Tim Rühsen
7ec15b9c92 Remove SIZEOF_WGINT as wgint is always int64_t
* src/http.c (test_parse_range_header): Remove use of SIZEOF_WGINT.
* src/utils.c (human_readable): Remove superfluous HR_NUMTYPE,
*   (number_to_string): Remove use of SIZEOF_WGINT.
* src/utils.h: Remove use of SIZEOF_WGINT and HR_NUMTYPE.
* src/wget.h: Remove #define SIZEOF_WGINT.
2020-12-29 12:44:20 +00:00
Tim Rühsen
015afd7cc7 * src/http.c (http_cleanup): Reset wget_cookie_jar after freeing
This silences the wget_options_fuzzer which triggered #28610 on
OSS-Fuzz. This issue can not happen with the Wget utility.
The fuzzer runs main(),...,cleanup() in a loop which the Wget utility
never does.
2020-12-13 18:23:39 +01:00
Tim Rühsen
4221b5656a * src/http.c (check_auth): Fix memory leak 2020-03-06 12:09:02 +01:00
Tim Rühsen
6377ae1647 * src/http.c: Don't cast 3rd param to request_set_header. 2020-03-05 19:19:42 +01:00
Tim Rühsen
eb02512a40 * src/http.c (gethttp): Remove unneeded free of hstat.(message|error) 2020-03-03 19:33:53 +01:00
Tim Rühsen
1ac3812e8a * src/http.c (gethttp): Fix memleaks
Reported-by: Вячеслав Петрищев
2020-03-03 18:38:41 +01:00
Tim Rühsen
910c0abe7a * src/http.c (read_response_body): Fix memleak
Reported-by: Вячеслав Петрищев
2020-03-03 10:26:09 +01:00
Tim Rühsen
ee187476de * src/http.c (http_cleanup): Free hashtable basic_authed_hosts 2020-02-27 19:33:47 +01:00
Tim Rühsen
7146b9dc77 * src/http.c (http_cleanup): Call invalidate_persistent to free resources 2020-02-27 14:28:10 +01:00
Tim Rühsen
8010507d73 * src/http.c (http_loop): Avoid unneeded strdup 2020-02-27 11:35:50 +01:00
Вячеслав Петрищев
f747eea059 * src/http.c (gethttp): Fix memleak 2020-02-27 11:12:27 +01:00
Tim Rühsen
46a6e2957e * src/http.c: Do not reveal OS type to server via User-Agent: header
Following the "privacy by design" principle, which is now European law by
virtue of the GDPR since 2018, the client should not transmit the operating
system.

Fixes #57884
Reported-by: Bruno Haible
2020-02-23 18:51:10 +01:00
Tim Rühsen
8e130c32ba Split unique_name into two functions with just one arg
* src/http.c (check_file_output): Call unique_name_passthrough.
* src/metalink.c (badhash_suffix): Call unique_name.
* src/url.c (url_file_name): Call unique_name_passthrough.
* src/utils.c: Split unique_name, amend description.
* src/utils.h: Amend unique_name, add unique_name_passthrough.
2020-02-22 12:59:18 +01:00
Tim Rühsen
d08bdbc878 * src/http.c (print_response_line): Fixed comparison
Introduced in 68868bbb37 (13.2.2020).

Reported-by: Gisle Vanem
2020-02-22 08:47:20 +01:00
Tim Rühsen
1c16819558 * src/http.c (set_file_timestamp): Fix stack memory leak (Coverity #1419655) 2020-02-19 19:13:48 +01:00
Tim Rühsen
1dfd1aa1cf * src/http.c (metalink_from_http): Fix memleak (Coverity #1419661, #1419656) 2020-02-19 17:07:52 +01:00
Tim Rühsen
6d181cb8ef * src/http.c: Include xstrndup.h unconditionally 2020-02-13 19:34:36 +01:00
Tim Rühsen
cbea3d41dd * src/http.c (check_auth): Remove use of alloca 2020-02-13 16:27:38 +01:00
Tim Rühsen
68868bbb37 * src/http.c (print_response_line): Remove use of alloca 2020-02-13 16:11:51 +01:00
Tim Rühsen
a3f2ba708d * src/http.c (request_set_user_header): Remove use of alloca 2020-02-13 16:11:23 +01:00
Tim Rühsen
f1ce7f5bd6 * src/http.c (metalink_from_http): Remove use of alloca 2020-02-13 16:00:19 +01:00
Tim Rühsen
704208b766 * src/http.c (basic_authentication_encode): Remove use of alloca 2020-02-13 15:54:16 +01:00
Tim Rühsen
0bc0729bcc * src/http.c (gethttp): Remove use of alloca 2020-02-13 15:40:48 +01:00
Tim Rühsen
572f6f9b39 * src/http.c (gethttp): Remove use of alloca 2020-02-13 15:34:13 +01:00
Tim Rühsen
e22dbe5af0 * src/http.c (set_file_timestamp): Remove use of alloca 2020-02-13 15:30:50 +01:00
Tim Rühsen
233f982f9d Add buffer size param to warc_uuid_str
* src/http.c (gethttp): Add buffer size param to warc_uuid_str.
* src/warc.c: Likewise
* src/warc.h: Likewise
2020-02-13 11:28:18 +01:00
Ander Juaristi
928c54d27e Do not overwrite restval if len is smaller
* src/http.c (http_loop): overwrite 'restval' only if 'len' is
   greater than it. Else substract 'len' to 'restval'.

When retrying a request, only send a 'Range' header if the previous
request returned data.
2020-01-25 18:35:05 +01:00
Darshit Shah
221f1ba3ef Update copyright year to 2020 2020-01-10 12:56:08 +01:00
Darshit Shah
60dc09b507 Revert " Hi, Thank you again Darshit for your response. The RejectHeaderField rule rejects ANY header"
I accidentally commited this patch.

This reverts commit 4ce2f93600.
2019-05-30 11:19:46 +02:00
sulfastor
4ce2f93600 Hi, Thank you again Darshit for your response. The RejectHeaderField rule rejects ANY header
of the header field while RejectHeader rejects ONLY the specified full header.
	Since we wanted to be sure a header field is not sent to the server we wrote this rule.

	* doc/wget.texi: Added --disable-header documentation.
        * fuzz/wget_options_fuzzer.dict: Update with --disable-header inputs.
        * src/http.c (disabled_header): Checks for disabled headers
	(request_set_header): Doesn't let header to be set if disabled
	(gethttp): frees disabled header to let overriding
        * src/init.c (cmd_dis_header), (check_user_disabled_header) added new option disabled_headers.
        * src/main.c: added new option --disable-header, added help description
        * src/options.h: added new option --disable-header
	* src/utils.h (vec_remove_header)
	* src/utils.c (vec_remove_header) removes all header instances from vector
        * testenv/Makefile.am: Added new test files
        * testenv/server/http/http_server.py: Added new rule RejectHeaderField
        * testenv/conf/reject_header_field.py: Added new rule RejectHeaderField
        * testenv/README: Added help description for new rule
        * testenv/Test-disable-default-headers.py: Test without using --header
        * testenv/Test-disable-headers-after.py: Test using --header before --disable-header
        * testenv/Test-disable-headers-before.py: Test using --header after --disable-header

Signed-off-by: sulfastor <torresmoisesa@gmail.com>, adham elkarn <adhamelkarn@hotmail.com>
2019-05-30 10:01:24 +02:00
Tim Rühsen
5d87635c66 Fix corner case in processing server response
* src/http.c (response_head_terminator): Don't access uninitialized data
* fuzz/wget_read_hunk_fuzzer.c: Sync response_head_terminator()
2019-03-25 16:05:56 +01:00
Darshit Shah
8b975e83d3 Update copyright statements 2019-02-10 11:50:17 +01:00
Tim Rühsen
fdb7fe613c Fix typos detected by codespell (via contrib/spell-checker) 2018-12-28 18:58:15 +01:00
Tim Rühsen
3cdfb594cf Don't save user/pw with --xattr
Also the Referer info is reduced to scheme+host+port.

* src/ftp.c (getftp): Change params of set_file_metadata()
* src/http.c (gethttp): Change params of set_file_metadata()
* src/xattr.c (set_file_metadata): Remove user/password from origin URL,
  reduce Referer value to scheme/host/port.
* src/xattr.h: Change prototype of set_file_metadata()
2018-12-26 14:38:24 +01:00
Tim Rühsen
11fad3fa72 Revert "Bail out on unexpected 416 server errors"
This reverts commit 6f3b995993.

The code is obviously wrong, see https://savannah.gnu.org/bugs/?54963
Also, the example from the original post doesn't work any more.
With other words, the broken server behavior has been fixed meanwhile.
2018-11-09 16:16:43 +01:00
Tim Rühsen
0727b8f3a9 * src/http.c (resp_new): Fix code to avoid false positive by clang 2018-09-20 14:59:06 +02:00
Tomas Hozza
dfef92bac3 * src/http.c (http_loop): Fix RESOURCE LEAK found by Coverity
Error: RESOURCE_LEAK (CWE-772):
wget-1.19.5/src/http.c:4486: alloc_fn: Storage is returned from allocation function "url_string".
wget-1.19.5/src/url.c:2248:3: alloc_fn: Storage is returned from allocation function "xmalloc".
wget-1.19.5/lib/xmalloc.c:41:11: alloc_fn: Storage is returned from allocation function "malloc".
wget-1.19.5/lib/xmalloc.c:41:11: var_assign: Assigning: "p" = "malloc(n)".
wget-1.19.5/lib/xmalloc.c:44:3: return_alloc: Returning allocated memory "p".
wget-1.19.5/src/url.c:2248:3: var_assign: Assigning: "result" = "xmalloc(size)".
wget-1.19.5/src/url.c:2248:3: var_assign: Assigning: "p" = "result".
wget-1.19.5/src/url.c:2250:3: noescape: Resource "p" is not freed or pointed-to in function "memcpy". [Note: The source code implementation of the function has been overridden by a builtin model.]
wget-1.19.5/src/url.c:2253:7: noescape: Resource "p" is not freed or pointed-to in function "memcpy". [Note: The source code implementation of the function has been overridden by a builtin model.]
wget-1.19.5/src/url.c:2257:11: noescape: Resource "p" is not freed or pointed-to in function "memcpy". [Note: The source code implementation of the function has been overridden by a builtin model.]
wget-1.19.5/src/url.c:2264:3: noescape: Resource "p" is not freed or pointed-to in function "memcpy". [Note: The source code implementation of the function has been overridden by a builtin model.]
wget-1.19.5/src/url.c:2270:7: identity_transfer: Passing "p" as argument 1 to function "number_to_string", which returns an offset off that argument.
wget-1.19.5/src/utils.c:1776:11: var_assign_parm: Assigning: "p" = "buffer".
wget-1.19.5/src/utils.c:1847:3: return_var: Returning "p", which is a copy of a parameter.
wget-1.19.5/src/url.c:2270:7: noescape: Resource "p" is not freed or pointed-to in function "number_to_string".
wget-1.19.5/src/utils.c:1774:25: noescape: "number_to_string(char *, wgint)" does not free or save its parameter "buffer".
wget-1.19.5/src/url.c:2270:7: var_assign: Assigning: "p" = "number_to_string(p, url->port)".
wget-1.19.5/src/url.c:2273:3: noescape: Resource "p" is not freed or pointed-to in function "full_path_write".
wget-1.19.5/src/url.c:1078:47: noescape: "full_path_write(struct url const *, char *)" does not free or save its parameter "where".
wget-1.19.5/src/url.c:2287:3: return_alloc: Returning allocated memory "result".
wget-1.19.5/src/http.c:4486: var_assign: Assigning: "hurl" = storage returned from "url_string(u, URL_AUTH_HIDE_PASSWD)".
wget-1.19.5/src/http.c:4487: noescape: Resource "hurl" is not freed or pointed-to in "logprintf".
wget-1.19.5/src/http.c:4513: leaked_storage: Variable "hurl" going out of scope leaks the storage it points to.
\# 4511|               {
\# 4512|                 printwhat (count, opt.ntry);
\# 4513|->               continue;
\# 4514|               }
\# 4515|             else

There are two conditional branches, which call continue, without freeing memory potentially allocated and pointed to by"hurl" pointer. In fase "!opt.verbose" is True and some of the appropriate conditions in the following if/else if construction, in which "continue" is called, are also true, then the memory allocated to "hurl" will leak.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2018-08-27 13:24:24 +02:00
Tomas Hozza
b8be904ac7 * src/http.c (check_auth): Fix RESOURCE LEAK found by Coverity
Error: RESOURCE_LEAK (CWE-772):
wget-1.19.5/src/http.c:2434: alloc_fn: Storage is returned from allocation function "xmalloc".
wget-1.19.5/lib/xmalloc.c:41:11: alloc_fn: Storage is returned from allocation function "malloc".
wget-1.19.5/lib/xmalloc.c:41:11: var_assign: Assigning: "p" = "malloc(n)".
wget-1.19.5/lib/xmalloc.c:44:3: return_alloc: Returning allocated memory "p".
wget-1.19.5/src/http.c:2434: var_assign: Assigning: "auth_stat" = storage returned from "xmalloc(4UL)".
wget-1.19.5/src/http.c:2446: noescape: Resource "auth_stat" is not freed or pointed-to in "create_authorization_line".
wget-1.19.5/src/http.c:5203:70: noescape: "create_authorization_line(char const *, char const *, char const *, char const *, char const *, _Bool *, uerr_t *)" does not free or save its parameter "auth_err".
wget-1.19.5/src/http.c:2476: leaked_storage: Variable "auth_stat" going out of scope leaks the storage it points to.
\# 2474|                 /* Creating the Authorization header went wrong */
\# 2475|               }
\# 2476|->         }
\# 2477|         else
\# 2478|           {

Error: RESOURCE_LEAK (CWE-772):
wget-1.19.5/src/http.c:2431: alloc_fn: Storage is returned from allocation function "url_full_path".
wget-1.19.5/src/url.c:1105:19: alloc_fn: Storage is returned from allocation function "xmalloc".
wget-1.19.5/lib/xmalloc.c:41:11: alloc_fn: Storage is returned from allocation function "malloc".
wget-1.19.5/lib/xmalloc.c:41:11: var_assign: Assigning: "p" = "malloc(n)".
wget-1.19.5/lib/xmalloc.c:44:3: return_alloc: Returning allocated memory "p".
wget-1.19.5/src/url.c:1105:19: var_assign: Assigning: "full_path" = "xmalloc(length + 1)".
wget-1.19.5/src/url.c:1107:3: noescape: Resource "full_path" is not freed or pointed-to in function "full_path_write".
wget-1.19.5/src/url.c:1078:47: noescape: "full_path_write(struct url const *, char *)" does not free or save its parameter "where".
wget-1.19.5/src/url.c:1110:3: return_alloc: Returning allocated memory "full_path".
wget-1.19.5/src/http.c:2431: var_assign: Assigning: "pth" = storage returned from "url_full_path(u)".
wget-1.19.5/src/http.c:2446: noescape: Resource "pth" is not freed or pointed-to in "create_authorization_line".
wget-1.19.5/src/http.c:5203:40: noescape: "create_authorization_line(char const *, char const *, char const *, char const *, char const *, _Bool *, uerr_t *)" does not free or save its parameter "path".
wget-1.19.5/src/http.c:2476: leaked_storage: Variable "pth" going out of scope leaks the storage it points to.
\# 2474|                 /* Creating the Authorization header went wrong */
\# 2475|               }
\# 2476|->         }
\# 2477|         else
\# 2478|           {

Both "pth" and "auth_stat" are allocated in "check_auth()" function. These are used for creating the HTTP Authorization Request header via "create_authorization_line()" function. In case the creation went OK (auth_err == RETROK), then the memory previously allocated to "pth" and "auth_stat" is freed. However if the creation failed, then the memory is never freed and it leaks.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2018-08-27 13:23:52 +02:00
Tim Rühsen
4fc69950da * src/http.c (http_loop): Fix --retry-on-host-error 2018-06-13 20:16:22 +02:00