We may change the global environ variable in the child; when using
vfork() this also sets it in the parent. Preserve the parent's
environ in child_execute_job() so it takes effect for all callers.
Reported by Denis Excoffier <bug-tar@Denis-Excoffier.org>
Root cause found by Martin Dorey <Martin.Dorey@hitachivantara.com>
* src/job.c (start_job_command): Remove save/restore of the parent
environment.
(child_execute_job): Add save/restore of the parent environment,
if we use vfork().
* tests/scripts/functions/shell: Add a test the crashes if we don't
reset environ after we run $(shell ...).
If we detect a recursive variable reference when constructing the
environment for the shell function, return the original value from the
caller's environment. Other options such as failing, returning the
empty string, or returning the unexpanded make variable value have
been shown to not behave well in real-world environments. If the
variable doesn't exist in the caller's environment, return the empty
string.
Found by Sergei Trofimovich <slyich@gmail.com> when testing older
versions of autoconf.
* NEWS: Clarify this behavior.
* doc/make.texi (Shell Function): Ditto. Also add info about !=.
* src/expand.c (recursively_expand_for_file): Search the caller's
environment if we detect a recursive variable expansion.
* tests/scripts/functions/shell: Add tests for this behavior.
The fix for SV 10593 caused recursive expansion errors when exporting
a variable that contains a $(shell ...) invocation. If we see this
type of recursion, ignore it and expand to the empty string rather
than failing.
* src/variable.h (env_recursion): New global variable.
* src/variable.c (target_environment): If creating the environment
for a $(shell ...) function increment env_recursion. Remove the
check for expansion in a shell function context.
* src/expand.c (recursively_expand_for_file): Check for recursive
expansion in a $(shell ...) environment context and if present,
show the verbose message and return the empty string.
* tests/scripts/functions/shell: Add a test for this situation.
Some POSIX systems do not allow locks to be taken on non-files, such
as pipes. This is a problem since very often make is invoked with
its stdout redirected to a pipe. Also, if stdout is redirected to a
file that already has a lock on it for some other reason (perhaps a
shared file such as /dev/null) it can cause a hang.
This means our previous method of locking stdout, although it had some
nice advantages, is not portable enough. Instead, use a temporary
file and take the lock on that. We pass the name of the file to child
make processes. On Windows we continue to use a shared mutex for
output sync.
Remove POSIX emulation functions like fcntl from Windows; instead
follow the lead of the jobserver and create an interface in os.h for
output sync, and move the OS-specific content to posixos.c and
w32os.c.
* NEWS: Add a note.
* src/makeint.h (ALL_SET): Check that all bits are set.
* src/os.h: Add bits for checking the state of stdin/stdout/stderr.
Add prototypes for OS-specific output sync methods.
* src/posixos.c (check_io_state): Determine the status of stdin,
stdout, stderr an return a suite of bits describing them.
(osync_enabled): If the global variable holding the FD of the lock
file (osync_handle) is valid return true.
(osync_setup): Create a temporary file and remember its name in a
global variable (osync_tmpfile), and set osync_handle.
(osync_get_mutex): If output sync is enabled, return the filename
of the lock file prefixed with "fnm:" to denote a filename.
(osync_parse_mutex): If the provided filename has the wrong format
disable output sync. Else open the lock file and set osync_handle.
(osync_clear): Close osync_handle. If we're the parent make, then
also unlink the temporary file.
(osync_acquire): Take a lock on the osync_handle descriptor.
(osync_release): Release the lock on the osync_handle descriptor.
(fd_set_append): Add APPEND mode to a file descriptor.
* src/w32/w32os.c: Perform the same actions as posixos.c, copying
the details from src/w32/compat/posixfcn.c. Use a mutex rather
than locking a temporary file.
* src/output.h: Remove all the OS-specific content.
* src/output.c: Remove all the OS-specific content.
(set_append_mode): Remove and replace with fd_set_append().
(sync_init): Remove and replace with check_io_state().
(acquire_semaphore): Remove and replace with osync_acquire().
(release_semaphore): Remove and replace with osync_release().
(setup_tmpfile): If the IO state is not obtained, get it. If stdout
and/or stderr are valid, set up a tempfile to capture them.
(output_init): Set io_state if not set already, and check it when
deciding whether to close stdout on exit.
* src/main.c (main): If we're syncing, set up the mutex using the
new osync_setup() / osync_parse_mutex() methods.
(prepare_mutex_handl_string): Replace with osync_parse_mutex().
(die): Call osync_clear().
* src/w32/compat/posixfcn.c: Remove implementations of fcntl(),
record_sync_mutex(), create_mutex(), and same_stream().
Using anonymous pipes for jobserver support has some advantages:
for example there is nothing on disk that needs to be cleaned up.
However it has many obscure problems, related to the fact that in
order for it to work we need to ensure these resources are properly
passed through to child processes that want to use the jobserver.
At the same time we don't want to pass the pipe to process which
DON'T know about the jobserver.
Other processes can open file descriptors which we then think are
our jobserver, but aren't. And, we open the pipe file descriptors
in blocking mode which doesn't work for all users.
See issues such as SV 57178, SV 57242, and SV 62397
To avoid these issues, use named pipes (on systems where they are
available) instead of anonoymous pipes. This simplifies many things:
we never need to pass open file descriptors to our children; they
can open the jobserver named pipe. We don't need to worry about
recursive vs. non-recursive children. Users don't have to "pass
through" the resources if they are invoking sub-makes. Each child
can open its own file descriptor and set blocking as needed.
The downside is the named pipe exists on disk and so must be cleaned
up when the "top-level" make instance exits.
In order to allow make to continue to be used in build systems where
older versions of GNU make, or other tools that want to use the
jobserver, but don't understand named pipes, introduce a new option
--jobserver-style that allows the user to choose anonymous pipes.
* NEWS: Announce the change and the --jobserver-style option.
* doc/make.1: Add --jobserver-style documentation.
* doc/make.texi (Special Variables): Add missing items to .FEATURES.
(Options Summary): Add --jobserver-style.
(POSIX Jobserver): Named pipes, changes to --jobserver-auth, and the
--jobserver-style option.
(Windows Jobserver): Document --jobserver-style for Windows.
* configure.ac: Check for mkfifo.
* src/config.h-vms.template: Undefined HAVE_MKFIFO.
* src/config.h.W32.template: Ditto.
* src/main.c: Add jobserver-style as a new command line option.
(main): Add jobserver-fifo to .FEATURES if supported. Pass the style
option to jobserver_setup().
* src/os.h (jobserver_setup): Accept a style string option.
* src/posixos.c (enum js_type): Enumeration of the jobserver style.
(js_type): Which style we are currently using.
(fifo_name): The path to the named pipe (if in use).
(jobserver_setup): If no style is given, or "fifo" is given, set up a
named pipe: get a temporary file and use mkfifo() on it, then open it
for reading and writing. If something fails fall back to anonymous
pipes.
(jobserver_parse_auth): Parse jobserver-auth to determine the style.
If we are using a named pipe, open it. If we're using anonymous pipes
ensure they're valid as before.
(jobserver_get_invalid_auth): Don't invalidate the jobserver when
using named pipes.
(jobserver_clear): Clean up memory used for named pipes.
(jobserver_acquire_all): Unlink the named pipe when done.
* src/w32/w32os.c (jobserver_setup): Check the style argument.
* tests/scripts/features/jobserver: Use --jobserver-style to test
the anonymous pipe behavior, and also test named pipe/semaphore
behavior. Check invalid jobserver-style options.
* tests/scripts/functions/shell: Use --jobserver-style to test the
anonymous pipe behavior, and also test named pipe/semaphore
behavior.
* src/main.c (main): Don't reset the jobserver if the number of
slots has not changed.
(define_makeflags): Add all normal flags even when ALL is not set.
* tests/scripts/functions/shell: Test invoking make in $(shell ...).
* tests/scripts/variables/MAKEFLAGS: Test the value of MAKEFLAGS in
$(shell ...).
Export all variables, including exported makefile variables, when
invoking a shell for the $(shell ...) function. If we detect a
recursive variable expansion, silently ignore that variable and do
not export it. We do print a debug message.
* NEWS: Announce the potential backward-incompatibility.
* doc/make.texi (Shell Function): Document the export behavior.
* src/main.c (main): Add "shell-export" to .FEATURES.
* src/job.h: New function to free struct childbase.
* src/job.c (free_childbase): Implement it; call from free_child.
* src/function.c (func_shell_base): Use target_environment() to
obtain the proper environment for the shell function.
Use free_childbase() to free memory.
(windows32_openpipe): Don't reset the environment: the caller
already provided a proper PATH variable in envp.
* src/variable.c (target_environment): If we detect a recursive
expansion and we're called from func_shell, ignore the variable.
(sync_Path_environment): Simplify and reduce memory allocation.
* tests/scripts/functions/shell: Add tests for this.
If a $(shell ...) invocation failed due to a command-not-found error,
make wrote the stdout of that shell to our stderr for some reason.
That seems very wrong.
If the command's stderr was not redirected then its output would have
already been written to its stderr, and if it was redirected then we
shouldn't take it upon ourselves to force it to go to stderr!
* src/function.c (func_shell_base): Append shell stdout even if the
shell command failed.
* tests/run_make_tests.pl: Determine the error generated for
command-not-found situations.
* tests/scripts/functions/shell: Verify that redirecting stderr to
stdout will behave properly if the command is not found.
* src/function.c (func_shell_base): Use error() instead of recreating
the error output.
* src/job.c (exec_command): Show more standard error messages.
* src/load.c (unload_file): Fix whitespace in the error message.
* tests/scripts/features/errors: Add tests for starting non-
existent commands and new error message formats.
* tests/scripts/features/output-sync: New error message formats.
* tests/scripts/functions/shell: Ditto.
* tests/scripts/features/jobserver: Windows doesn't use pipes
* tests/scripts/functions/shell: Don't test kill -2 on Windows
* tests/scripts/misc/bs-nl: Windows doesn't handle single quotes
* tests/scripts/misc/general3: Ditto.
* main.c (main): Sanitize program name detection on Windows.
* makeint.h: 'program' is a const string on all platforms now.
* tests/run_make_tests.bat: Windows bat file to invoke tests
* tests/test_driver.pl: Obtain system-specific error messages.
(get_osname): Compute the $port_type here. Add more $osname checks
for different Windows Perl ports.
(_run_command): Rewrite the timeout capability to work properly
with Windows. Don't use Perl fork/exec; instead use system(1,...)
which allows a more reliable/proper kill operation.
Also, allow options to be given as a list instead of a string, to
allow more complex quoting of command-line arguments.
* tests/run_make_tests.pl (run_make_with_options): Allow options
to be provided as a list in addition to a simple string.
(set_more_defaults): Write sample makefiles and run make on them
instead of trying to run echo and invoking make with -f-, to avoid
relying on shell and echo to get basic configuration values. Also
create a $sh_name variable instead of hard-coding /bin/sh.
* tests/scripts/features/archives: Skip on Windows.
* tests/scripts/features/escape: Use list method for passing options.
* tests/scripts/features/include: Use system-specific error messages.
* tests/scripts/features/output-sync: "Command not found" errors
generate very different / odd output on Windows. This needs to be
addressed but for now disable these tests on Windows.
* tests/scripts/functions/abspath: Disable on Windows.
* tests/scripts/functions/file: Use system-specific error messages.
* tests/scripts/functions/shell: "Command not found" errors generate
very different / odd output on Windows. This needs to be addressed
but for now disable these tests on Windows.
* tests/scripts/misc/close_stdout: Disable on Windows.
* tests/scripts/options/dash-k: Use system-specific error messages.
* tests/scripts/options/dash-l: Disable on Windows.
* tests/scripts/options/eval: Use list method for passing options.
* tests/scripts/options/general: Skip some non-portable tests.
* tests/scripts/targets/ONESHELL: Skip some non-portable tests.
* tests/scripts/targets/POSIX: Skip some non-portable tests.
* tests/scripts/variables/MAKEFILES: Skip some non-portable tests.
* tests/scripts/variables/SHELL: Use a makefile not -f- for testing.
* NEWS: Document the change, as a backward-incompatible change.
* main.c (main): Add 'nocomment' to the .FEATURES variable.
* read.c (remove_comments): Skip variable references during remove.
(find_char_unquote): Fix comments for new STOPMAP support.
* tests/scripts/features/escape: Test new escape syntax.
* tests/scripts/functions/guile: Ditto.
* tests/scripts/functions/shell: Ditto.
While displaying line numbers, show the relevant line number inside
the recipe not just the first line of the entire recipe.
Sample changes suggested by Brian Vandenberg <phantall@gmail.com>
* gnumake.h (gmk_floc): Add an 'offset' to track the recipe offset.
* read.c (eval, eval_makefile, eval_buffer): Initialize 'offset'.
(record_files, install_pattern_rule): Ditto.
* job.c (new_job, job_next_command): Update 'offset' based on the
line of the recipe we're expanding or invoking.
(child_error): Add 'offset' when showing the line number.
* function.c (func_shell_base): Ditto.
* output.c (error, fatal): Ditto.
* NEWS: Mention the new ability.
* tests/scripts/features/errors: Check the line number on errors.
* tests/scripts/functions/warning: Check the line number on warnings.
* tests/scripts/features/output-sync,
tests/scripts/features/parallelism, tests/scripts/functions/shell,
tests/scripts/functions/error: Update line numbers.
Add a new variable .SHELLSTATUS which holds the exit status of the
last-invoked shell function or != assignment.
* NEWS, doc/make.texi: Document the change.
* function.c (shell_completed, msdos_openpipe, func_shell_base): Add
shell_completed() to handle the completion of the shell, by setting
.SHELLSTATUS. Call it where needed.
* job.c (child_handler): Call shell_completed().
* tests/scripts/functions/shell: Add tests for .SHELLSTATUS.
