In this mode we still collect all the output from a given target and
dump it at once. However we don't treat recursive lines any differently
from non-recursive lines. Also we don't print enter/leave messages
after every dump. However we do ensure that we always print them once
to stdout, so the parent make will collect it properly.
Create a new file, output.c, and collect functions that generate output there.
We introduce a new global context specifying where output should go (to stdout
or to a sync file), and the lowest level output generator chooses where to
write output based on that context.
This allows us to set the context globally, and all operations that write
output (including functions like $(info ...) etc.) will use it.
Removed the "--trace=dir" capability. It was too confusing. If you have
directory tracking enabled then output sync will print the enter/leave message
for each synchronized block. If you don't want that, disable directory
tracking.
We tried to get some efficiency by avoiding a parse_file_seq() for simple
pattern prerequisites, but this also means no wildcard expansion was
happening, so add it back. Add regression tests for wildcards in target and
prerequisite lists.
POSIX does not guarantee that writes will be atomic if a file is
opened for normal (non-append) output. That means if multiple processes
are writing to the same file, output could be lost. I can't think of
a real use-case where we would NOT want append for stdout/stderr, so
force it if we can.
main.c (find_and_set_default_shell): Don't use file_exists_p or
dir_file_exists_p, as those call readdir, which can fail if PATH
includes directories with non-ASCII characters, and that would
cause Make to fail at startup with confusing diagnostics. See
https://sourceforge.net/mailarchive/message.php?msg_id=30846737
for the details.
In various places we were passing flags and characters to compare, then
using complex conditionals to see where to stop in string searches.
Performance numbers reveal that we were spending as much as 23% of our
processing time in these functions, most of it in the comparison lines.
Instead create a character map and use a single bitwise comparison to
determine if this is any one of the stop characters.
Ensure all simple flags are kept in the initial batch of flags.
Do not allow any flags with options in that batch.
If there are only non-simple flags ensure MAKEFLAGS begins with " ".
Don't let MFLAGS start with "- ".
If -R is set in the makefile and not the command line, then go through all the
default variables and undefine them. If -r is set in the makefile and not in
the command line, then remove all .SUFFIX prefixes (unless the user set it)
and SUFFIX variable setting. In -p mode don't print builtins.
This allows you to write portable makefiles that set GNU make-specific command
line options in the environment or makefile: add them to GNUMAKEFLAGS instead
of MAKEFLAGS and they will be seen by GNU make but ignored by other
implementations of make.
This mode replaces the previous heuristic setting enabled with -O, where we
would log directory enter/leave for each synchronized output. Now we only
do that if --trace=dir is given.
A new flag to the -O/--output-sync, "job", selects a per-job (that is, per
line of a recipe) output synchronization. To support this move the close of
the temp file out of the sync_output() function and don't do it until we free
the child, since we may call sync_output() multiple times in a given recipe.
When we set up for a new temp file, if we're in per-job mode we truncate the
file and seek to the beginning to re-use it for every job.
w32/compat/posixfcn.c: New file, with emulations of Posix
functions and Posix functionality for MS-Windows.
w32/subproc/sub_proc.c: Include io.h.
(process_noinherit): New function, forces a file descriptor to not
be inherited by child processes.
(process_easy): Accept two additional arguments, and use them to
set up the standard output and standard error handles of the child
process.
w32/include/sub_proc.h (process_easy): Adjust prototype.
(process_noinherit): Add prototype.
read.c [WINDOWS32]: Include windows.h and sub_proc.h.
makeint.h (LOCALEDIR) [WINDOWS32}: Define to NULL if not
defined. This is needed because the MS-Windows build doesn't have
a canonical place for LOCALEDIR.
(WIN32_LEAN_AND_MEAN) [WINDOWS32]: Define, to avoid getting from
windows.h header too much stuff that could conflict with the code.
main.c <sync_mutex>: New static variable.
<switches>: Add support for "--sync-mutex" switch.
(decode_output_sync_flags): Decode the --sync-mutex= switch.
(prepare_mutex_handle_string) [WINDOWS32]: New function.
(main): Add "output-sync" to .FEATURES.
job.h (CLOSE_ON_EXEC) [WINDOWS32]: Define to call
process_noinherit.
(F_GETFD, F_SETLKW, F_WRLCK, F_UNLCK, struct flock) [WINDOWS32]:
New macros.
(RECORD_SYNC_MUTEX): New macro, a no-op for Posix platforms.
(sync_handle_t): New typedef.
job.c <sync_handle>: Change type to sync_handle_t.
(FD_NOT_EMPTY): Seek to the file's end. Suggested by Frank
Heckenbach <f.heckenbach@fh-soft.de>.
(pump_from_tmp_fd) [WINDOWS32]: Switch to_fd to binary mode for
the duration of this function, and then change back before
returning.
(start_job_command) [WINDOWS32]: Support output_sync mode on
MS-Windows. Use a system-wide mutex instead of locking
stdout/stderr. Call process_easy with two additional arguments:
child->outfd and child->errfd.
(exec_command) [WINDOWS32]: Pass two additional arguments, both
-1, to process_easy, to adjust for the changed function signature.
function.c (windows32_openpipe) [WINDOWS32]: This function now
returns an int, which is -1 if it fails and zero otherwise. It
also calls 'error' instead of 'fatal', to avoid exiting
prematurely.
(func_shell_base) [WINDOWS32]: Call perror_with_name if
windows32_openpipe fails, now that it always returns. This avoids
a compiler warning that error_prefix is not used in the MS-Windows
build.
config.h.W32.template (OUTPUT_SYNC): Define.
build_w32.bat: Add w32/compat/posixfcn.c to compilation and
linking commands.
From Frank Heckenbach <f.heckenbach@fh-soft.de>:
job.c (sync_output): Don't discard the output if
acquire_semaphore fails; instead, dump the output unsynchronized.
Provide a simple API for loaded objects to interact with GNU make. I still
won't guarantee that this API won't change but it's much closer to something
that's supported and provides easy-to-use interfaces with a public header
file.
On configure-enabled systems, configure will detect Guile installed
(using pkg-config, which is how GNU Guile is distributed) and enable
it if so.
On all non-configure-enabled systems, currently, the default is for
Guile support to be disabled.
(exec_command) [WINDOWS32]: Use pid2str instead of non-portable
%Id.
main.c (handle_runtime_exceptions): Use %p to print addresses,
to DTRT on both 32-bit and 64-bit hosts. Savannah bug #27809.
job.c (w32_kill, start_job_command, create_batch_file): Use
pid_t for process IDs and intptr_t for the 1st arg of
_open_osfhandle.
function.c (windows32_openpipe): Use pid_t for process IDs and
intptr_t for the 1st arg of _open_osfhandle.
(func_shell): Use pid_t for process IDs.
main.c (main) [WINDOWS32]: Pacify the compiler.
config.h.W32.template (pid_t): Add a definition for 64-bit
Windows builds that don't use GCC.
Savannah bug #27809. Patch by Ozkan Sezer <sezeroz@gmail.com>
backward-incompatible change in the 2008 POSIX specification.
- Add the .SHELLFLAGS variable so people can choose their own shell flags.
- Add tests for this.
- Add documentation for this.
- Fix memory errors found by valgrind
- Remove multi_glob() and empower parse_file_seq() to do its job:
the goal here is to remove the confusing reverse/re-reverse we do on
the file lists: needed for future fixes.
- Add a prefix arg to parse_file_seq()
- Make concat() variadic so it can take arbitrary #'s of strings
Allows the user to reset the prefix character for introducing recipe lines
from the default (tab) to any other single character, and back again.
Also, reworked the manual to consistently use the word "recipe" to describe
the set of commands we use to update a target, instead of the various
phrases used in the past: "commands", "command lines", "command scripts",
etc.
comparison functions to always use POSIX strcasecmp(). For non-POSIX
systems that use other functions (strcmpi or stricmp) use a macro to alias
strcasecmp to those. If we can't find any of them (VMS, plus whatever
UNIX doesn't have them) then define our own version in misc.c.
string into the strcache. As a side-effect, many more structure members and
function arguments can/should be declared const.
As mentioned in the changelog, unfortunately measurement shows that this
change does not yet reduce memory. The problem is with secondary expansion:
because of this we store all the prerequisites in the string cache twice.
First we store the prerequisite string after initial expansion but before
secondary expansion, then we store each individual file after secondary
expansion and expand_deps(). I plan to change expand_deps() to be callable
in either context (eval or snap_deps) then have non-second-expansion
targets call expand_deps() during eval, so that we only need to store that
dependency list once.
A few changes from char* to void* where appropriate, and removing of
unnecessary casts.
Much more work on const-ifying the codebase. This round involves some code
changes to make it correct. NOTE!! There will almost certainly be problems
on the non-POSIX ports that will need to be addressed after the const changes
are finished: they will need to be const-ified properly and there may need to
be some changes to allocate memory, etc. as well.
The next (last?) big push for this, still to come, is const-ifying the
filenames in struct file, struct dep, etc. This will allow us to store file
names in the string cache and finally resolve Savannah bug #15182 (make uses
too much memory), among other advantages.
- Add more warnings.
- Rename variables that mask out-scope vars with the same name.
- Remove all casts of return values from xmalloc, xrealloc, and alloca.
- Remove casts of the first argument to xrealloc.
- Convert all bcopy/bzero/bcmp invocations to use memcp/memmove/memset/memcmp.
I decided this feature was too impacting to make the permanent default
behavior. This set of changes makes the default behavior of make the
old behavior (no second expansion). If you want second expansion, you
must define the .SECONDEXPANSION: special target before the first target
that needs it.
This set of changes ONLY fixes explicit and static pattern rules to work
like this. Implicit rules still have second expansion enabled all the
time: I'll work on that next.
Note that there is still a backward-incompatibility: now to get the old
SysV behavior using $$@ etc. in the prerequisites list you need to set
.SECONDEXPANSION: as well.
check for this and exit with an error.
The closeout.c version from gnulib pulls in too much other stuff, and
gnulib requires an ANSI C 89 compliant compiler, while GNU make (so far)
still wants to work on K&R.
cleanups.
If we find a make error (invalid makefile syntax or something like that)
write back any tokens we have before we exit.
If we have waiting jobs (using -j + -l) set an alarm before we sleep on
the read() system call, so we can wake up to check the load and start
waiting jobs, if there are long-running jobs we would otherwise be
waiting for. Suggested by Grant Taylor.
Taylor. There are two forms of this: first, it was possible to lose
tokens when using -j and -l at the same time, because waiting jobs were
not checked when determining whether any jobs were outstanding. Second,
if you had an exported recursive variable that contained a $(shell ...)
function there is a possibility to lose tokens, since a token was taken
but the child list was not updated until after the shell function was
complete.
To resolve this I introduced a new variable that counted the number of
tokens we have obtained, rather than checking whether there were any
children on the list. I also added some sanity checks to make sure we
weren't writing back too many or not enough tokens. And, the master
make will drain the token pipe before exiting and compare the count of
tokens at the end to what was written there at the beginning.
Also:
* Ensure a bug in the environment (missing "=") doesn't cause make to core.
* Rename the .DEFAULT_TARGET variable to .DEFAULT_GOAL, to match the
terminology in the documentation and other variables like MAKECMDGOALS.
* Add documentation of the .DEFAULT_GOAL special variable.
Still need to document the secondary expansion stuff...
I did this by adding intelligence into the algorithm such that the
second expansion was only actually performed when the prerequisite list
contained at least one "$", so we knew it is actually needed.
Without this we were using up a LOT more memory, since every single
target (even ones never used by make) had their file variables
initialized. This also used a lot more CPU, since we needed to create
and populate a new variable hash table for every target.
There is one issue remaining with this feature: it leaks memory. In
pattern_search() we now initialize the file variables for every pattern
target, which allocates a hash table, etc. However, sometimes we
recursively invoke pattern_search() (for intermediate files) with an
automatic variable (alloca() I believe) as the file. When that function
returns, obviously, the file variable hash memory is lost.
