Introduce non-deterministic ordering into goal and prerequisite
traversal to help tease out inconsistent failures that may happen
when running in parallel build mode.
Introduce second order into each dependency chain:
1. Existing order is syntactic order reachable via 'dep->next'
2. New order is shuffled order stored as 'dep->shuf' in each 'dep'
When updating goals and prerequisites and '--shuffle' is provided,
use the shuffled order to walk the graph. When automatic variable
are set always use the syntactic order of parameters.
* Makefile.am: Add new src/shuffle.c and src/shuffle.h file.
* build_w32.bat: Ditto.
* builddos.bat: Ditto.
* makefile.com: Ditto.
* po/POTFILES.in: Ditto.
* doc/make.texi: Add documentation for --shuffle.
* doc/make.1: Ditto.
* src/dep.h (DEP): Add the shuf pointer.
* src/filedef.h (struct file): Add was_shuffled flag.
* src/main.c: (shuffle_mode): Global flag for the shuffle mode.
(usage): Add the --shuffle option.
(switches): Ditto.
(main): Set shuffle_mode based on the command line parameter.
Reshuffle prerequisites if requested.
* src/remake.c (update_goal_chain): Walk the shuffled list if enabled.
(update_file_1): Ditto.
* src/shuffle.h: Provide an interface for shuffling prerequisites.
* src/shuffle.c: Implement option parsing and prerequisite shuffling.
* tests/scripts/options/shuffle: Test shuffle option and modes.
If -R is given on the command line it implies -r as well; make this
same assumption if -R is provided in MAKEFLAGS set in the makefile.
* src/main.c (main): Check no_builtin_variables_flag after reading
makefiles.
* tests/scripts/options/dash-r: Add a test.
The -f, -file, and --makefile options were not properly handled when
re-exec'ing due to makefile updates. This problem, plus a patch and
tests, was reported by Dmitry Goncharov <dgoncharov@users.sf.net>.
While examining this I found another bug: after re-exec we forgot the
batch file was temporary and never deleted it.
I decided to fix all these problems at once using a different fix
than Dmitry's: I created a new internal-only command-line option,
--temp-stdin. When reconstructing the make options for a re-exec,
replace the -f/--file/--makefile option that reads from stdin with
--temp-stdin=<filename> so that the re-exec'd version of make knows
it's a temporary batch file and will delete it.
We no longer need to add the -o options because the re-exec'd make
knows this is a temporary makefile and treats it as such.
To simplify, replace the --file and --makefile options taking a
filename, with just -f<filename> on re-exec.
Some examples of the rewrite:
User command line Re-exec command line
----------------- --------------------
-f- --temp-stdin=<batch>
--file - --temp-stdin=<batch>
-f - --makefile a.mk --temp-stdin=<batch> -fa.mk
--file=a.mk -fa.mk
-fa.mk -fa.mk
-Rf a.mk -Rf a.mk
-Rf- -R --temp-stdin=<batch>
* src/main.c (stdin_offset): Remember the offset into the makefiles
list of the batch file read from stdin. Remove stdin_nm.
(struct command_switch): Create a new --temp-stdin option, which
also updates the makefiles list.
(main): Add the temporary filename to the string cache.
Move the tempfile handling after checking makefile arguments for "-"
so that files provided via --temp-stdin are also handled specially.
When rewriting re-exec options, we may need one more than we had
originally so create a new argv list. Walk through the original
list and convert it to the new list, following the above process.
(decode_switches): Set the stdin_offset flag if we see --temp-stdin.
* tests/scripts/options/dash-f: Add many more tests, provided by
Dmitry Goncharov <dgoncharov@users.sf.net>.
We already use undef makefile strings to mean "re-use the previous
makefile", so if the string is empty we'll assume it means "don't
use a makefile at all" (don't add -f).
* tests/run_make_tests.pl (run_make_test): If the makefile string
is empty, don't create a makefile or add -f.
* tests/scripts/features/include: Change empty makefile to "\n".
* tests/scripts/misc/close_stdout: Ditto.
* tests/scripts/options/dash-r: Ditto.
* tests/scripts/options/dash-I: Use string comparison operator.
* tests/test_driver.pl (cmd2str): New method to convert a
command line to a string with proper quoting.
* tests/run_make_tests.pl (create_command): Call cmd2str() to format
the command. Add the filename/line number to the output.
When MAKEFLAGS is set in a makefile, reparse it immediately rather
than waiting until after all makefiles have been read and parsed.
This change doesn't actually fix the SV bug referenced because, even
though we do reparse MAKEFLAGS, we don't handle the -r or -R options
immediately. Doing this will require more effort.
* NEWS: Announce the change.
* src/makeint.h: Publish reset_switches() and decode_env_switches()
from main.c
* src/main.c (main): Don't call construct_include_path(); it will be
invoked decode_switches().
Preserve the old values of builtin_rules, builtin_variables, and
job_slots before we read makefiles since they can be changed now.
(reset_switches): Publish (remove static). Set the initial value of
the stringlist list to NULL.
(decode_switches): Call construct_include_path() after decoding.
(decode_env_switches): Publish (remove static).
(define_makeflags): Set the MAKEFLAGS variable for special handling.
* src/read.c (eval_makefile): Check for empty include_directories.
(construct_include_path): Clear any old value of .INCLUDE_DIRS before
appending new values. Free the previous include_directories.
* src/variable.c (lookup_special_var): When MAKEFLAGS is set, first
reset the switches then re-parse the variable.
* tests/run_make_tests.pl: Memo-ize some default variable values.
* tests/scripts/options/dash-r: Create tests for setting -r and -R.
* tests/scripts/variables/MAKEFLAGS: Test that resetting -I from
within the makefile takes effect immediately.
Previously if an included makefile was rebuilt as a prerequisite of
another included makefile which didn't need to be rebuilt, make would
not realize that it needed to re-exec itself.
Ensure that if any included makefile target is rebuilt we re-exec.
Also ensure that if an included makefile is not readable, and our rule
for rebuilding it doesn't actually change it, we will still fail.
* src/remake.c (update_goal_chain): If a goal's update was successful
then check its status, even if no actual commands were run because it
was already up to date.
(show_goal_error): Remove superfluous cast.
* src/main.c (main): If the makefile remake did nothing, check that we
were able to successfully include all the makefiles we care about; if
not fail. When generating error messages about included makefiles be
sure to show the filename/linenumber information.
* test/scripts/features/reinvoke: Add tests for this behavior.
* test/scripts/options/dash-k: Update error messages.
Accept a "-" directory value to the -I option to clear the set of
directories to be searched up to that point, including the default
directories.
* NEWS: Announce the change.
* doc/make.texi (Summary of Options): Add documentation.
* src/read.c (construct_include_path): Check for '-' and if found,
clear the list of directories to be searched.
* tests/scripts/options/dash-I: Add tests for -I-.
* tests/scripts/variables/INCLUDE_DIRS: Add tests for -I-.
Rather than having an %extraENV that is added to the default %ENV
and resetting %ENV _before_ each test, allow the test setup to
modify %ENV directly as needed then reset %ENV _after_ each test.
* tests/test_driver.pl: Remove unused %extraENV.
(resetENV): Don't add in %extraENV.
(_run_command): Reset after we run the command rather than before.
* tests/scripts/features/export: Convert %extraENV to %ENV
* tests/scripts/features/jobserver: Ditto
* tests/scripts/features/parallelism: Ditto
* tests/scripts/features/targetvars: Ditto
* tests/scripts/functions/eval: Ditto
* tests/scripts/functions/foreach: Ditto
* tests/scripts/functions/origin: Ditto
* tests/scripts/misc/general4: Ditto
* tests/scripts/options/dash-e: Ditto
* tests/scripts/targets/POSIX: Ditto
* tests/scripts/variables/GNUMAKEFLAGS: Ditto
* tests/scripts/variables/SHELL: Ditto
Previously if --no-print-directory was seen anywhere even once
(environment, command line, etc.) it would always take precedence
over any --print-directory option. Change this so that the last
seen option (which will be the command line, if present there) takes
precedence.
* NEWS: Mark this change in behavior.
* src/makeint.h (print_directory): A new variable to control printing.
* src/output.c (output_dump): Use the new variable.
(output_start): Ditto.
* src/main.c: Add a new variable print_directory. Use -1 for
print_directory_flag so we know of the option was seen or not. Add a
new default_print_directory_flag set to -1 to keep options from being
added.
(switches): Use flag_off for --no-print-directory, rather than a
separate inhibit_print_directory_flag.
(main): If print_directory_flag was set by the user, use that for
print_directory. If not, compute the print_directory value based on
-s, -C, and sub-makes as before.
* tests/scripts/variables/GNUMAKEFLAGS: -w is not added automatically
* tests/scripts/options/print-directory: Add tests for overriding
print-directory options.
This reverts commit 6264deece3.
Further investigation discovers that the real issue is that
GNU Emacs compile mode doesn't have a matching regex for GNU
make error messages generated when targets fail. I submitted
a patch to GNU Emacs adding a matcher for compile mode.
Go through both run_make_tests.pl and test_driver.pl and slightly
modernize the Perl and clean up indentation etc. Fix a number of
warnings in the test scripts detected by running with -w.
* tests/test_driver.pl: Move make error string detection out of the
base test driver.
(run_all_tests): Ensure that we always look for tests in the cwd.
* tests/run_make_tests.pl: Use File::Spec for path manipulations.
Correctly use setlocale() when detecting error strings.
Get configuration from the config-flags.pm file not config.status.
* tests/scripts/features/archives: Use new $cwddir variable.
* tests/scripts/features/reinvoke: Add missing semicolon.
* tests/scripts/features/vpath2: Avoid non-existent variable.
* tests/scripts/functions/foreach: Escape variables.
* tests/scripts/misc/bs-nl: Remove non-existing \v escape sequence.
* tests/scripts/misc/general4: Use handy create_file().
* tests/scripts/options/dash-C: Use Cwd/$cwddir.
* tests/scripts/options/dash-I: Use subst_make_string() and #PWD#.
* tests/scripts/options/symlinks: Use File::Spec.
* tests/scripts/targets/DEFAULT: Use create_file and run_make_test.
* tests/scripts/variables/CURDIR: Use run_make_test.
* tests/scripts/variables/automatic: Remove extraneous "\".
* tests/scripts/vms/library: Remove extra "my" and extraneous "\".
* main.c (main): Sanitize program name detection on Windows.
* makeint.h: 'program' is a const string on all platforms now.
* tests/run_make_tests.bat: Windows bat file to invoke tests
* tests/test_driver.pl: Obtain system-specific error messages.
(get_osname): Compute the $port_type here. Add more $osname checks
for different Windows Perl ports.
(_run_command): Rewrite the timeout capability to work properly
with Windows. Don't use Perl fork/exec; instead use system(1,...)
which allows a more reliable/proper kill operation.
Also, allow options to be given as a list instead of a string, to
allow more complex quoting of command-line arguments.
* tests/run_make_tests.pl (run_make_with_options): Allow options
to be provided as a list in addition to a simple string.
(set_more_defaults): Write sample makefiles and run make on them
instead of trying to run echo and invoking make with -f-, to avoid
relying on shell and echo to get basic configuration values. Also
create a $sh_name variable instead of hard-coding /bin/sh.
* tests/scripts/features/archives: Skip on Windows.
* tests/scripts/features/escape: Use list method for passing options.
* tests/scripts/features/include: Use system-specific error messages.
* tests/scripts/features/output-sync: "Command not found" errors
generate very different / odd output on Windows. This needs to be
addressed but for now disable these tests on Windows.
* tests/scripts/functions/abspath: Disable on Windows.
* tests/scripts/functions/file: Use system-specific error messages.
* tests/scripts/functions/shell: "Command not found" errors generate
very different / odd output on Windows. This needs to be addressed
but for now disable these tests on Windows.
* tests/scripts/misc/close_stdout: Disable on Windows.
* tests/scripts/options/dash-k: Use system-specific error messages.
* tests/scripts/options/dash-l: Disable on Windows.
* tests/scripts/options/eval: Use list method for passing options.
* tests/scripts/options/general: Skip some non-portable tests.
* tests/scripts/targets/ONESHELL: Skip some non-portable tests.
* tests/scripts/targets/POSIX: Skip some non-portable tests.
* tests/scripts/variables/MAKEFILES: Skip some non-portable tests.
* tests/scripts/variables/SHELL: Use a makefile not -f- for testing.
* make.1: Document the -E and --eval options.
* doc/make.texi: Document the -E option.
* tests/scripts/options/eval: Test the -E option and MAKEFILES.
* NEWS: Add information about the new option.
* make.1: Document the new flag.
* doc/make.texi: Document the new flag. Remove suggestions that the
.SILENT special target is deprecated or should not be used.
* tests/scripts/options/dash-s: Test the -s and --no-silent options.
* NEWS: Add information about the new option.
* tests/test_driver.pl: Save error strings for later comparison.
* tests/run_make_tests.pl: Create portable commands for later use.
* tests/*: Use these new variables.
Delay the generation of error messages for included files until we
are sure that we can't rebuild that included file.
* dep.h (struct dep): Don't reuse "changed"; make a separate field
to keep "flags". Get rid of dontcare and use the flag.
(struct goaldep): Create a new structure for goal prereqs
that tracks an errno value and the floc where the include happened.
Rework the structures to ensure they are supersets as expected.
In maintainer mode with GCC, use inline to get type checking.
* read.c (eval_makefile): Return a struct goaldep for the new
makefile. Ensure errno is set properly to denote a failure.
(read_all_makefiles): Switch to goaldep and check errno.
(eval): Don't show included file errors; instead remember them.
* remake.c (update_goal_chain): Set global variables to the current
goaldep we're building, and the entire chain.
(show_goal_error): Check if the current failure is a consequence
of building an included makefile and if so print an error.
(complain): Call show_goal_error() on rule failure.
* job.c (child_error): Call show_goal_error() on child error.
* main.c (main): Switch from struct dep to goaldep.
* misc.c (free_dep_chain): Not used; make into a macro.
* tests/scripts/features/include: Update and include new tests.
* tests/scripts/options/dash-B, tests/scripts/options/dash-W,
tests/scripts/options/print-directory,
tests/scripts/variables/MAKE_RESTARTS: Update known-good-output.
* job.h (struct child): New bit to mark recursive command lines.
* job.c (start_job_command): Set the recursive command line bit.
(reap_children): If the child is a recursive command and it exits
with 1 during question mode, don't print an error and exit with 1.
* tests/scripts/options/dash-q: Add a regression test.
* remake.c (update_file, update_file_1, check_dep): Return an enum
update_status value instead of an int, and keep the highest value we
find as we walk the graph so that the ultimate status is correct.
* tests/scripts/options/dash-q: Add a test for updating prerequisites.
Add a new feature to the test suite suggested by Icarus Sparry:
set a timer before invoking a test, so that if it loops infinitely we
will wake up and have a chance to kill the process and continue.
Revert a fix for $? including non-existent files as it shows a bug
in the Linux kernel build. Give them a release to fix this.
Add some changes from Eli Z. for Windows changes.
Solaris system with an EMC NFS storage solution. Still get some odd
errors here unfortunately related to sub-second timestamps that I just
can't figure out. It all works if we run the tests in /tmp instead
though :-/.
