make/tests
Dmitry Goncharov dc2d963989 [SV 63347] Always add command line variable assignments to MAKEFLAGS
This commit introduces two visible changes:
1. Keep command line variable assignments in MAKEFLAGS at all times,
   even while parsing makefiles.
2. Define makeflags immediately when a makefile modifies MAKEFLAGS.

The new MAKEFLAGS and MAKEOVERRIDES initialization procedure:
1. decode_switches (argc, argv, o_command) is called to parse command
   line variable assignments.
2. Command line variable assignments go through quote_for_env.
   Initialize -*-command-variables-*- to the quoted values.
3. MAKEOVERRIDES is initialized to refer to -*-command-variables-*-
   with origin o_env to keep the definitions in the database intact.
4. define_makeflags() is called which adds MAKEOVERRIDES to MAKEFLAGS.
5. Makefiles are parsed.  If a makefile modifies MAKEFLAGS, the new
   value of MAKEFLAGS is defined right away.
6. Env switches are decoded again as o_env.  The definitions set by
   decode_switches at step 1 stay intact, as o_command beats o_env.

We must preserve the original intact definitions in order to detect
failure cases; for example:
  $ cat makefile
  all:; $(hello)
  $ make hello='$(world'
  makefile:1: *** unterminated variable reference.  Stop.

* src/makeint.h: Declare enum variable_origin, struct variable and
define_makeflags().  Add parameter origin to decode_env_switches().
* src/main.c (define_makeflags): Remove "all". If a variable is
assigned on the command line then append MAKEOVERRIDES to MAKEFLAGS.
(decode_env_switches): Replace parameter env with origin.
(decode_switches): Replace parameter env with origin.
Treat origin == o_command as env == 0.
(handle_non_switch_argument): Replace parameter env with origin.
Treat origin == o_command as env == 0.
(main): Call decode_switches() with origin==o_command before parsing
makefiles.  Call decode_switches() with origin==o_env after parsing
makefiles.
* src/variable.c (set_special_var): Define makeflags at parse time,
each time a makefile modifies MAKEFLAGS.
(do_variable_definition): Strip command line variable assignments from
MAKEFLAGS before appending extra flags.  set_special_var() adds them
back.
* tests/scripts/variables/MAKEFLAGS: Add tests.
2022-11-28 10:50:55 -05:00
..
scripts [SV 63347] Always add command line variable assignments to MAKEFLAGS 2022-11-28 10:50:55 -05:00
.gitignore Collect failure details when the regression tests fail 2022-10-22 22:40:26 -04:00
ChangeLog.1 Update URLs to use https rather than http 2022-10-18 14:37:47 -04:00
config_flags_pm.com Update URLs to use https rather than http 2022-10-18 14:37:47 -04:00
config-flags.pm.in * tests/config-flags.pm.in: Add USE_SYSTEM_GLOB. 2019-09-16 08:25:33 -04:00
config-flags.pm.W32 tests: Convert %CONFIG_FLAGS to get_config() 2019-10-05 17:47:52 -04:00
guile.supp Fix Savannah bug #35410: handle escape chars in filter/filter-out 2012-03-03 22:12:46 +00:00
mkshadow Update URLs to use https rather than http 2022-10-18 14:37:47 -04:00
NEWS Update URLs to use https rather than http 2022-10-18 14:37:47 -04:00
README tests: Avoid the use of File::Temp->newdir() 2022-10-29 13:36:21 -04:00
run_make_tests * Added the test suite to the main distribution. 1999-09-14 02:03:19 +00:00
run_make_tests.bat Update URLs to use https rather than http 2022-10-18 14:37:47 -04:00
run_make_tests.com Update URLs to use https rather than http 2022-10-18 14:37:47 -04:00
run_make_tests.pl tests: Don't convert \ to / when checking regex's 2022-11-13 10:39:48 -05:00
test_driver.pl tests: Don't convert \ to / when checking regex's 2022-11-13 10:39:48 -05:00
thelp.pl [SV 63315] tests: Simplify TERM signaling 2022-11-13 10:39:48 -05:00

The test suite was originally written by Steve McGee and Chris Arthur.
It is covered by the GNU General Public License (Version 2), described
in the file COPYING.  It has been maintained as part of GNU make proper
since GNU make 3.78.

This entire test suite, including all test files, are copyright and
distributed under the following terms:

 -----------------------------------------------------------------------------
 Copyright (C) 1992-2022 Free Software Foundation, Inc.
 This file is part of GNU Make.

 GNU Make is free software; you can redistribute it and/or modify it under the
 terms of the GNU General Public License as published by the Free Software
 Foundation; either version 3 of the License, or (at your option) any later
 version.

 GNU Make is distributed in the hope that it will be useful, but WITHOUT ANY
 WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR
 A PARTICULAR PURPOSE.  See the GNU General Public License for more details.

 You should have received a copy of the GNU General Public License along with
 this program.  If not, see <http://www.gnu.org/licenses/>.
 -----------------------------------------------------------------------------

The test suite requires Perl.  These days, you should have at least Perl 5.6.
Newer versions may be required: I don't test regularly with older versions
than what is installed by default on my development systems.

The test suite assumes that the first "diff" it finds on your PATH is
GNU diff, but that only matters if a test fails.

To run the test suite on a UNIX system, use "perl ./run_make_tests"
(or just "./run_make_tests" if you have a perl on your PATH).

To run the test suite on Windows NT or DOS systems, use
"perl.exe ./run_make-tests.pl".

By default, the test engine picks up the first executable called "make"
that it finds in your path.  You may use the -make_path option (i.e.,
"perl run_make_tests -make_path /usr/local/src/make-3.78/make") if
you want to run a particular copy.  This now works correctly with
relative paths and when make is called something other than "make" (like
"gmake").

Tests cannot end with a "~" character, as the test suite will ignore any
that do (I was tired of having it run my Emacs backup files as tests :))

Also, sometimes the tests may behave strangely on networked
filesystems.  You can use mkshadow to create a copy of the test suite in
/tmp or similar, and try again.  If the error disappears, it's an issue
with your network or file server, not GNU make (I believe).  This
shouldn't happen very often anymore: I've done a lot of work on the
tests to reduce the impacts of this situation.

A directory named "work" will be created when the tests are run which
will contain any makefiles and "diff" files of tests that fail so that
you may look at them afterward to see the output of make and the
expected result.

There is a -help option which will give you more information about the
other possible options for the test suite.


Open Issues
-----------

The test suite has a number of problems which should be addressed.  One VERY
serious one is that there is no real documentation.  You just have to see the
existing tests.  Use the newer tests: many of the tests haven't been updated
to use the latest/greatest test methods.  See the ChangeLogs for pointers.

The second serious problem is that it's not relocatable: when you build out of
the source tree it creates symlinks, which doesn't work on every system and is
just bogus to boot.

The third serious problem is that it's not parallelizable: it scribbles all
over its installation directory and so can only test one make at a time.

The fourth serious problem is that since the tests scribble all over the same
directory (a) they can interfere with each other and (b) we cannot preserve
the full environment for every test, if it involves creating temporary files
etc. as they must be deleted before the next test.

To solve these the suite should create a separate directory for EVERY test,
local to the build directory, and all temporary files should exist in that
directory.  The directory can be preserved on error, or removed if the test
succeeds (unless --keep is given).


Bugs
----

Any complaints/suggestions/bugs/etc. for the test suite itself should be
handled the same way as normal GNU make bugs/problems (see the README for GNU
make).


                                                Paul D. Smith
						Chris Arthur