Add support for per-job output sync.

A new flag to the -O/--output-sync, "job", selects a per-job (that is, per
line of a recipe) output synchronization.  To support this move the close of
the temp file out of the sync_output() function and don't do it until we free
the child, since we may call sync_output() multiple times in a given recipe.
When we set up for a new temp file, if we're in per-job mode we truncate the
file and seek to the beginning to re-use it for every job.
This commit is contained in:
Paul Smith 2013-04-28 01:19:19 -04:00
parent 30843dea3a
commit 7f01830927
11 changed files with 319 additions and 104 deletions

View File

@ -37,7 +37,7 @@ GNU make porting efforts:
Port to MS-DOS (DJGPP), OS/2, and MS-Windows (native/MinGW) by:
DJ Delorie <dj@delorie.com>
Rob Tulloh <rob_tulloh@tivoli.com>
Eli Zaretskii <eliz@is.elta.co.il>
Eli Zaretskii <eliz@gnu.org>
Jonathan Grant <jg@jguk.org>
Andreas Beuning <andreas.buening@nexgo.de>
Earnie Boyd <earnie@uses.sf.net>

View File

@ -1,3 +1,21 @@
2013-04-28 Paul Smith <psmith@gnu.org>
Implement a "per-job" output synchronization option.
* main.c (decode_output_sync_flags): Recognize the new option.
* makeint.h (OUTPUT_SYNC_JOB): Add new values for "job"
* job.c (assign_child_tempfiles): In per-job mode, truncate the
temp file for re-use by the next job.
(sync_output): Don't close the temp files as we may still use them.
(free_child): Close the temp files here as we definitely don't
need them.
(new_job): In per-job output mode, sync_output() after each job.
* job.h (struct child): Avoid ifdefs.
* make.1: Add new options to the man page.
* doc/make.texi (Parallel Output): Break documentation on input
and output into separate sections for readability. Document the
new "job" and "none" modes.
2013-04-27 Paul Smith <psmith@gnu.org>
* job.c (construct_command_argv_internal): Fix oneshell support

12
NEWS
View File

@ -1,6 +1,6 @@
GNU make NEWS -*-indented-text-*-
History of user-visible changes.
16 April 2013
27 April 2013
See the end of this file for copyrights and conditions.
@ -31,13 +31,17 @@ http://sv.gnu.org/bugs/index.php?group=make&report_id=111&fix_release_id=101&set
* New command line option: --output-sync (-O) enables grouping of output by
target or by recursive make. This is useful during parallel builds to avoid
mixing output from different jobs together giving hard-to-understand results.
Original implementation by David Boyce <dsb@boyski.com>. Patch was reworked
by Frank Heckenbach <f.heckenbach@fh-soft.de>.
mixing output from different jobs together giving hard-to-understand
results. Original implementation by David Boyce <dsb@boyski.com>.
Reworked and enhanced by Frank Heckenbach <f.heckenbach@fh-soft.de>.
Windows support by Eli Zaretskii <eliz@gnu.org>.
* New feature: The "job server" capability is now supported on Windows.
Implementation contributed by Troy Runkel <Troy.Runkel@mathworks.com>
* New feature: The .ONESHELL capability is now supported on Windows. Support
added by Eli Zaretskii <eliz@gnu.org>.
* New feature: "!=" shell assignment operator as an alternative to the
$(shell ...) function. Implemented for compatibility with BSD makefiles.
WARNING: Backward-incompatibility!

View File

@ -219,6 +219,11 @@ Recipe Execution
* Choosing the Shell:: How @code{make} chooses the shell used
to run recipes.
Parallel Execution
* Parallel Output:: Handling output during parallel execution
* Parallel Input:: Handling input during parallel execution
Recursive Use of @code{make}
* MAKE Variable:: The special effects of using @samp{$(MAKE)}.
@ -4057,48 +4062,16 @@ If there is nothing looking like an integer after the @samp{-j} option,
there is no limit on the number of job slots. The default number of job
slots is one, which means serial execution (one thing at a time).
When running several recipes simultaneously the output from each
recipe appears as soon as it is generated, with the result that
messages from different recipes may be interspersed. To avoid this
you can use the @samp{--output-sync} (@samp{-O}) option; if this
option is provided then the output from each recipe will be saved
until the recipe is complete, then written all at once. This ensures
that output from different recipes is not mixed together.
Another problem is that two processes cannot both take input from the
same device; so to make sure that only one recipe tries to take input
from the terminal at once, @code{make} will invalidate the standard
input streams of all but one running recipe. This means that
attempting to read from standard input will usually be a fatal error (a
@samp{Broken pipe} signal) for most child processes if there are
several.
@cindex broken pipe
@cindex standard input
It is unpredictable which recipe will have a valid standard input stream
(which will come from the terminal, or wherever you redirect the standard
input of @code{make}). The first recipe run will always get it first, and
the first recipe started after that one finishes will get it next, and so
on.
We will change how this aspect of @code{make} works if we find a better
alternative. In the mean time, you should not rely on any recipe using
standard input at all if you are using the parallel execution feature; but
if you are not using this feature, then standard input works normally in
all recipes.
Finally, handling recursive @code{make} invocations raises issues. For
more information on this, see
@ref{Options/Recursion, ,Communicating Options to a Sub-@code{make}}.
Handling recursive @code{make} invocations raises issues for parallel
execution. For more information on this, see @ref{Options/Recursion,
,Communicating Options to a Sub-@code{make}}.
If a recipe fails (is killed by a signal or exits with a nonzero
status), and errors are not ignored for that recipe
(@pxref{Errors, ,Errors in Recipes}),
the remaining recipe lines to remake the same target will not be run.
If a recipe fails and the @samp{-k} or @samp{--keep-going}
option was not given
(@pxref{Options Summary, ,Summary of Options}),
@code{make} aborts execution. If make
status), and errors are not ignored for that recipe (@pxref{Errors,
,Errors in Recipes}), the remaining recipe lines to remake the same
target will not be run. If a recipe fails and the @samp{-k} or
@samp{--keep-going} option was not given (@pxref{Options Summary,
,Summary of Options}), @code{make} aborts execution. If make
terminates for any reason (including a signal) with child processes
running, it waits for them to finish before actually exiting.@refill
@ -4131,6 +4104,105 @@ average goes below that limit, or until all the other jobs finish.
By default, there is no load limit.
@menu
* Parallel Output:: Handling output during parallel execution
* Parallel Input:: Handling input during parallel execution
@end menu
@node Parallel Output, Parallel Input, Parallel, Parallel
@subsection Output During Parallel Execution
@cindex output during parallel execution
@cindex parallel execution, output during
When running several recipes in parallel the output from each
recipe appears as soon as it is generated, with the result that
messages from different recipes may be interspersed, sometimes even
appearing on the same line. This can make reading the output very
difficult.
@cindex @code{--output-sync}
@cindex @code{-O}
To avoid this you can use the @samp{--output-sync} (@samp{-O}) option.
This option instructs @code{make} to save the output from the commands
it invokes and print it all once the commands are completed.
Additionally, if there are multiple recursive @code{make} invocations
running in parallel, they will communicate so that only one of them is
generating output at a time.
There are four levels of granularity when synchronizing output,
specified by giving an argument to the option (e.g., @samp{-Ojob} or
@samp{--output-sync=make}).
@table @code
@item none
The is the default: all output is sent directly as it is generated and
no synchronization is performed.
@item job
Output from each individual line of the recipe is grouped and printed
as soon as that line is complete. If a recipe consists of multiple
lines, they may be interspersed with lines from other recipes.
@item target
Output from the entire recipe for each target is grouped and printed
once the target is complete. This is the default if the
@code{--output-sync} or @code{-O} option is given with no argument.
@item make
Output from each recursive invocation of @code{make} is grouped and
printed once the recursive invocation is complete.
@end table
Regardless of the mode chosen, the total build time will be the same.
The only difference is in how the output appears.
The @samp{make} mode provides the most comprehensive grouping,
allowing output from all targets built by a given makefile to appear
together. However, there will be long interludes during the build
where no output appears while a recursive @code{make} is running,
followed by a burst of output. This mode is best for builds being
performed in the background, where the output will be examined later.
The @samp{job} mode is mainly useful for front-ends that may be
watching the output of @code{make} and looking for certain generated
output to determine when recipes are started and completed.
You should be aware that some programs may act differently when they
determine they're writing output to a terminal versus a file
(typically described as ``interactive'' vs. ``non-interactive''
modes). If your makefile invokes a program like this then using the
output synchronization options will cause the program to believe it's
running in ``non-interactive'' mode even when it's writing to the
terminal. Of course, invoking @code{make} with output redirected to a
file will elicit the same behavior.
@node Parallel Input, , Parallel Output, Parallel
@subsection Input During Parallel Execution
@cindex input during parallel execution
@cindex parallel execution, input during
@cindex standard input
Two processes cannot both take input from the same device at the same
time. To make sure that only one recipe tries to take input from the
terminal at once, @code{make} will invalidate the standard input
streams of all but one running recipe. If another recipe attempts to
read from standard input it will usually incur a fatal error (a
@samp{Broken pipe} signal).
@cindex broken pipe
It is unpredictable which recipe will have a valid standard input stream
(which will come from the terminal, or wherever you redirect the standard
input of @code{make}). The first recipe run will always get it first, and
the first recipe started after that one finishes will get it next, and so
on.
We will change how this aspect of @code{make} works if we find a better
alternative. In the mean time, you should not rely on any recipe using
standard input at all if you are using the parallel execution feature; but
if you are not using this feature, then standard input works normally in
all recipes.
@node Errors, Interrupts, Parallel, Recipes
@section Errors in Recipes
@cindex errors (in recipes)
@ -8628,16 +8700,19 @@ uninterrupted sequence. This option is only useful when using the
(@pxref{Parallel, ,Parallel Execution}) Without this option output
will be displayed as it is generated by the recipes.@refill
With no type or the type @samp{target}, output from each individual
target is grouped together. With the type @samp{make}, the output
from an entire recursive make is grouped together. The latter
achieves better grouping of output from related jobs, but causes
longer delay since messages do not appear until the entire recursive
make has completed (this does not increase the total build time,
though). In general @samp{target} mode is useful when watching the
output while make runs, and @samp{make} mode is useful when running a
complex parallel build in the background and checking its output
afterwards.
With no type or the type @samp{target}, output from the entire recipe
of each target is grouped together. With the type @samp{job}, output
from each job in the recipe is grouped together. With the type
@samp{make}, the output from an entire recursive make is grouped
together. The latter achieves better grouping of output from related
jobs, but causes longer delay since messages do not appear until the
entire recursive make has completed (this does not increase the total
build time, though). In general @samp{target} mode is useful when
watching the output while make runs, and @samp{make} mode is useful
when running a complex parallel build in the background and checking
its output afterwards. The @samp{job} mode may be helpful for tools
which watch the output to determine when recipes have started or
stopped.
@item -q
@cindex @code{-q}
@ -10886,7 +10961,7 @@ will load the dynamic object @file{../mk_funcs.so}. After the object
is loaded, @code{make} will invoke the function @code{init_mk_func}.
Regardless of how many times an object file appears in a @code{load}
directive, it will only be loaded (and it's setup function will only
directive, it will only be loaded (and its setup function will only
be invoked) once.
@vindex .LOADED

67
job.c
View File

@ -557,24 +557,41 @@ child_handler (int sig UNUSED)
static int
assign_child_tempfiles (struct child *c, int combined)
{
/* If we already have a temp file assigned we're done. */
if (c->outfd != -1 && c->errfd != -1)
return 1;
if (STREAM_OK (stdout))
/* If we don't have a temp file, get one. */
if (c->outfd < 0 && c->errfd < 0)
{
c->outfd = open_tmpfd ();
CLOSE_ON_EXEC (c->outfd);
if (STREAM_OK (stdout))
{
c->outfd = open_tmpfd ();
CLOSE_ON_EXEC (c->outfd);
}
if (STREAM_OK (stderr))
{
if (c->outfd >= 0 && combined)
c->errfd = c->outfd;
else
{
c->errfd = open_tmpfd ();
CLOSE_ON_EXEC (c->errfd);
}
}
return 1;
}
if (STREAM_OK (stderr))
/* We already have a temp file. On per-job output, truncate and reset. */
if (output_sync == OUTPUT_SYNC_JOB)
{
if (c->outfd >= 0 && combined)
c->errfd = c->outfd;
else
if (c->outfd >= 0)
{
c->errfd = open_tmpfd ();
CLOSE_ON_EXEC (c->errfd);
lseek(c->outfd, 0, SEEK_SET);
ftruncate(c->outfd, 0);
}
if (c->errfd >= 0 && c->errfd != c->outfd)
{
lseek(c->errfd, 0, SEEK_SET);
ftruncate(c->outfd, 0);
}
}
@ -690,14 +707,8 @@ sync_output (struct child *c)
/* Exit the critical section. */
if (sem)
release_semaphore (sem);
release_semaphore (sem);
}
if (c->outfd >= 0)
close (c->outfd);
if (c->errfd >= 0)
close (c->errfd);
c->outfd = c->errfd = -1;
}
#endif /* OUTPUT_SYNC */
@ -995,6 +1006,11 @@ reap_children (int block, int err)
c->sh_batch_file = NULL;
}
#ifdef OUTPUT_SYNC
if (output_sync == OUTPUT_SYNC_JOB)
sync_output (c);
#endif
/* If this child had the good stdin, say it is now free. */
if (c->good_stdin)
good_stdin_used = 0;
@ -1073,7 +1089,7 @@ reap_children (int block, int err)
#ifdef OUTPUT_SYNC
/* Synchronize parallel output if requested */
if (output_sync)
if (output_sync > OUTPUT_SYNC_JOB)
sync_output (c);
#endif /* OUTPUT_SYNC */
@ -1131,6 +1147,11 @@ reap_children (int block, int err)
static void
free_child (struct child *child)
{
if (child->outfd >= 0)
close (child->outfd);
if (child->errfd >= 0 && child->errfd != child->outfd)
close (child->errfd);
if (!jobserver_tokens)
fatal (NILF, "INTERNAL: Freeing child %p (%s) but no tokens left!\n",
child, child->file->name);
@ -1613,7 +1634,7 @@ start_job_command (struct child *child)
/* If it still looks like we can synchronize, create a temp
file to hold stdout (and one for stderr if separate). */
if (output_sync == OUTPUT_SYNC_MAKE
|| (output_sync == OUTPUT_SYNC_TARGET && !(flags & COMMANDS_RECURSE)))
|| (output_sync && !(flags & COMMANDS_RECURSE)))
{
if (!assign_child_tempfiles (child, combined_output))
output_sync = 0;
@ -2035,9 +2056,7 @@ new_job (struct file *file)
c->file = file;
c->command_lines = lines;
c->sh_batch_file = NULL;
#ifdef OUTPUT_SYNC
c->outfd = c->errfd = -1;
#endif
/* Cache dontcare flag because file->dontcare can be changed once we
return. Check dontcare inheritance mechanism for details. */

8
job.h
View File

@ -99,16 +99,14 @@ struct child
#endif
unsigned int command_line; /* Index into command_lines. */
pid_t pid; /* Child process's ID number. */
int outfd; /* File descriptor for saving stdout */
int errfd; /* File descriptor for saving stderr */
pid_t pid; /* Child process's ID number. */
unsigned int remote:1; /* Nonzero if executing remotely. */
unsigned int noerror:1; /* Nonzero if commands contained a '-'. */
unsigned int good_stdin:1; /* Nonzero if this child has a good stdin. */
unsigned int deleted:1; /* Nonzero if targets have been deleted. */
unsigned int dontcare:1; /* Saved dontcare flag. */
#ifdef OUTPUT_SYNC
int outfd; /* File descriptor for saving stdout */
int errfd; /* File descriptor for saving stderr */
#endif
};
extern struct child *children;

6
main.c
View File

@ -695,7 +695,11 @@ decode_output_sync_flags (void)
{
const char *p = *pp;
if (streq (p, "target"))
if (streq (p, "none"))
output_sync = OUTPUT_SYNC_NONE;
else if (streq (p, "job"))
output_sync = OUTPUT_SYNC_JOB;
else if (streq (p, "target"))
output_sync = OUTPUT_SYNC_TARGET;
else if (streq (p, "make"))
output_sync = OUTPUT_SYNC_MAKE;

13
make.1
View File

@ -227,11 +227,20 @@ other jobs. If
.I type
is not specified or is
.B target
output is grouped together on a per-target basis. If
the output from the entire recipe for each target is grouped together. If
.I type
is
.B job
the output from each job within a recipe is grouped together.
If
.I type
is
.B make
output from an entire recursive make is grouped together.
output from an entire recursive make is grouped together. If
.I type
is
.B none
output synchronization is disabled.
.TP 0.5i
\fB\-p\fR, \fB\-\-print\-data\-base\fR
Print the data base (rules and variable values) that results from

View File

@ -538,8 +538,10 @@ int strncasecmp (const char *s1, const char *s2, int n);
# define OUTPUT_SYNC
#endif
#define OUTPUT_SYNC_TARGET 1
#define OUTPUT_SYNC_MAKE 2
#define OUTPUT_SYNC_NONE 0
#define OUTPUT_SYNC_JOB 1
#define OUTPUT_SYNC_TARGET 2
#define OUTPUT_SYNC_MAKE 3
extern const gmk_floc *reading_file;
extern const gmk_floc **expanding_var;

View File

@ -1,3 +1,8 @@
2013-04-28 Paul Smith <psmith@gnu.org>
* scripts/features/output-sync (output_sync_set): Add tests for
the per-job syntax mode.
2013-04-15 Paul Smith <psmith@gnu.org>
* scripts/features/output-sync (output_sync_set): New arg syntax.

View File

@ -15,6 +15,18 @@ else {
$sleep_command = "sleep";
}
# The following subdirectories with Makefiles are used in several
# of the following tests. The model is:
# foo/Makefile - has a "foo" target that waits for the bar target
# bar/Makefile - has a "bar" target that runs immediately
# - has a "baz" target that waits for the foo target
#
# So, you start the two sub-makes in parallel and first the "bar" target is
# built, followed by "foo", followed by "baz". The trick is that first each
# target prints a "start" statement, then waits (if appropriate), then prints
# an end statement. Thus we can tell if the -O flag is working, since
# otherwise these statements would be mixed together.
@syncfiles = ();
sub output_sync_clean {
@ -36,19 +48,21 @@ sub output_sync_set {
return "date > ../mksync.$_[0]";
}
@syncfiles = qw(mksync.foo mksync.bar);
@syncfiles = qw(mksync.foo mksync.foo_start mksync.bar mksync.bar_start);
# The following subdirectories with Makefiles are used in several
# of the following tests.
output_sync_clean();
mkdir('foo', 0777);
mkdir('bar', 0777);
$set_foo = output_sync_set('foo');
$set_bar = output_sync_set('bar');
$set_foo_start = output_sync_set('foo_start');
$set_bar_start = output_sync_set('bar_start');
$wait_foo = output_sync_wait('foo');
$wait_bar = output_sync_wait('bar');
$wait_foo_start = output_sync_set('foo_start');
$wait_bar_start = output_sync_set('bar_start');
open(MAKEFILE,"> foo/Makefile");
print MAKEFILE <<EOF;
@ -56,9 +70,25 @@ all: foo
foo: foo-base ; \@$set_foo
foo-base: ; \@echo foo: start; $wait_bar ; echo foo: end
foo-base:
\t\@echo foo: start
\t\@$wait_bar
\t\@echo foo: end
foo-fail: ; \@$wait_bar ; false
foo-job: foo-job-base ; \@$set_foo
foo-job-base:
\t\@$wait_bar_start
\t\@echo foo: start
\t\@$set_foo_start
\t\@$wait_bar
\t\@echo foo: end
foo-fail:
\t\@echo foo-fail: start
\t\@$wait_bar
\t\@echo foo-fail: end
\t\@false
EOF
close(MAKEFILE);
@ -66,15 +96,28 @@ open(MAKEFILE,"> bar/Makefile");
print MAKEFILE <<EOF;
all: bar baz
bar: ; \@echo bar: start; echo bar: end; $set_bar
bar: bar-base ; \@$set_bar
bar-base:
\t\@echo bar: start
\t\@echo bar: end
bar-job: bar-job-base ; \@$set_bar
bar-job-base:
\t\@echo bar: start
\t\@$set_bar_start
\t\@$wait_foo_start
\t\@echo bar: end
baz: baz-base
baz-base: ; \@echo baz: start; $wait_foo; echo baz: end
baz-base:
\t\@echo baz: start
\t\@$wait_foo
\t\@echo baz: end
EOF
close(MAKEFILE);
# Test coarse synchronization.
# Test per-make synchronization.
unlink(@syncfiles);
run_make_test(qq!
all: make-foo make-bar
@ -102,7 +145,7 @@ baz: end
#MAKE#[1]: Leaving directory '#PWD#/bar'
#MAKE#[1]: Leaving directory '#PWD#/bar'\n", 0, 6);
# Test fine synchronization.
# Test per-target synchronization.
# Note we have to sleep again here after starting the foo makefile before
# starting the bar makefile, otherwise the "entering/leaving" messages for the
# submakes might be ordered differently than we expect.
@ -154,16 +197,54 @@ bar: end
#MAKE#[1]: Leaving directory '#PWD#/bar'
#MAKE#[1]: Leaving directory '#PWD#/bar'
#MAKE#[1]: Entering directory '#PWD#/foo'
Makefile:7: recipe for target 'foo-fail' failed
#MAKE#[1]: Leaving directory '#PWD#/foo'
#MAKE#[1]: Entering directory '#PWD#/foo'
#MAKE#[1]: *** [foo-fail] Error 1
Makefile:20: recipe for target 'foo-fail' failed
make[1]: Leaving directory '/home/psmith/src/make/make/tests/foo'
make[1]: Entering directory '/home/psmith/src/make/make/tests/foo'
make[1]: *** [foo-fail] Error 1
make[1]: Leaving directory '/home/psmith/src/make/make/tests/foo'
make[1]: Entering directory '/home/psmith/src/make/make/tests/foo'
foo-fail: start
foo-fail: end
#MAKE#[1]: Leaving directory '#PWD#/foo'
#MAKE#[1]: Leaving directory '#PWD#/foo'
#MAKEFILE#:4: recipe for target 'make-foo-fail' failed
#MAKE#: *** [make-foo-fail] Error 2\n",
512);
# Test the per-job synchronization.
# For this we'll have bar-job:
# print start, invoke bar-start, wait for foo-start, print end, print-bar-end
# And foo-job:
# wait for bar-start, print foo-start, wait for bar-end, print end
unlink(@syncfiles);
run_make_test(qq!
all: make-foo make-bar
make-foo: ; \$(MAKE) -C foo foo-job
make-bar: ; $sleep_command 1 ; \$(MAKE) -C bar bar-job!,
'-j --output-sync=job',
"#MAKEPATH# -C foo foo-job
$sleep_command 1 ; #MAKEPATH# -C bar bar-job
#MAKE#[1]: Entering directory '#PWD#/foo'
#MAKE#[1]: Entering directory '#PWD#/foo'
foo: start
#MAKE#[1]: Leaving directory '#PWD#/foo'
#MAKE#[1]: Entering directory '#PWD#/bar'
#MAKE#[1]: Entering directory '#PWD#/bar'
bar: start
#MAKE#[1]: Leaving directory '#PWD#/bar'
#MAKE#[1]: Entering directory '#PWD#/bar'
bar: end
#MAKE#[1]: Leaving directory '#PWD#/bar'
#MAKE#[1]: Leaving directory '#PWD#/bar'
#MAKE#[1]: Entering directory '#PWD#/foo'
foo: end
#MAKE#[1]: Leaving directory '#PWD#/foo'
#MAKE#[1]: Leaving directory '#PWD#/foo'\n", 0, 6);
# Remove temporary directories and contents.
output_sync_clean();