mirror of
https://github.com/mirror/make.git
synced 2025-01-14 06:10:12 +08:00
Support implementing the jobserver using named pipes
Using anonymous pipes for jobserver support has some advantages: for example there is nothing on disk that needs to be cleaned up. However it has many obscure problems, related to the fact that in order for it to work we need to ensure these resources are properly passed through to child processes that want to use the jobserver. At the same time we don't want to pass the pipe to process which DON'T know about the jobserver. Other processes can open file descriptors which we then think are our jobserver, but aren't. And, we open the pipe file descriptors in blocking mode which doesn't work for all users. See issues such as SV 57178, SV 57242, and SV 62397 To avoid these issues, use named pipes (on systems where they are available) instead of anonoymous pipes. This simplifies many things: we never need to pass open file descriptors to our children; they can open the jobserver named pipe. We don't need to worry about recursive vs. non-recursive children. Users don't have to "pass through" the resources if they are invoking sub-makes. Each child can open its own file descriptor and set blocking as needed. The downside is the named pipe exists on disk and so must be cleaned up when the "top-level" make instance exits. In order to allow make to continue to be used in build systems where older versions of GNU make, or other tools that want to use the jobserver, but don't understand named pipes, introduce a new option --jobserver-style that allows the user to choose anonymous pipes. * NEWS: Announce the change and the --jobserver-style option. * doc/make.1: Add --jobserver-style documentation. * doc/make.texi (Special Variables): Add missing items to .FEATURES. (Options Summary): Add --jobserver-style. (POSIX Jobserver): Named pipes, changes to --jobserver-auth, and the --jobserver-style option. (Windows Jobserver): Document --jobserver-style for Windows. * configure.ac: Check for mkfifo. * src/config.h-vms.template: Undefined HAVE_MKFIFO. * src/config.h.W32.template: Ditto. * src/main.c: Add jobserver-style as a new command line option. (main): Add jobserver-fifo to .FEATURES if supported. Pass the style option to jobserver_setup(). * src/os.h (jobserver_setup): Accept a style string option. * src/posixos.c (enum js_type): Enumeration of the jobserver style. (js_type): Which style we are currently using. (fifo_name): The path to the named pipe (if in use). (jobserver_setup): If no style is given, or "fifo" is given, set up a named pipe: get a temporary file and use mkfifo() on it, then open it for reading and writing. If something fails fall back to anonymous pipes. (jobserver_parse_auth): Parse jobserver-auth to determine the style. If we are using a named pipe, open it. If we're using anonymous pipes ensure they're valid as before. (jobserver_get_invalid_auth): Don't invalidate the jobserver when using named pipes. (jobserver_clear): Clean up memory used for named pipes. (jobserver_acquire_all): Unlink the named pipe when done. * src/w32/w32os.c (jobserver_setup): Check the style argument. * tests/scripts/features/jobserver: Use --jobserver-style to test the anonymous pipe behavior, and also test named pipe/semaphore behavior. Check invalid jobserver-style options. * tests/scripts/functions/shell: Use --jobserver-style to test the anonymous pipe behavior, and also test named pipe/semaphore behavior.
This commit is contained in:
Paul Smith
parent
09cce75c30
commit
7ad2593b2d
13
NEWS
13
NEWS
@ -74,6 +74,19 @@ https://sv.gnu.org/bugs/index.php?group=make&report_id=111&fix_release_id=109&se
|
||||
described in the makefile.
|
||||
Implementation provided by Sergei Trofimovich <siarheit@google.com>
|
||||
|
||||
* New feature: The --jobserver-style command line option and named pipes
|
||||
A new jobserver method is used on systems where mkfifo(3) is supported.
|
||||
This solves a number of obscure issues related to using the jobserver
|
||||
and recursive invocations of GNU make. This change means that sub-makes
|
||||
will connect to the jobserver even if they are not marked as recursive.
|
||||
It also means that other tools that want to participate in the jobserver
|
||||
will need to be enhanced as described in the GNU make manual.
|
||||
You can force GNU make to use the simple pipe-based jobserver (perhaps if
|
||||
you are integrating with other tools or older versions of GNU make) by
|
||||
adding the '--jobserver-style=pipe' option to the command line of the
|
||||
top-level invocation of GNU make, or via MAKEFLAGS or GNUMAKEFLAGS.
|
||||
To detect this change search for 'jobserver-fifo' in the .FEATURES variable.
|
||||
|
||||
* GNU make has sometimes chosen unexpected, and sub-optimal, chains of
|
||||
implicit rules due to the definition of "ought to exist" in the implicit
|
||||
rule search algorithm, which considered any prerequisite mentioned in the
|
||||
|
||||
@ -133,7 +133,7 @@ AS_IF([test "$ac_cv_func_gettimeofday" = yes],
|
||||
AC_CHECK_FUNCS([strtoll strdup strndup stpcpy memrchr mempcpy umask mkstemp \
|
||||
mktemp fdopen dup dup2 getcwd realpath sigsetmask sigaction \
|
||||
getgroups seteuid setegid setlinebuf setreuid setregid \
|
||||
getrlimit setrlimit setvbuf pipe strerror strsignal \
|
||||
mkfifo getrlimit setrlimit setvbuf pipe strerror strsignal \
|
||||
lstat readlink atexit isatty ttyname pselect posix_spawn \
|
||||
posix_spawnattr_setsigmask])
|
||||
|
||||
|
||||
10
doc/make.1
10
doc/make.1
@ -205,6 +205,16 @@ option is given without an argument,
|
||||
.BR make
|
||||
will not limit the number of jobs that can run simultaneously.
|
||||
.TP 0.5i
|
||||
\fB\--jobserver-style=\fR\fIstyle\fR
|
||||
The style of jobserver to use. The
|
||||
.I style
|
||||
may be one of
|
||||
.BR fifo ,
|
||||
.BR pipe ,
|
||||
or
|
||||
.B sem
|
||||
(Windows only).
|
||||
.TP 0.5i
|
||||
\fB\-k\fR, \fB\-\-keep\-going\fR
|
||||
Continue as much as possible after an error.
|
||||
While the target that failed, and those that depend on it, cannot
|
||||
|
||||
163
doc/make.texi
163
doc/make.texi
@ -6800,10 +6800,33 @@ Supports the @code{-L} (@code{--check-symlink-times}) flag.
|
||||
Supports ``else if'' non-nested conditionals. @xref{Conditional
|
||||
Syntax, ,Syntax of Conditionals}.
|
||||
|
||||
@item extra-prereqs
|
||||
Supports the @code{.EXTRA_PREREQS} special target.
|
||||
|
||||
@item grouped-target
|
||||
Supports grouped target syntax for explicit rules. @xref{Multiple Targets,
|
||||
,Multiple Targets in a Rule}.
|
||||
|
||||
@item guile
|
||||
Has GNU Guile available as an embedded extension language.
|
||||
@xref{Guile Integration, ,GNU Guile Integration}.
|
||||
|
||||
@item jobserver
|
||||
Supports ``job server'' enhanced parallel builds. @xref{Parallel,
|
||||
,Parallel Execution}.
|
||||
|
||||
@item jobserver-fifo
|
||||
Supports ``job server'' enhanced parallel builds using named pipes.
|
||||
@xref{Integrating make, ,Integrating GNU @code{make}}.
|
||||
|
||||
@item load
|
||||
Supports dynamically loadable objects for creating custom extensions.
|
||||
@xref{Loading Objects, ,Loading Dynamic Objects}.
|
||||
|
||||
@item notintermediate
|
||||
Supports the @code{.NOTINTERMEDIATE} special target.
|
||||
@xref{Integrating make, ,Integrating GNU @code{make}}.
|
||||
|
||||
@item oneshell
|
||||
Supports the @code{.ONESHELL} special target. @xref{One Shell, ,Using
|
||||
One Shell}.
|
||||
@ -6812,9 +6835,16 @@ One Shell}.
|
||||
Supports order-only prerequisites. @xref{Prerequisite Types, ,Types
|
||||
of Prerequisites}.
|
||||
|
||||
@item output-sync
|
||||
Supports the @code{--output-sync} command line option. @xref{Options Summary,
|
||||
,Summary of Options}.
|
||||
|
||||
@item second-expansion
|
||||
Supports secondary expansion of prerequisite lists.
|
||||
|
||||
@item shell-export
|
||||
Supports exporting @code{make} variables to @code{shell} functions.
|
||||
|
||||
@item shortest-stem
|
||||
Uses the ``shortest stem'' method of choosing which pattern, of
|
||||
multiple applicable options, will be used. @xref{Pattern Match, ,How
|
||||
@ -6826,14 +6856,6 @@ Supports target-specific and pattern-specific variable assignments.
|
||||
|
||||
@item undefine
|
||||
Supports the @code{undefine} directive. @xref{Undefine Directive}.
|
||||
|
||||
@item guile
|
||||
Has GNU Guile available as an embedded extension language.
|
||||
@xref{Guile Integration, ,GNU Guile Integration}.
|
||||
|
||||
@item load
|
||||
Supports dynamically loadable objects for creating custom extensions.
|
||||
@xref{Loading Objects, ,Loading Dynamic Objects}.
|
||||
@end table
|
||||
|
||||
@vindex .INCLUDE_DIRS @r{(list of include directories)}
|
||||
@ -9286,6 +9308,15 @@ If there is more than one @samp{-j} option, the last one is effective.
|
||||
@xref{Parallel, ,Parallel Execution}, for more information on how
|
||||
recipes are run. Note that this option is ignored on MS-DOS.
|
||||
|
||||
@item --jobserver-style=[@var{style}]
|
||||
@cindex @code{--jobserver-style}
|
||||
Chooses the style of jobserver to use. This option only has effect if
|
||||
parallel builds are enabled (@pxref{Parallel, ,Parallel Execution}). On POSIX
|
||||
systems @var{style} can be one of @code{fifo} (the default) or @code{pipe}.
|
||||
On Windows the only acceptable @var{style} is @code{sem} (the default). This
|
||||
option is useful if you need to use an older versions of GNU @code{make}, or a
|
||||
different tool that requires a specific jobserver style.
|
||||
|
||||
@item -k
|
||||
@cindex @code{-k}
|
||||
@itemx --keep-going
|
||||
@ -12025,20 +12056,20 @@ number of active jobs across recursive invocations. The actual
|
||||
implementation of the jobserver varies across different operating
|
||||
systems, but some fundamental aspects are always true.
|
||||
|
||||
First, only command lines that @code{make} understands to be recursive
|
||||
invocations of @code{make} (@pxref{MAKE Variable, ,How the @code{MAKE}
|
||||
Variable Works}) will have access to the jobserver. When writing
|
||||
makefiles you must be sure to mark the command as recursive (most
|
||||
commonly by prefixing the command line with the @code{+} indicator
|
||||
(@pxref{Recursion, ,Recursive Use of @code{make}}).
|
||||
@cindex @code{--jobserver-auth}
|
||||
First, @code{make} will provide information necessary for accessing the
|
||||
jobserver through the environment to its children, in the @code{MAKEFLAGS}
|
||||
environment variable. Tools which want to participate in the jobserver
|
||||
protocol will need to parse this environment variable and find the word
|
||||
starting with @code{--jobserver-auth=}. The value of this option will
|
||||
describe how to communicate with the jobserver. The interpretation of this
|
||||
value is described in the sections below.
|
||||
|
||||
Second, @code{make} will provide information necessary for accessing
|
||||
the jobserver through the environment to its children, in the
|
||||
@code{MAKEFLAGS} environment variable. Tools which want to
|
||||
participate in the jobserver protocol will need to parse this
|
||||
environment variable, as described in subsequent sections.
|
||||
Be aware that the @code{MAKEFLAGS} variable may contain multiple instances of
|
||||
the @code{--jobserver-auth=} option. Only the @emph{last} instance is
|
||||
relevant.
|
||||
|
||||
Third, every command @code{make} starts has one implicit job slot
|
||||
Second, every command @code{make} starts has one implicit job slot
|
||||
reserved for it before it starts. Any tool which wants to participate
|
||||
in the jobserver protocol should assume it can always run one job
|
||||
without having to contact the jobserver at all.
|
||||
@ -12075,54 +12106,75 @@ the jobserver.
|
||||
@subsection POSIX Jobserver Interaction
|
||||
@cindex jobserver on POSIX
|
||||
|
||||
On POSIX systems the jobserver is implemented as a simple UNIX pipe.
|
||||
The pipe will be pre-loaded with one single-character token for each
|
||||
available job. To obtain an extra slot you must read a single
|
||||
character from the jobserver pipe; to release a slot you must write a
|
||||
single character back into the jobserver pipe. Note that the read
|
||||
side of the jobserver pipe is set to ``blocking'' mode.
|
||||
On POSIX systems the jobserver is implemented in one of two ways: on systems
|
||||
that support it, GNU @code{make} will create a named pipe and use that for the
|
||||
jobserver. In this case the auth option will have the form
|
||||
@code{--jobserver-auth=fifo:PATH} where @samp{PATH} is the pathname of the
|
||||
named pipe. To access the jobserver you should open the named pipe path and
|
||||
read/write to it as described below.
|
||||
|
||||
To access the pipe you must parse the @code{MAKEFLAGS} variable and
|
||||
look for the argument string @code{--jobserver-auth=R,W} where
|
||||
@samp{R} and @samp{W} are non-negative integers representing file
|
||||
descriptors: @samp{R} is the read file descriptor and @samp{W} is the
|
||||
write file descriptor.
|
||||
@cindex @code{--jobserver-style}
|
||||
If the system doesn't support named pipes, or if the user provided the
|
||||
@code{--jobserver-style} option and specified @samp{pipe}, then the jobserver
|
||||
will be implemented as a simple UNIX pipe. In this case the auth option will
|
||||
have the form @code{--jobserver-auth=R,W} where @samp{R} and @samp{W} are
|
||||
non-negative integers representing file descriptors: @samp{R} is the read file
|
||||
descriptor and @samp{W} is the write file descriptor. If either or both of
|
||||
these file descriptors are negative, it means the jobserver is disabled for
|
||||
this process.
|
||||
|
||||
It's important that when you release the job slot, you write back the
|
||||
same character you read from the pipe for that slot. Don't assume
|
||||
that all tokens are the same character; different characters may have
|
||||
different meanings to GNU @code{make}. The order is not important,
|
||||
since @code{make} has no idea in what order jobs will complete anyway.
|
||||
When using a simple pipe, only command lines that @code{make} understands to
|
||||
be recursive invocations of @code{make} (@pxref{MAKE Variable, ,How the
|
||||
@code{MAKE} Variable Works}) will have access to the jobserver. When writing
|
||||
makefiles you must be sure to mark the command as recursive (most commonly by
|
||||
prefixing the command line with the @code{+} indicator (@pxref{Recursion,
|
||||
,Recursive Use of @code{make}}). Note that the read side of the jobserver
|
||||
pipe is set to ``blocking'' mode. This should not be changed.
|
||||
|
||||
In both implementations of the jobserver, the pipe will be pre-loaded with one
|
||||
single-character token for each available job. To obtain an extra slot you
|
||||
must read a single character from the jobserver; to release a slot you must
|
||||
write a single character back into the jobserver.
|
||||
|
||||
It's important that when you release the job slot, you write back the same
|
||||
character you read. Don't assume that all tokens are the same character;
|
||||
different characters may have different meanings to GNU @code{make}. The
|
||||
order is not important, since @code{make} has no idea in what order jobs will
|
||||
complete anyway.
|
||||
|
||||
There are various error conditions you must consider to ensure your
|
||||
implementation is robust:
|
||||
|
||||
@itemize @bullet
|
||||
@item
|
||||
Usually you will have a command-line argument controlling the parallel
|
||||
operation of your tool. Consider whether your tool should detect
|
||||
situations where both the jobserver and the command-line argument are
|
||||
specified, and how it should react.
|
||||
If you have a command-line argument controlling the parallel operation of your
|
||||
tool, consider whether your tool should detect situations where both the
|
||||
jobserver and the command-line argument are specified, and how it should
|
||||
react.
|
||||
|
||||
@item
|
||||
If your tool determines that the @code{--jobserver-auth} option is
|
||||
available in @code{MAKEFLAGS} but that the file descriptors specified
|
||||
are closed, this means that the calling @code{make} process did not
|
||||
think that your tool was a recursive @code{make} invocation (e.g., the
|
||||
command line was not prefixed with a @code{+} character). You should
|
||||
notify your users of this situation.
|
||||
If your tool does not recognize the format of the @code{--jobserver-auth}
|
||||
string, it should assume the jobserver is using a different style and it
|
||||
cannot connect.
|
||||
|
||||
@item
|
||||
Your tool should also examine the first word of the @code{MAKEFLAGS}
|
||||
variable and look for the character @code{n}. If this character is
|
||||
present then @code{make} was invoked with the @samp{-n} option and
|
||||
your tool should stop without performing any operations.
|
||||
If your tool determines that the @code{--jobserver-auth} option references a
|
||||
simple pipe but that the file descriptors specified are closed, this means
|
||||
that the calling @code{make} process did not think that your tool was a
|
||||
recursive @code{make} invocation (e.g., the command line was not prefixed with
|
||||
a @code{+} character). You should notify your users of this situation.
|
||||
|
||||
@item
|
||||
Your tool should be sure to write back the tokens it read, even under
|
||||
error conditions. This includes not only errors in your tool but also
|
||||
outside influences such as interrupts (@code{SIGINT}), etc. You may
|
||||
want to install signal handlers to manage this write-back.
|
||||
Your tool should be sure to write back the tokens it read, even under error
|
||||
conditions. This includes not only errors in your tool but also outside
|
||||
influences such as interrupts (@code{SIGINT}), etc. You may want to install
|
||||
signal handlers to manage this write-back.
|
||||
|
||||
@item
|
||||
Your tool may also examine the first word of the @code{MAKEFLAGS} variable and
|
||||
look for the character @code{n}. If this character is present then
|
||||
@code{make} was invoked with the @samp{-n} option and your tool may want to
|
||||
stop without performing any operations.
|
||||
@end itemize
|
||||
|
||||
@node Windows Jobserver, , POSIX Jobserver, Job Slots
|
||||
@ -12139,6 +12191,9 @@ look for the argument string @code{--jobserver-auth=NAME} where
|
||||
@samp{NAME} is the name of the named semaphore. Use this name with
|
||||
@code{OpenSemaphore} to create a handle to the semaphore.
|
||||
|
||||
@cindex @code{--jobserver-style} for Windows
|
||||
The only valid style for @code{--jobserver-style} is @samp{sem}.
|
||||
|
||||
There are various error conditions you must consider to ensure your
|
||||
implementation is robust:
|
||||
|
||||
|
||||
@ -260,6 +260,9 @@ this program. If not, see <http://www.gnu.org/licenses/>. */
|
||||
/* Define to 1 if you have the memmove function. */
|
||||
#define HAVE_MEMMOVE 1
|
||||
|
||||
/* Define to 1 if you have the 'mkfifo' function. */
|
||||
/* #undef HAVE_MKFIFO */
|
||||
|
||||
/* Define to 1 if you have the mktemp function. */
|
||||
#define HAVE_MKTEMP 1
|
||||
|
||||
|
||||
@ -195,6 +195,9 @@ this program. If not, see <http://www.gnu.org/licenses/>. */
|
||||
/* Define to 1 if you have the <minix/config.h> header file. */
|
||||
/* #undef HAVE_MINIX_CONFIG_H */
|
||||
|
||||
/* Define to 1 if you have the 'mkfifo' function. */
|
||||
/* #undef HAVE_MKFIFO */
|
||||
|
||||
/* Define to 1 if you have the 'mkstemp' function. */
|
||||
/* #undef HAVE_MKSTEMP */
|
||||
|
||||
|
||||
11
src/main.c
11
src/main.c
@ -234,6 +234,9 @@ static const int inf_jobs = 0;
|
||||
|
||||
char *jobserver_auth = NULL;
|
||||
|
||||
/* Style for the jobserver. */
|
||||
static char *jobserver_style = NULL;
|
||||
|
||||
/* Shuffle mode for goals and prerequisites. */
|
||||
|
||||
static char *shuffle_mode = NULL;
|
||||
@ -342,6 +345,8 @@ static const char *const usage[] =
|
||||
N_("\
|
||||
-j [N], --jobs[=N] Allow N jobs at once; infinite jobs with no arg.\n"),
|
||||
N_("\
|
||||
--jobserver-style=STYLE Select the style of jobserver to use.\n"),
|
||||
N_("\
|
||||
-k, --keep-going Keep going when some targets can't be made.\n"),
|
||||
N_("\
|
||||
-l [N], --load-average[=N], --max-load[=N]\n\
|
||||
@ -488,6 +493,7 @@ static const struct command_switch switches[] =
|
||||
/* There is special-case handling for this in decode_switches() as well. */
|
||||
{ TEMP_STDIN_OPT, filename, &makefiles, 0, 0, 0, 0, 0, "temp-stdin" },
|
||||
{ CHAR_MAX+11, string, &shuffle_mode, 1, 1, 0, "random", 0, "shuffle" },
|
||||
{ CHAR_MAX+12, string, &jobserver_style, 1, 0, 0, 0, 0, "jobserver-style" },
|
||||
{ 0, 0, 0, 0, 0, 0, 0, 0, 0 }
|
||||
};
|
||||
|
||||
@ -1347,6 +1353,9 @@ main (int argc, char **argv, char **envp)
|
||||
#endif
|
||||
#ifdef MAKE_JOBSERVER
|
||||
" jobserver"
|
||||
#ifdef HAVE_MKFIFO
|
||||
" jobserver-fifo"
|
||||
#endif
|
||||
#endif
|
||||
#ifndef NO_OUTPUT_SYNC
|
||||
" output-sync"
|
||||
@ -2094,7 +2103,7 @@ main (int argc, char **argv, char **envp)
|
||||
submakes it's the token they were given by their parent. For the top
|
||||
make, we just subtract one from the number the user wants. */
|
||||
|
||||
if (job_slots > 1 && jobserver_setup (job_slots - 1))
|
||||
if (job_slots > 1 && jobserver_setup (job_slots - 1, jobserver_style))
|
||||
{
|
||||
/* Fill in the jobserver_auth for our children. */
|
||||
jobserver_auth = jobserver_get_auth ();
|
||||
|
||||
38
src/os.h
38
src/os.h
@ -20,10 +20,10 @@ this program. If not, see <http://www.gnu.org/licenses/>. */
|
||||
#ifdef MAKE_JOBSERVER
|
||||
|
||||
/* Returns 1 if the jobserver is enabled, else 0. */
|
||||
unsigned int jobserver_enabled (void);
|
||||
unsigned int jobserver_enabled ();
|
||||
|
||||
/* Called in the master instance to set up the jobserver initially. */
|
||||
unsigned int jobserver_setup (int job_slots);
|
||||
unsigned int jobserver_setup (int job_slots, const char *style);
|
||||
|
||||
/* Called in a child instance to connect to the jobserver.
|
||||
Return 1 if we got a valid auth, else 0. */
|
||||
@ -37,16 +37,16 @@ char *jobserver_get_auth ();
|
||||
const char *jobserver_get_invalid_auth ();
|
||||
|
||||
/* Clear this instance's jobserver configuration. */
|
||||
void jobserver_clear (void);
|
||||
void jobserver_clear ();
|
||||
|
||||
/* Recover all the jobserver tokens and return the number we got. */
|
||||
unsigned int jobserver_acquire_all (void);
|
||||
unsigned int jobserver_acquire_all ();
|
||||
|
||||
/* Release a jobserver token. If it fails and is_fatal is 1, fatal. */
|
||||
void jobserver_release (int is_fatal);
|
||||
|
||||
/* Notify the jobserver that a child exited. */
|
||||
void jobserver_signal (void);
|
||||
void jobserver_signal ();
|
||||
|
||||
/* Get ready to start a non-recursive child. */
|
||||
void jobserver_pre_child (int);
|
||||
@ -55,7 +55,7 @@ void jobserver_pre_child (int);
|
||||
void jobserver_post_child (int);
|
||||
|
||||
/* Set up to acquire a new token. */
|
||||
void jobserver_pre_acquire (void);
|
||||
void jobserver_pre_acquire ();
|
||||
|
||||
/* Wait until we can acquire a jobserver token.
|
||||
TIMEOUT is 1 if we have other jobs waiting for the load to go down;
|
||||
@ -66,18 +66,18 @@ unsigned int jobserver_acquire (int timeout);
|
||||
|
||||
#else
|
||||
|
||||
#define jobserver_enabled() (0)
|
||||
#define jobserver_setup(_slots) (0)
|
||||
#define jobserver_parse_auth(_auth) (0)
|
||||
#define jobserver_get_auth() (NULL)
|
||||
#define jobserver_clear() (void)(0)
|
||||
#define jobserver_release(_fatal) (void)(0)
|
||||
#define jobserver_acquire_all() (0)
|
||||
#define jobserver_signal() (void)(0)
|
||||
#define jobserver_pre_child(_r) (void)(0)
|
||||
#define jobserver_post_child(_r) (void)(0)
|
||||
#define jobserver_pre_acquire() (void)(0)
|
||||
#define jobserver_acquire(_tmout) (0)
|
||||
#define jobserver_enabled() (0)
|
||||
#define jobserver_setup(_slots, _style) (0)
|
||||
#define jobserver_parse_auth(_auth) (0)
|
||||
#define jobserver_get_auth() (NULL)
|
||||
#define jobserver_clear() (void)(0)
|
||||
#define jobserver_release(_fatal) (void)(0)
|
||||
#define jobserver_acquire_all() (0)
|
||||
#define jobserver_signal() (void)(0)
|
||||
#define jobserver_pre_child(_r) (void)(0)
|
||||
#define jobserver_post_child(_r) (void)(0)
|
||||
#define jobserver_pre_acquire() (void)(0)
|
||||
#define jobserver_acquire(_tmout) (0)
|
||||
|
||||
#endif
|
||||
|
||||
@ -85,7 +85,7 @@ unsigned int jobserver_acquire (int timeout);
|
||||
#if defined(VMS) || defined(WINDOWS32) || defined(_AMIGA) || defined(__MSDOS__)
|
||||
# define get_bad_stdin() (-1)
|
||||
#else
|
||||
int get_bad_stdin (void);
|
||||
int get_bad_stdin ();
|
||||
#endif
|
||||
|
||||
/* Set a file descriptor to close/not close in a subprocess. */
|
||||
|
||||
192
src/posixos.c
192
src/posixos.c
@ -20,10 +20,15 @@ this program. If not, see <http://www.gnu.org/licenses/>. */
|
||||
|
||||
#ifdef HAVE_FCNTL_H
|
||||
# include <fcntl.h>
|
||||
# define FD_OK(_f) (fcntl ((_f), F_GETFD) != -1)
|
||||
#elif defined(HAVE_SYS_FILE_H)
|
||||
# include <sys/file.h>
|
||||
#endif
|
||||
|
||||
#if !defined(FD_OK)
|
||||
# define FD_OK(_f) 1
|
||||
#endif
|
||||
|
||||
#if defined(HAVE_PSELECT) && defined(HAVE_SYS_SELECT_H)
|
||||
# include <sys/select.h>
|
||||
#endif
|
||||
@ -34,6 +39,8 @@ this program. If not, see <http://www.gnu.org/licenses/>. */
|
||||
|
||||
#ifdef MAKE_JOBSERVER
|
||||
|
||||
#define FIFO_PREFIX "fifo:"
|
||||
|
||||
/* This section provides OS-specific functions to support the jobserver. */
|
||||
|
||||
/* These track the state of the jobserver pipe. Passed to child instances. */
|
||||
@ -47,8 +54,21 @@ static int job_rfd = -1;
|
||||
/* Token written to the pipe (could be any character...) */
|
||||
static char token = '+';
|
||||
|
||||
/* The type of jobserver we're using. */
|
||||
enum js_type
|
||||
{
|
||||
js_none = 0, /* No jobserver. */
|
||||
js_pipe, /* Use a simple pipe as the jobserver. */
|
||||
js_fifo /* Use a named pipe as the jobserver. */
|
||||
};
|
||||
|
||||
static enum js_type js_type = js_none;
|
||||
|
||||
/* The name of the named pipe (if used). */
|
||||
static char *fifo_name = NULL;
|
||||
|
||||
static int
|
||||
make_job_rfd (void)
|
||||
make_job_rfd ()
|
||||
{
|
||||
#ifdef HAVE_PSELECT
|
||||
/* Pretend we succeeded. */
|
||||
@ -81,13 +101,57 @@ set_blocking (int fd, int blocking)
|
||||
}
|
||||
|
||||
unsigned int
|
||||
jobserver_setup (int slots)
|
||||
jobserver_setup (int slots, const char *style)
|
||||
{
|
||||
int r;
|
||||
|
||||
EINTRLOOP (r, pipe (job_fds));
|
||||
if (r < 0)
|
||||
pfatal_with_name (_("creating jobs pipe"));
|
||||
#if HAVE_MKFIFO
|
||||
if (style == NULL || strcmp (style, "fifo") == 0)
|
||||
{
|
||||
fifo_name = get_tmppath ();
|
||||
|
||||
EINTRLOOP (r, mkfifo (fifo_name, 0600));
|
||||
if (r < 0)
|
||||
{
|
||||
perror_with_name("jobserver mkfifo: ", fifo_name);
|
||||
free (fifo_name);
|
||||
}
|
||||
else
|
||||
{
|
||||
/* We have to open the read side in non-blocking mode, else it will
|
||||
hang until the write side is open. */
|
||||
EINTRLOOP (job_fds[0], open (fifo_name, O_RDONLY|O_NONBLOCK));
|
||||
if (job_fds[0] < 0)
|
||||
OSS (fatal, NILF, _("Cannot open jobserver %s: %s"),
|
||||
fifo_name, strerror (errno));
|
||||
|
||||
EINTRLOOP (job_fds[1], open (fifo_name, O_WRONLY));
|
||||
if (job_fds[0] < 0)
|
||||
OSS (fatal, NILF, _("Cannot open jobserver %s: %s"),
|
||||
fifo_name, strerror (errno));
|
||||
|
||||
DB (DB_JOBS,
|
||||
(_("Jobserver setup (fifo %s)\n"), fifo_name));
|
||||
|
||||
js_type = js_fifo;
|
||||
}
|
||||
}
|
||||
#endif
|
||||
|
||||
if (js_type == js_none)
|
||||
{
|
||||
if (style && strcmp (style, "pipe") != 0)
|
||||
OS (fatal, NILF, _("Unknown jobserver auth style '%s'"), style);
|
||||
|
||||
EINTRLOOP (r, pipe (job_fds));
|
||||
if (r < 0)
|
||||
pfatal_with_name (_("creating jobs pipe"));
|
||||
|
||||
DB (DB_JOBS,
|
||||
(_("Jobserver setup (fds %d,%d)\n"), job_fds[0], job_fds[1]));
|
||||
|
||||
js_type = js_pipe;
|
||||
}
|
||||
|
||||
/* By default we don't send the job pipe FDs to our children.
|
||||
See jobserver_pre_child() and jobserver_post_child(). */
|
||||
@ -113,34 +177,63 @@ jobserver_setup (int slots)
|
||||
unsigned int
|
||||
jobserver_parse_auth (const char *auth)
|
||||
{
|
||||
int rfd, wfd;
|
||||
|
||||
/* Given the command-line parameter, parse it. */
|
||||
if (sscanf (auth, "%d,%d", &job_fds[0], &job_fds[1]) != 2)
|
||||
OS (fatal, NILF,
|
||||
_("internal error: invalid --jobserver-auth string '%s'"), auth);
|
||||
|
||||
DB (DB_JOBS,
|
||||
(_("Jobserver client (fds %d,%d)\n"), job_fds[0], job_fds[1]));
|
||||
/* First see if we're using a named pipe. */
|
||||
if (strncmp (auth, FIFO_PREFIX, CSTRLEN (FIFO_PREFIX)) == 0)
|
||||
{
|
||||
fifo_name = xstrdup (auth + CSTRLEN (FIFO_PREFIX));
|
||||
|
||||
if (job_fds[0] == -2 || job_fds[1] == -2)
|
||||
return 0;
|
||||
EINTRLOOP (job_fds[0], open (fifo_name, O_RDONLY));
|
||||
if (job_fds[0] < 0)
|
||||
OSS (fatal, NILF,
|
||||
_("Cannot open jobserver %s: %s"), fifo_name, strerror (errno));
|
||||
|
||||
#ifdef HAVE_FCNTL_H
|
||||
# define FD_OK(_f) (fcntl ((_f), F_GETFD) != -1)
|
||||
#else
|
||||
# define FD_OK(_f) 1
|
||||
#endif
|
||||
EINTRLOOP (job_fds[1], open (fifo_name, O_WRONLY));
|
||||
if (job_fds[0] < 0)
|
||||
OSS (fatal, NILF,
|
||||
_("Cannot open jobserver %s: %s"), fifo_name, strerror (errno));
|
||||
|
||||
/* Make sure our pipeline is valid, and (possibly) create a duplicate pipe,
|
||||
that will be closed in the SIGCHLD handler. If this fails with EBADF,
|
||||
the parent has closed the pipe on us because it didn't think we were a
|
||||
submake. If so, warn and default to -j1. */
|
||||
js_type = js_fifo;
|
||||
}
|
||||
/* If not, it must be a simple pipe. */
|
||||
else if (sscanf (auth, "%d,%d", &rfd, &wfd) == 2)
|
||||
{
|
||||
DB (DB_JOBS,
|
||||
(_("Jobserver client (fds %d,%d)\n"), rfd, wfd));
|
||||
|
||||
if (!FD_OK (job_fds[0]) || !FD_OK (job_fds[1]) || make_job_rfd () < 0)
|
||||
/* The parent overrode our FDs because we aren't a recursive make. */
|
||||
if (rfd == -2 || wfd == -2)
|
||||
return 0;
|
||||
|
||||
/* Make sure our pipeline is valid. */
|
||||
if (!FD_OK (rfd) || !FD_OK (wfd))
|
||||
return 0;
|
||||
|
||||
job_fds[0] = rfd;
|
||||
job_fds[1] = wfd;
|
||||
|
||||
js_type = js_pipe;
|
||||
}
|
||||
/* Who knows what it is? */
|
||||
else
|
||||
{
|
||||
OS (error, NILF, _("invalid --jobserver-auth string '%s'"), auth);
|
||||
return 0;
|
||||
}
|
||||
|
||||
/* Create a duplicate pipe, if needed, that will be closed in the SIGCHLD
|
||||
handler. If this fails with EBADF, the parent closed the pipe on us as
|
||||
it didn't think we were a submake. If so, warn and default to -j1. */
|
||||
|
||||
if (make_job_rfd () < 0)
|
||||
{
|
||||
if (errno != EBADF)
|
||||
pfatal_with_name (_("jobserver pipeline"));
|
||||
pfatal_with_name ("jobserver readfd");
|
||||
|
||||
job_fds[0] = job_fds[1] = -1;
|
||||
jobserver_clear ();
|
||||
|
||||
return 0;
|
||||
}
|
||||
@ -159,27 +252,40 @@ jobserver_parse_auth (const char *auth)
|
||||
char *
|
||||
jobserver_get_auth ()
|
||||
{
|
||||
char *auth = xmalloc ((INTSTR_LENGTH * 2) + 2);
|
||||
sprintf (auth, "%d,%d", job_fds[0], job_fds[1]);
|
||||
char *auth;
|
||||
|
||||
if (js_type == js_fifo) {
|
||||
auth = xmalloc (strlen (fifo_name) + CSTRLEN (FIFO_PREFIX) + 1);
|
||||
sprintf (auth, FIFO_PREFIX "%s", fifo_name);
|
||||
} else {
|
||||
auth = xmalloc ((INTSTR_LENGTH * 2) + 2);
|
||||
sprintf (auth, "%d,%d", job_fds[0], job_fds[1]);
|
||||
}
|
||||
|
||||
return auth;
|
||||
}
|
||||
|
||||
const char *
|
||||
jobserver_get_invalid_auth ()
|
||||
{
|
||||
/* If we're using a named pipe we don't need to invalidate the jobserver. */
|
||||
if (js_type == js_fifo) {
|
||||
return NULL;
|
||||
}
|
||||
|
||||
/* It's not really great that we are assuming the command line option
|
||||
here but other alternatives are also gross. */
|
||||
return " --" JOBSERVER_AUTH_OPT "=-2,-2";
|
||||
}
|
||||
|
||||
unsigned int
|
||||
jobserver_enabled (void)
|
||||
jobserver_enabled ()
|
||||
{
|
||||
return job_fds[0] >= 0;
|
||||
return js_type != js_none;
|
||||
}
|
||||
|
||||
void
|
||||
jobserver_clear (void)
|
||||
jobserver_clear ()
|
||||
{
|
||||
if (job_fds[0] >= 0)
|
||||
close (job_fds[0]);
|
||||
@ -189,6 +295,11 @@ jobserver_clear (void)
|
||||
close (job_rfd);
|
||||
|
||||
job_fds[0] = job_fds[1] = job_rfd = -1;
|
||||
|
||||
free (fifo_name);
|
||||
fifo_name = NULL;
|
||||
|
||||
js_type = js_none;
|
||||
}
|
||||
|
||||
void
|
||||
@ -205,8 +316,9 @@ jobserver_release (int is_fatal)
|
||||
}
|
||||
|
||||
unsigned int
|
||||
jobserver_acquire_all (void)
|
||||
jobserver_acquire_all ()
|
||||
{
|
||||
int r;
|
||||
unsigned int tokens = 0;
|
||||
|
||||
/* Use blocking reads to wait for all outstanding jobs. */
|
||||
@ -219,19 +331,25 @@ jobserver_acquire_all (void)
|
||||
while (1)
|
||||
{
|
||||
char intake;
|
||||
int r;
|
||||
EINTRLOOP (r, read (job_fds[0], &intake, 1));
|
||||
if (r != 1)
|
||||
return tokens;
|
||||
break;
|
||||
++tokens;
|
||||
}
|
||||
|
||||
if (fifo_name)
|
||||
EINTRLOOP (r, unlink (fifo_name));
|
||||
|
||||
DB (DB_JOBS, ("Acquired all %u jobserver tokens.\n", tokens));
|
||||
|
||||
return tokens;
|
||||
}
|
||||
|
||||
/* Prepare the jobserver to start a child process. */
|
||||
void
|
||||
jobserver_pre_child (int recursive)
|
||||
{
|
||||
if (recursive && job_fds[0] >= 0)
|
||||
if (recursive && js_type == js_pipe)
|
||||
{
|
||||
fd_inherit (job_fds[0]);
|
||||
fd_inherit (job_fds[1]);
|
||||
@ -242,7 +360,7 @@ jobserver_pre_child (int recursive)
|
||||
void
|
||||
jobserver_post_child (int recursive)
|
||||
{
|
||||
if (recursive && job_fds[0] >= 0)
|
||||
if (recursive && js_type == js_pipe)
|
||||
{
|
||||
fd_noinherit (job_fds[0]);
|
||||
fd_noinherit (job_fds[1]);
|
||||
@ -250,7 +368,7 @@ jobserver_post_child (int recursive)
|
||||
}
|
||||
|
||||
void
|
||||
jobserver_signal (void)
|
||||
jobserver_signal ()
|
||||
{
|
||||
if (job_rfd >= 0)
|
||||
{
|
||||
@ -260,7 +378,7 @@ jobserver_signal (void)
|
||||
}
|
||||
|
||||
void
|
||||
jobserver_pre_acquire (void)
|
||||
jobserver_pre_acquire ()
|
||||
{
|
||||
/* Make sure we have a dup'd FD. */
|
||||
if (job_rfd < 0 && job_fds[0] >= 0 && make_job_rfd () < 0)
|
||||
@ -460,7 +578,7 @@ jobserver_acquire (int timeout)
|
||||
|
||||
/* Create a "bad" file descriptor for stdin when parallel jobs are run. */
|
||||
int
|
||||
get_bad_stdin (void)
|
||||
get_bad_stdin ()
|
||||
{
|
||||
static int bad_stdin = -1;
|
||||
|
||||
|
||||
@ -34,10 +34,13 @@ static char jobserver_semaphore_name[MAX_PATH + 1];
|
||||
static HANDLE jobserver_semaphore = NULL;
|
||||
|
||||
unsigned int
|
||||
jobserver_setup (int slots)
|
||||
jobserver_setup (int slots, const char *style)
|
||||
{
|
||||
/* sub_proc.c is limited in the number of objects it can wait for. */
|
||||
|
||||
if (style && strcmp (style, "sem") != 0)
|
||||
OS (fatal, NILF, _("Unknown jobserver auth style '%s'"), style);
|
||||
|
||||
if (slots > process_table_usable_size())
|
||||
{
|
||||
slots = process_table_usable_size();
|
||||
|
||||
@ -82,21 +82,29 @@ unlink('inc.mk');
|
||||
# Test recursion which is hidden from make.
|
||||
# See Savannah bug #39934
|
||||
# Or Red Hat bug https://bugzilla.redhat.com/show_bug.cgi?id=885474
|
||||
# Windows doesn't use a pipe, and doesn't close access, so this won't happen.
|
||||
# Environments that don't use a pipe won't close access, so this won't happen.
|
||||
if ($port_type ne 'W32') {
|
||||
open(MAKEFILE,"> Makefile2");
|
||||
print MAKEFILE '
|
||||
vpath %.c ../
|
||||
foo:
|
||||
';
|
||||
close(MAKEFILE);
|
||||
create_file('Makefile2', "vpath %.c ../\n", "foo:\n");
|
||||
|
||||
run_make_test(q!
|
||||
default: ; @ #MAKEPATH# -f Makefile2
|
||||
!,
|
||||
"--jobserver-style=pipe -j2 $np",
|
||||
"#MAKE#[1]: warning: jobserver unavailable: using -j1. Add '+' to parent make rule.
|
||||
#MAKE#[1]: Nothing to be done for 'foo'.");
|
||||
|
||||
rmfiles('Makefile2');
|
||||
}
|
||||
|
||||
# For Windows and named pipes, we don't need to worry about recursion
|
||||
if ($port_type eq 'W32' || exists $FEATURES{'jobserver-fifo'}) {
|
||||
create_file('Makefile2', "vpath %.c ../\n", "foo:\n");
|
||||
|
||||
run_make_test(q!
|
||||
default: ; @ #MAKEPATH# -f Makefile2
|
||||
!,
|
||||
"-j2 $np",
|
||||
"#MAKE#[1]: warning: jobserver unavailable: using -j1. Add '+' to parent make rule.
|
||||
#MAKE#[1]: Nothing to be done for 'foo'.");
|
||||
"#MAKE#[1]: Nothing to be done for 'foo'.");
|
||||
|
||||
rmfiles('Makefile2');
|
||||
}
|
||||
@ -114,4 +122,13 @@ hi
|
||||
#MAKE#[1]: Leaving directory '#PWD#'
|
||||
#MAKE#: Leaving directory '#PWD#'\n");
|
||||
|
||||
# Check for invalid jobserver-style options
|
||||
|
||||
run_make_test(q!
|
||||
all: a
|
||||
all a: ; @echo $@
|
||||
!,
|
||||
'--jobserver-style=foo -j8',
|
||||
"#MAKE#: *** Unknown jobserver auth style 'foo'. Stop.", 512);
|
||||
|
||||
1;
|
||||
|
||||
@ -154,7 +154,24 @@ else
|
||||
default:;: $(ELT)
|
||||
endif
|
||||
!,
|
||||
'--no-print-directory -j2', "#MAKE#[2]: warning: jobserver unavailable: using -j1. Add '+' to parent make rule.\n: 2\n: 1");
|
||||
'--no-print-directory -j2 --jobserver-style=pipe', "#MAKE#[2]: warning: jobserver unavailable: using -j1. Add '+' to parent make rule.\n: 2\n: 1");
|
||||
}
|
||||
|
||||
# If we're not using pipes for jobserver, then they are available in sub-makes
|
||||
# invoked by $(shell ...)
|
||||
if ($port_type eq 'W32' || exists $FEATURES{'jobserver-fifo'}) {
|
||||
run_make_test(q!
|
||||
ifeq ($(ELT),)
|
||||
default:; @$(MAKE) -f #MAKEFILE# ELT=1
|
||||
else ifeq ($(ELT),1)
|
||||
OUTPUT := $(shell $(MAKE) -f #MAKEFILE# ELT=2)
|
||||
$(info $(OUTPUT))
|
||||
default:;: $(ELT)
|
||||
else
|
||||
default:;: $(ELT)
|
||||
endif
|
||||
!,
|
||||
'--no-print-directory -j2', ": 2\n: 1");
|
||||
}
|
||||
|
||||
1;
|
||||
|
||||
Loading…
Reference in New Issue
Block a user