github-repos/make
1
0
Fork 0
mirror of https://github.com/mirror/make.git synced 2025-01-27 21:00:22 +08:00
Code Issues Packages Projects Releases Wiki Activity

* Bugfix for NT archive support.

Browse Source 
* Rework how the jobserver stuff works.
This commit is contained in:
Paul Smith 1999-07-23 22:46:47 +00:00
parent b05cb1d99e
commit 4e7ee4fc01
5 changed files with 264 additions and 111 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

90
ChangeLog
View File

@ -1,3 +1,60 @@
1999-07-23 Paul D. Smith <psmith@gnu.org>
* job.c (my_job_token): This variable controls whether we've
handed our personal token to a subprocess or not. Note we could
probably infer this from the value of job_slots_used, but it's
clearer to just keep it separately. Job_slots_used isn't really
relevant when running the job server.
(free_job_token): New function: free a job token. If we don't
have one, no-op. If we have the personal token, reclaim it. If
we have another token, write it back to the pipe.
(reap_children): Call free_job_token.
(free_child): Call free_job_token.
(start_job_command): Remove duplicate test for '+' in the command.
If we don't appear to be running a recursive make, close the
jobserver filedescriptors.
(start_waiting_job): If our personal token is available, use that
instead of going to the server pipe.
(*): Add the token value to many debugging statements, and print
the child target name in addition to the ptr hex value.
Change the default "no token" value from '\0' to '-' so it looks
better in the output.
* main.c (main): Install the child_handler with sigaction()
instead of signal() if we have it. On SysV systems, signal() uses
SysV semantics which are a pain. But sigaction() always does what
we want.
(main): If we got job server FDs from the environment, test them
to see if they're open. If not, the parent make closed them
because it didn't think we were a submake. Print a warning and
suggestion to use "+" on the submake invocation, and hard-set to
-j1 for this instance of make.
(main): Change the algorithm for assigning slots to be more
robust. Previously make checked to see if it thought a subprocess
was a submake and if so, didn't give it a token. Since make's
don't consume tokens we could spawn many of makes fighting for a
small number of tokens. Plus this is unreliable because submakes
might not be recognized by the parent (see above) then all the
tokens could be used up by unrecognized makes, and no one could
run. Now every make consumes a token from its parent. However,
the make can also use this token to spawn a child. If the make
wants more than one, it goes to the jobserver pipe. Thus there
will never be more than N makes running for -jN, and N*2 processes
(N makes and their N children). Every make can always run at
least one job, and we'll never deadlock. (Note the closing of the
pipe for non-submakes also solves this, but this is still a better
algorithm.) So! Only put N-1 tokens into the pipe, since the
topmost make keeps one for itself.
* configure.in: Find sigaction. Disable job server support unless
the system provides it, in addition to either waitpid() or
wait3().
1999-07-22 Rob Tulloh <rob_tulloh@dev.tivoli.com>
* arscan.c (ar_member_touch) [WINDOWS32]: The ar_date field is a
string on Windows, not a timestamp.
1999-07-21 Paul D. Smith <psmith@gnu.org> 1999-07-21 Paul D. Smith <psmith@gnu.org>
* Version 3.77.90 released. * Version 3.77.90 released.
@ -111,21 +168,24 @@
already have one; if we're waiting on the load to go down already have one; if we're waiting on the load to go down
start_waiting_job() might get called twice on the same file. start_waiting_job() might get called twice on the same file.
* remake.c (update_goal_chain): If we try to update a goal and it * filedef.h (struct file): Add new field, mtime_before_update.
doesn't complete (e.g., parallel builds) remember that by setting When notice_finished_file runs it assigns the cached last_mtime to
the `deferred' flag in the goal structure. Later when we're this field.
determining the return value we consider a goal updated if either * remake.c (update_goal_chain): Notice that a file wasn't updated
the mtime has changed _or_ this flag was set. We need this by asking if it changed (g->changed) and comparing the current
because the mtime doesn't change during the update_file() function cached time (last_mtime) with the previous one, stored in
if we started a job running; instead it's set during the mtime_before_update. The previous check ("did last_mtime changed
reap_children() call. So, the code doesn't know it was updated during the run of update_file?") fails for parallel builds because
and returns a status of -1 (nothing done). This is OK during last_mtime is set during reap_children, before update_file is run.
"normal" builds since our caller (main) treats these cases This causes update_goal_chain to always return -1 (nothing
identically in that case, but if you're building makefiles the rebuilt) when running parallel (-jN). This is OK during "normal"
difference is very important (whether we re-exec or not). builds since our caller (main) treats these cases identically in
that case, but if when rebuilding makefiles the difference is very
* dep.h: Add a `deferred' flag to track whether a goal was run but important, as it controls whether we re-exec or not.
not completed (parallel builds). * file.c (file_hash_enter): Copy the mtime_before_update field.
(snap_deps): Initialize mtime_before_update to -1.
* main.c (main): Initialize mtime_before_update on old (-o) and
new (-W) files.
1999-07-08 Paul D. Smith <psmith@gnu.org> 1999-07-08 Paul D. Smith <psmith@gnu.org>

2
arscan.c
View File

@ -780,7 +780,7 @@ ar_member_touch (arname, memname)
#else #else
fstat (fd, &statbuf); fstat (fd, &statbuf);
#endif #endif
#if defined(ARFMAG) || defined(ARFZMAG) || defined(AIAMAG) #if defined(ARFMAG) || defined(ARFZMAG) || defined(AIAMAG) || defined(WINDOWS32)
/* Advance member's time to that time */ /* Advance member's time to that time */
for (i = 0; i < sizeof ar_hdr.ar_date; i++) for (i = 0; i < sizeof ar_hdr.ar_date; i++)
ar_hdr.ar_date[i] = ' '; ar_hdr.ar_date[i] = ' ';

40
configure.in
View File

@ -56,8 +56,9 @@ AC_MSG_RESULT($ac_cv_check_symbol_$1)])dnl
AC_CHECK_LIB(posix4, clock_gettime) AC_CHECK_LIB(posix4, clock_gettime)
AC_CHECK_FUNCS(memmove strdup psignal mktemp pstat_getdynamic \ AC_CHECK_FUNCS(memmove strdup psignal mktemp pstat_getdynamic \
clock_gettime dup2 getcwd sigsetmask getgroups setlinebuf \ clock_gettime dup2 getcwd sigsetmask sigaction getgroups \
seteuid setegid setreuid setregid strerror strsignal pipe) setlinebuf seteuid setegid setreuid setregid \
strerror strsignal pipe)
AC_CHECK_SYMBOL(sys_siglist) AC_CHECK_SYMBOL(sys_siglist)
AC_FUNC_ALLOCA AC_FUNC_ALLOCA
AC_FUNC_VFORK AC_FUNC_VFORK
@ -129,11 +130,16 @@ dnl See if we can handle the job server feature, and if the user wants it.
AC_ARG_ENABLE(job-server, AC_ARG_ENABLE(job-server,
[ --disable-job-server Disallow recursive make communication during -jN], [ --disable-job-server Disallow recursive make communication during -jN],
[make_cv_job_server="$enableval"], [make_cv_job_server="$enableval" user_job_server="$enableval"],
[make_cv_job_server="yes"]) [make_cv_job_server="yes"])
case "$ac_cv_func_pipe $make_cv_job_server" in has_wait_nohang=yes
"yes yes") AC_DEFINE(MAKE_JOBSERVER) ;; case "$ac_cv_func_waitpid/$ac_cv_func_wait3" in
no/no) has_wait_nohang=no ;;
esac
case "$ac_cv_func_pipe/$ac_cv_func_sigaction/$has_wait_nohang/$make_cv_job_server" in
yes/yes/yes/yes) AC_DEFINE(MAKE_JOBSERVER) ;;
esac esac
dnl Allow building with dmalloc dnl Allow building with dmalloc
@ -232,13 +238,23 @@ case "$with_customs" in
fi ;; fi ;;
esac esac
case "$ac_cv_func_waitpid/$ac_cv_func_wait3" in case "$has_wait_nohang" in
no/no) echo no) echo
echo "WARNING: Your system has neither waitpid() nor wait3()." echo "WARNING: Your system has neither waitpid() nor wait3()."
echo " Without one of these, signal handling is unreliable." echo " Without one of these, signal handling is unreliable."
echo " You should be aware that running GNU make with -j" echo " You should be aware that running GNU make with -j"
echo " could result in erratic behavior." echo " could result in erratic behavior."
echo ;; echo ;;
esac
case "$make_cv_job_server/$user_job_server" in
no/yes) echo
echo "WARNING: Make job server requires a POSIX-ish system that"
echo " supports the pipe(), sigaction(), and either"
echo " waitpid() or wait3() functions. Your system doesn't"
echo " appear to provide one or more of those."
echo " Disabling job server support."
echo ;;
esac esac
dnl Local Variables: dnl Local Variables:

177
job.c
View File

@ -191,6 +191,11 @@ unsigned int job_slots_used = 0;
static int good_stdin_used = 0; static int good_stdin_used = 0;
/* Specifies whether the current process's reserved job token is in use.
'+' means it's available, '-' means it isn't. */
static char my_job_token = '+';
/* Chain of children waiting to run until the load average goes down. */ /* Chain of children waiting to run until the load average goes down. */
static struct child *waiting_jobs = 0; static struct child *waiting_jobs = 0;
@ -198,6 +203,9 @@ static struct child *waiting_jobs = 0;
/* Non-zero if we use a *real* shell (always so on Unix). */ /* Non-zero if we use a *real* shell (always so on Unix). */
int unixy_shell = 1; int unixy_shell = 1;
/* #define debug_flag 1 */
#ifdef WINDOWS32 #ifdef WINDOWS32
/* /*
@ -209,6 +217,40 @@ int w32_kill(int pid, int sig)
} }
#endif /* WINDOWS32 */ #endif /* WINDOWS32 */
#ifndef MAKE_JOBSERVER
# define free_job_token(c)
#else
static void
free_job_token (child)
struct child *child;
{
switch (child->job_token)
{
case '-':
/* If this child doesn't have a token, punt. */
return;
case '+':
/* If this child has the reserved token, take it back. */
my_job_token = '+';
break;
default:
/* Write any other job tokens back to the pipe. */
write (job_fds[1], &child->job_token, 1);
break;
}
if (debug_flag)
printf ("Released token `%c' for child 0x%08lx (%s).\n",
child->job_token, (unsigned long int) child, child->file->name);
child->job_token = '-';
}
#endif
/* Write an error message describing the exit status given in /* Write an error message describing the exit status given in
EXIT_CODE, EXIT_SIG, and COREDUMP, for the target TARGET_NAME. EXIT_CODE, EXIT_SIG, and COREDUMP, for the target TARGET_NAME.
Append "(ignored)" if IGNORED is nonzero. */ Append "(ignored)" if IGNORED is nonzero. */
@ -342,9 +384,10 @@ reap_children (block, err)
any_remote |= c->remote; any_remote |= c->remote;
any_local |= ! c->remote; any_local |= ! c->remote;
if (debug_flag) if (debug_flag)
printf ("Live child 0x%08lx PID %ld%s\n", printf ("Live child 0x%08lx (%s) PID %ld token %c%s\n",
(unsigned long int) c, (unsigned long int) c, c->file->name,
(long) c->pid, c->remote ? " (remote)" : ""); (long) c->pid, c->job_token,
c->remote ? " (remote)" : "");
#ifdef VMS #ifdef VMS
break; break;
#endif #endif
@ -507,10 +550,10 @@ reap_children (block, err)
else else
{ {
if (debug_flag) if (debug_flag)
printf ("Reaping %s child 0x%08lx PID %ld%s\n", printf ("Reaping %s child 0x%08lx PID %ld token %c%s\n",
child_failed ? "losing" : "winning", child_failed ? "losing" : "winning",
(unsigned long int) c, (unsigned long int) c, (long) c->pid, c->job_token,
(long) c->pid, c->remote ? " (remote)" : ""); c->remote ? " (remote)" : "");
if (c->sh_batch_file) { if (c->sh_batch_file) {
if (debug_flag) if (debug_flag)
@ -607,9 +650,9 @@ reap_children (block, err)
notice_finished_file (c->file); notice_finished_file (c->file);
if (debug_flag) if (debug_flag)
printf ("Removing child 0x%08lx PID %ld%s from chain.\n", printf ("Removing child 0x%08lx PID %ld token %c%s from chain.\n",
(unsigned long int) c, (unsigned long int) c, (long) c->pid, c->job_token,
(long) c->pid, c->remote ? " (remote)" : ""); c->remote ? " (remote)" : "");
/* Block fatal signals while frobnicating the list, so that /* Block fatal signals while frobnicating the list, so that
children and job_slots_used are always consistent. Otherwise children and job_slots_used are always consistent. Otherwise
@ -618,18 +661,9 @@ reap_children (block, err)
live and call reap_children again. */ live and call reap_children again. */
block_sigs (); block_sigs ();
#ifdef MAKE_JOBSERVER
/* If this job has a token out, return it. */ /* If this job has a token out, return it. */
if (c->job_token) free_job_token(c);
{
assert(job_slots_used > 0);
write (job_fds[1], &c->job_token, 1);
if (debug_flag)
printf ("Released token `%c' for child 0x%08lx.\n",
c->job_token, (unsigned long int) c);
c->job_token = 0;
}
#endif
/* There is now another slot open. */ /* There is now another slot open. */
if (job_slots_used > 0) if (job_slots_used > 0)
--job_slots_used; --job_slots_used;
@ -680,18 +714,10 @@ free_child (child)
free ((char *) child->environment); free ((char *) child->environment);
} }
#ifdef MAKE_JOBSERVER
/* If this child has a token it hasn't relinquished, give it up now. /* If this child has a token it hasn't relinquished, give it up now.
This can happen if the job completes immediately, mainly because This can happen if the job completes immediately, mainly because
all the command lines evaluated to empty strings. */ all the command lines evaluated to empty strings. */
if (child->job_token) free_job_token(child);
{
write (job_fds[1], &child->job_token, 1);
if (debug_flag)
printf ("Freed token `%c' for child 0x%08lx.\n",
child->job_token, (unsigned long int) child);
}
#endif
free ((char *) child); free ((char *) child);
} }
@ -764,7 +790,7 @@ start_job_command (child)
flags |= COMMANDS_RECURSE; flags |= COMMANDS_RECURSE;
else if (*p == '-') else if (*p == '-')
child->noerror = 1; child->noerror = 1;
else if (!isblank (*p) && *p != '+') else if (!isblank (*p))
break; break;
++p; ++p;
} }
@ -986,6 +1012,15 @@ start_job_command (child)
{ {
/* We are the child side. */ /* We are the child side. */
unblock_sigs (); unblock_sigs ();
/* If we aren't running a recursive command and we have a jobserver
pipe, close it before exec'ing. */
if (!(flags & COMMANDS_RECURSE) && job_fds[0] >= 0)
{
close (job_fds[0]);
close (job_fds[1]);
}
child_execute_job (child->good_stdin ? 0 : bad_stdin, 1, child_execute_job (child->good_stdin ? 0 : bad_stdin, 1,
argv, child->environment); argv, child->environment);
} }
@ -1139,47 +1174,54 @@ start_waiting_job (c)
if (!c->remote) if (!c->remote)
{ {
#ifdef MAKE_JOBSERVER #ifdef MAKE_JOBSERVER
/* If this is not a recurse command and we are controlling /* If we are controlling multiple jobs, and we don't yet have one,
multiple jobs, and we don't yet have one, obtain a token before * obtain a token before starting the child. */
starting child. */ if (job_fds[0] >= 0)
if (job_fds[0] >= 0 && !f->cmds->any_recurse && !c->job_token) {
{ while (c->job_token == '-')
fd_set rfds; /* If the reserved token is available, just use that. */
if (my_job_token != '-')
{
c->job_token = my_job_token;
my_job_token = '-';
}
/* Read a token. We set the non-blocking bit on this earlier, so
if there's no token to be read we'll return immediately. */
else if (read (job_fds[0], &c->job_token, 1) < 1)
{
fd_set rfds;
int r;
FD_ZERO(&rfds); FD_ZERO(&rfds);
FD_SET(job_fds[0], &rfds); FD_SET(job_fds[0], &rfds);
/* Read a token. We set the non-blocking bit on this earlier, /* The select blocks until (a) there's data to read, in which
so if there's no token to be read we'll fall through to the case we come back around and try to grab the token before
select. The select blocks until (a) there's data to read, someone else does, or (b) a signal (SIGCHLD), is reported
in which case we come back around and try to grab the token (because we installed a handler for it). If the latter,
before someone else does, or (b) a signal, such as SIGCHLD, call reap_children() to try to free up some slots. */
is caught (because we installed a handler for it). If the
latter, call reap_children() to try to free up some slots. */
while (read (job_fds[0], &c->job_token, 1) < 1) r = select (job_fds[0]+1, SELECT_FD_SET_CAST &rfds,
{ NULL, NULL, NULL);
int r = select (job_fds[0]+1, SELECT_FD_SET_CAST &rfds,
NULL, NULL, NULL);
if (r < 0) if (r < 0)
{ {
#ifdef EINTR #ifdef EINTR
if (errno != EINTR) if (errno != EINTR)
/* We should definitely handle this more gracefully! /* We should definitely handle this more gracefully!
What kinds of things can happen here? ^C closes the What kinds of things can happen here? ^C closes the
pipe? Something else closes it? */ pipe? Something else closes it? */
pfatal_with_name ("read jobs pipe"); pfatal_with_name ("read jobs pipe");
#endif #endif
/* We were interrupted; handle any dead children. */ /* We were interrupted; handle any dead children. */
reap_children (1, 0); reap_children (1, 0);
} }
} }
assert(c->job_token != 0); assert(c->job_token != '-');
if (debug_flag) if (debug_flag)
printf ("Obtained token `%c' for child 0x%08lx.\n", printf ("Obtained token `%c' for child 0x%08lx (%s).\n",
c->job_token, (unsigned long int) c); c->job_token, (unsigned long int) c, c->file->name);
} }
#endif #endif
/* If we are running at least one job already and the load average /* If we are running at least one job already and the load average
@ -1203,9 +1245,10 @@ start_waiting_job (c)
case cs_running: case cs_running:
c->next = children; c->next = children;
if (debug_flag) if (debug_flag)
printf ("Putting child 0x%08lx PID %ld%s on the chain.\n", printf ("Putting child 0x%08lx (%s) PID %ld token %c%s on the chain.\n",
(unsigned long int) c, (unsigned long int) c, c->file->name,
(long) c->pid, c->remote ? " (remote)" : ""); (long) c->pid, c->job_token,
c->remote ? " (remote)" : "");
children = c; children = c;
/* One more job slot is in use. */ /* One more job slot is in use. */
++job_slots_used; ++job_slots_used;
@ -1368,7 +1411,7 @@ new_job (file)
c->command_ptr = 0; c->command_ptr = 0;
c->environment = 0; c->environment = 0;
c->sh_batch_file = NULL; c->sh_batch_file = NULL;
c->job_token = 0; c->job_token = '-';
/* Fetch the first command line to be run. */ /* Fetch the first command line to be run. */
job_next_command (c); job_next_command (c);

66
main.c
View File

@ -1172,19 +1172,32 @@ int main (int argc, char ** argv)
functionality here and rely on the signal handler and counting functionality here and rely on the signal handler and counting
children. children.
Also, if we're using the jobs pipe we need a signal handler so that If we're using the jobs pipe we need a signal handler so that
SIGCHLD is not ignored; we need it to interrupt select(2) in SIGCHLD is not ignored; we need it to interrupt select(2) in
job.c:start_waiting_job() if we're waiting on the pipe for a token. */ job.c:start_waiting_job() if we're waiting on the pipe for a token.
Use sigaction where possible as it's more reliable. */
{ {
extern RETSIGTYPE child_handler PARAMS ((int sig)); extern RETSIGTYPE child_handler PARAMS ((int sig));
/* Set up to handle children dying. This must be done before /* Set up to handle children dying. This must be done before
reading in the makefiles so that `shell' function calls will work. */ reading in the makefiles so that `shell' function calls will work. */
#ifndef HAVE_SIGACTION
# define HANDLESIG(s) signal(s, child_handler)
#else
# define HANDLESIG(s) sigaction(s, &sa, NULL)
struct sigaction sa;
bzero ((char *)&sa, sizeof (struct sigaction));
sa.sa_handler = child_handler;
#endif
# if defined SIGCHLD # if defined SIGCHLD
(void) signal (SIGCHLD, child_handler); (void) HANDLESIG (SIGCHLD);
# endif # endif
# if defined SIGCLD && SIGCLD != SIGCHLD # if defined SIGCLD && SIGCLD != SIGCHLD
(void) signal (SIGCLD, child_handler); (void) HANDLESIG (SIGCLD);
# endif # endif
} }
#endif #endif
@ -1267,21 +1280,38 @@ int main (int argc, char ** argv)
#ifdef MAKE_JOBSERVER #ifdef MAKE_JOBSERVER
/* If extended jobs are available then the -j option can have one of 4 /* If extended jobs are available then the -j option can have one of 4
formats: (1) not specified: default is "1"; (2) specified with no formats: (1) not specified: default is "1"; (2) specified with no value:
value: default is "0" (infinite); (3) specified with a single default is "0" (infinite); (3) specified with a single value: this means
value: this means the user wants N job slots; or (4) specified with the user wants N job slots; or (4) specified with 2 values separated by
2 values separated by commas. The latter means we're a submake and a comma. The latter means we're a submake; the two values are the read
the two values are the read and write FDs, respectively, for the and write FDs, respectively, for the pipe. Note this last form is
pipe. Note this last form is undocumented for the user! undocumented for the user! */
*/
sscanf(job_slots_str, "%d", &job_slots); sscanf(job_slots_str, "%d", &job_slots);
{ {
char *cp = index(job_slots_str, ','); char *cp = index(job_slots_str, ',');
/* In case #4, get the FDs. */
if (cp && sscanf(cp+1, "%d", &job_fds[1]) == 1) if (cp && sscanf(cp+1, "%d", &job_fds[1]) == 1)
{ {
job_fds[0] = job_slots; /* Set up the first FD and set job_slots to 0. The combination of a
job_slots = 0; pipe + !job_slots means we're using the jobserver. If !job_slots
and we don't have a pipe, we can start infinite jobs. */
job_fds[0] = job_slots;
job_slots = 0;
/* Make sure the pipe is open! The parent might have closed it
because it didn't think we were a submake. If so, print a warning
then default to -j1. */
if (fcntl (job_fds[0], F_GETFL, 0) < 0
|| fcntl (job_fds[1], F_GETFL, 0) < 0)
{
error (NILF,
"warning: jobserver unavailable (using -j1). Add `+' to parent make rule.");
job_slots = 1;
job_fds[0] = job_fds[1] = -1;
job_slots_str = "1";
}
} }
} }
@ -1293,14 +1323,18 @@ int main (int argc, char ** argv)
char buf[(sizeof("1024")*2)+1]; char buf[(sizeof("1024")*2)+1];
char c = '0'; char c = '0';
if (pipe(job_fds) < 0) if (pipe (job_fds) < 0)
pfatal_with_name("creating jobs pipe"); pfatal_with_name ("creating jobs pipe");
/* Set the read FD to nonblocking; we'll use select() to wait /* Set the read FD to nonblocking; we'll use select() to wait
for it in job.c. */ for it in job.c. */
fcntl (job_fds[0], F_SETFL, O_NONBLOCK); fcntl (job_fds[0], F_SETFL, O_NONBLOCK);
for (; job_slots; --job_slots) /* Every make assumes that it always has one job it can run. For the
submakes it's the token they were given by their parent. For the
top make, we just subtract one from the number the user wants. */
while (--job_slots)
{ {
write(job_fds[1], &c, 1); write(job_fds[1], &c, 1);
if (c == '9') if (c == '9')