Sharing Job Slots with GNU make

GNU make has the ability to run multiple recipes in parallel (see Parallel Execution) and to cap the total number of parallel jobs even across recursive invocations of make (see Communicating Options to a Sub-make). Tools that make invokes which are also able to run multiple operations in parallel, either using multiple threads or multiple processes, can be enhanced to participate in GNU make’s job management facility to ensure that the total number of active threads/processes running on the system does not exceed the maximum number of slots provided to GNU make.

GNU make uses a method called the “jobserver” to control the number of active jobs across recursive invocations. The actual implementation of the jobserver varies across different operating systems, but some fundamental aspects are always true.

First, only command lines that make understands to be recursive invocations of make (see How the MAKE Variable Works) will have access to the jobserver. When writing makefiles you must be sure to mark the command as recursive (most commonly by prefixing the command line with the + indicator (see Recursive Use of make).

Second, make will provide information necessary for accessing the jobserver through the environment to its children, in the MAKEFLAGS environment variable. Tools which want to participate in the jobserver protocol will need to parse this environment variable, as described in subsequent sections.

Third, every command make starts has one implicit job slot reserved for it before it starts. Any tool which wants to participate in the jobserver protocol should assume it can always run one job without having to contact the jobserver at all.

Finally, it’s critical that tools that participate in the jobserver protocol return the exact number of slots they obtained from the jobserver back to the jobserver before they exit, even under error conditions. Remember that the implicit job slot should not be returned to the jobserver! Returning too few slots means that those slots will be lost for the rest of the build process; returning too many slots means that extra slots will be available. The top-level make command will print an error message at the end of the build if it detects an incorrect number of slots available in the jobserver.

As an example, suppose you are implementing a linker which provides for multithreaded operation. You would like to enhance the linker so that if it is invoked by GNU make it can participate in the jobserver protocol to control how many threads are used during link. First you will need to modify the linker to determine if the MAKEFLAGS environment variable is set. Next you will need to parse the value of that variable to determine if the jobserver is available, and how to access it. If it is available then you can access it to obtain job slots controlling how much parallelism your tool can use. Once done your tool must return those job slots back to the jobserver.

• POSIX Jobserver		Using the jobserver on POSIX systems.
• Windows Jobserver		Using the jobserver on Windows systems.