include_server - Online in the Cloud

PROGRAM:

NAME

include_server.py - conservative approximation of include dependencies for C/C++

SYNOPSIS

include_server --port INCLUDE_SERVER_PORT [OPTIONS]

DESCRIPTION

include_server.py starts an include server process. This process answers queries from
distcc(1) clients about what files to include in C/C++ compilations. The include_server.py
command itself terminates as soon as the include server has been spawned.

The INCLUDE_SERVER_PORT argument is the name of a socket used for all communication
between distcc clients and the include server. The distcc-pump(1) command is responsible
for creating the socket location, for passing it to this script, and for passing it to all
distcc clients via the environment variable named INCLUDE_SERVER_PORT.

The protocol used by the include server uses distcc's RPC implementation. Each distcc
request consists of (1) the current directory and (2) the list of arguments of the
compilation command.

If the include server is able to process the request, then it answers the distcc client by
sending a list of filepaths. The filepaths are those of the compressed source and header
files found to be necessary for compilation through include analysis. The list also
comprises symbolic links and even dummy files needed for the compilation server to
construct an accurate replica of the parts of the filesystem needed for compilation. In
this way, a needed header file like /path/foo.h is compressed, renamed, and stored in a
temporary location, such as /dev/shm/tmpiAvfGv.include_server-9368-1/path/foo.h.lzo. The
distcc client will pass these files on to a compilation server, where they will be
uncompressed and mounted temporarily.

If the include server is not able to process the request, then it returns the empty list
to the distcc client.

There are two kinds of failures that relate to the include server. The include server may
fail to compute the includes or fail in other ways, see section INCLUDE SERVER SYMPTOMS.
Also, the compilation on the remove server may fail due to inadequacy of the calculated
include closure, but then succeed when locally retried, see section DISTCC DISCREPANCY
SYMPTOMS.

OPTION SUMMARY

The following options are understood by include_server.py.

-dPAT, --debug_pattern=PAT
Bit vector for turning on warnings and debugging
1 = warnings
2 = trace some functions
other powers of two: see include_server/basics.py.

-e, --email
Send email to 'distcc-pump-errors' or if defined, the value of enviroment variable
DISTCC_EMAILLOG_WHOM_TO_BLAME, when include server gets in trouble. The default is
to not send email.

--email_bound NUMBER
Maximal number of emails to send (in addition to a final email). Default: 3.

--no-email
Do not send email. This is the default.

--path_observation_re=RE
Issue warning message whenever a filename is resolved to a realpath that is matched
by RE, which is a regular expression in Python syntax. This is useful for finding
out where files included actually come from. Use RE="" to find them all. Note:
warnings must be enabled with at least -d1.

--pid_file FILEPATH
The pid of the include server is written to file FILEPATH. This allows a script
such a distcc-pump to tear down the include server.

-s, --statistics
Print information to stdout about include analysis.

--stat_reset_triggers=LIST
Flush stat caches when the timestamp of any filepath in LIST changes or the
filepath comes in or out of existence. LIST is a colon separated string of
filepaths, possibly containing simple globs (as allowed by Python's glob module).
Print a warning whenever such a change happens (if warnings are enabled). This
option allows limited exceptions to distcc-pump's normal assumption that source
files are not modified during the build.

-t, --time
Print elapsed, user, and system time to stderr.

--unsafe_absolute_includes
Do preprocessing on the compilation server even if includes of absolute filepaths
are encountered. Normally the include-server will fall back on local preprocessing
if it detects any absolute includes. Thus, this flag is useful for preventing such
fallbacks when the absolute includes are a false alarm, either because the absolute
include is discarded during preprocessing or because the absolutely included file
exists on the compilation servers.

More precisely, with --unsafe_absolute_includes absolute includes are ignored for
the purposes of gathering the include closure. Using this option may lead to
incorrect results because (1) the header may actually be included on the
compilation server and it may not be the same as on the client, (2) the include
directives of the header are not further analyzed.

The option is useful for compiling code that has such hardcoded absolute locations
of header files inside conditional directives (e.g. "#ifdef") that render the
includes irrelevant. More precisely, these includes must be eliminated during
preprocessing for the actual configuration. Then the question of existence of the
header file is moot and the remote compilation is sound. This is often the case if
such includes are meant for unusual configurations different from the actual
configuration.

-v, --verify
Verify that files in CPP closure are contained in closure calculated by include
processor.

-w, --write_include_closure
Write a .d_approx file which lists all the included files calculated by the include
server; with -x, additionally write the included files as calculated by CPP to a
.d_exact file.

-x, --exact_analysis
Use CPP instead, do not omit system headers files.

INCLUDE SERVER SYMPTOMS AND ISSUES

The most likely messages and warnings to come from the include processor are listed below.

Preprocessing locally. Include server not covering: Couldn't determine default system
include directories
To determine the default system header directories, the include server runs the
compiler once for each language needed during its session. This message indicates
that the compiler specified to distcc is not present on the client.

Preprocessing locally. Include server not covering: Bailing out because include server
spent more than ...s user time handling request
In uncommon situations, the include server fails to analyze very complicated macro
expressions. The distcc client will use plain distcc mode.

Warning: Filepath must be relative but isn't
The include server does not accept absolute filepaths, such as
/usr/include/stdio.h, in include directives, because there is no guarantee that
this header on the compilation server machine will be the same as that on the
client. The include server gives up analyzing the include closure. The distcc
client cannot use pump-mode.

To overcome this problem in a not always reliable way, set the environment variable
INCLUDE_SERVER_ARGS='--unsafe_absolute_includes' when invoking the distcc-pump
script to pass the --unsafe_absolute_includes option to the include server.

Warning: Absolute filepath ... was IGNORED
The --unsafe_absolute_includes is in use. This situation happens under the same
circumstances as when "Filepath must be relative but isn't" is issued, but in this
case the include will provide an answer to the distcc client.

Warning: Path '/PATH/FILE' changed/came into existence/no longer exists
These warnings are issued when using stat reset triggers. Because /PATH/FILE
changed, the include server clears its caches; the new version of the file (or the
lack of it) renders the include analysis invalid. This message can usually be
ignored; it does signify a somewhat precarious use of files by the build system.
It is recommended to fix the build system so that files are not rewritten.

Warning: For translation unit ..., lookup of file ... resolved to ... whose realpath is
...
This warning occurs with --path_observation_re when a new realpath matching a
source or header file is observed.

DISTCC DISCREPANCY SYMPTOMS

The interactions between the build system, distcc, and the include server is somewhat
complex. When a distcc commands receives a failing compilation from the remote server it
retries the compilation locally. This section discusses the causes of discrepancies
between remote and local compilation. These are flagged by the demotion message:

__________Warning: ... pump-mode compilation(s) failed on server, but succeeded locally.
__________Distcc-pump was demoted to plain mode. See the Distcc Discrepancy Symptoms
section in the include_server(1) man page.

The distcc-pump script issues this message at the end of the build. This means that for at
least one distcc invocation a local compilation succeeded after the remote compilation
failed. Each distcc invocation for which such a discrepancy occurred in turn also issues a
message such as:

Warning: remote compilation of '...' failed, retried locally and got a different result.

The demotion makes subsequent distcc invocations use plain distcc mode. Thus
preprocessing will take place on the local machine for the remainder of the build. This
technique prevents very slow builds where all compilations end up on the local machine
after failing remotely.

Of course, if the local compilations fails after the remote failures, then the distcc
invocation exits with the non-zero status of the local compilation. The error messages
printed are also those of the local compilation.

The fallback behavior for distcc-pump mode to local compilation can be disabled by setting
the environment variable DISTCC_FALLBACK to 1, which makes the distcc command fail as soon
as the remote compilation has failed. This setting is very useful for debugging why the
remote compilation went wrong, because now the output from the server will be printed.

Next we discuss the possible causes of discrepancies.

The user changed a source or header file during the build.
This yields inconsistent results of course.

A source or header file changed during the build.
The build system rewrites a file. For Linux kernel 2.6, this happens for
'include/linux/compile.h' and 'include/asm/asm-offsets.h'. This condition is fixed
by letting the include server know that it must reset its caches when a stat of any
of the files changes. Practically, this is done by gathering the files in a colon-
separated list and then setting the INCLUDE_SERVER_ARGS command as in:

include_server_args="--stat_reset_triggers=include/linux/compile.h:include/asm/asm-
offsets.h"

A header file is potentially included, but does not exist, and is then later included.

This occurs when some header foo.h includes another header file trick.h, but the
trick.h file has not yet been generated and the inclusion is actually ignored
because of preprocessing directives. The include server will probe for the
existence of trick.h, because it overapproximates all possible ways directives
actually evaluate. The file trick.h is determined not to exist. If it is later
generated, and then really included, then the include server will falsely believe
that the file still does not exist. The solution to this problem is to make the
build system generate trick.h before the first time any header file is included
that makes a syntactic reference to trick.h

The include server was started with --unsafe_absolute_includes.
This is a problem if there are header files locally that do not exist remotely and
that are actually used. Such includes are often protected by conditional
directives that evaluate so that are actually used on only specific and often
uncommon platforms. If you are not compiling for such a platform, then it may be
correct to use --unsafe_absolute_include.

The include server has calculated the wrong includes.
We do not know of such a situation.

EXIT CODES

The exit code of include_server.py is usually 0. That the include server has been started
properly is communicated through the existence of the pid_file.

ENVIRONMENT VARIABLES

DISTCC_EMAILLOG_WHOM_TO_BLAME The email address to use for include server automated
emails. The default is 'distcc-pump-errors' (which is an email address that probably will
not exist in your domain).

Additionally, the invocation of the compiler may use additional environment variables.

Use include_server online using onworks.net services