ccache - Online in the Cloud

PROGRAM:

NAME

ccache - a fast C/C++ compiler cache

SYNOPSIS

ccache [options]
ccache compiler [compiler options]
compiler [compiler options] (via symbolic link)

DESCRIPTION

ccache is a compiler cache. It speeds up recompilation by caching the result of previous
compilations and detecting when the same compilation is being done again. Supported
languages are C, C++, Objective-C and Objective-C++.

ccache has been carefully written to always produce exactly the same compiler output that
you would get without the cache. The only way you should be able to tell that you are
using ccache is the speed. Currently known exceptions to this goal are listed under BUGS.
If you ever discover an undocumented case where ccache changes the output of your
compiler, please let us know.

Features
· Keeps statistics on hits/misses.

· Automatic cache size management.

· Can cache compilations that generate warnings.

· Easy installation.

· Low overhead.

· Optionally uses hard links where possible to avoid copies.

· Optionally compresses files in the cache to reduce disk space.

Limitations
· Only knows how to cache the compilation of a single C/C++/Objective-C/Objective-C++
file. Other types of compilations (multi-file compilation, linking, etc) will silently
fall back to running the real compiler.

· Only works with GCC and compilers that behave similar enough.

· Some compiler flags are not supported. If such a flag is detected, ccache will
silently fall back to running the real compiler.

RUN MODES

There are two ways to use ccache. You can either prefix your compilation commands with
ccache or you can let ccache masquerade as the compiler by creating a symbolic link (named
as the compiler) to ccache. The first method is most convenient if you just want to try
out ccache or wish to use it for some specific projects. The second method is most useful
for when you wish to use ccache for all your compilations.

To use the second method on a Debian system, it's easiest to just prepend /usr/lib/ccache
to your PATH. /usr/lib/ccache contains symlinks for all compilers currently installed as
Debian packages.

Alternatively, you can create any symlinks you like yourself like this:

ln -s /usr/bin/ccache /usr/local/bin/gcc
ln -s /usr/bin/ccache /usr/local/bin/g++
ln -s /usr/bin/ccache /usr/local/bin/cc
ln -s /usr/bin/ccache /usr/local/bin/c++

And so forth. This will work as long as the directory with symlinks comes before the path
to the compiler (which is usually in /usr/bin). After installing you may wish to run
“which gcc” to make sure that the correct link is being used.

Warning
The technique of letting ccache masquerade as the compiler works well, but currently
doesn’t interact well with other tools that do the same thing. See USING CCACHE WITH
OTHER COMPILER WRAPPERS.

Warning
Do not use a hard link, use a symbolic link. A hard link will cause “interesting”
problems.

OPTIONS

These options only apply when you invoke ccache as “ccache”. When invoked as a compiler
(via a symlink as described in the previous section), the normal compiler options apply
and you should refer to the compiler’s documentation.

-c, --cleanup
Clean up the cache by removing old cached files until the specified file number and
cache size limits are not exceeded. This also recalculates the cache file count and
size totals. Normally, there is no need to initiate cleanup manually as ccache keeps
the cache below the specified limits at runtime and keeps statistics up to date on
each compilation. Forcing a cleanup is mostly useful if you manually modify the cache
contents or believe that the cache size statistics may be inaccurate.

-C, --clear
Clear the entire cache, removing all cached files, but keeping the configuration file.

-F, --max-files=N
Set the maximum number of files allowed in the cache. Use 0 for no limit. The value is
stored in a configuration file in the cache directory and applies to all future
compilations.

-h, --help
Print an options summary page.

-M, --max-size=SIZE
Set the maximum size of the files stored in the cache. SIZE should be a number
followed by an optional suffix: k, M, G, T (decimal), Ki, Mi, Gi or Ti (binary). The
default suffix is G. Use 0 for no limit. The value is stored in a configuration file
in the cache directory and applies to all future compilations.

-o, --set-config=KEY=VALUE
Set configuration KEY to VALUE. See CONFIGURATION for more information.

-p, --print-config
Print current configuration options and from where they originate (environment
variable, configuration file or compile-time default).

-s, --show-stats
Print the current statistics summary for the cache.

-V, --version
Print version and copyright information.

-z, --zero-stats
Zero the cache statistics (but not the configuration options).

EXTRA OPTIONS

When run as a compiler, ccache usually just takes the same command line options as the
compiler you are using. The only exception to this is the option --ccache-skip. That
option can be used to tell ccache to avoid interpreting the next option in any way and to
pass it along to the compiler as-is. Note: --ccache-skip currently only tells ccache not
to interpret the next option as a special compiler option — the option will still be
included in the direct mode hash.

The reason this can be important is that ccache does need to parse the command line and
determine what is an input filename and what is a compiler option, as it needs the input
filename to determine the name of the resulting object file (among other things). The
heuristic ccache uses when parsing the command line is that any argument that exists as a
file is treated as an input file name. By using --ccache-skip you can force an option to
not be treated as an input file name and instead be passed along to the compiler as a
command line option.

Another case where --ccache-skip can be useful is if ccache interprets an option specially
but shouldn’t, since the option has another meaning for your compiler than what ccache
thinks.

CONFIGURATION

ccache’s default behavior can be overridden by configuration file settings, which in turn
can be overridden by environment variables with names starting with CCACHE_. ccache
normally reads configuration from two files: first a system-level configuration file and
secondly a cache-specific configuration file. The priority of configuration settings is as
follows (where 1 is highest):

1. Environment variables.

2. The cache-specific configuration file <ccachedir>/ccache.conf (typically
$HOME/.ccache/ccache.conf).

3. The system-wide configuration file <sysconfdir>/ccache.conf (typically
/etc/ccache.conf or /usr/local/etc/ccache.conf).

4. Compile-time defaults.

As a special case, if the environment variable CCACHE_CONFIGPATH is set, ccache reads
configuration from the specified path instead of the default paths.

Configuration file syntax
Configuration files are in a simple “key = value” format, one setting per line. Lines
starting with a hash sign are comments. Blank lines are ignored, as is whitespace
surrounding keys and values. Example:

# Set maximum cache size to 10 GB:
max_size = 10G

Boolean values
Some settings are boolean values (i.e. truth values). In a configuration file, such values
must be set to the string true or false. For the corresponding environment variables, the
semantics are a bit different: a set environment variable means “true” regardless of the
value (even if set to the empty string), and an unset environment variable means “false”.
Each boolean environment variable also has a negated form starting with CCACHE_NO. For
example, CCACHE_COMPRESS can be set to force compression and CCACHE_NOCOMPRESS can be set
to force no compression.

Configuration settings
Below is a list of available configuration settings. The corresponding environment
variable name is indicated in parentheses after each configuration setting key. Boolean
options are indicated with “[boolean]”

base_dir (CCACHE_BASEDIR)
This setting should be an absolute path to a directory. ccache then rewrites absolute
paths into relative paths before computing the hash that identifies the compilation,
but only for paths under the specified directory. If set to the empty string (which is
the default), no rewriting is done. See also the discussion under COMPILING IN
DIFFERENT DIRECTORIES.

cache_dir (CCACHE_DIR)
This setting specifies where ccache will keep its cached compiler outputs. It will
only take effect if set in the system-wide configuration file or as an environment
variable. The default is $HOME/.ccache.

cache_dir_levels (CCACHE_NLEVELS)
This setting allows you to choose the number of directory levels in the cache
directory. The default is 2. The minimum is 1 and the maximum is 8.

compiler (CCACHE_CC)
This setting can be used to force the name of the compiler to use. If set to the empty
string (which is the default), ccache works it out from the command line.

compiler_check (CCACHE_COMPILERCHECK)
By default, ccache includes the modification time (“mtime”) and size of the compiler
in the hash to ensure that results retrieved from the cache are accurate. This setting
can be used to select another strategy. Possible values are:

content
Hash the content of the compiler binary. This makes ccache very slightly slower
compared to the mtime setting, but makes it cope better with compiler upgrades
during a build bootstrapping process.

mtime
Hash the compiler’s mtime and size, which is fast. This is the default.

none
Don’t hash anything. This may be good for situations where you can safely use the
cached results even though the compiler’s mtime or size has changed (e.g. if the
compiler is built as part of your build system and the compiler’s source has not
changed, or if the compiler only has changes that don’t affect code generation).
You should only use the none setting if you know what you are doing.

string:value
Use value as the string to calculate hash from. This can be the compiler revision
number you retrieved earlier and set here via environment variable.

a command string
Hash the standard output and standard error output of the specified command. The
string will be split on whitespace to find out the command and arguments to run.
No other interpretation of the command string will be done, except that the
special word %compiler% will be replaced with the path to the compiler. Several
commands can be specified with semicolon as separator. Examples:

· %compiler% -v

· %compiler% -dumpmachine; %compiler% -dumpversion

You should make sure that the specified command is as fast as possible since it
will be run once for each ccache invocation.

Identifying the compiler using a command is useful if you want to avoid cache
misses when the compiler has been rebuilt but not changed.

Another case is when the compiler (as seen by ccache) actually isn’t the real
compiler but another compiler wrapper — in that case, the default mtime method
will hash the mtime and size of the other compiler wrapper, which means that
ccache won’t be able to detect a compiler upgrade. Using a suitable command to
identify the compiler is thus safer, but it’s also slower, so you should consider
continue using the mtime method in combination with the prefix_command setting if
possible. See USING CCACHE WITH OTHER COMPILER WRAPPERS.

compression (CCACHE_COMPRESS) [boolean]
If true, ccache will compress object files and other compiler output it puts in the
cache. However, this setting has no effect on how files are retrieved from the cache;
compressed and uncompressed results will still be usable regardless of this setting.
The default is false.

compression_level (CCACHE_COMPRESSLEVEL)
This setting determines the level at which ccache will compress object files. It only
has effect if compression is enabled. The value defaults to 6, and must be no lower
than 1 (fastest, worst compression) and no higher than 9 (slowest, best compression).

cpp_extension (CCACHE_EXTENSION)
This setting can be used to force a certain extension for the intermediate
preprocessed file. The default is to automatically determine the extension to use for
intermediate preprocessor files based on the type of file being compiled, but that
sometimes doesn’t work. For example, when using the “aCC” compiler on HP-UX, set the
cpp extension to i.

direct_mode (CCACHE_DIRECT) [boolean]
If true, the direct mode will be used. The default is true. See THE DIRECT MODE.

disable (CCACHE_DISABLE) [boolean]
When true, ccache will just call the real compiler, bypassing the cache completely.
The default is false.

extra_files_to_hash (CCACHE_EXTRAFILES)
This setting is a list of paths to files that ccache will include in the the hash sum
that idetifies the build. The list separator is semicolon on Windows systems and colon
on other systems.

hard_link (CCACHE_HARDLINK) [boolean]
If true, ccache will attempt to use hard links from the cache directory when creating
the compiler output rather than using a file copy. Using hard links may be slightly
faster in some situations, but can confuse programs like “make” that rely on
modification times. Another thing to keep in mind is that if the resulting object file
is modified in any way, this corrupts the cached object file as well. Hard links are
never made for compressed cache files. This means that you should not enable
compression if you want to use hard links. The default is false.

hash_dir (CCACHE_HASHDIR) [boolean]
If true, ccache will include the current working directory in the hash that is used to
distinguish two compilations. This prevents a problem with the storage of the current
working directory in the debug info of a object file, which can lead ccache to give a
cached object file that has the working directory in the debug info set incorrectly.
This option is off by default as the incorrect setting of this debug info rarely
causes problems. If you strike problems with GDB not using the correct directory then
enable this option.

log_file (CCACHE_LOGFILE)
If set to a file path, ccache will write information on what it is doing to the
specified file. This is useful for tracking down problems.

max_files (CCACHE_MAXFILES)
This option specifies the maximum number of files to keep in the cache. Use 0 for no
limit (which is the default).

max_size (CCACHE_MAXSIZE)
This option specifies the maximum size of the cache. Use 0 for no limit. The default
value is 5G. Available suffixes: k, M, G, T (decimal) and Ki, Mi, Gi, Ti (binary). The
default suffix is "G".

path (CCACHE_PATH)
If set, ccache will search directories in this list when looking for the real
compiler. The list separator is semicolon on Windows systems and colon on other
systems. If not set, ccache will look for the first executable matching the compiler
name in the normal PATH that isn’t a symbolic link to ccache itself.

prefix_command (CCACHE_PREFIX)
This option adds a list of prefixes (separated by space) to the command line that
ccache uses when invoking the compiler. See also USING CCACHE WITH OTHER COMPILER
WRAPPERS.

read_only (CCACHE_READONLY) [boolean]
If true, ccache will attempt to use existing cached object files, but it will not to
try to add anything new to the cache. If you are using this because your ccache
directory is read-only, then you need to set temporary_dir as otherwise ccache will
fail to create temporary files.

read_only_direct (CCACHE_READONLY_DIRECT) [boolean]
Just like read_only except that ccache will only try to retrieve results from the
cache using the direct mode, not the preprocessor mode. See documentation for
read_only regarding using a read-only ccache directory.

recache (CCACHE_RECACHE) [boolean]
If true, ccache will not use any previously stored result. New results will still be
cached, possibly overwriting any pre-existing results.

run_second_cpp (CCACHE_CPP2) [boolean]
If true, ccache will not use the optimisation of avoiding the second call to the
preprocessor by compiling the preprocessed output that was used for finding the hash
in the case of a cache miss. This is primarily a debugging option, although it is
possible that some unusual compilers will have problems with compiling the
preprocessed output, in which case this option could allow ccache to be used anyway.

sloppiness (CCACHE_SLOPPINESS)
By default, ccache tries to give as few false cache hits as possible. However, in
certain situations it’s possible that you know things that ccache can’t take for
granted. This setting makes it possible to tell ccache to relax some checks in order
to increase the hit rate. The value should be a comma-separated string with options.
Available options are:

file_macro
Ignore __FILE__ being present in the source.

file_stat_matches
ccache normally examines a file’s contents to determine whether it matches the
cached version. With this option set, ccache will consider a file as matching its
cached version if the sizes, mtimes and ctimes match.

include_file_ctime
By default, ccache also will not cache a file if it includes a header whose ctime
is too new. This option disables that check.

include_file_mtime
By default, ccache will not cache a file if it includes a header whose mtime is
too new. This option disables that check.

pch_defines
Be sloppy about #defines when precompiling a header file. See PRECOMPILED HEADERS
for more information.

time_macros
Ignore __DATE__ and __TIME__ being present in the source code.

See the discussion under TROUBLESHOOTING for more information.

stats (CCACHE_STATS) [boolean]
If true, ccache will update the statistics counters on each compilation. The default
is true.

temporary_dir (CCACHE_TEMPDIR)
This setting specifies where ccache will put temporary files. The default is
<cache_dir>/tmp.

Note
In previous versions of ccache, CCACHE_TEMPDIR had to be on the same filesystem as
the CCACHE_DIR path, but this requirement has been relaxed.)

umask (CCACHE_UMASK)
This setting specifies the umask for ccache and all child processes (such as the
compiler). This is mostly useful when you wish to share your cache with other users.
Note that this also affects the file permissions set on the object files created from
your compilations.

unify (CCACHE_UNIFY) [boolean]
If true, ccache will use a C/C++ unifier when hashing the preprocessor output if the
-g option is not used. The unifier is slower than a normal hash, so setting this
environment variable loses a little bit of speed, but it means that ccache can take
advantage of not recompiling when the changes to the source code consist of
reformatting only. Note that enabling the unifier changes the hash, so cached
compilations produced when the unifier is enabled cannot be reused when the unifier is
disabled, and vice versa. Enabling the unifier may result in incorrect line number
information in compiler warning messages and expansions of the __LINE__ macro. Also
note that enabling the unifier implies turning off the direct mode.

CACHE SIZE MANAGEMENT

By default, ccache has a five gigabyte limit on the total size of files in the cache and
no maximum number of files. You can set different limits using the -M/--max-size and
-F/--max-files options. Use ccache -s/--show-stats to see the cache size and the currently
configured limits (in addition to other various statistics).

CACHE COMPRESSION

ccache can optionally compress all files it puts into the cache using the compression
library zlib. While this may involve a tiny performance slowdown, it increases the number
of files that fit in the cache. You can turn on compression with the compression
configuration setting and you can also tweak the compression level with compression_level.

HOW CCACHE WORKS

The basic idea is to detect when you are compiling exactly the same code a second time and
reuse the previously produced output. The detection is done by hashing different kinds of
information that should be unique for the compilation and then using the hash sum to
identify the cached output. ccache uses MD4, a very fast cryptographic hash algorithm, for
the hashing. (MD4 is nowadays too weak to be useful in cryptographic contexts, but it
should be safe enough to be used to identify recompilations.) On a cache hit, ccache is
able to supply all of the correct compiler outputs (including all warnings, dependency
file, etc) from the cache.

ccache has two ways of doing the detection:

· the direct mode, where ccache hashes the source code and include files directly

· the preprocessor mode, where ccache runs the preprocessor on the source code and
hashes the result

The direct mode is generally faster since running the preprocessor has some overhead.

Common hashed information
For both modes, the following information is included in the hash:

· the extension used by the compiler for a file with preprocessor output (normally .i
for C code and .ii for C++ code)

· the compiler’s size and modification time (or other compiler-specific information
specified by the compiler_check setting)

· the name of the compiler

· the current directory (if the hash_dir setting is enabled)

· contents of files specified by the extra_files_to_hash setting (if any)

The direct mode
In the direct mode, the hash is formed of the common information and:

· the input source file

· the command line options

Based on the hash, a data structure called “manifest” is looked up in the cache. The
manifest contains:

· references to cached compilation results (object file, dependency file, etc) that were
produced by previous compilations that matched the hash

· paths to the include files that were read at the time the compilation results were
stored in the cache

· hash sums of the include files at the time the compilation results were stored in the
cache

The current contents of the include files are then hashed and compared to the information
in the manifest. If there is a match, ccache knows the result of the compilation. If there
is no match, ccache falls back to running the preprocessor. The output from the
preprocessor is parsed to find the include files that were read. The paths and hash sums
of those include files are then stored in the manifest along with information about the
produced compilation result.

There is a catch with the direct mode: header files that were used by the compiler are
recorded, but header files that were not used, but would have been used if they existed,
are not. So, when ccache checks if a result can be taken from the cache, it currently
can’t check if the existence of a new header file should invalidate the result. In
practice, the direct mode is safe to use in the absolute majority of cases.

The direct mode will be disabled if any of the following holds:

· the configuration setting direct_mode is false

· a modification time of one of the include files is too new (needed to avoid a race
condition)

· the unifier is enabled (the configuration setting unify is true)

· a compiler option not supported by the direct mode is used:

· a -Wp,X compiler option other than -Wp,-MD,path and -Wp,-MMD,path

· -Xpreprocessor

· the string “__TIME__” is present in the source code

The preprocessor mode
In the preprocessor mode, the hash is formed of the common information and:

· the preprocessor output from running the compiler with -E

· the command line options except options that affect include files (-I, -include, -D,
etc; the theory is that these options will change the preprocessor output if they have
any effect at all)

· any standard error output generated by the preprocessor

Based on the hash, the cached compilation result can be looked up directly in the cache.

COMPILING IN DIFFERENT DIRECTORIES

Some information included in the hash that identifies a unique compilation may contain
absolute paths:

· The preprocessed source code may contain absolute paths to include files if the
compiler option -g is used or if absolute paths are given to -I and similar compiler
options.

· Paths specified by compiler options (such as -I, -MF, etc) may be absolute.

· The source code file path may be absolute, and that path may substituted for __FILE__
macros in the source code or included in warnings emitted to standard error by the
preprocessor.

This means that if you compile the same code in different locations, you can’t share
compilation results between the different build directories since you get cache misses
because of the absolute build directory paths that are part of the hash. To mitigate this
problem, you can specify a “base directory” in the configuration setting base_dir to an
absolute path to the directory. ccache will then rewrite absolute paths that are under the
base directory (i.e., paths that have the base directory as a prefix) to relative paths
when constructing the hash. A typical path to use as the base directory is your home
directory or another directory that is a parent of your build directories. (Don’t use / as
the base directory since that will make ccache also rewrite paths to system header files,
which doesn’t gain anything.)

The drawbacks of using a base directory are:

· If you specify an absolute path to the source code file, __FILE__ macros will be
expanded to a relative path instead.

· If you specify an absolute path to the source code file and compile with -g, the
source code path stored in the object file may point to the wrong directory, which may
prevent debuggers like GDB from finding the source code. Sometimes, a work-around is
to change the directory explicitly with the “cd” command in GDB.

PRECOMPILED HEADERS

ccache has support for GCC’s precompiled headers. However, you have to do some things to
make it work properly:

· You must set sloppiness to pch_defines,time_macros. The reason is that ccache can’t
tell whether __TIME__ or __DATE__ is used when using a precompiled header. Further, it
can’t detect changes in #defines in the source code because of how preprocessing works
in combination with precompiled headers.

· You must either:

· use the -include compiler option to include the precompiled header (i.e., don’t
use #include in the source code to include the header); or

· (for the Clang compiler) use the -include-pch compiler option to include the PCH
file generated from the precompiled header; or

· add the -fpch-preprocess compiler option when compiling.

If you don’t do this, either the non-precompiled version of the header file will be
used (if available) or ccache will fall back to running the real compiler and increase
the statistics counter “preprocessor error” (if the non-precompiled header file is not
available).

SHARING A CACHE

A group of developers can increase the cache hit rate by sharing a cache directory. To
share a cache without unpleasant side effects, the following conditions should to be met:

· Use the same cache directory.

· Make sure that the configuration setting hard_link is false (which is the default).

· Make sure that all users are in the same group.

· Set the configuration setting umask to 002. This ensures that cached files are
accessible to everyone in the group.

· Make sure that all users have write permission in the entire cache directory (and that
you trust all users of the shared cache).

· Make sure that the setgid bit is set on all directories in the cache. This tells the
filesystem to inherit group ownership for new directories. The command “find
$CCACHE_DIR -type d | xargs chmod g+s” might be useful for this.

The reason to avoid the hard link mode is that the hard links cause unwanted side effects,
as all links to a cached file share the file’s modification timestamp. This results in
false dependencies to be triggered by timestamp-based build systems whenever another user
links to an existing file. Typically, users will see that their libraries and binaries are
relinked without reason.

You may also want to make sure that a base directory is set appropriately, as discussed in
a previous section.

SHARING A CACHE ON NFS

It is possible to put the cache directory on an NFS filesystem (or similar filesystems),
but keep in mind that:

· Having the cache on NFS may slow down compilation. Make sure to do some benchmarking
to see if it’s worth it.

· ccache hasn’t been tested very thoroughly on NFS.

A tip is to set temporary_dir to a directory on the local host to avoid NFS traffic for
temporary files.

USING CCACHE WITH OTHER COMPILER WRAPPERS

The recommended way of combining ccache with another compiler wrapper (such as “distcc”)
is by letting ccache execute the compiler wrapper. This is accomplished by defining the
configuration setting prefix_command, for example by setting the environment variable
CCACHE_PREFIX to the name of the wrapper (e.g. distcc). ccache will then prefix the
command line with the specified command when running the compiler. To specify several
prefix commands, set prefix_command to a colon-separated list of commands.

Unless you set compiler_check to a suitable command (see the description of that
configuration option), it is not recommended to use the form ccache anotherwrapper
compiler args as the compilation command. It’s also not recommended to use the
masquerading technique for the other compiler wrapper. The reason is that by default,
ccache will in both cases hash the mtime and size of the other wrapper instead of the real
compiler, which means that:

· Compiler upgrades will not be detected properly.

· The cached results will not be shared between compilations with and without the other
wrapper.

Another minor thing is that if prefix_command is used, ccache will not invoke the other
wrapper when running the preprocessor, which increase performance.

Use ccache online using onworks.net services