The MakeXS Build System Explained

by Gianni Mariani

V1.5

9-May-2004

Introduction

MakeXS is a build system that enables a set of functions/targets consistently across an entire source tree using a base Makefile named MakeXS. Each of these functions/targets can be tweaked locally by setting various values in the local folder's Makefile minimizing the need to build complex Makefiles in each folder of the source tree.

MakeXS makes extensive use of GNU make and GNU m4 and some other system utilities. All systems that work with these tools and have a compliant Bourne shell and system utilities should work with MakeXS. Other build tools that may provide similar functionality, MakeXS uses the philosophy that the most common tools are sufficient and builds upon them. Other potential tools include:

• Ant

• bras

• Cons

• Cook

• imake

• Jam

In some ways MakeXS is similar to Autoconf however, MakeXS does not require to perform an initial configure to create all the requireed Makefiles which reduces the time to maintain the build environment. Autoconf has many more features than MakeXS but the goal for MakeXS is strictly confined to the build system. Automatic configuration can and should be performed as part of the build step.

A list of other similar tools may be found at the Open Directory web site: http://dmoz.org/Computers/Software/Build_Management/Make_Tools/ .

MakeXS Basics

The provenance of MakeXS is the simplification of the build process. Simplistically, one should simply change directory to the root of the tree and type 'make'. If configuration is required it should be performed automaticaly or a special configuration file should be provided.

The MakeXS Design Goals

MakeXS is the result of attempting to optimize the following set of features.

• Automate as much as possible. Computers are much faster at doing mundane tasks. MakeXS attempts to create a structure that enables the make tools to eliminate error prone mundane work. When new files or folders are added to the project, minimal changes to the Makefiles should be needed.

• Provide simple debugging facilities. Remove guess work in the Makefile behavior. Often it is difficult to know what value an environment variable has, providing more information about various environment variables aids in the debugging process.

• Utilize simple Makefiles. Provide a standard set of features that enable the use of simple consistent Makefiles. Also, in theory, these 'simple' Makefiles could be used as input for a different build system.

• Common behavior defined in a single place. Avoid having to modify all Makefiles if a change of behavior is desired.

• Allow architecture customization. Allow easy customization for different architectures.

• Cross platform builds within a single tree A single tree should be able to be built on multiple architectures. This reduces the need to check in changes without having done builds on other platforms.

GNU make Basics

There are many make tools available but one that is available everywhere and is consistent is the GNU make tool. More information about GNU make can be found using the "info" utility (info make). GNU make is robust and performs well, including on windows systems using cygwin.

The "make" system consists of a rule processor and a sophisticated macro language that enables the manipulation of the rule sets. These rule sets are located in a file called "Makefile" and referred to as a Makefile. A rule is invoked if the requested target can be created with that rule. The make tool will traverse a set of rules and discover the set of rules that can be invoked to build a target and then perform the commands at each step. The make tool will also detect if a target is stale by looking at all the components of a target and verifying that the time stamp of the target is newer than all the components. The make system will also remove any components built by intermediate rules if not specifically requested.

GNU make is also capable of performing builds in parallel by attempting to discover multiple rules that may be executed in parallel.

A Reference Point or MXI_ROOT

A source tree consists of various folders containing sources. During the build process, the locations of various folders are important as different components of the source tree refer to one another. MakeXS defines an environment variable named BUILD_ROOT for creating paths to various source tree components.

To minimize chances of error with incorrectly set environment variables, the MakeXS system will automatically locate the build tree's root directory. Because one may choose to build in any folder, the tree root location algorithm must be placed within every folder containing a Makefile. This locator script is itself a make(1) source file called "Makefile.xsi". Makefile.xsi locates the tree root, sets the "MXI_ROOT" environment variable and includes a base Makefile named "MakeXS" located in a folder named "make" off the tree root.

To summarize, the environment variable MXI_ROOT is computed by "Makefile.xsi". Makefile.xsi also includes $(MXI_ROOT)/make/MakeXS is expected to be included in all Makefiles in the source tree except for the file "make/Makefile" which is used to generate MakeXS.

Standard Rules

The file MakeXS contains a set of basic build rules and macro definitions that can be modified in each folder by the local Makefile. The aim is to have each local Makefile specify modifications to the MakeXS rules through various macros that are specific for that folder. The MakeXS file contains a comment describing all the available parameters.

The file MakeXS contains rules for recursive execution of Makefiles in subdirectories. Since each folder may be interdependent, it important to provide a mechanism for a local Makefile to specify those dependencies. To achieve this, MakeXS provides targets for each well-known target in all subdirectories containing a Makefile.

The example below indicates that the target all in the src subdirectory is dependant on the all targets in the subdirectories tools, external, and libs. This will force make to descend into tools, external and libs folders for the 'all' target before attempting to build the 'all' target in the src folder.

src/all : tools/all external/all libs/all

The next example indicates that the target "distclean" in the make folder is dependant on all other distclean folders. This would force make to execute the distclean target in the 'make' folder only after all other folder's distclean target's are done.

    make/distclean : $(subst make/distclean,,$(MXS_MAKEDIRS:%=%/distclean))

Note the use of the variable MXS_MAKEDIRS that is a list of subdirectories containing a file named 'Makefile'.

The standard rules currently defined in MakeXS are:

For development builds.

all perform regular build - create all build files

setup on a top level make it will build a directory structure

test run unit test

clean remove intermediate build files

mostlyclean like clean

distclean revert to checked out state

docs build whatever docs need building

install targets that remove from installation

cvsadd_all add all files to CVS - do this after a distclean

For release builds.

r_all perform regular build - create all build files

r_setup on a top level make it will build a directory structure

r_test run unit test

r_clean remove intermediate build files

r_mostlyclean like clean

r_distclean revert to checked out state

r_docs build whatever docs need building

r_install targets that remove from installation

r_cvsadd_all add all files to CVS - do this after a distclean

Build Architecture and Build Folders

It is often desirable to place intermediate within their own folders to eliminate pollution of the source folders and to enable cross-compiles within the same tree. MakeXS attempts to discover the architecture if not specified. The environment variable BUILD_ARCH can be set to the desired build architecture.

Include files

MakeXS will automatically discover the directories in the tree that contain ".h" files and will create compiler include parameters for compilers.

MakeXS also provides a helper rule to help locate a header file given the generated list. Simply issue the 'make' command with a whereis/ followed by the name of the header file. e.g.

$ make whereis/stdio.h

/usr/local/lib/gcc-lib/i686-pc-linux-gnu/3.0/include/stdio.h

Note that whenever header files are added in new directories in the tree, the header file list needs to be re-evaluated. This is simply done by removing the MakeXS file. Upon the next invocation of a Makefile, the MakeXS file is re-created and the include list is re-evaluated.

Helper rules

• build_arch, Print the environment variable MXI_ARCH. This variable is discovered in MakeXS if not defined before invocation of make.

• build_prefix, The prefix string pre-pended to all generated files.

• build_root, The value of the MXI_ROOT environment variable.

• env, The environment rule. e.g.

$ make env/MXI_ARCH

gx86

• whereis The locate header file rule. e.g.

$ make whereis/stdio.h
/usr/local/lib/gcc-lib/i686-pc-linux-gnu/3.0/include/stdio.h

MakeXS Explained

There are 6 files and 2 folders in a minimal MakeXS build system. These include the root level Makefile and Makefile.xsi which create the basic Makefile.

root

|

+ Makefile > Root level Makefile

+ Makefile.xsi > The common Makefile.xsi

|

+ make

|

+ Makefile > Special root/make/Makefile for building MakeXS

+ MakeXS.m4 > MakeXS source file

+ MakeXSConf.m4 > Application defiles

+ doxyscript > Script to create doxygen config file

Below is an example of a simple MakeXS Makefile. This Makefile will enable basic recursion.

#

# Basic MakeXS Makefile

include Makefile.xsi

A root level Makefile should implement the distclean rule shown below and discard the MXI_ROOT variable. This distclean rule is required because all directories require the MakeXS file and if it is cleaned, it will be automatically re-created and hence the make directory will be unclean. To enable multiple source trees to be built within a larger source tree, it is important to isolate the meaning of MXI_ROOT, this is done by defining MXI_ROOT as the empty string. Below is an example of a top level Makefile.

#

# This is the top level Makefile

#

#

# MXI_ROOT is different for different places. This forces the

# MXI_ROOT to be re-evaluated at the top level in case this tree

# is used within other trees.

#

MXI_ROOT=

include Makefile.xsi

make/distclean : $(subst make/distclean,,$(MXS_MAKEDIRS:%=%/distclean))

The structure of MakeXS Makefiles is in three components. The first component are the local folder's defines, the second is simply the inclusion of the Makefile.xsi the third is for any folder specific make rules. The order of these three components is important.

#

# Makefile based on MakeXS.

#

<place your build specific options here>

include Makefile.xsi

<place your folder specific make rules here>

MakeXS will also include any files named Makefile.defs and Makefile.<MXI_ARCH>.defs. These allow architecture dependant Makefiles.

Architecture Dependant Make Files

Some system architectures may require vastly different rules. Many of these rules may be difficult to alter for a different architecture. MakeXS provides two facilities for this case. The first is substitution of a Makefile based on MXI_ARCH and the other is the inclusion of a build specific architecture defs file.

Files named Makefile.<MXI_ARCH>.defs and Makefile.<MXI_ARCH> can change the behaviour of MakeXS in an architecture dependant way. If Makefile.<MXI_ARCH> exists, it is used instead of the file named Makefile. This is only done if executed from the parent folder because 'make' by default executes the Makefile hence, it is an undesirable option. Makefile.<MXI_ARCH>.defs is included by MakeXS after the build architecture is determined and so can make various changes to make variables that can't be made in the base Makefile itself.

To override default rules in a file, you may override the target that is being referenced. The environment variable MXS_BASE_TARGET_ALL defaults to all_default. If this is changed then the dependency that links to the all target is removed and those targets are no longer built.

Below is an example of a Makefile.<MXI_ARCH>.defs file that disables the default actions of the all as far as the targets described by targets in the variable MXS_TARGETS_ALL.

#

# This defs file should avoid any default "all"

# targets in this directory

#

#

.PHONY: all_dont_build

IN_BASE_TARGET_ALL=all_dont_build

MSDEV (Visual C++ 6.0 builds)

MSDEV projects can be easily built using MakeXS by using the MakeXS built-in MSDEV rules. By simply adding 'IN_TARGETS_ALL=msdev32_build' within an architecture dependant Makefile, a build of the tree may be performed. It is important that the MSDEV project file (".dsp" file) follow a set of conventions.

1. The output target directory for all targets within the project file must be the same as $(MXS_BUILDDIR).

2. The target base name must be the same name as the project file base name. (e.g. name.dsp -> name.exe)

3. The extension of the target must be specified like - IN_MSDEV32_RESULT_EXTENSION=exe . (lib is the default)

DEVENV (Visual C++ .NET builds)

Visual Studio has adopted an XML format for project (.vcproj) files. The MakeXS system is capable of creating .vcproj files in order to make it easier to create consistent project files across the entire tree. A number of rules and environment variable will tweak how MakeXS generates .vcproj files.

The assumed source tree organization for the vcproj build rules is as below. (this may not be visible in the rtf version of this document)

Illustration 1 - Tree Hierarchy

The following environment variables are available for creating .vcproj files given a project hierarchy.

IN?_MSVS_BUILD_TYPE Oprtion – CONSOLE or LIBRARY

The following rules are available from a MakeXS system.

%.vcproj This will create a Visual C++ project file with source files from the following directories. The "%" is matched to the project name.
. the current directory
../../../code the main source directory
../../code the win32 specific code directory

win32/projects/%/vcproj_exe This will create the directory hierarchy as shown above including MakeXS files as needed. The make variable IN_MSVS_BUILD_TYPE will be set to "CONSOLE".

win32/projects/%/vcproj_lib This will create the directory hierarchy as shown above including MakeXS files as needed. The make variable IN_MSVS_BUILD_TYPE will be set to "LIBRARY".

win32/projects/%/vcproj_test This will create the directory hierarchy as shown above including MakeXS files as needed. The make variable IN_MSVS_BUILD_TYPE will be set to "CONSOLE" and the sources in both test directories will be placed in the project.

make_Libdirs This will recreate the cached list of directories that contain libraries. This is used in vcproj files as the list of directories to search. This would need to be made for both release and non release builds.

make_Libfiles This will recreate the cached list of .lib files. This is used in vcproj files for linking options. This would need to be made for both release and non release builds.

make_all_cache_files This will recreate cached lista of all .h, .lib files and lib directories for both release and debug builds.

Note that when creating .vcproj files ( ...vcproj_exe, ...vcproj_lib, ...vcproj_test ) the cached directories files for both release and debug paths are used so library builds for both types will be needed to create the correct lib and libpath caches.

The rules to actually build a VC++ project are not supported yet. It is assumned that a Developer Studio "solution" (*.sln) will be used to manage the final build. In essance, this could also be generated by MakeXS at some future point in time.

Doxygen

Doxygen is and advanced javadoc like tool. It requires creating a configuration file. MakeXS supports the "dox", "dox_here" and "dox_gui" targets.

MakeXS target:

dox will automatically create doxygen configuration files and run doxygen

dox_config will create the doxygen configuration file (also run by dox target)

dox_gui will run the file brower over the dox/html directory

A number of environment variables will change the Doxygen configuration file. These are

IN?_DOX_PROJECT_NAME Option - Doxygen configuration PROJECT_NAME

IN?_DOX_PROJECT_NUMBER Option - Doxygen configuration PROJECT_NUMBER

IN?_DOX_OUTPUT_DIRECTORY Option - Doxygen configuration OUTPUT_DIRECTORY

IN?_DOX_BIN_ABSPATH Option - Doxygen configuration BIN_ABSPATH

IN?_DOX_EXT_DOC_PATHS Option - Doxygen configuration EXT_DOC_PATHS

IN?_DOX_INPUT Option - Doxygen configuration INPUT

IN?_DOX_INCLUDE_PATH Option - Doxygen configuration INCLUDE_PATH

IN?_DOX_HAVE_DOT Option - Doxygen configuration HAVE_DOT

IN?_DOX_BASE_CONFIG_SCRIPT Option - This script takes creates a doxygen config file from environment vars

IN?_DOX_SUBDIR Option - The subdirectory that is selected for doxygen scanning.

Dox rules for running doxygen on a subdirectory also exist, these are dox_here, dox_here_config and dox_here_gui. These can be used to create a doxygen subset.

By default, "make dox" will run doxygen over all directories under the directory "src". This may be changed by IN?_DOX_SUBDIR.

Packaging

MakeXS provides an “install” mechanism for creating release distributions. To do this, one needs to define an m4 macro, build_directories, that describes the list of directories. These directories are the targets for the build. Below is an example of the hello_world product install hierarchy.

define(`install_directories',`(`hello_world',`hello_world/docs',`hello_world/testing')')

This will make MakeXS sensitive to the following macros.

# IN?_INSTABLE_HELLO_WORLD Installable files for hello_world
# IN?_INSTABLE_HELLO_WORLD_DOCS Installable files for hello_world/docs
# IN?_INSTABLE_HELLO_WORLD_TESTING Installable files for hello_world/testing

The ‘INSTABLE’ macros allow the declaration of files that can be installed for those target directories. For example:

IN_INSTABLE_HELLO_WORLD_DOCS=$(wildcard *.rtf)

This will target for make install to install all the .rtf files into hello_world/docs.

MakeXS Environment

MakeXS provides a large number of environment variables that can be changed to enable different behaviour. These variables are all described in the generated MakeXS file. The following is a snap of the relevant comments in the MakeXS file. Check out the generated MakeXS file for the latest documentation.

Variable prefix conventions

IN_	defined in the host Makefile
INL_	defined in the user's environment
INS_	defined for subsystem
ING_	defined globally - usually defined in MakeXSConf.m4
MXS_	Makefile variables in MakeXS
M4X_	M4 macros defined in MakeXS.m4 (private ones)
MIN_	M4 macros defined in MakeXS.m4 (public ones)
IN4_	M4 macros defined in MakeXSConf.m4

The following targets are defined by MakeXS. To perform an operation on a target you will need to define IN_TARGETS_* - e.g.:

                        IN_TARGETS_ALL=XFile

Similarly if you want XFile removed on when you run 'make clean' do this e.g.:

                        IN_FILES_CLEAN=XFile

Or if you have your own clean target "myclean", you can have the default clean target depend on it - e.g.:

                        IN_TARGETS_CLEAN=myclean

Defining IN?_SUBMAKEFILES with a list of Makfiles that also need to be executed will cause a list of targets to be generated that can be used. For example :

                        IN_SUBMAKEFILES=foo/Makefile bar/Makefile

Will cause targets "foo/all foo/setup ... bar/all etc" to also be defined - these will also have default actions that will cause the sub-directories to be built. ING_SUBMAKEFILES by default is defined like:

                        ING_SUBMAKEFILES=$(wildcard */Makefile)

IN?_SUBMAKEDIRS operates in much the same way as ING_SUBMAKEFILES except that it is a list of directories.

The target "mxs_version" will echo the MakeXS version number.

List of all MakeXS common targets.

all	Perform regular build - create all build files
setup	On build a directory structure or whatever pre-build setup is required.
test	Build and run unit test
clean	Remove intermediate build files
mostlyclean	Like clean
distclean	Revert to checked out state.
docs	Build documents
install	Targets that install files previously built by all
uninstall	Targets that remove from installation
cvsadd_all	Add all files to CVS - do this after a distclean
rebuild_vcproj	Recreate visual studio project files

Environment variables

The list of environment variables below change the behaviour of this Makefile. The '?' in the "IN?_" can be one of these:

*empty*	Define these in the input Makefile for the directory
G	These are global defines - usually defined in this file.
L	Defined in the user's environment
S	Defined in the subsystem

For option like variables there is a priority order for these defines (INL_*, IN_*, INS_*, ING_*).

As mentioned earlier, the file MakeXS is generated and the variables supported may be different to the ones described here. Refer to the MakeXS for the variables supported by your version of MakeXS.

Derivation and License

MakeXS is similar to other make based systems however there are a number of original ideas, including root auto discovery and automatic include specifications. MakeXS is based an original work the author developed to create a number of open source initiatives and is covered by the LPGL.

Bugs

1) GNU make aggressively caches directory entries and the non-existence of various directories. This can cause problems with the auto-creation of the work directory. It may be necessary to create that directory for invoking make. make can be easily patched to remove this cache behaviour and result in expected behaviour. This is not an issue if requested targets are placed in the work directory.