Makefiles are really good at one thing, managing dependencies between files. In
other words, make makes sure all files that depend on another file are
updated when that file changes.
We tell make how to do this by declaring rules. A typical rule looks like
this:
A Simple Makefile
# Makefile # Create bundle.js by concatenating jquery.js lib.js and main.js bundle.js: jquery.js lib.js main.js cat $^ > $@
There are three parts to this rule:
- The target,
bundle.js, before the colon (:). - The prerequisites (what the target depends on),
jquery.js lib.js main.js, after the colon (:). - The command, cat $^ > $@, on the next line after a leading tab, (\t).
There are two "automatic" variables in this command.
- $@ - filename representing the target, in this case
bundle.js. - $^ - filenames representing the list of the prerequisites (with duplicates removed).
"Automatic" means that the variables are automatically populated with relevant
filenames. This will make more sense when we get into patterns later.
Here are some more variables that are useful.
- $< - filename representing the first prerequisite.
- $? - filenames representing the list of the prerequisites that are newer than the target.
$*- filename representing the stem of the target, in the above casebundle.
The Make Manual contains the full list of automatic variables
Execution
Running make with the above Makefile results in the following execution.
$ make cat jquery.js lib.js main.js > bundle.js $ make make: 'bundle.js' is up to date.
make runs the first target it finds in the file if none is given on the
command line. In this case it is the only target.
The second run didn't do anything since bundle.js is up to date. To be up to
date means that the last-modified time of bundle.js is newer than any of
its prerequisites' last-modified times. Simple but powerful! When we create
new targets all we have to worry about is making sure that our targets know
what files it depends on, and what files they depend on, and so on.
It is possible to enter many targets on the left of the colon (:). make will
treat them as separate rules and the automatic variable will make sure that the
correct files are built.
But, since make treats the rules as separate rules, it will only build the
first of them, the default target.
# Makefile bundle.js bundle2.js: jquery.js lib.js main.js cat $^ > $@
$ make make: 'bundle.js' is up to date.
If we want to build bundle2.js, we can do it by explicitly telling make to do
it by giving the target as command line parameter.
$ make bundle2.js cat jquery.js lib.js main.js > bundle2.js
To get make to build both targets at once, we need to add a new, .PHONY:,
target.
# Makefile .PHONY: bundles bundles: bundle.js bundle2.js bundle.js bundle2.js: jquery.js lib.js main.js cat $^ > $@
Running make now results in (after removing bundle*)
$ make cat jquery.js lib.js main.js > bundle.js cat jquery.js lib.js main.js > bundle2.js
A .PHONY: target is a target without a corresponding file for make to check
last modified time on. This means the target will always be run, forcing make
to check if all the target's prerequisites needs to be built. The .PHONY
label is not strictly necessary. If it is left out, make will check to see if
there is a file called bundles and since there isn't one it will build it
anyway.
Here's an illustration:
# Makefile build: echo 'Running build'
# Makefile2 .PHONY build: echo 'Running build'
$ touch build $ make make: 'build' is up to date. $ make -f Makefile2 echo 'Running build'
Marking a target that doesn't represent a file as .PHONY: is easy to do and
avoids annoying problems once your Makefile grows.
.PHONY: clean
Conventionally every Makefile contains a clean target to remove all the
artifacts that are built. In the above case it would contain something like:
# Makefile .PHONY: clean clean: rm -f bundle*.js
make clean will now clean out all files created by the Makefile.
Directories
Directories in make usually needs a bit of special treatment. Let's say we want
the bundles above to end up in a build directory. The following Makefile
illustrates a problem.
# Makefile bundles: build/bundle.js build/bundle2.js build/bundle.js build/bundle2.js: jquery.js lib.js main.js cat $^ > $@
Running make illustrates the problem:
cat jquery.js lib.js main.js > build/bundle.js /bin/sh: build/bundle.js: No such file or directory make: *** [build/bundle.js] Error 1
The directory is not automatically created by cat. There are three ways to
solve this and one is better than the others.
- Add
mkdir -pto all rules creating files in the directory. - Add a prerequisite to create the directory on the
bundlestarget. - Add an ordering prerequisite (|) to the rules creating the files in the directory.
1. is not good because the directory will be created more than once, one for
each bundle (this is why the -p is needed).
2. is not good because the build directory is not a prerequisite target
for bundles.
3. is good because the build directory is clearly a prerequisite of the rule
that creates the bundles in this directory.
The reason we have to use an ordering prerequisite instead of a normal
prerequisite is that cat would fail otherwise. Here's the resulting good
Makefile.
# Makefile bundles: build build/bundle.js build/bundle2.js build: mkdir build build/bundle.js build/bundle2.js: jquery.js lib.js main.js | build cat $^ > $@ clean: rm -rf build
Patterns
Now, we know the basics of Makefiles. We can create rules with targets, prerequisites and commands that are run when needed. But, we have been working with named files all this time. This works fine for small examples like above, but when we have hundreds of files this quickly gets out of hand. Patterns to the rescue.
Let's say that we have a bunch of images that we would like to optimize by
running them through an optimizer. The images are in the images/ directory
and the optimized images are built into build/images. The naive (and not
working) way to do this is shown below. (I'm faking optimize with a simple
copy, cp.) The % sign is glob matched with the part of the filename that is
not literal.
# Makefile (NOT WORKING) optimize: build/images/* build/images/%: images/% | build/images cp $< $@ build/images: mkdir -p $@
To see why this is not a viable Makefile, we try to run it with make.
$ make
mkdir -p build/images
cp images/a.png build/images/*.
$ tree build
build/
└── images
└── *
What is going on here? Why is only one image copied and why is it copied as
name build/images/*? The problem is that the target files don't exist yet and
the * is interpreted literally. If we copy the files into the build directory
and touch the source files, it works the way we want.
# Copy the image directory into build $ cp -r images build # Touch the orignal images $ touch images/* # Build works since build/image/* evaluates to the list of images $ make cp images/a.png build/images/a.png cp images/b.png build/images/b.png
Here is the main rule to know about patterns. The target file list has to
be created from the available source files. To do this we have help of a
number of functions, including wildcard, shell, etc. shell will allow us
to call anything that we can call from the shell This is very powerful!
How do we solve the above problem? We can do this by getting a list of source images and transforming this list into a list of target images. This is easily done.
# Makefile # 1. Get the souce list of images images := $(wildcard images/*.png) # 2. Tranform the source list into the target list target_images := $(images:%=build/%) # 3. Our default target, optimize, depends on all the target_images optimize: $(target_images) # 4. Build the targets from the sources, make sure build/images exist $(target_images): build/% : % | build/images cp $< $@ build/images: mkdir -p $@
The first line introduces both variables and functions.
Variables can be declared in a number of
ways, but the
:=-declaration is the simplest. It evaluates the value on the right and sets
the value on the left to the result, like variables in most programming
languages.
Functions are called with the $() construct, and wildcard is a function
that evaluates a shell filename pattern and returns a list of filenames.
The full line above populates images with the .png files from the images
directory.
The second line converts the source images into the target images. Variables
are evaluated the same way as function calls, with the $() construct. By
adding a colon-equals expression, a variable substitution reference, after the
variable name we can substitute a pattern for another. Example
files := "src/a.java src/b.java" # Pattern replaces the files into "target/a.class target/b.class" class_files := $(files:src/%.java:target/%.class)
The third line tells make that our optimize target depends on all the
targets existing. This makes sure that all the targets are built.
The fourth line sets up the targets `$(target_images) and its prerequisites with a static pattern rule. The pattern does the opposite of the variable substitution reference above, it deconstructs a single target into the source it depends on. The final part of the of this line is an order prerequisite on the rule to create the directory.
A Recipe for Creating Makefiles
- Create a list of targets that you want to create from the sources. You have
the full power of
bash,python,awk, etc. at your disposal. - Create a static pattern rule to convert a single target into the source it can be created from.
- Add order prerequisites to make sure directories are automatically created.
- Add a callable target that depends on all the target files that you want to create.
Commented Example
Here's a more exotic example of what you can use make for. We have a directory
of Javascript source files in lib. The corresponding test files are in
test. There may be multiple directories below both lib and test. The
testfiles are named like the source files with an added .spec after the stem
of the filename.
We want to use a makefile to help us run only the tests that are relevant based
on the files that are changed. To keep track of what tests have been run we're
going to use marker files and touch them every time a test is run.
# Create the list of test files by using the shell function and find test_files := $(shell find test -name '*.spec.js' -print) # Convert the test files into marker files with variable substitution # A marker files looks like this tmp/model/person_test.marker marker_files := $(test_files:%.js=tmp/%.marker) # Do the same thing for the test directories test_dirs := $(shell find test -type d -print) # The marker directories have their normal names, no special ending marker_dirs := $(test_dirs:%=tmp/%) # test is the default target .PHONY: test test: $(marker_files) # The marker files order depend on the marker directories $(marker_files): | $(marker_dirs) # Marker files depend on the source files # Deconstruct a marker file into a source file $(marker_files): tmp/test/%.spec.marker : lib/%.js # Marker files depend on the test files # Deconstruct the marker file into a test file # When any prerequisite changes, run the tests and then touch the marker file $(marker_files): tmp/%.marker : %.js mocha $< @touch $@ # Create the marker dirs $(marker_dirs): @mkdir -p $@ # Clean the project by removing the entire tmp directory .PHONY: clean clean: rm -rf tmp
Now whenever you run make, it will run only the relevant tests.
# Modify a test file
$ touch test/models/passbook.spec.js
$ make
mocha test/models/passbook.spec.js
passbook
✓ generate pass strips image names
✓ doesnt crash with no store number
2 passing (21ms)
# Modify a source file
$ touch lib/models/passbook.js
$ make
mocha test/models/passbook.spec.js
passbook
✓ generates pass strip image names
✓ doesnt crash with no store number
2 passing (22ms)
Makefiles are really good at one thing: building only stale files. If that
is our problem, we should give make a try.
