Changes to define variables, removing sources does not cause rebuild or relink

[matthew.herrmann@…]

1. If globbing for source files, and a file is removed, the link step is not invalidated when rebuilding.

2. If a <define> is changed in the Jamfile, all sources in the project should be rebuilt, but are not.

Because Boost.Build does not keep track of the commands used to build targets at the time, it is not possible to respond to changes in the Jamfile itself, either through globbing or changed parameters.

To resolve this, when a lib or exe target is built, a .exe.parm or .o.parm file should also be generated, simply containing the command line used to build the application. Before reusing a target, not only should the timestamp be checked, but the contents of the .parm file should be read to ensure it matches what would have been invoked to rebuild the target.

This resolved the two previous issues in the following way:

1. The parameters to the linker will change if source files are added or removed, forcing a rebuild.

2. The parameters to the compiler will change in the case of a modified <define> directive, leading to recompilation.

Optimisations

Several processes may be possible to reduce the overhead of opening, say, 13,000 .o.parm files when scanning dependencies.

Firstly, all dependencies for a folder may be stored in a single file rather than one per file. For example: ./bin/gcc/debug/link-static/.jamparams

This would allow the I/O to be buffered and would speed things up somewhat.

An incorrect but possibly useful optimisation is not to consider the contents of the passed strings, but merely to consider whether the size of the .parm file matches the argument string contents. Most changes would still be captured by this trick, though it is clearly not perfect. This would have to be a configurable mode that could be turned off for reliable semantics at the expense of speed. A size check of .jamparams against an internal list's length would also be a good heuristic that would be faster than checking each entry if the consolidated storage approach was used.

In general, however, the interpreted nature of bjam makes 100% valid optimisations difficult. For example, it is possible to create a bjam script that outputs files to a random filename. Caching, say, the contents of the Jamfile used to generate a project and the input files would still lead to incorrect semantics.

Another way to look at the problem is from the idea of ccache and macro expansion. Ignoring the problem of semantics for random generators raised in the previous paragraph, if it is possible to output a Jamfile with all function outputs and globbing resolved, i.e.:

inputs = [ glob *.cpp ]

becoming:

inputs = blah.cpp other.cpp one_more.cpp

then the contents of the expanded files could be compared relatively quickly. Only if the contents were different would a file-by-file check be undertaken. This is quite complex, however, and I prefer the previously mentioned optimisations using argument length.