Boost.Build Python Port Design

This document outlines internal design of the Boost.Build V2 port to Python.

Constraints

The primary goal of the port is to replicate exactly the same functionality, and the same design and the same code, and get all the tests still work. Therefore, for modules in the 'build' and 'tools' directory only minimal design changes are allowed.

• No globals -- for the 'build' modules all data that is global in existing code will be kept inside various Python classes. There will be a top-level Manager object keeping all important data together. For 'tools' modules, using globals at this point does not seem reasonable, so we'll provide functions that return the 'global' instances of classes.
  • Example: See the boost.build.build.project.ProjectRegistry class
• Build actions in Python -- in case where jam code, for a given command line, has both a command line proper and procedural code to setup something, said procedural code will be moved in Python -- while the command line will be declared using Jam syntax.
  • Example: See how boost.build.tools.common.py defines the 'mkdir' function -- only the actual command is defined at Jam level
• Using of Python facilities. Python datatypes and builtin modules will be used.
• Using of the property_set class. Unless required for compatibility with Jamfiles, we'll be using property_set in all remaining cases where raw property list is used now.
• Path normalization. Jam code uses 'path.make' and 'path.native' to convert from native to normalized pathnames. Since Python has robust path manipulation functions, this logic will be dropped -- we'll use os.path everywhere.

Python style

Please try to adhere to the Python style guidelines at: ​http://www.python.org/dev/peps/pep-0008/.

The current code base breaks against a lot of these rules. If you encounter any of these problems while working on the code, please fix them as you go.

The most common problems are extraneous whitespace.

NO

__declared_subfeature [str (t, name)] = True
m = __re__before_first_dash.match (toolset)

YES

__declared_subfeature[str(t, name)] = True
m = __re__before_first_dash.match(toolset)

This is bad style. No whitespace should be used before brackets or parentheses. Sometimes whitespace is also used inside braces:

NO

{ 'foo': 'bar' }

YES

{'foo': 'bar'}

These whitespace should also be removed. Whitespace should also not be present in argument defaults:

NO

def __init__(self, name, project, manager = None):

YES

def __init__(self, name, project, manager=None):

Another common issue is using has_key to determine if a key is present in a dictionary. Instead operator in should be used:

NO

if foo.has_key('bar')

YES

if 'bar' in foo:

Don't use [] as an argument default without making absolutely sure the argument isn't mutated inside the function. Instead, use something like:

def foo(x=None):
    if x is None:
        x = []

Don't use len() to test for empty sequences:

NO

if len(x):
if not len(x):

YES

if x:
if not x:

Python has booleans. Don't use 0 or 1 instead of False and True.

NO

def add_constant(self, name, value, path=0):

YES

def add_constant(self, name, value, path=False):

Design: Physical structure

The entire code base is divided in two parts -- Boost.Build proper and bjam. Boost.Build is written in Python. bjam is a build engine, written in C, and provides two major features:

• Parsing of project description files, called Jamfiles
• Build engine functionality -- detecting out-of-date targets and running commands to make them out of date.

The entry point is bjam, which is linked to Python interpreter. On startup, bjam imports Python main function and calls it, from which point Python is in control. We can alternatively create an extension library from bjam, and call that from Python, but this is not important design aspect.

Design: Logical structure

There's one top-level object -- manager -- of type boost.build.Manager. It holds all global definitions -- set of tools, features, project tree.

Project loading level

Porting is in progress yet. Check back later.

Metatargets level

Porting is in progress yet. Check back later.

Virtual targets level

Porting is in progress yet. Check back later.

Bjam engine level

On the bjam engine level, we have bjam targets and bjam actions. Bjam target is just a string. It can have arbitrary set of (variable-name,value) pairs associated with it. An action has a name, and a body. Body actually specifies the command that should be executed, and body can make use of the variables defined on targets. See bjam docs for more details.

In Python port, the bjam engine level is represented by the boost.build.engine.Engine class. The basic operation is to specify that certain targets are produced from other other targets using specified action. This is done by the 'set_update_action' method.

The targets are strings, they don't need any declarations. The actions however do need declarations. A declaration provides an name, and tells how the action is actually run. The action names are global to the engine instance, and used by other parts of Boost.Build. There are two methods to declare an action:

• The 'register_action' provides a command string to be executed. It's also possible to provide a Python function that will be executed before the command, and can set variables that are subsequently substituted in the command string.
• The 'register_bjam_action' specifies that a given action is completely defined in Jam code. This is only useful when user has defined some actions himself in a Jamfile. No part of Boost.Build will ever define action this way, it will

Design: Bjam-Python interface

Bjam side

The EXTRA_PYTHONPATH variable, in the global module, should be a list with directories where Python modules are searched, in addition to PYTHONPATH.

From Bjam side, a single extra rule is available:

PYTHON_IMPORT_RULE <python-module> : <callable> : <bjam-module> : <name> ;

This imports a python callable object from the specified module into bjam.

Python side

The following functions are avaiable from python, in the bjam module.

call <name>, <param0>, <param1>, .... Looksup <name> in bjam module 'python_interface'. If found, calls that rule, passsing <param0>, <paramN>. Each <paramI> should be a list of strings.

import_rule <bjam module>, <bjam name>, <callable> Adds new rule <bjam name> to <bjam module>. Calling that rule will call back to Python <callable> -- which should accept lists of strings as parameters.

define_action action_name, action_body, bind_list, flags. Defines new action with specified name, body, list if bound variables and flags. The action is created in the global module.

variable name <name> Obtains the value of a variable 'name' in Jam's global module.