Boost Internal Regression Test Suite

Boost's internal regression test suite produces the compiler status tables.

Although not ordinarily run by Boost library users, it is documented here for the benefit of Boost developers, and for Boost users porting to a new platform.

Boost is transitioning to a new version of the regression tests. During the transition, both versions are available.

Documentation for version 3
Documentation for version 2

Requirements

The test suite has been designed to meet to the following requirements.

Pure ISO C++, no tools required except a C++ compiler.
Support for tests which expect an error
Flexible configuration, independent of the target platform or compiler.
HTML output

These requirements rule out any script-based approach such as dejagnu (requires Tcl and expect) or even shell scripts.

Regression Tests - Version Two

The implementation is provided in a single source file named regression.cpp.

You should be able to compile and link this file using whatever C++ compiler is at your disposition. However, you may need to configure both the compiler and the standard library to use "strict" ISO compliance mode. Also, you need to extend the search path for include files with the main boost directory so that the header file boost/config.hpp can be found. This header file is required to work around compiler deficiencies.

You can then start the resulting executable to run regression tests. By default, the regression test program reads the file "compiler.cfg" in the current directory to determine the list of compilers and their invocation syntax. After that, it reads the file "regression.cfg" in the current directory to determine the regression tests to run. The results of the regression tests are written in an HTML formatted text file. This file is by default named "cs-OS.html" in the current directory, where "OS" is a placeholder for the name of the operating system (e.g., "linux" or "win32").

To generate the compiler status tables, boost uses the files status/compiler.cfg and status/regression.cfg.

The regression test program accepts some command-line options to alter its behavior.

-h or --help prints a help message

--config file Use file instead of "compiler.cfg" as the compiler configuration file. This allows for private compiler setups.

--tests file Use file instead of "regression.cfg" as the tests configuration file. This allows individual libraries to specify additional tests not to be published in the main compiler status tables.

--boost path Use path as the filesystem path to the main boost directory. The default is "..", i.e. the parent directory.

--output file
-o file Write the HTML output to file instead of the default "cs-OS.html".

--compiler name Run the tests only with compiler name. The name must be defined in the second line of an applicable compiler configuration (see below). The default is to run the tests with all compilers suitable for the platform.

--diff Read the HTML output file before writing it. In the HTML output, highlight differences in test outcomes compared to the previous run.

test Run only the named test. The syntax is the same as in the configuration file (see below).

When running only a selected test, you must also provide an alternate HTML output filename with "--output" so that the full test output is not accidentally overwritten.

You should redirect the output (std::cout) and error (std::cerr) channels to a suitable log file for later inspection.

Configuration Files

In both configuration files, single-line comments starting with "//" at the leftmost column are ignored.

Compiler Configuration

The compiler configuration file can contain descriptions for an arbitrary number of compilers. Each compiler is configured by a block of six consecutive text lines.

Name of the operating system for which the entry is applicable (e.g., "linux" or "win32").
Name of the compiler; should be unique within one operating system. The name of the compiler should not contain the version number, because it is expected that regression tests are always run with the most recent compiler version available.
Name and version number of the compiler. This is printed on std::cout prior to running a test with that compiler.
Command-line invocation of the compiler to compile a single source file to an object file.
Command-line invocation of the compiler to compile a single source file to an executable.
Identification of the compiler for inclusion in the HTML output; may contain HTML tags such as <br>.

The two command-lines are subject to the following substitutions:

Any string starting with a dollar symbol "$", then containing an arbitrary number of underscores, digits, or uppercase letters, is replaced by the value of the associated environment variable.
%include is replaced by the path given by the "--boost" command line option (".." by default). This should be used to extend the search path of your compiler so that the boost header files are found.
%source is the name of the source file to be compiled, qualified with the path given by the "--boost" command-line option (".." by default).

Test Configuration

The test configuration file can contain descriptions for an arbitrary number of tests. Each test is described by a single line. The first word (up to the first space) is the type of the test, the next word gives the filename of the test relative the directory to be given by the "--boost" command-line option. Optionally, additional words are passed on as command-line arguments when the test is executed (only for types "run" and "run-fail"). In these arguments, "%boost" is replaced by the path given by the "--boost" command-line option (".." by default).

The following test types are available:

compile: The given test file should compile successfully. Otherwise, the test fails.
compile-fail: The given test file should not compile successfully, but instead the compiler should give an error message. If the test does compile successfully, the test fails.
link: The given test file should compile and link successfully. Otherwise, the test fails. In particular, a main function must be present in the test file.
link-fail: The given test file should not compile and link successfully. It is not specified whether the compile or the link should not succeed. If the test does compile and link successfully, the test fails.
run: The given test file should compile and link successfully. After that, the resulting executable is invoked and should return a zero exit code. If any of these steps fail, the test fails.
run-fail: The given test file is compiled, linked, and, if an executable was successfully generated, it is invoked. Either compiling or linking should fail or the resulting executable should return a non-zero exit code. If compiling and linking succeeds and the resulting executable returns a zero exit code, the test fails.

Adapting for a New Platform

In order to adapt the regression test suite for a new platform, a few changes to the regression.cpp are required.

Add an appropriate check for your platform in get_host() and return a unique string identifying the platform (this string is used to filter the compile configuration file).
Verify that get_system_configuration() is appropriately defined for your platform. For a Unix platform, it most likely is. The function may return a verbose HTML string describing the platform.

You also need to configure the compilers available on your platform in "compiler.cfg" as described above.

You may need to add an entry for the compiler to boost/config.hpp, but only if entries for the compiler is not already present, and the compiler doesn't fully conform to the ISO C++ Standard.

Regression Tests - Version Three

Version 3 of the Boost regression testing framework is based on Boost.Build, and uses bjam to actually run the tests. Because Boost.Build does dependency analysis, only tests for which some dependency has change are rerun.

The reporting of test results as HTML files is accomplished by separate C++ programs which process the residue and log files from the bjam run.

Preparation

Install the following programs on your system, in some location suitable for program executables. Normally that location will be a directory which is part of your search path for executables.

bjam - Executables and sources are available; see Boost.Build docs. Before worrying about regression tests, you might want to verify your bjam installation works by building the boost-root/status/Jamfile libraries.
process_jam_log and compiler_status - Sources available in the Boost Sandbox. These two post-bjam processing programs are currently only available in the sandbox libs/filesystem/example directory, since they depend on the Filesystem Library, which is not scheduled for Boost formal review until late September, 2002. Once the Filesystem Library has been accepted, they will become part of the regular Boost distribution.

Execution

These examples assume several environment variables have been set:

BOOST_ROOT is set to the directory where the unzipped Boost distribution or CVS working copy resides. For example:

set BOOST_ROOT=/boost_1_29_0
TOOLS is set to the compiler toolsets you wish to use. For example:

set TOOLS=borland gcc intel-win32 metrowerks vc7
Any environment variables required for particular compilers. You might wish to test first with one compiler at a time to make sure each toolset is fully operational.

A full set of tests and status tables can then be run thusly on a Window 2000 system:

cd %BOOST_ROOT%\status
md bin 2>nul
bjam test >bin\regr.log 2>&1
start notepad bin\regr.log
process_jam_log <bin\regr.log
compiler_status %BOOST_ROOT% cs-win32.html
rem Specify links file, even though it will be overwritten, so report html includes links
rem This works because the generated bookmark names are the same regardless of other settings
compiler_status --ignore-pass --no-warn %BOOST_ROOT% cs-win32-fail.html cs-win32-links.html
compiler_status --ignore-pass %BOOST_ROOT% cs-win32-warn-or-fail.html cs-win32-links.html
compiler_status %BOOST_ROOT% cs-win32-full.html cs-win32-links.html

Modulo syntax adjustments, the same procedure should work on other operating systems. Rename the cs-win-xxx output files as appropriate.

If you execute compiler_status without arguments, you can see the available options and tailor your own favorite report.

If you want to run just a single test, specify it as the bjam target rather than "test". For example, to debug configurations, it might be useful to just run the config_info test, with a switch to force even up-to-date programs be rebuilt:

bjam -a config_info

Adding a new test

Overall Boost regression tests

Adding tests is as simple as adding a single line to the boost-root/status/Jamfile:

run libs/mylib/test/mytest.cpp ;

Don't forget that Jam is white-space sensitive; delete the space before the semi-colon in the example above, and you will get an introduction to Jam error messages.

For creating more complex tests and test-suites, look at examples in the Jamfile. The bind and config test-suites are simple examples, while the regex and threads test suites show more of the power of Jam based testing.

Private tests for a specific library

A library can have its own private set of tests by creating a Jamfile in one of the library's own sub-directories. For an example of this, see boost-root/libs/test/test/Jamfile.

History

The version 3 testing.jam and status/Jamfile foundation was contributed by Dave Abrahams. The post-bjam processing programs were contributed by Beman Dawes.

The version 2 regression.cpp test program was contributed by Jens Maurer, generalizing and improving an earlier version 1 program by Beman Dawes.

2001-01-30
Jens Maurer

-h or --help	prints a help message
--config file	Use file instead of "compiler.cfg" as the compiler configuration file. This allows for private compiler setups.
--tests file	Use file instead of "regression.cfg" as the tests configuration file. This allows individual libraries to specify additional tests not to be published in the main compiler status tables.
--boost path	Use path as the filesystem path to the main boost directory. The default is "..", i.e. the parent directory.
--output file -o file	Write the HTML output to file instead of the default "cs-OS.html".
--compiler name	Run the tests only with compiler name. The name must be defined in the second line of an applicable compiler configuration (see below). The default is to run the tests with all compilers suitable for the platform.
--diff	Read the HTML output file before writing it. In the HTML output, highlight differences in test outcomes compared to the previous run.
test	Run only the named test. The syntax is the same as in the configuration file (see below).