A microbenchmark support library
Go to file
Nicholas Junge b93f5a5929
Add pre-commit config and GitHub Actions job (#1688)
* Add pre-commit config and GitHub Actions job

Contains the following hooks:
* buildifier - for formatting and linting Bazel files.
* mypy, ruff, isort, black - for Python typechecking, import hygiene,
static analysis, and formatting.

The pylint CI job was changed to be a pre-commit CI job, where pre-commit
is bootstrapped via Python.

Pylint is currently no longer part of the
code checks, but can be re-added if requested. The reason to drop was
that it does not play nicely with pre-commit, and lots of its
functionality and responsibilities are actually covered in ruff.

* Add dev extra to pyproject.toml for development installs

* Clarify that pre-commit contains only Python and Bazel hooks

* Add one-line docstrings to Bazel modules

* Apply buildifier pre-commit fixes to Bazel files

* Apply pre-commit fixes to Python files

* Supply --profile=black to isort to prevent conflicts

* Fix nanobind build file formatting

* Add tooling configs to `pyproject.toml`

In particular, set line length 80 for all Python files.

* Reformat all Python files to line length 80, fix return type annotations

Also ignores the `tools/compare.py` and `tools/gbench/report.py` files
for mypy, since they emit a barrage of errors which we can deal with
later. The errors are mostly related to dynamic classmethod definition.
2023-10-30 15:35:37 +00:00
.github Add pre-commit config and GitHub Actions job (#1688) 2023-10-30 15:35:37 +00:00
bazel Add pre-commit config and GitHub Actions job (#1688) 2023-10-30 15:35:37 +00:00
bindings/python Add pre-commit config and GitHub Actions job (#1688) 2023-10-30 15:35:37 +00:00
cmake missing cmake include 2023-07-10 17:58:01 +01:00
docs add logo to github pages 2023-08-21 14:31:58 +01:00
include/benchmark correct cli param in docs 2023-10-08 11:08:46 +01:00
src Increase the kMaxIterations limit (#1668) 2023-10-17 17:13:59 +01:00
test Add no-unititialized to tests (#1683) 2023-10-23 08:54:08 +00:00
tools Add pre-commit config and GitHub Actions job (#1688) 2023-10-30 15:35:37 +00:00
_config.yml enable markdown rendering on github pages 2021-04-28 16:44:33 +01:00
.clang-format Fix .clang-format 2019-04-08 12:38:11 +03:00
.clang-tidy clang-tidy: readability-redundant and performance (#1298) 2021-12-06 11:18:04 +00:00
.gitignore add .DS_Store to .gitignore 2021-08-17 21:46:11 +01:00
.pre-commit-config.yaml Add pre-commit config and GitHub Actions job (#1688) 2023-10-30 15:35:37 +00:00
.travis.yml Enable various sanitizer builds in github actions (#1167) 2021-06-03 19:45:02 +01:00
.ycm_extra_conf.py Add pre-commit config and GitHub Actions job (#1688) 2023-10-30 15:35:37 +00:00
appveyor.yml CMake: use full add_test(NAME <> COMMAND <>) signature (#901) 2019-11-05 22:46:13 +03:00
AUTHORS Audit MSVC references in cmake files to consider clang++ (#1669) 2023-09-26 12:31:24 +01:00
BUILD.bazel Add pre-commit config and GitHub Actions job (#1688) 2023-10-30 15:35:37 +00:00
CMakeLists.txt Audit MSVC references in cmake files to consider clang++ (#1669) 2023-09-26 12:31:24 +01:00
CONTRIBUTING.md Add information about CLAs. 2014-02-12 18:51:08 -05:00
CONTRIBUTORS Audit MSVC references in cmake files to consider clang++ (#1669) 2023-09-26 12:31:24 +01:00
LICENSE Stop generating the export header and just check it in (#1435) 2022-07-20 20:34:39 +01:00
MODULE.bazel Add pre-commit config and GitHub Actions job (#1688) 2023-10-30 15:35:37 +00:00
pyproject.toml Add pre-commit config and GitHub Actions job (#1688) 2023-10-30 15:35:37 +00:00
README.md remove icon from main README 2023-08-21 14:24:38 +01:00
setup.py Add pre-commit config and GitHub Actions job (#1688) 2023-10-30 15:35:37 +00:00
WORKSPACE Add pre-commit config and GitHub Actions job (#1688) 2023-10-30 15:35:37 +00:00
WORKSPACE.bzlmod Add support for bzlmod (excluding Python bindings) (#1615) 2023-06-27 13:03:39 +01:00

Benchmark

build-and-test bazel pylint test-bindings Coverage Status

Discord

A library to benchmark code snippets, similar to unit tests. Example:

#include <benchmark/benchmark.h>

static void BM_SomeFunction(benchmark::State& state) {
  // Perform setup here
  for (auto _ : state) {
    // This code gets timed
    SomeFunction();
  }
}
// Register the function as a benchmark
BENCHMARK(BM_SomeFunction);
// Run the benchmark
BENCHMARK_MAIN();

Getting Started

To get started, see Requirements and Installation. See Usage for a full example and the User Guide for a more comprehensive feature overview.

It may also help to read the Google Test documentation as some of the structural aspects of the APIs are similar.

Resources

Discussion group

IRC channels:

Additional Tooling Documentation

Assembly Testing Documentation

Building and installing Python bindings

Requirements

The library can be used with C++03. However, it requires C++11 to build, including compiler and standard library support.

The following minimum versions are required to build the library:

  • GCC 4.8
  • Clang 3.4
  • Visual Studio 14 2015
  • Intel 2015 Update 1

See Platform-Specific Build Instructions.

Installation

This describes the installation process using cmake. As pre-requisites, you'll need git and cmake installed.

See dependencies.md for more details regarding supported versions of build tools.

# Check out the library.
$ git clone https://github.com/google/benchmark.git
# Go to the library root directory
$ cd benchmark
# Make a build directory to place the build output.
$ cmake -E make_directory "build"
# Generate build system files with cmake, and download any dependencies.
$ cmake -E chdir "build" cmake -DBENCHMARK_DOWNLOAD_DEPENDENCIES=on -DCMAKE_BUILD_TYPE=Release ../
# or, starting with CMake 3.13, use a simpler form:
# cmake -DCMAKE_BUILD_TYPE=Release -S . -B "build"
# Build the library.
$ cmake --build "build" --config Release

This builds the benchmark and benchmark_main libraries and tests. On a unix system, the build directory should now look something like this:

/benchmark
  /build
    /src
      /libbenchmark.a
      /libbenchmark_main.a
    /test
      ...

Next, you can run the tests to check the build.

$ cmake -E chdir "build" ctest --build-config Release

If you want to install the library globally, also run:

sudo cmake --build "build" --config Release --target install

Note that Google Benchmark requires Google Test to build and run the tests. This dependency can be provided two ways:

  • Checkout the Google Test sources into benchmark/googletest.
  • Otherwise, if -DBENCHMARK_DOWNLOAD_DEPENDENCIES=ON is specified during configuration as above, the library will automatically download and build any required dependencies.

If you do not wish to build and run the tests, add -DBENCHMARK_ENABLE_GTEST_TESTS=OFF to CMAKE_ARGS.

Debug vs Release

By default, benchmark builds as a debug library. You will see a warning in the output when this is the case. To build it as a release library instead, add -DCMAKE_BUILD_TYPE=Release when generating the build system files, as shown above. The use of --config Release in build commands is needed to properly support multi-configuration tools (like Visual Studio for example) and can be skipped for other build systems (like Makefile).

To enable link-time optimisation, also add -DBENCHMARK_ENABLE_LTO=true when generating the build system files.

If you are using gcc, you might need to set GCC_AR and GCC_RANLIB cmake cache variables, if autodetection fails.

If you are using clang, you may need to set LLVMAR_EXECUTABLE, LLVMNM_EXECUTABLE and LLVMRANLIB_EXECUTABLE cmake cache variables.

To enable sanitizer checks (eg., asan and tsan), add:

 -DCMAKE_C_FLAGS="-g -O2 -fno-omit-frame-pointer -fsanitize=address -fsanitize=thread -fno-sanitize-recover=all"
 -DCMAKE_CXX_FLAGS="-g -O2 -fno-omit-frame-pointer -fsanitize=address -fsanitize=thread -fno-sanitize-recover=all "  

Stable and Experimental Library Versions

The main branch contains the latest stable version of the benchmarking library; the API of which can be considered largely stable, with source breaking changes being made only upon the release of a new major version.

Newer, experimental, features are implemented and tested on the v2 branch. Users who wish to use, test, and provide feedback on the new features are encouraged to try this branch. However, this branch provides no stability guarantees and reserves the right to change and break the API at any time.

Usage

Basic usage

Define a function that executes the code to measure, register it as a benchmark function using the BENCHMARK macro, and ensure an appropriate main function is available:

#include <benchmark/benchmark.h>

static void BM_StringCreation(benchmark::State& state) {
  for (auto _ : state)
    std::string empty_string;
}
// Register the function as a benchmark
BENCHMARK(BM_StringCreation);

// Define another benchmark
static void BM_StringCopy(benchmark::State& state) {
  std::string x = "hello";
  for (auto _ : state)
    std::string copy(x);
}
BENCHMARK(BM_StringCopy);

BENCHMARK_MAIN();

To run the benchmark, compile and link against the benchmark library (libbenchmark.a/.so). If you followed the build steps above, this library will be under the build directory you created.

# Example on linux after running the build steps above. Assumes the
# `benchmark` and `build` directories are under the current directory.
$ g++ mybenchmark.cc -std=c++11 -isystem benchmark/include \
  -Lbenchmark/build/src -lbenchmark -lpthread -o mybenchmark

Alternatively, link against the benchmark_main library and remove BENCHMARK_MAIN(); above to get the same behavior.

The compiled executable will run all benchmarks by default. Pass the --help flag for option information or see the User Guide.

Usage with CMake

If using CMake, it is recommended to link against the project-provided benchmark::benchmark and benchmark::benchmark_main targets using target_link_libraries. It is possible to use find_package to import an installed version of the library.

find_package(benchmark REQUIRED)

Alternatively, add_subdirectory will incorporate the library directly in to one's CMake project.

add_subdirectory(benchmark)

Either way, link to the library as follows.

target_link_libraries(MyTarget benchmark::benchmark)