github-repos/benchmark
1
0
Fork 0
mirror of https://github.com/google/benchmark.git synced 2025-02-07 01:30:16 +08:00
Code Issues Packages Projects Releases Wiki Activity
6f094ba13e
benchmark/docs/random_interleaving.md
haih-g a6a738c1cc
Implementation of random interleaving. (#1105) 
* Implementation of random interleaving. See
http://github.com/google/benchmark/issues/1051 for the feature requests.

Committer: Hai Huang (http://github.com/haih-g)

On branch fr-1051
Changes to be committed:
modified:   include/benchmark/benchmark.h
modified:   src/benchmark.cc
new file:   src/benchmark_adjust_repetitions.cc
new file:   src/benchmark_adjust_repetitions.h
modified:   src/benchmark_api_internal.cc
modified:   src/benchmark_api_internal.h
modified:   src/benchmark_register.cc
modified:   src/benchmark_runner.cc
modified:   src/benchmark_runner.h
modified:   test/CMakeLists.txt
new file:   test/benchmark_random_interleaving_gtest.cc

* Fix benchmark_random_interleaving_gtest.cc for fr-1051

Committer: Hai Huang <haih@google.com>

On branch fr-1051
Your branch is up to date with 'origin/fr-1051'.

Changes to be committed:
modified:   src/benchmark.cc
modified:   src/benchmark_runner.cc
modified:   test/benchmark_random_interleaving_gtest.cc

* Fix macos build for fr-1051

Committer: Hai Huang <haih@google.com>

On branch fr-1051
Your branch is up to date with 'origin/fr-1051'.

Changes to be committed:
modified:   src/benchmark_api_internal.cc
modified:   src/benchmark_api_internal.h
modified:   src/benchmark_runner.cc

* Fix macos and windows build for fr-1051.

Committer: Hai Huang <haih@google.com>

On branch fr-1051
Your branch is up to date with 'origin/fr-1051'.

Changes to be committed:
modified:   src/benchmark_runner.cc

* Fix benchmark_random_interleaving_test.cc for macos and windows in fr-1051

Committer: Hai Huang <haih@google.com>

On branch fr-1051
Your branch is up to date with 'origin/fr-1051'.

Changes to be committed:
modified:   test/benchmark_random_interleaving_gtest.cc

* Fix int type benchmark_random_interleaving_gtest for macos in fr-1051

Committer: Hai Huang <haih@google.com>

On branch fr-1051
Your branch is up to date with 'origin/fr-1051'.

Changes to be committed:
modified:   test/benchmark_random_interleaving_gtest.cc

* Address dominichamon's comments 03/29 for fr-1051

Committer: Hai Huang <haih@google.com>

On branch fr-1051
Your branch is up to date with 'origin/fr-1051'.

Changes to be committed:
modified:   src/benchmark.cc
modified:   src/benchmark_api_internal.cc
modified:   src/benchmark_api_internal.h
modified:   test/benchmark_random_interleaving_gtest.cc

* Address dominichamon's comment on default min_time / repetitions for fr-1051.
Also change sentinel of random_interleaving_repetitions to -1. Hopefully it
fixes the failures on Windows.

Committer: Hai Huang <haih@google.com>

On branch fr-1051
Your branch is up to date with 'origin/fr-1051'.

Changes to be committed:
modified:   src/benchmark.cc
modified:   src/benchmark_api_internal.cc
modified:   src/benchmark_api_internal.h

* Fix windows test failures for fr-1051

Committer: Hai Huang <haih@google.com>

On branch fr-1051
Your branch is up to date with 'origin/fr-1051'.

Changes to be committed:
modified:   src/benchmark_api_internal.cc
modified:   src/benchmark_runner.cc

* Add license blurb for fr-1051.

Committer: Hai Huang <haih@google.com>

On branch fr-1051
Your branch is up to date with 'origin/fr-1051'.

Changes to be committed:
modified:   src/benchmark_adjust_repetitions.cc
modified:   src/benchmark_adjust_repetitions.h

* Switch to std::shuffle() for fr-1105.

Committer: Hai Huang <haih@google.com>

On branch fr-1051
Your branch is up to date with 'origin/fr-1051'.

Changes to be committed:
modified:   src/benchmark.cc

* Change to 1e-9 in fr-1105

Committer: Hai Huang <haih@google.com>

On branch fr-1051
Your branch is up to date with 'origin/fr-1051'.

Changes to be committed:
modified:   src/benchmark_adjust_repetitions.cc

* Fix broken build caused by bad merge for fr-1105.

Committer: Hai Huang <haih@google.com>

On branch fr-1051
Your branch is up to date with 'origin/fr-1051'.

Changes to be committed:
modified:   src/benchmark_api_internal.cc
modified:   src/benchmark_runner.cc

* Fix build breakage for fr-1051.

Committer: Hai Huang <haih@google.com>

On branch fr-1051
Your branch is up to date with 'origin/fr-1051'.

Changes to be committed:
modified:   src/benchmark.cc
modified:   src/benchmark_api_internal.cc
modified:   src/benchmark_api_internal.h
modified:   src/benchmark_register.cc
modified:   src/benchmark_runner.cc

* Print out reports as they come in if random interleaving is disabled (fr-1051)

Committer: Hai Huang <haih@google.com>

On branch fr-1051
Your branch is up to date with 'origin/fr-1051'.

Changes to be committed:
modified:   src/benchmark.cc

* size_t, int64_t --> int in benchmark_runner for fr-1051.

Committer: Hai Huang <haih@google.com>

On branch fr-1051
Your branch is up to date with 'origin/fr-1051'.

Changes to be committed:
modified:   src/benchmark_runner.cc
modified:   src/benchmark_runner.h

* Address comments from dominichamon for fr-1051

Committer: Hai Huang <haih@google.com>

On branch fr-1051
Your branch is up to date with 'origin/fr-1051'.

Changes to be committed:
modified:   src/benchmark.cc
modified:   src/benchmark_adjust_repetitions.cc
modified:   src/benchmark_adjust_repetitions.h
modified:   src/benchmark_api_internal.cc
modified:   src/benchmark_api_internal.h
modified:   test/benchmark_random_interleaving_gtest.cc

* benchmar_indices --> size_t to make CI pass: fr-1051

Committer: Hai Huang <haih@google.com>

On branch fr-1051
Your branch is up to date with 'origin/fr-1051'.

Changes to be committed:
modified:   src/benchmark.cc

* Fix min_time not initialized issue for fr-1051.

Committer: Hai Huang <haih@google.com>

On branch fr-1051
Your branch is up to date with 'origin/fr-1051'.

Changes to be committed:
modified:   src/benchmark_api_internal.cc
modified:   src/benchmark_api_internal.h

* min_time --> MinTime in fr-1051.

Committer: Hai Huang <haih@google.com>

On branch fr-1051
Your branch is up to date with 'origin/fr-1051'.

Changes to be committed:
modified:   src/benchmark_api_internal.cc
modified:   src/benchmark_api_internal.h
modified:   src/benchmark_runner.cc

* Add doc for random interleaving for fr-1051

Committer: Hai Huang <haih@google.com>

On branch fr-1051
Your branch is up to date with 'origin/fr-1051'.

Changes to be committed:
modified:   README.md
new file:   docs/random_interleaving.md

Co-authored-by: Dominic Hamon <dominichamon@users.noreply.github.com>
2021-05-20 17:09:16 +01:00

1.2 KiB
Raw Blame History

Random Interleaving

Random Interleaving is a technique to lower run-to-run variance. It breaks the execution of a microbenchmark into multiple chunks and randomly interleaves them with chunks from other microbenchmarks in the same benchmark test. Data shows it is able to lower run-to-run variance by 40% on average.

To use, set --benchmark_enable_random_interleaving=true.

It's a known issue that random interleaving may increase the benchmark execution time, if:

  1. A benchmark has costly setup and / or teardown. Random interleaving will run setup and teardown many times and may increase test execution time significantly.
  2. The time to run a single benchmark iteration is larger than the desired time per repetition (i.e., benchmark_min_time / benchmark_repetitions).

The overhead of random interleaving can be controlled by --benchmark_random_interleaving_max_overhead. The default value is 0.4 meaning the total execution time under random interlaving is limited by 1.4 x original total execution time. Set it to inf for unlimited overhead.