From 1c26d8a3371dfd988223c6b86e3397751ded1f7c Mon Sep 17 00:00:00 2001 From: Matt Armstrong Date: Fri, 9 Sep 2022 07:41:10 -0700 Subject: [PATCH] Discuss sources of variance in the user guide (#1481) * Discuss sources of variance in the user guide * Mention cpufreq/boost * Pull variance material into a new document Add reducing_variance.md as a place to discuss things related to variance and, in the future, statistical interpretation of benchmark results. Co-authored-by: Dominic Hamon --- docs/reducing_variance.md | 100 ++++++++++++++++++++++++++++++++++++++ docs/user_guide.md | 37 +++----------- 2 files changed, 106 insertions(+), 31 deletions(-) create mode 100644 docs/reducing_variance.md diff --git a/docs/reducing_variance.md b/docs/reducing_variance.md new file mode 100644 index 00000000..f8b75758 --- /dev/null +++ b/docs/reducing_variance.md @@ -0,0 +1,100 @@ +# Reducing Variance + + + +## Disabling CPU Frequency Scaling + +If you see this error: + +``` +***WARNING*** CPU scaling is enabled, the benchmark real time measurements may be noisy and will incur extra overhead. +``` + +you might want to disable the CPU frequency scaling while running the +benchmark, as well as consider other ways to stabilize the performance of +your system while benchmarking. + +See [Reducing Variance](reducing_variance.md) for more information. + +Exactly how to do this depends on the Linux distribution, +desktop environment, and installed programs. Specific details are a moving +target, so we will not attempt to exhaustively document them here. + +One simple option is to use the `cpupower` program to change the +performance governor to "performance". This tool is maintained along with +the Linux kernel and provided by your distribution. + +It must be run as root, like this: + +```bash +sudo cpupower frequency-set --governor performance +``` + +After this you can verify that all CPUs are using the performance governor +by running this command: + +```bash +cpupower frequency-info -o proc +``` + +The benchmarks you subsequently run will have less variance. + + + +## Reducing Variance in Benchmarks + +The Linux CPU frequency governor [discussed +above](user_guide#disabling-cpu-frequency-scaling) is not the only source +of noise in benchmarks. Some, but not all, of the sources of variance +include: + +1. On multi-core machines not all CPUs/CPU cores/CPU threads run the same + speed, so running a benchmark one time and then again may give a + different result depending on which CPU it ran on. +2. CPU scaling features that run on the CPU, like Intel's Turbo Boost and + AMD Turbo Core and Precision Boost, can temporarily change the CPU + frequency even when the using the "performance" governor on Linux. +3. Context switching between CPUs, or scheduling competition on the CPU the + benchmark is running on. +4. Intel Hyperthreading or AMD SMT causing the same issue as above. +5. Cache effects caused by code running on other CPUs. +6. Non-uniform memory architectures (NUMA). + +These can cause variance in benchmarks results within a single run +(`--benchmark_repetitions=N`) or across multiple runs of the benchmark +program. + +Reducing sources of variance is OS and architecture dependent, which is one +reason some companies maintain machines dedicated to performance testing. + +Some of the easier and and effective ways of reducing variance on a typical +Linux workstation are: + +1. Use the performance governer as [discussed +above](user_guide#disabling-cpu-frequency-scaling). +1. Disable processor boosting by: + ```sh + echo 0 | sudo tee /sys/devices/system/cpu/cpufreq/boost + ``` + See the Linux kernel's + [boost.txt](https://www.kernel.org/doc/Documentation/cpu-freq/boost.txt) + for more information. +2. Set the benchmark program's task affinity to a fixed cpu. For example: + ```sh + taskset -c 0 ./mybenchmark + ``` +3. Disabling Hyperthreading/SMT. This can be done in the Bios or using the + `/sys` file system (see the LLVM project's [Benchmarking + tips](https://llvm.org/docs/Benchmarking.html)). +4. Close other programs that do non-trivial things based on timers, such as + your web browser, desktop environment, etc. +5. Reduce the working set of your benchmark to fit within the L1 cache, but + do be aware that this may lead you to optimize for an unrelistic + situation. + +Further resources on this topic: + +1. The LLVM project's [Benchmarking + tips](https://llvm.org/docs/Benchmarking.html). +1. The Arch Wiki [Cpu frequency +scaling](https://wiki.archlinux.org/title/CPU_frequency_scaling) page. diff --git a/docs/user_guide.md b/docs/user_guide.md index dde1f0e9..3c2e8f7e 100644 --- a/docs/user_guide.md +++ b/docs/user_guide.md @@ -58,8 +58,11 @@ [A Faster KeepRunning Loop](#a-faster-keep-running-loop) +## Benchmarking Tips + [Disabling CPU Frequency Scaling](#disabling-cpu-frequency-scaling) +[Reducing Variance in Benchmarks](reducing_variance.md) @@ -1243,35 +1246,7 @@ If you see this error: ``` you might want to disable the CPU frequency scaling while running the -benchmark. Exactly how to do this depends on the Linux distribution, -desktop environment, and installed programs. Specific details are a moving -target, so we will not attempt to exhaustively document them here. +benchmark, as well as consider other ways to stabilize the performance of +your system while benchmarking. -One simple option is to use the `cpupower` program to change the -performance governor to "performance". This tool is maintained along with -the Linux kernel and provided by your distribution. - -It must be run as root, like this: - -```bash -sudo cpupower frequency-set --governor performance -``` - -After this you can verify that all CPUs are using the performance governor -by running this command: - -```bash -cpupower frequency-info -o proc -``` - -The benchmarks you subsequently run will have less variance. - -Note that changing the governor in this way will not persist across -reboots. To set the governor back, run the first command again with the -governor your system usually runs with, which varies. - -If you find yourself doing this often, there are probably better options -than running the commands above. Some approaches allow you to do this -without root access, or by using a GUI, etc. The Arch Wiki [Cpu frequency -scaling](https://wiki.archlinux.org/title/CPU_frequency_scaling) page is a -good place to start looking for options. +See [Reducing Variance](reducing_variance.md) for more information.