mirror of
https://github.com/google/benchmark.git
synced 2025-03-23 15:40:07 +08:00
7d03f2df49
As previously discussed, let's flip the switch ^^.
This exposes the problem that it will now be run
for everyone, even if one did not read the help
about the recommended repetition count.
This is not good. So i think we can do the smart thing:
```
$ ./compare.py benchmarks gbench/Inputs/test3_run{0,1}.json
Comparing gbench/Inputs/test3_run0.json to gbench/Inputs/test3_run1.json
Benchmark Time CPU Time Old Time New CPU Old CPU New
--------------------------------------------------------------------------------------------------------
BM_One -0.1000 +0.1000 10 9 100 110
BM_Two +0.1111 -0.0111 9 10 90 89
BM_Two +0.2500 +0.1125 8 10 80 89
BM_Two_pvalue 0.2207 0.6831 U Test, Repetitions: 2. WARNING: Results unreliable! 9+ repetitions recommended.
BM_Two_stat +0.0000 +0.0000 8 8 80 80
```
(old screenshot)
Or, in the good case (noise omitted):
```
s$ ./compare.py benchmarks /tmp/run{0,1}.json
Comparing /tmp/run0.json to /tmp/run1.json
Benchmark Time CPU Time Old Time New CPU Old CPU New
---------------------------------------------------------------------------------------------------------------------------------
<99 more rows like this>
./_T012014.RW2/threads:8/real_time +0.0160 +0.0596 46 47 10 10
./_T012014.RW2/threads:8/real_time_pvalue 0.0000 0.0000 U Test, Repetitions: 100
./_T012014.RW2/threads:8/real_time_mean +0.0094 +0.0609 46 47 10 10
./_T012014.RW2/threads:8/real_time_median +0.0104 +0.0613 46 46 10 10
./_T012014.RW2/threads:8/real_time_stddev -0.1160 -0.1807 1 1 0 0
```
(old screenshot)
40 lines
719 B
JSON
40 lines
719 B
JSON
{
|
|
"context": {
|
|
"date": "2016-08-02 17:44:46",
|
|
"num_cpus": 4,
|
|
"mhz_per_cpu": 4228,
|
|
"cpu_scaling_enabled": false,
|
|
"library_build_type": "release"
|
|
},
|
|
"benchmarks": [
|
|
{
|
|
"name": "BM_One",
|
|
"iterations": 1000,
|
|
"real_time": 10,
|
|
"cpu_time": 100,
|
|
"time_unit": "ns"
|
|
},
|
|
{
|
|
"name": "BM_Two",
|
|
"iterations": 1000,
|
|
"real_time": 9,
|
|
"cpu_time": 90,
|
|
"time_unit": "ns"
|
|
},
|
|
{
|
|
"name": "BM_Two",
|
|
"iterations": 1000,
|
|
"real_time": 8,
|
|
"cpu_time": 80,
|
|
"time_unit": "ns"
|
|
},
|
|
{
|
|
"name": "short",
|
|
"iterations": 1000,
|
|
"real_time": 8,
|
|
"cpu_time": 80,
|
|
"time_unit": "ns"
|
|
}
|
|
]
|
|
}
|