Commit Graph

403 Commits

Author SHA1 Message Date
Roman Lebedev
507c06e636
Aggregates: use non-aggregate count as iteration count. (#706)
It is incorrect to say that an aggregate is computed over
run's iterations, because those iterations already got averaged.
Similarly, if there are N repetitions with 1 iterations each,
an aggregate will be computed over N measurements, not 1.
Thus it is best to simply use the count of separate reports.

Fixes #586.
2018-10-18 17:17:14 +03:00
Roman Lebedev
99d1356c04
[NFC] BenchmarkRunner: always populate *_report_aggregates_only bools. (#708)
It is better to let the RunBenchmarks(), report() decide
whether to actually *only* output aggregates or not,
depending on whether there are actually aggregates.

It's subtle indeed.

Previously, `BenchmarkRunner()` always said that "if there are no repetitions,
then you should never output only the repetitions". And the `report()` simply assumed
that the `report_aggregates_only` bool it received makes sense, and simply used it.

Now, the logic is the same, but the blame has shifted.
`BenchmarkRunner()` always propagates what those benchmarks would have wanted
to happen wrt the aggregates. And the `report()` lambda has to actually consider
both the `report_aggregates_only` bool, and it's meaningfulness.

To put it in the context of the patch series - if the repetition count was `1`,
but `*_report_aggregates_only` was set to `true`, and we capture each iteration separately,
then we will compute the aggregates, but then output everything, both the iteration,
and aggregates, despite `*_report_aggregates_only` being set to `true`.
2018-10-18 15:08:59 +03:00
Roman Lebedev
9cacec8e78
[NFC] RunBenchmarks(): s/has_repetitions/might_have_aggregates/ (#707)
That is the real purpose of that bool. A follow-up change will
make it consider something else other than repetitions.
2018-10-18 15:03:17 +03:00
Ilya A. Kriveshko
8503dfe537 benchmark_color: fix auto option (#559) (#699)
As prevously written, "--benchmark_color=auto" was treated as true,
because IsTruthyFlagValue("auto") returned true.  The fix is to
rely on IsColorTerminal test only if the flag value is "auto",
and fall back to IsTruthyFlagValue otherwise.  I also integrated
force_no_color check into the same block.
2018-10-08 09:33:21 +01:00
Roman Lebedev
a8082de5df
[NFC] Refactor RunBenchmark() (#690)
Ok, so, i'm still trying to get to the state when it will be a trivial change to report all the separate iterations.
The old code (LHS of the diff) was rather convoluted i'd say.
I have tried to refactor it a bit into *small* logical chunks, with proper comments.
As far as i can tell, i preserved the intent of the code, what it was doing before.
The road forward still isn't clear, but i'm quite sure it's not with the old code :)
2018-10-01 17:51:08 +03:00
Dominic Hamon
edc77a3669
Make State constructor private. (#650)
The State constructor should not be part of the public API. Adding a
utility method to BenchmarkInstance allows us to avoid leaking the
RunInThread method into the public API.
2018-09-28 12:28:43 +01:00
Martin Storsjö
439d6b1c2a Include sys/time.h for cycleclock.h when building on MinGW (#680)
When building for ARM, there is a fallback codepath that uses
gettimeofday, which requires sys/time.h.

The Windows SDK doesn't have this header, but MinGW does have it.
Thus, this fixes building for Windows on ARM with MinGW
headers/libraries, while Windows on ARM with the Windows SDK still
is broken.
2018-09-19 11:52:05 +01:00
Martin Storsjö
5261307982 [benchmark] Lowercase windows specific includes (#679)
The windows SDK headers don't have self-consistent casing anyway,
and many projects consistently use lowercase for them, in order
to fix crosscompilation with mingw headers.
2018-09-18 09:42:20 +01:00
Roman Lebedev
1b44120cd1
Un-deprecate [SG]et{Item,Byte}sProcessed, re-implement as custom counters. (#676)
As discussed with @dominichamon and @dbabokin, sugar is nice.
Well, maybe not for the health, but it's sweet.
Alright, enough puns.

A special care needs to be applied not to break csv reporter. UGH.
We end up shedding some code over this.
We no longer specially pretty-print them, they are printed just like the rest of custom counters.

Fixes #627.
2018-09-13 22:03:47 +03:00
Roman Lebedev
58588476ce
Track two more details about runs - the aggregate name, and run name. (#675)
This is related to @BaaMeow's work in https://github.com/google/benchmark/pull/616 but is not based on it.

Two new fields are tracked, and dumped into JSON:
* If the run is an aggregate, the aggregate's name is stored.
  It can be RMS, BigO, mean, median, stddev, or any custom stat name.
* The aggregate-name-less run name is additionally stored.
  I.e. not some name of the benchmark function, but the actual
  name, but without the 'aggregate name' suffix.

This way one can group/filter all the runs,
and filter by the particular aggregate type.

I *might* need this for further tooling improvement.
Or maybe not.
But this is certainly worthwhile for custom tooling.
2018-09-13 15:08:15 +03:00
Roman Lebedev
c614dfc0d4
*Display* aggregates only. (#665)
There is a flag 
d9cab612e4/src/benchmark.cc (L75-L78)
and a call
d9cab612e4/include/benchmark/benchmark.h (L837-L840)
But that affects everything, every reporter, destination:
d9cab612e4/src/benchmark.cc (L316)


It would be quite useful to have an ability to be more picky.


More specifically, i would like to be able to only see the aggregates in the on-screen output,
but for the file output to still contain everything. The former is useful in case of a lot of repetition
(or even more so if every iteration is reported separately), while the former is **great** for tooling.

Fixes https://github.com/google/benchmark/issues/664
2018-09-12 16:26:17 +03:00
Roman Lebedev
f0901417c8 GetCacheSizesMacOSX(): use consistent types. (#667)
I have absolutely no way to test this, but this looks obviously-good.

This was reported by Tim Northover @TNorthover in
http://lists.llvm.org/pipermail/llvm-commits/Week-of-Mon-20180903/584223.html

> I think this breaks some 32-bit configurations (well, mine at least).
> I was using Clang (from Xcode 10 beta) on macOS and got a bunch of
> errors referencing sysinfo.cc:292 and onwards:

> /Users/tim/llvm/llvm-project/llvm/utils/benchmark/src/sysinfo.cc:292:47:
> error: non-constant-expression cannot be narrowed from type
> 'std::__1::array<unsigned long long, 4>::value_type' (aka 'unsigned
> long long') to 'size_t' (aka 'unsigned long') in initializer list
> [-Wc++11-narrowing]
>   } Cases[] = {{"hw.l1dcachesize", "Data", 1, CacheCounts[1]},
>                                               ^~~~~~~~~~~~~~
>
> The same happens when self-hosting ToT. Unfortunately I couldn't
> reproduce the issue on Debian (Clang 6.0.1) even with libc++; I'm not
> sure what the difference is.
2018-09-05 12:20:18 +01:00
pseyfert
fbfc495d7f add missing closing bracket in --help message (#666) 2018-09-03 19:45:09 +03:00
Roman Lebedev
caa2fcb19c
Counter(): add 'one thousand' param. (#657)
* Counter(): add 'one thousand' param.

Needed for https://github.com/google/benchmark/pull/654

Custom user counters are quite custom. It is not guaranteed
that the user *always* expects for these to have 1k == 1000.
If the counter represents bytes/memory/etc, 1k should be 1024.

Some bikeshedding points:
1. Is this sufficient, or do we really want to go full on
   into custom types with names?
   I think just the '1000' is sufficient for now.
2. Should there be a helper benchmark::Counter::Counter{1000,1024}()
   static 'constructor' functions, since these two, by far,
   will be the most used?
3. In the future, we should be somehow encoding this info into JSON.

* Counter(): use std::pair<> to represent 'one thousand'

* Counter(): just use a new enum with two values 1000 vs 1024.

Simpler is better. If someone comes up with a real reason
to need something more advanced, it can be added later on.

* Counter: just store the 1000 or 1024 in the One_K values directly

* Counter: s/One_K/OneK/
2018-08-29 21:11:06 +03:00
Roman Lebedev
d9cab612e4
[NFC] s/console_reporter/display_reporter/ (#663)
There are two destinations:
* display (console, terminal) and
* file.

And each of the destinations can be poplulated with one of the reporters:
* console - human-friendly table-like display
* json
* csv (deprecated)

So using the name console_reporter is confusing.
Is it talking about the console reporter in the sense of
table-like reporter, or in the sense of display destination?
2018-08-29 14:58:54 +03:00
Roman Lebedev
8688c5c4cf
Track 'type' of the run - is it an actual measurement, or an aggregate. (#658)
This is *only* exposed in the JSON. Not in CSV, which is deprecated.

This *only* supposed to track these two states.
An additional field could later track which aggregate this is,
specifically (statistic name, rms, bigo, ...)

The motivation is that we already have ReportAggregatesOnly,
but it affects the entire reports, both the display,
and the reporters (json files), which isn't ideal.

It would be very useful to have a 'display aggregates only' option,
both in the library's console reporter, and the python tooling,
This will be especially needed for the 'store separate iterations'.
2018-08-28 18:11:36 +03:00
Roman Lebedev
9a179cb93f
[NFC] Prefix "report(_)?mode" with Aggregation. (#656)
This only specifically represents handling of reporting of aggregates.
Not of anything else. Making it more specific makes the name less generic.

This is an issue because i want to add "iteration report mode",
so the naming would be conflicting.
2018-08-28 17:19:25 +03:00
BaaMeow
af441fc114 properly escape json names (#652) 2018-08-16 09:47:09 -07:00
Kirill Bobyrev
f85304e4e3 Remove redundant default which causes failures (#649)
* Remove redundant default which causes failures

* Fix old GCC warnings caused by poor analysis

* Use __builtin_unreachable

* Use BENCHMARK_UNREACHABLE()

* Pull __has_builtin to benchmark.h too

* Also move compiler identification macro to main header

* Move custom compiler identification macro back
2018-08-08 14:39:57 +01:00
Dominic Hamon
f965eab508
Memory management and reporting hooks (#625)
* Introduce memory manager interface

* Add memory stats to JSON reporter and a test

* Add comments and switch json output test to int
2018-07-24 15:57:15 +01:00
Federico Ficarelli
0c21bc369a Fix build with Intel compiler (#631)
* Set -Wno-deprecated-declarations for Intel

Intel compiler silently ignores -Wno-deprecated-declarations
so warning no. 1786 must be explicitly suppressed.

* Make std::int64_t → double casts explicit

While std::int64_t → double is a perfectly conformant
implicit conversion, Intel compiler warns about it.
Make them explicit via static_cast<double>.

* Make std::int64_t → int casts explicit

Intel compiler warns about emplacing an std::int64_t
into an int container. Just make the conversion explicit
via static_cast<int>.

* Cleanup Intel -Wno-deprecated-declarations workaround logic
2018-07-09 11:45:10 +01:00
Federico Ficarelli
5946795e82 Disable Intel invalid offsetof warning (#629) 2018-07-03 10:13:22 +01:00
Roman Lebedev
b123abdcf4 Add Iteration-related Counter::Flags. Fixes #618 (#621)
Inspired by these [two](a1ebe07bea) [bugs](0891555be5) in my code due to the lack of those i have found fixed in my code:
* `kIsIterationInvariant` - `* state.iterations()`
  The value is constant for every iteration, and needs to be **multiplied** by the iteration count.
* `kAvgIterations` - `/ state.iterations()`
  The is global over all the iterations, and needs to be **divided** by the iteration count.

They play nice with `kIsRate`:
* `kIsIterationInvariantRate`
* `kAvgIterationsRate`.

I'm not sure how  meaningful they are when combined with `kAvgThreads`.
I guess the `kIsThreadInvariant` can be added, too, for symmetry with `kAvgThreads`.
2018-06-27 15:45:30 +01:00
Marat Dukhan
505be96ab2 Avoid using CMake 3.6 feature list(FILTER ...) (#612)
list(FILTER ...) is a CMake 3.6 feature, but benchmark targets CMake 2.8.12
2018-06-06 12:32:42 +01:00
Sergiu Deitsch
1301f53e31 cmake: use numeric version in package config (#611) 2018-06-05 15:01:44 +01:00
Marat Dukhan
7fb3c564e5 Fix compilation on Android with GNU STL (#596)
* Fix compilation on Android with GNU STL

GNU STL in Android NDK lacks string conversion functions from C++11, including std::stoul, std::stoi, and std::stod.
This patch reimplements these functions in benchmark:: namespace using C-style equivalents from C++03.

* Avoid use of log2 which doesn't exist in Android GNU STL

GNU STL in Android NDK lacks log2 function from C99/C++11.
This patch replaces their use in the code with double log(double) function.
2018-06-05 11:36:26 +01:00
BaaMeow
4c2af07889 (clang-)format all the things (#610)
* format all documents according to contributor guidelines and specifications
use clang-format on/off to stop formatting when it makes excessively poor decisions

* format all tests as well, and mark blocks which change too much
2018-06-01 11:14:19 +01:00
Dominic Hamon
4fbfa2f336
Some platforms and environments don't pass a valid argc/argv. (#607)
Specifically some iOS targets.
2018-05-30 13:17:41 +01:00
Alex Strelnikov
e776aa0275 Add benchmark_main target. (#601)
* Add benchmark_main library with support for Bazel.

* fix newline at end of file

* Add CMake support for benchmark_main.

* Mention optionally using benchmark_main in README.
2018-05-25 11:18:58 +01:00
Nan Xiao
e90801ae47 Remove unnecessary memset functions. (#591) 2018-05-09 10:31:24 +01:00
Sam Clegg
8986839e4a Use __EMSCRIPTEN__ (rather then EMSCRIPTEN) to check for emscripten (#583)
The old EMSCRIPTEN macro is deprecated and not enabled when
EMCC_STRICT is set.

Also fix a typo in EMSCRIPTN (not sure how this ever worked).
2018-05-03 09:34:26 +01:00
Nan Xiao
ea5551e7b3 Porting into OpenBSD (#582) 2018-05-02 11:26:43 +01:00
Tim Bradgate
ed1bac8434 Issue 571: Allow support for negative regex filtering (#576)
* Allow support for negative regex filtering

This patch allows one to apply a negation to the entire regex filter
by appending it with a '-' character, much in the same style as
GoogleTest uses.

* Address issues in PR

* Add unit tests for negative filtering
2018-04-26 10:56:06 +01:00
Victor Costan
64d4805dd7 Fix precision loss warning in MSVC. (#574) 2018-04-23 11:58:02 +01:00
Dominic Hamon
c4858d8012
Report the actual iterations run. (#572)
Before this change, we would report the number of requested iterations
passed to the state. After, we will report the actual number run. As a
side-effect, instead of multiplying the expected iterations by the
number of threads to get the total number, we can report the actual
number of iterations across all threads, which takes into account the
situation where some threads might run more iterations than others.
2018-04-19 18:40:08 +01:00
Dominic Hamon
64e5a13fa0
Ensure 64-bit truncation doesn't happen for complexity_n (#569)
* Ensure 64-bit truncation doesn't happen for complexity results

* One more complexity_n 64-bit fix

* Missed another vector of int

* Piping through the int64_t
2018-04-12 15:40:24 +01:00
Fred Tingaud
50ffc781b1 Optimize by using nth_element instead of partial_sort to find the median. (#565) 2018-04-09 13:40:58 +01:00
Dominic Hamon
9913418d32
Allow AddRange to work with int64_t. (#548)
* Allow AddRange to work with int64_t.

Fixes #516

Also, tweak how we manage per-test build needs, and create a standard
_gtest suffix for googletest to differentiate from non-googletest tests.

I also ran clang-format on the files that I changed (but not the
benchmark include or main src as they have too many clang-format
issues).

* Add benchmark_gtest to cmake

* Set(Items|Bytes)Processed now take int64_t
2018-04-03 23:12:47 +01:00
Dominic Hamon
df60aeb266
Rely on compiler intrinsics to identify regex engine. (#555)
Having the copts set on a per-target level can lead to ODR violations
in some cases. Avoid this by ensuring the regex engine is picked
through compiler intrinsics in the header directly.
2018-03-23 11:45:15 +00:00
Eric Fiselier
e668e2a1ba Fix #552 - GCC and Clang warn on possibly invalid offsetof usage.
This patch disables the -Winvalid-offsetof warning for GCC and Clang
when using it to check the cache lines of the State object.

Technically this usage of offsetof is undefined behavior until C++17.
However, all major compilers support this application as an extension,
as demonstrated by the passing static assert (If a compiler encounters UB
during evaluation of a constant expression, that UB must be diagnosed).
Unfortunately, Clang and GCC also produce a warning about it.

This patch temporarily suppresses the warning using #pragma's in the
source file (instead of globally suppressing the warning in the build systems).
This way the warning is ignored for both CMake and Bazel builds without
having to modify either build system.
2018-03-21 13:47:25 -06:00
Dominic Hamon
674d0498b8
Move thread classes out to clean up monolithic code (#554) 2018-03-16 10:14:38 +00:00
Wink Saville
61497236dd Make string_util naming more consistent (#547)
* Rename StringXxx to StrXxx in string_util.h and its users

This makes the naming consistent within string_util and moves is the
Abseil convention.

* Style guide is 2 spaces before end of line "//" comments

* Rename StrPrintF/StringPrintF to StrFormat for absl compatibility.
2018-03-07 11:20:06 +00:00
Wink Saville
f48a28d12a Do not let StrCat be renamed to lstrcatA (#546)
On Windows the Shlwapi.h file has a macro:

  #define StrCat lstrcatA

And benchmark/src/string_util.h defines StrCat and it is renamed to
lstrcatA if we don't undef the macro in Shlwapi.h. This is an innocuous
bug if string_util.h is included after Shlwapi.h, but it is a compile
error if string_util.h is included before Shlwapi.h.

This fixes issue #545.
2018-03-06 18:15:03 +00:00
Wink Saville
69a52cff4f Spelling fixes (#543)
Upstream spelling fix changes from Pony, ec47ba8f565726414552f4bbf97d7,
by ka7@la-evento.com that effected google/benchmark.
2018-03-06 11:44:25 +00:00
alekseyshl
47df49e573 Add Solaris support (#539)
* Add Solaris support

Define BENCHMARK_OS_SOLARIS for Solaris.

Platform specific implementations added:
* number of CPUs detection
* CPU cycles per second detection
* Thread CPU usage
* Process CPU usage

* Remove the special case for per process CPU time for Solaris, it's the same as the default.
2018-03-02 03:53:58 -08:00
Robert Guo
ff2c255af5 Use STCK to get the CPU clock on s390x (#540) 2018-03-02 03:22:03 -08:00
Eric
56f52ee228 Print the executable name as part of the context. (#534)
* Print the executable name as part of the context.

A common use case of the library is to run two different
versions of a benchmark to compare them. In my experience
this often means compiling a benchmark twice, renaming
one of the executables, and then running the executables
back-to-back. In this case the name of the executable
is important contextually information.  Unfortunately the
benchmark does not report this information.

This patch adds the executable name to the context reported
by the benchmark.

* attempt to fix tests on Windows

* attempt to fix tests on Windows
2018-02-21 08:43:57 -08:00
Ian McKellar
6ecf8a8e80 Don't include <sys/resource.h> on Fuchsia. (#531)
* Don't include <sys/resource.h> on Fuchsia.

It doesn't support POSIX resource measurement and timing APIs.

Change-Id: Ifab4bac4296575f042c699db1ce5a4f7c2d82893

* Add BENCHMARK_OS_FUCHSIA for Fuchsia

Change-Id: Ic536f9625e413270285fbfd08471dcb6753ddad1
2018-02-14 14:17:12 -07:00
Eric
207b9c7aec
Improve State packing: put important members on first cache line. (#527)
* Improve State packing: put important members on first cache line.

This patch does a few different things to ensure commonly accessed
data is on the first cache line of the `State` object.

First, it moves the `error_occurred_` member to reside after
the `started_` and `finished_` bools, since there was internal
padding there that was unused.

Second, it moves `batch_leftover_` and `max_iterations` further up
in the struct declaration. These variables are used in the calculation
of `iterations()` which users might call within the loop. Therefore
it's more important they exist on the first cache line.

Finally, this patch turns the bool members into bitfields. Although
this shouldn't have much of an effect currently, because padding is
still needed between the last bool and the first size_t, it should
help in future changes that require more "bool like" members.

* Remove bitfield change for now

* Move bools (and their padding) to end of "first cache line" vars.

I think it makes the most sense to move the padding required
following the group of bools to the end of the variables we want
on the first cache line.

This also means that the `total_iterations_` variable, which is the
most accessed, has the same address as the State object.

* Fix static assertion after moving bools
2018-02-14 13:44:41 -07:00
Samuel Panzer
296ec5693e Support State::KeepRunningBatch(). (#521)
* Support State::KeepRunningBatch().

State::KeepRunning() can take large amounts of time relative to quick
operations (on the order of 1ns, depending on hardware). For such
sensitive operations, it is recommended to run batches of repeated
operations.

This commit simplifies handling of total_iterations_. Rather than
predecrementing such that total_iterations_ == 1 signals that
KeepRunning() should exit, total_iterations_ == 0 now signals the
intention for the benchmark to exit.

* Create better fast path in State::KeepRunningBatch()

* Replace int parameter with size_t to fix signed mismatch warnings

* Ensure benchmark State has been started even on error.

* Simplify KeepRunningBatch()
2018-02-09 21:57:04 -07:00