* Json reporter: passthrough fp, don't cast it to int; adjust tooling
Json output format is generally meant for further processing
using some automated tools. Thus, it makes sense not to
intentionally limit the precision of the values contained
in the report.
As it can be seen, FormatKV() for doubles, used %.2f format,
which was meant to preserve at least some of the precision.
However, before that function is ever called, the doubles
were already cast to the integer via RoundDouble()...
This is also the case for console reporter, where it makes
sense because the screen space is limited, and this reporter,
however the CSV reporter does output some( decimal digits.
Thus i can only conclude that the loss of the precision
was not really considered, so i have decided to adjust the
code of the json reporter to output the full fp precision.
There can be several reasons why that is the right thing
to do, the bigger the time_unit used, the greater the
precision loss, so i'd say any sort of further processing
(like e.g. tools/compare_bench.py does) is best done
on the values with most precision.
Also, that cast skewed the data away from zero, which
i think may or may not result in false- positives/negatives
in the output of tools/compare_bench.py
* Json reporter: FormatKV(double): address review note
* tools/gbench/report.py: skip benchmarks with different time units
While it may be useful to teach it to operate on the
measurements with different time units, which is now
possible since floats are stored, and not the integers,
but for now at least doing such a sanity-checking
is better than providing misinformation.
* Make Benchmark a single header library (but not header-only)
This patch refactors benchmark into a single header, to allow
for slightly easier usage.
The initial reason for the header split was to keep C++ library
components from being included by benchmark_api.h, making that
part of the library STL agnostic. However this has since changed
and there seems to be little reason to separate the reporters from
the rest of the library.
* Fix internal_macros.h
* Remove more references to macros.h
* Add ClearRegisteredBenchmark() function.
Since benchmarks can be registered at runtime using the RegisterBenchmark(...)
functions, it makes sense to have a ClearRegisteredBenchmarks() function too,
that can be used at runtime to clear the currently registered benchmark and
re-register an entirely new set.
This allows users to run a set of registered benchmarks, get the output using
a custom reporter, and then clear and re-register new benchmarks based on the
previous results.
This fixes issue #400, at least partially.
* Remove unused change
* Fix#342: DoNotOptimize causes compile errors on older GCC versions.
DoNotOptimize uses inline assembly contraints to tell
the compiler what the type of the input variable. The 'g'
operand allows the input to be any register, memory, or
immediate integer operand. However this constraint seems
to be too weak on older GCC versions, and certain inputs
will cause compile errors.
This patch changes the constraint to 'X', which is documented
as "any operand whatsoever is allowed". This appears to fix
the issues with older GCC versions.
However Clang doesn't seem to like "X", and will attempt
to put the input into a register even when it can't/shouldn't;
causing a compile error. However using "g" seems to work like
"X" with GCC, so for this reason Clang still uses "g".
* Try alternative formulation to placate GCC
This thing with the pragma ignore was getting out of hand: now
MinGW (and probably GCC) was erroring too. So I chose to move
the definition of IsZero() out of the anonymous namespace into
benchmark.cc.
This is needed for examining the values of user counters (needed
for #348). It is also needed for checking the values of standard
benchmark results like items_processed or complexities (for example,
checking the standard deviation is needed for unit testing #357
as discussed in #362).
The tests are still missing a way to check actual validity of
numerical results; this will be done next. As they currently are,
the tests pass, but the problem detected with #378 is still
standing and the results with non-standard counters are wrong.
* fix android compilation
* checking __GLIBCXX__ and __GLIBCPP__ macro in addition to __ANDROID__
* using vsnprintf instead of std::vsnprintf to compile on Android
* removed __GLIBCPP__ check on Android
* StringPrintF instead of std::to_string for Android
* Add Benchmark::Iterations for explicitly specifying the number of iterations to use.
* Document that benchmark::Iterations should not be used to limit benchmark runtimes
In non-debug builds CMake automatically adds -DNDEBUG, this means
that uses of `assert` in the tests are disabled for non-debug builds.
Obviously we want these tests to run, regardless of configuration.
This patch strips -DNDEBUG during non-debug builds and adds
-UNDEBUG just to be sure.
fixes#354
The build fails with ICC17 because of warnings and Werror. What is the correct solution to fix it?
Should a patch
disable Werror for ICC (or maybe all non known compilers)
disable the false postive warnings for all files. This could be done using:
add_cxx_compiler_flag(-wd2102) #ICC17u2: Many false positives for Wstrict-aliasing
add_cxx_compiler_flag(-wd2259) #ICC17u2: non-pointer conversion from "long" to "int" may lose significant bits (even for explicit static cast, sleep.cc(44))
add_cxx_compiler_flag(-wd654) #ICC17u2: overloaded virtual function "benchmark::Fixture::SetUp" is only partially overridden (because of deprecated overload)
disable warnings at file level or some other granularity
* Added user counters, and move use of bytes_processed and items_processed to user counter logic.
Each counter is a string-value pair. The counters were
made available through the State class. Two helper virtual
methods were added to the Fixture class to allow convenient
initialization and termination of the counters: InitState()
and TerminateState(). The reporting of the counters is buggy
and is still a work in progress, to be completed in the next commits.
* fix bad removal of BenchmarkCounters code during the merge
* add myself to AUTHORS/CONTRIBUTORS
* fix printing to std::cout in csv_reporter
* bytes_per_second and items_per_second are now in the UserCounters class
* add user counters to json reporter
* moving bytes_per_second and items_per_second to their old state
* console reporter dealing ok with user counters.
* update unit tests for user counters
* CSVReporter now prints user counters too.
* cleanup user counters
* reverted changes to cmake files which should have gone into later commits
* fixture_test: fix gcc 4.6 compilation
* remove ctor with default argument
see https://github.com/google/benchmark/pull/262#discussion_r72298055
* use (auto-defined) BENCHMARK_HAS_CXX11 instead of BENCHMARK_INITLIST.
https://github.com/google/benchmark/pull/262#discussion_r72298310
* leanify counters API
Discussions:
API complexity: https://github.com/google/benchmark/pull/262#discussion_r72298731
remove std::string dependency (WIP): https://github.com/google/benchmark/pull/262#discussion_r72298142
spacing & alignment: https://github.com/google/benchmark/pull/262#discussion_r72298422
* remove std::string dependency on public API - changed counter name storage to char*
* Counter ctor: use overloads instead of default arguments
discussion:
https://github.com/google/benchmark/pull/262#discussion_r72298055
* Use raw pointers to remove dependency on std::vector from public API .
For more info, see discussion at https://github.com/google/benchmark/pull/262#discussion_r72319678 .
* Move counter implementation from benchmark.cc to counter.cc.
See discussion: https://github.com/google/benchmark/pull/262#discussion_r72298980 .
* Remove unused (commented-out) code.
* Moved thread counters to ThreadStats.
* Counters: fixed copy and move constructors.
* Counter: use an inplace buffer for small names.
* benchmark_test: move counters test out of CXX11 preprocessor conditional.
* Counter: fix VS2013 compilation error in char[] initialization.
* Fix typo.
* Expose counters from State.
See discussion: https://github.com/google/benchmark/pull/262#issuecomment-237156951
* Changed counters interface to map-like.
* Fix printing of user counters in ConsoleReporter.
* Applied clang-format to counter.cc and console_reporter.cc.
Command was `clang-format -style=Google -i counter.cc console_reporter.cc`
I also applied to all other files, but the changes were very
far-reaching so I rolled those back.
* Rename Counter::Flags_e to Counter::Flags
* Fix use of reserved names in Counter and BenchmarkCounters.
* Counter: Fix move ctor bug + change order of members.
* Fixture: remove tentative methods InitState() and TerminateState().
* Update fixture_test to the new Fixture interface.
* BenchmarkCounters: fixed a bug in the move ctor. Remove call to CHECK_LT().
CHECK_LT() was making the size_t lookup take ~double the time of a string lookup!
* BenchmarkCounters: add option to not print zero counters (defaults to false).
* Add test to compare counter storage and access with std::map.
* README: clarify cost of counter access modes.
* move counter access test to an own test.
* BenchmarkCounters: add move Insert()
* Counters access test: add accelerated lookup by name.
* Fix old range syntax.
* Fix missing include of cstdio
* Fix Visual Studio warning
* VS2013 and lower: fix use of snprintf()
* VS2013: fix use of char[] as a member of std::pair<>.
* change counter storage to std::map
* Remove skipZeroCounters logic
* Fix VS compilation error.
* Implemented request changes to PR #262.
* PR #262: More requested changes.
* README: cleanup counter text.
* PR #262: remove clang-format changes for preexisting code
* Complexity+Counters: fix counter flags which were being ignored.
* Document all Counter::Flag members
* fixed loss of counter values
* ConsoleReporter: remove tabular printing of user counters.
* ConsoleReporter: header printing should not be contingent on user counter names.
* Minor white space and alignment fixes.
* cxx03_test + counters: reuse the BM_empty() function.
* user counters: add note to README on how counters are gathered across threads
* Test bytes_per_second and items_per_second.
* Test SetLabel.
* Reformat.
* Make State::error_occurred_ private.
* Fix tests with floats.
* Merge private blocks
In the `Ranges(...)` generation code a "control" vector which stores
the current index for each range passed to `Ranges`. Previously this vector
was incorrectly initialized to the size of the subranges not the number
of subranges.
Additionally this patch suppresses unused warnings generated by
`stream_init_anchor`.
* refactor
* Move default substitutions into library
* Move default substitutions to the *right* place in the library
* Fix init order issues that caused test failures
* improve diagnostics
* add missing include
* general cleanup
* Address review comments
Currently the Appveyor bot is a PIT. It never passes and it often hangs
or gives very poor output. This patch rewrites the configuration.
This patch also attempts to fix a flaky complexity test as a drive-by.
This patch adds new builders that test against GCC 6 and Clang 3.8 respectivly.
They also enable both address and undefined sanitizer. MSAN currently won't work
since it requires a sanitized STL.
VC 2013 injects valid when assigning an initializer list to std::set.
This attempts to work around this issue by using std::set's constructors
instead of the assignment operators.
* Support multiple ranges in the benchmark
google-benchmark library allows to provide up to two ranges to the
benchmark method (range_x and range_y). However, in many cases it's not
sufficient. The patch introduces multi-range features, so user can easily
define multiple ranges by passing a vector of integers, and access values
through the method range(i).
* Remove redundant API
Functions State::range_x() and State::range_y() have been removed. They should
be replaced by State::range(0) and State::range(1).
Functions Benchmark::ArgPair() and Benchmark::RangePair() have been removed.
They should be replaced by Benchmark::Args() and Benchmark::Ranges().
* Add RegisterBenchmark
* fix test inputs
* fix UB caused by unitialized value
* Add RegisterBenchmark
* fix test inputs
* fix UB caused by unitialized value
* Work around GCC 4.6/4.7/4.8 bug
These options allow you to write the output of a benchmark to the specified
file and with the specified format. The goal of this change is to help support
tooling.
GCC 4.6 doesn't provide std::chrono::steady_clock and GCC 4.7 doesn't provide
std::this_thread::sleep_for. I would prefer to support GCC 4.7 but I'm
reverting this since the bots are GCC 4.6.
This reverts commit c5f454957d.
* Move ComputeStats call out of the reporters
* Cleanup adjusted time calculations in reporters
* Move ComputeBigO call out of reporters
* Remove ReportComplexity interface using ReportRuns instead
* Factor out reporting of basic context information
* Attempt to fix GCC 4.6 build errors
* Move ComputeStats to complexity.cc
* Add lambda benchmarks
* Remove lambda capture since the lambda is not at a block scope
* Remove LambdaBenchmark helper since FunctionBenchmark can be used with non-capturing lambas
* Add lambda benchmarks
* Remove lambda capture since the lambda is not at a block scope
* Remove LambdaBenchmark helper since FunctionBenchmark can be used with non-capturing lambas
* Add more docs for BENCHMARK_CAPTURE.
* Fix use of misnamed parameter
* Guard BENCHMARK_CAPTURE tests against non-c++11 compilers
* Move tests out of basic_test.cc
* Add test for reporter output.
* setup err_stream tests
* Fix warnings in tests
* whitespace
* Fix build errors caused by super pedantic compilers
* Pass streams by pointer not non-const reference
MSVC++ before 2015 Update 2 has a bug in sleep_for where it tries to
implicitly += the input with a nanoseconds variable. Work around this by
using nanoseconds directly (which can be implicitly +='d with
chrono::nanoseconds).
Having access to the thread count from within a benchmark is useful,
for when one wants to distribute a workload dynamically among the
benchmarks running in parallel e.g when using ThreadRange() or
ThreadPerCpu().
This patch adds a `coverage` target that allows coverage statisitics to be
retrieved for the project. It requires that lcov and gcov is installed and
that the generator is unix makefiles but this can be improved upon in
future releases.
To make it work use the coverage build type:
```
cmake -DCMAKE_BUILD_TYPE=Coverage .
make coverage
```
This patch adopts a new internal structure for how timings are performed.
Currently every iteration of a benchmark checks to see if it has been running
for an appropriate amount of time. Checking the clock introduces noise into
the timings and this can cause inconsistent output from each benchmark.
Now every iteration of a benchmark only checks an iteration count to see if it
should stop running. The iteration count is determined before hand by testing
the benchmark on a series of increasing iteration counts until a suitable count
is found. This increases the amount of time it takes to run the actual benchmarks
but it also greatly increases the accuracy of the results.
This patch introduces some breaking changes. The notable breaking changes are:
1. Benchmarks run on multiple threads no generate a report per thread. Instead
only a single report is generated.
2. ::benchmark::UseRealTime() was removed and replaced with State::UseRealTime().
Fixes#89
UseRealTime was defined in the internal namespace by mistake.
Similarly, documentation suggested that benchmark::SetLabel should be
used to set a label, and a function was declared but not defined, while
actually the call should be 'state.SetLabel'.
This patch does two things:
1. It overhalls the static initialization in Walltime to be simplier. It uses
a static variable inside WallTime::Now() to initialize the timer.
2. Add a logging mechanism so that the -v flag actually has meaning and
reimplement the CHECK macros to allow extra messages to be streamed in.
This patch cleans up our use of generic macros and also merges changes in the
build system.
It adds options -DBENCHMARK_ENABLE_TESTING and -DBENCHMARK_ENABLE_SHARED.
Previously the benchmark_test program executed the benchmark tests to make sure
the API was working but was not checking the number of tests that were
completed. If the regular expression matching breaks, zero tests could be ran.
Similarly, if one of the APIs breaks and doesn't run the correct amount of tests
then `make test` will catch this.
Benchmark library builds and runs but only single-threaded. Multithreaded
support needs a bit more love.
Currently requires some C++11 support (g++ 4.6.3 seems to work).
