aboutsummaryrefslogtreecommitdiff
path: root/src/bench/bench.cpp
AgeCommit message (Collapse)Author
2024-03-07test: test_bitcoin: allow -testdatadir=<datadir>Larry Ruane
Specifying this argument overrides the path location for test_bitcoin; it becomes <datadir>/test_common_Bitcoin Core/<testname>/datadir. Also, this directory isn't removed after the test completes. This can make it easier for developers to study the results of a test (see the state of the data directory after the test runs), and also (for example) have an editor open on debug.log to monitor it across multiple test runs instead of having to re-open a different pathname each time. Example usage (note the "--" is needed): test_bitcoin --run_test=getarg_tests/boolarg -- \ -testdatadir=/somewhere/mydatadir This will create (if necessary) and use the data directory: /somewhere/mydatadir/test_common_Bitcoin Core/getarg_tests/boolarg/datadir Co-authored-by: furszy <mfurszy@protonmail.com>
2023-03-23refactor: Move fs.* to util/fs.*TheCharlatan
The fs.* files are already part of the libbitcoin_util library. With the introduction of the fs_helpers.* it makes sense to move fs.* into the util/ directory as well.
2022-12-29Merge bitcoin/bitcoin#26481: bench: Suppress output when running with ↵MarcoFalke
`-sanity-check` option f1e89597c803001ab9d5afd7e173184fe6886d1d test: Drop no longer required bench output redirection (Hennadii Stepanov) 4dbcdf26a301f54c60c85ceab1aaa4dae43f6aeb bench: Suppress output when running with `-sanity-check` option (Hennadii Stepanov) Pull request description: This change allows to simplify CI tests, and makes it easier to integrate the `bench_bitcoin` binary into CMake custom [targets](https://cmake.org/cmake/help/latest/command/add_custom_target.html) or [commands](https://cmake.org/cmake/help/latest/command/add_custom_command.html), as `COMMAND` does not support output redirection. ACKs for top commit: aureleoules: tACK f1e89597c803001ab9d5afd7e173184fe6886d1d. Ran as expected and is more practical than using an output redirection. Tree-SHA512: 29086d428cccedcfd031c0b4514213cbc1670e35f955e8fd35cee212bc6f9616cf9f20d0cb984495390c4ae2c50788ace616aea907d44e0d6a905b9dda1685d8
2022-12-24scripted-diff: Bump copyright headersHennadii Stepanov
-BEGIN VERIFY SCRIPT- ./contrib/devtools/copyright_header.py update ./ -END VERIFY SCRIPT- Commits of previous years: - 2021: f47dda2c58b5d8d623e0e7ff4e74bc352dfa83d7 - 2020: fa0074e2d82928016a43ca408717154a1c70a4db - 2019: aaaaad6ac95b402fe18d019d67897ced6b316ee0
2022-11-10bench: Suppress output when running with `-sanity-check` optionHennadii Stepanov
This change allows to simplify CI tests, and makes it easier to integrate the `bench_bitcoin` binary into CMake custom targets or commands, as `COMMAND` does not support output redirection
2022-10-20bench: add "priority level" to the benchmark frameworkfurszy
Will allow us to run certain benchmarks while skip non-prioritized ones in 'make check'.
2022-09-28bench: place benchmark implementation inside benchmark namespacefurszy
2022-05-25Fixup option name in bench messageBen Woosley
2022-05-17bench: Add `--sanity-check` flag, use it in `make check`laanwj
The benchmarks are run as part of `make check` for a minimum sanity check. The actual results are being ignored. So only run them for one iteration. This makes the `bench_bitcoin` part take 2m00 instead of 5m20 here. Which is still too long (imo), but this needs to be solved in the `WalletLoading*` benchmarks which take that long per iteration.
2022-02-04bench: Represents paths with fs::path instead of std::stringRyan Ofsky
Also uses fs::path quoting in bench printed strings and fixes a misleading error message. Originally suggested https://github.com/bitcoin/bitcoin/pull/20744#issuecomment-1022486215 Co-authored-by: Hennadii Stepanov <32963518+hebasto@users.noreply.github.com>
2022-02-03refactor: replace boost::filesystem with std::filesystemKiminuo
Warning: Replacing fs::system_complete calls with fs::absolute calls in this commit may cause minor changes in behaviour because fs::absolute no longer strips trailing slashes; however these changes are believed to be safe. Co-authored-by: Russell Yanofsky <russ@yanofsky.org> Co-authored-by: Hennadii Stepanov <32963518+hebasto@users.noreply.github.com>
2022-01-26fs: consistently use fsbridge for {i,o}fstreamfanquake
Part of #20744, but this can be done now, and will simplify the diff.
2022-01-11test: parse the command line arguments in unit testsVasil Dimov
Retrieve the command line arguments from boost and pass them to `BasicTestingSetup` so that we gain extra flexibility of passing any config options on the test command line, e.g.: ``` test_bitcoin -- -printtoconsole=1 -checkaddrman=5 ```
2021-12-30scripted-diff: Bump copyright headersHennadii Stepanov
-BEGIN VERIFY SCRIPT- ./contrib/devtools/copyright_header.py update ./ -END VERIFY SCRIPT- Commits of previous years: * 2020: fa0074e2d82928016a43ca408717154a1c70a4db * 2019: aaaaad6ac95b402fe18d019d67897ced6b316ee0
2021-09-21bench: clean up includesJon Atack
Drops unneeded and adds missing includes
2021-09-21bench: introduce -min_time argumentMartin Ankerl
When it is not easily possible to stabilize benchmark machine and code the argument -min_time can be used to specify a minimum duration that a benchmark should take. E.g. choose -min_time=1000 if you are willing to wait about 1 second for each benchmark result. The default is now set to 10ms instead of 0, which should make runs on fast machines more stable with negligible slowdown.
2020-08-14bench: Allow skip benchmarkHennadii Stepanov
Co-authored-by: Martin Ankerl <Martin.Ankerl@gmail.com>
2020-06-13Replace current benchmarking framework with nanobenchMartin Ankerl
This replaces the current benchmarking framework with nanobench [1], an MIT licensed single-header benchmarking library, of which I am the autor. This has in my opinion several advantages, especially on Linux: * fast: Running all benchmarks takes ~6 seconds instead of 4m13s on an Intel i7-8700 CPU @ 3.20GHz. * accurate: I ran e.g. the benchmark for SipHash_32b 10 times and calculate standard deviation / mean = coefficient of variation: * 0.57% CV for old benchmarking framework * 0.20% CV for nanobench So the benchmark results with nanobench seem to vary less than with the old framework. * It automatically determines runtime based on clock precision, no need to specify number of evaluations. * measure instructions, cycles, branches, instructions per cycle, branch misses (only Linux, when performance counters are available) * output in markdown table format. * Warn about unstable environment (frequency scaling, turbo, ...) * For better profiling, it is possible to set the environment variable NANOBENCH_ENDLESS to force endless running of a particular benchmark without the need to recompile. This makes it to e.g. run "perf top" and look at hotspots. Here is an example copy & pasted from the terminal output: | ns/byte | byte/s | err% | ins/byte | cyc/byte | IPC | bra/byte | miss% | total | benchmark |--------------------:|--------------------:|--------:|----------------:|----------------:|-------:|---------------:|--------:|----------:|:---------- | 2.52 | 396,529,415.94 | 0.6% | 25.42 | 8.02 | 3.169 | 0.06 | 0.0% | 0.03 | `bench/crypto_hash.cpp RIPEMD160` | 1.87 | 535,161,444.83 | 0.3% | 21.36 | 5.95 | 3.589 | 0.06 | 0.0% | 0.02 | `bench/crypto_hash.cpp SHA1` | 3.22 | 310,344,174.79 | 1.1% | 36.80 | 10.22 | 3.601 | 0.09 | 0.0% | 0.04 | `bench/crypto_hash.cpp SHA256` | 2.01 | 496,375,796.23 | 0.0% | 18.72 | 6.43 | 2.911 | 0.01 | 1.0% | 0.00 | `bench/crypto_hash.cpp SHA256D64_1024` | 7.23 | 138,263,519.35 | 0.1% | 82.66 | 23.11 | 3.577 | 1.63 | 0.1% | 0.00 | `bench/crypto_hash.cpp SHA256_32b` | 3.04 | 328,780,166.40 | 0.3% | 35.82 | 9.69 | 3.696 | 0.03 | 0.0% | 0.03 | `bench/crypto_hash.cpp SHA512` [1] https://github.com/martinus/nanobench * Adds support for asymptotes This adds support to calculate asymptotic complexity of a benchmark. This is similar to #17375, but currently only one asymptote is supported, and I have added support in the benchmark `ComplexMemPool` as an example. Usage is e.g. like this: ``` ./bench_bitcoin -filter=ComplexMemPool -asymptote=25,50,100,200,400,600,800 ``` This runs the benchmark `ComplexMemPool` several times but with different complexityN settings. The benchmark can extract that number and use it accordingly. Here, it's used for `childTxs`. The output is this: | complexityN | ns/op | op/s | err% | ins/op | cyc/op | IPC | total | benchmark |------------:|--------------------:|--------------------:|--------:|----------------:|----------------:|-------:|----------:|:---------- | 25 | 1,064,241.00 | 939.64 | 1.4% | 3,960,279.00 | 2,829,708.00 | 1.400 | 0.01 | `ComplexMemPool` | 50 | 1,579,530.00 | 633.10 | 1.0% | 6,231,810.00 | 4,412,674.00 | 1.412 | 0.02 | `ComplexMemPool` | 100 | 4,022,774.00 | 248.58 | 0.6% | 16,544,406.00 | 11,889,535.00 | 1.392 | 0.04 | `ComplexMemPool` | 200 | 15,390,986.00 | 64.97 | 0.2% | 63,904,254.00 | 47,731,705.00 | 1.339 | 0.17 | `ComplexMemPool` | 400 | 69,394,711.00 | 14.41 | 0.1% | 272,602,461.00 | 219,014,691.00 | 1.245 | 0.76 | `ComplexMemPool` | 600 | 168,977,165.00 | 5.92 | 0.1% | 639,108,082.00 | 535,316,887.00 | 1.194 | 1.86 | `ComplexMemPool` | 800 | 310,109,077.00 | 3.22 | 0.1% |1,149,134,246.00 | 984,620,812.00 | 1.167 | 3.41 | `ComplexMemPool` | coefficient | err% | complexity |--------------:|-------:|------------ | 4.78486e-07 | 4.5% | O(n^2) | 6.38557e-10 | 21.7% | O(n^3) | 3.42338e-05 | 38.0% | O(n log n) | 0.000313914 | 46.9% | O(n) | 0.0129823 | 114.4% | O(log n) | 0.0815055 | 133.8% | O(1) The best fitting curve is O(n^2), so the algorithm seems to scale quadratic with `childTxs` in the range 25 to 800.
2020-04-17bench: Remove requirement that all benches use RegTestingSetupMarcoFalke
2020-04-16scripted-diff: Bump copyright headersMarcoFalke
-BEGIN VERIFY SCRIPT- ./contrib/devtools/copyright_header.py update ./ -END VERIFY SCRIPT-
2020-01-28Fix benchmarks filtersElichai Turkel
2020-01-02test: Show debug log on unit test failureMarcoFalke
2019-12-23rpc: Remove mempool global from minerMarcoFalke
2019-11-06scripted-diff: test: Move setup_common to test libraryMarcoFalke
-BEGIN VERIFY SCRIPT- # Move files for f in $(git ls-files src/test/lib/); do git mv $f src/test/util/; done git mv src/test/setup_common.cpp src/test/util/ git mv src/test/setup_common.h src/test/util/ # Replace Windows paths sed -i -e 's|\\setup_common|\\util\\setup_common|g' $(git grep -l '\\setup_common') sed -i -e 's|src\\test\\lib\\|src\\test\\util\\|g' build_msvc/test_bitcoin/test_bitcoin.vcxproj # Everything else sed -i -e 's|/setup_common|/util/setup_common|g' $(git grep -l 'setup_common') sed -i -e 's|test/lib/|test/util/|g' $(git grep -l 'test/lib/') # Fix include guard sed -i -e 's|BITCOIN_TEST_SETUP_COMMON_H|BITCOIN_TEST_UTIL_SETUP_COMMON_H|g' ./src/test/util/setup_common.h sed -i -e 's|BITCOIN_TEST_LIB_|BITCOIN_TEST_UTIL_|g' $(git grep -l 'BITCOIN_TEST_LIB_') -END VERIFY SCRIPT-
2019-10-30test: Add RegTestingSetup to setup_commonMarcoFalke
2019-05-13[refactor] interfaces: Add missing LockAnnotation for cs_mainMarcoFalke
2019-05-03scripted-diff: replace chainActive -> ::ChainActive()James O'Beirne
Though at the moment ChainActive() simply references `g_chainstate.m_chain`, doing this change now clears the way for multiple chainstate usage and allows us to script the diff. -BEGIN VERIFY SCRIPT- git grep -l "chainActive" | grep -E '(h|cpp)$' | xargs sed -i '/chainActive =/b; /extern CChain& chainActive/b; s/\(::\)\{0,1\}chainActive/::ChainActive()/g' -END VERIFY SCRIPT-
2019-04-11scripted-diff: Bump copyright headers in test, benchMarcoFalke
-BEGIN VERIFY SCRIPT- ./contrib/devtools/copyright_header.py update ./src/bench/ ./contrib/devtools/copyright_header.py update ./src/test/ -END VERIFY SCRIPT-
2019-04-11scripted-diff: Rename test_bitcoin to test/setup_commonMarcoFalke
-BEGIN VERIFY SCRIPT- sed -i --regexp-extended -e 's/test_bitcoin\.(h|cpp)/setup_common.\1/g' $(git grep -l test_bitcoin) git mv ./src/test/test_bitcoin.h ./src/test/setup_common.h git mv ./src/test/test_bitcoin.cpp ./src/test/setup_common.cpp sed -i -e 's/BITCOIN_TEST_TEST_BITCOIN_H/BITCOIN_TEST_SETUP_COMMON_H/g' ./src/test/setup_common.h -END VERIFY SCRIPT-
2019-04-10test: Use test_bitcoin setup in bench, Add test utilsMarcoFalke
2018-07-27Update copyright headers to 2018DrahtBot
2018-04-18benchmark: Removed bench/perf.cppThomas Snider
2018-01-16Log debug build status and warn when running benchmarksWladimir J. van der Laan
Log whether the starting instance of bitcoin core is a debug or release build (--enable-debug). Also warn when running the benchmarks with a debug build, to prevent mistakes comparing debug to non-debug results.
2018-01-03Increment MIT Licence copyright header year on files modified in 2017Akira Takizawa
2017-12-23Improved microbenchmarking with multiple features.Martin Ankerl
* inline performance critical code * Average runtime is specified and used to calculate iterations. * Console: show median of multiple runs * plot: show box plot * filter benchmarks * specify scaling factor * ignore src/test and src/bench in command line check script * number of iterations instead of time * Replaced runtime in BENCHMARK makro number of iterations. * Added -? to bench_bitcoin * Benchmark plotly.js URL, width, height can be customized * Fixed incorrect precision warning
2017-11-16scripted-diff: Replace #include "" with #include <> (ryanofsky)MeshCollider
-BEGIN VERIFY SCRIPT- for f in \ src/*.cpp \ src/*.h \ src/bench/*.cpp \ src/bench/*.h \ src/compat/*.cpp \ src/compat/*.h \ src/consensus/*.cpp \ src/consensus/*.h \ src/crypto/*.cpp \ src/crypto/*.h \ src/crypto/ctaes/*.h \ src/policy/*.cpp \ src/policy/*.h \ src/primitives/*.cpp \ src/primitives/*.h \ src/qt/*.cpp \ src/qt/*.h \ src/qt/test/*.cpp \ src/qt/test/*.h \ src/rpc/*.cpp \ src/rpc/*.h \ src/script/*.cpp \ src/script/*.h \ src/support/*.cpp \ src/support/*.h \ src/support/allocators/*.h \ src/test/*.cpp \ src/test/*.h \ src/wallet/*.cpp \ src/wallet/*.h \ src/wallet/test/*.cpp \ src/wallet/test/*.h \ src/zmq/*.cpp \ src/zmq/*.h do base=${f%/*}/ relbase=${base#src/} sed -i "s:#include \"\(.*\)\"\(.*\):if test -e \$base'\\1'; then echo \"#include <\"\$relbase\"\\1>\\2\"; else echo \"#include <\\1>\\2\"; fi:e" $f done -END VERIFY SCRIPT-
2017-11-09Require a steady clock for bench with at least micro precisionMatt Corallo
2017-11-07bench: switch to std::chrono for time measurementsCory Fields
std::chrono removes portability issues. Rather than storing doubles, store the untouched time_points. Then convert to nanoseconds for display. This allows for maximum precision, while keeping results comparable between differing hardware/operating systems. Also, display full nanosecond counts rather than sub-second floats.
2017-09-11Remove countMaskInv caching in bench frameworkMatt Corallo
We were saving a div by caching the inverse as a float, but this ended up requiring a int -> float -> int conversion, which takes almost as much time as the difference between float mul and div. There are lots of other more pressing issues with the bench framework which probably require simply removing the adaptive iteration count stuff anyway.
2017-08-14Merge #10483: scripted-diff: Use the C++11 keyword nullptr to denote the ↵Wladimir J. van der Laan
pointer literal instead of the macro NULL 90d4d89 scripted-diff: Use the C++11 keyword nullptr to denote the pointer literal instead of the macro NULL (practicalswift) Pull request description: Since C++11 the macro `NULL` may be: * an integer literal with value zero, or * a prvalue of type `std::nullptr_t` By using the C++11 keyword `nullptr` we are guaranteed a prvalue of type `std::nullptr_t`. For a more thorough discussion, see "A name for the null pointer: nullptr" (Sutter & Stroustrup), http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2007/n2431.pdf With this patch applied there are no `NULL` macro usages left in the repo: ``` $ git grep NULL -- "*.cpp" "*.h" | egrep -v '(/univalue/|/secp256k1/|/leveldb/|_NULL|NULLDUMMY|torcontrol.*NULL|NULL cert)' | wc -l 0 ``` The road towards `nullptr` (C++11) is split into two PRs: * `NULL` → `nullptr` is handled in PR #10483 (scripted, this PR) * `0` → `nullptr` is handled in PR #10645 (manual) Tree-SHA512: 3c395d66f2ad724a8e6fed74b93634de8bfc0c0eafac94e64e5194c939499fefd6e68f047de3083ad0b4eff37df9a8a3a76349aa17d55eabbd8e0412f140a297
2017-08-07scripted-diff: Use the C++11 keyword nullptr to denote the pointer literal ↵practicalswift
instead of the macro NULL -BEGIN VERIFY SCRIPT- sed -i 's/\<NULL\>/nullptr/g' src/*.cpp src/*.h src/*/*.cpp src/*/*.h src/qt/*/*.cpp src/qt/*/*.h src/wallet/*/*.cpp src/wallet/*/*.h src/support/allocators/*.h sed -i 's/Prefer nullptr, otherwise SAFECOOKIE./Prefer NULL, otherwise SAFECOOKIE./g' src/torcontrol.cpp sed -i 's/tor: Using nullptr authentication/tor: Using NULL authentication/g' src/torcontrol.cpp sed -i 's/METHODS=nullptr/METHODS=NULL/g' src/test/torcontrol_tests.cpp src/torcontrol.cpp sed -i 's/nullptr certificates/NULL certificates/g' src/qt/paymentserver.cpp sed -i 's/"nullptr"/"NULL"/g' src/torcontrol.cpp src/test/torcontrol_tests.cpp -END VERIFY SCRIPT-
2017-07-31Restore default format state of cout after printing with std::fixed/setprecisionpracticalswift
2017-05-13Replace boost::function with std::function (C++11)practicalswift
2017-03-06Merge #9547: bench: Assert that division by zero is unreachableWladimir J. van der Laan
db07f91 Assert that what might look like a possible division by zero is actually unreachable (practicalswift) Tree-SHA512: f1652eb37196a5b72f356503a1fbb44fb98aa8a94954ad1765f86d81ebf41a2337d4eb58c4f19937fda3752f5d2d642756e44afdbd438015b87ac20801246bff
2017-02-07bench: Fix initialization order in registrationWladimir J. van der Laan
The initialization order of global data structures in different implementation units is undefined. Making use of this is essentially gambling on what the linker does, the so-called [Static initialization order fiasco](https://isocpp.org/wiki/faq/ctors#static-init-order). In this case it apparently worked on Linux but failed on OpenBSD and FreeBSD. To create it on first use, make the registration structure local to a function. Fixes #8910.
2017-02-02Assert that what might look like a possible division by zero is actually ↵practicalswift
unreachable
2017-01-05Merge #9281: Refactor: Remove using namespace <xxx> from bench/ & test/ sourcesMarcoFalke
73f4119 Refactoring: Removed using namespace <xxx> from bench/ and test/ source files. (Karl-Johan Alm)
2017-01-02Refactoring: Removed using namespace <xxx> from bench/ and test/ source files.Karl-Johan Alm
2016-12-31Increment MIT Licence copyright header year on files modified in 2016isle2983
Edited via: $ contrib/devtools/copyright_header.py update .
2016-11-22bench: Add support for measuring CPU cyclesWladimir J. van der Laan
This adds cycle min/max/avg to the statistics. Supported on x86 and x86_64 (natively through rdtsc), as well as Linux (perf syscall).
generated by cgit v1.2.3 (git 2.26.2) at 2024-09-18 06:36:42 +0000