aboutsummaryrefslogtreecommitdiff
path: root/contrib/devtools
AgeCommit message (Collapse)Author
2021-07-07scripts: adjust test-symbol-check for guix release environmentfanquake
Now that our release binaries are build in a glibc 2.24 and 2.27 environment, we can't use a symbol from glibc 2.28 to test our checks. Replace renameat2() with nextup(), which was introduced in 2.24. Note that this also means re-disabling the test for RISC-V, however RISC-V is built in a glibc 2.27 environment, and our minimum required glibc for that binary is 2.27.
2021-07-07scripts: more robustly test macOS symbol checksfanquake
2021-07-07build: Use and test PE binutils with --reloc-sectionCarl Dong
Also fix test-security-check.py to account for new PE PIE failure indication.
2021-07-07devtools: Improve *-check.py tool detectionCarl Dong
This is important to make sure that we're not testing tools different from the one we're building with. Introduce determine_wellknown_cmd, which encapsulates how we should handle well-known tools specification (IFS splitting, env override, etc.).
2021-07-06script: remove gitian reference from symbol-check.pyfanquake
2021-06-21Merge bitcoin/bitcoin#22244: devtools: Correctly extract symbol versions in ↵W. J. van der Laan
symbol-check e8cd3700eeb27437f5ea435869c9d61214285fdd devtools: Integrate ARCH_MIN_GLIBC_VER table into MAX_VERSIONS in symbol-check.py (W. J. van der Laan) a33381acf5ae2b43616fffaf26b1c8962e8ef0bb devtools: Add xkb version to symbol-check (W. J. van der Laan) 19e598bab0a1cb5ad93321eb9fa25d1a58d5e276 devtools: Fix verneed section parsing in pixie (W. J. van der Laan) Pull request description: I misunderstood the ELF specification for version symbols (verneed): The `vn_aux` pointer is relative to the main verneed record, not the start of the section. This caused many symbols to not be versioned properly in the return value of `elf.dyn_symbols`. This was discovered in #21454. Fix it by correcting the offset computation. - xkb versions symbols (using the prefix `V`), as this library is used by bitcoin-qt, add it to the valid versions in `symbol-check.py` This unfortunately brings to light some symbols that have been introduced since and weren't caught (from a gitian compile of master): ``` bitcoin-cli: symbol getrandom from unsupported version GLIBC_2.25 bitcoin-cli: failed IMPORTED_SYMBOLS bitcoind: symbol getrandom from unsupported version GLIBC_2.25 bitcoind: symbol log from unsupported version GLIBC_2.29 bitcoind: symbol fcntl64 from unsupported version GLIBC_2.28 bitcoind: symbol pow from unsupported version GLIBC_2.29 bitcoind: symbol exp from unsupported version GLIBC_2.29 bitcoind: failed IMPORTED_SYMBOLS bitcoin-qt: symbol exp from unsupported version GLIBC_2.29 bitcoin-qt: symbol fcntl64 from unsupported version GLIBC_2.28 bitcoin-qt: symbol log from unsupported version GLIBC_2.29 bitcoin-qt: symbol pow from unsupported version GLIBC_2.29 bitcoin-qt: symbol statx from unsupported version GLIBC_2.28 bitcoin-qt: symbol getrandom from unsupported version GLIBC_2.25 bitcoin-qt: symbol renameat2 from unsupported version GLIBC_2.28 bitcoin-qt: symbol getentropy from unsupported version GLIBC_2.25 bitcoin-qt: failed IMPORTED_SYMBOLS bitcoin-wallet: symbol exp from unsupported version GLIBC_2.29 bitcoin-wallet: symbol log from unsupported version GLIBC_2.29 bitcoin-wallet: symbol fcntl64 from unsupported version GLIBC_2.28 bitcoin-wallet: failed IMPORTED_SYMBOLS test_bitcoin: symbol getrandom from unsupported version GLIBC_2.25 test_bitcoin: symbol log from unsupported version GLIBC_2.29 test_bitcoin: symbol fcntl64 from unsupported version GLIBC_2.28 test_bitcoin: symbol pow from unsupported version GLIBC_2.29 test_bitcoin: symbol exp from unsupported version GLIBC_2.29 test_bitcoin: failed IMPORTED_SYMBOLS ``` ACKs for top commit: hebasto: ACK e8cd3700eeb27437f5ea435869c9d61214285fdd Tree-SHA512: 8c15e3478eb642f01a1ddaadef03f80583f088f9fa8e3bf171ce16b0ec05ffb4675ec147d7ffc6a4360637ed47fca517c6ca2bac7bb30d794c03783cfb964b79
2021-06-18devtools: Integrate ARCH_MIN_GLIBC_VER table into MAX_VERSIONS in ↵W. J. van der Laan
symbol-check.py The (ancient) versions specified here were deceptive. Entries older than MAX_VERSIONS['GLIBC'], which is 2.17, are ignored here. So reorganize the code to avoid confusion for other people reading this code.
2021-06-14devtools: Add xkb version to symbol-checkW. J. van der Laan
xkb versions symbols (using the prefix `V`), as this library is used by bitcoin-qt, add it to the valid versions in `symbol-check.py`.
2021-06-14devtools: Fix verneed section parsing in pixieW. J. van der Laan
I misunderstood the ELF specification for version symbols (verneed): The `vn_aux` pointer is relative to the main verneed record, not the start of the section. This caused many symbols to not be versioned properly in the return value of `elf.dyn_symbols`. This was discovered in #21454. Fix it by correcting the offset computation.
2021-06-10scripts: check macOS SDK version is setfanquake
Clangs Darwin driver should infer the SDK version used during compilation, and forward that through to the linker. Add a check that this has been done, and the expected SDK version is set. Should help prevent issues like #21771 in future.
2021-06-10scripts: check minimum required Windows version is setfanquake
We use linker flags (-Wl,--major/minor-subsystem-version) to set the minimum required version of Windows needed to run our binaries. This adds a sanity check that the version is being set as expected.
2021-06-10scripts: check minimum required macOS vesion is setfanquake
We use a compile flag (-mmacosx-version-min) to set the minimum required version of macOS needed to run our binaries. This adds a sanity check that the version is being set as expected.
2021-05-09scripts: test for MACHO control flow instrumentationfanquake
2021-05-04contrib: consolidate PIE and NX security checksfanquake
2021-05-04contrib: use LIEF in PE symbol checksfanquake
2021-05-04contrib: use LIEF in macOS symbol checksfanquake
2021-05-04contrib: use f strings in symbol-check.pyfanquake
2021-05-04contrib: add PE PIE check to security checksfanquake
2021-05-04contrib: use LIEF for PE security checksfanquake
2021-05-04contrib: use LIEF for macOS security checksfanquake
2021-05-04contrib: use f strings in security-check.pyfanquake
2021-03-19Doc: Tell howto install clang-format on Debian/Ubuntuwodry
Because only macOS wasy mentioned, I was unsure if this would be a macOS specific tool. I guess Linux is more used than Mac, so Linux guide should be there, too.
2021-03-13test: Drop unused get_machine functionHennadii Stepanov
The removed get_machine function is no longer used since #21255.
2021-03-13test: Cleanup test files in test-{security,symbol}-check.pyHennadii Stepanov
2021-03-10build: qt 5.12.10fanquake
remove fix_configure_mac.patch Fixed upstream: https://bugreports.qt.io/browse/QTBUG-67286 remove fix_riscv64_arch.patch Was fixed upstream in 6a39e49a6cdeb28a04a3657bb6a22f848d5dfa9d remove fix_rcc_determinism.patch Fixed upstream in https://bugreports.qt.io/browse/QTBUG-62511 remove freetype_back_compat.patch By the time we ship a release with Qt 5.12, we'll certainly no-longer be supporting Ubuntu 14.04 and Ubuntu 16.04 ships with FreeType 2.6.1, which is new enough that using the symbol is no-longer an issue. The renaming of FT_Get_X11_Font_Format() happened in FreeType 2.6 remove xkb-default.patch This was removed upstream in d5abf545971da717014d316127045fc19edbcd65 Co-authored-by: Hennadii Stepanov <32963518+hebasto@users.noreply.github.com>
2021-03-10build: Add xkbcommon 0.8.4Hennadii Stepanov
Co-authored-by: fanquake <fanquake@gmail.com>
2021-02-22contrib: run test-symbol check for RISCVfanquake
Now that we are using Focal for Gitian building (glibc 2.31), we can user a newer introduced symbol, and include RISCV in this test.
2021-01-12doc: Add manual page generation for bitcoin-utilWladimir J. van der Laan
- Add `-version` option to `bitcoin-util` - Add `bitcoin-util` call to `gen-manpages.sh` - Add stub manual page `bitcoin-util.1` - Add install of `bitcoin-util.1` to build system
2020-12-31scripted-diff: Bump copyright headersMarcoFalke
-BEGIN VERIFY SCRIPT- ./contrib/devtools/copyright_header.py update ./ -END VERIFY SCRIPT-
2020-12-28lint: run mypy over contrib/devtoolsfanquake
2020-12-18Merge #20434: contrib: Parse ELF directly for symbol and security checksWladimir J. van der Laan
a0a771843fc39c3cc2574a51f009c3391e1808e9 contrib: Changes to checks for PowerPC64 (Luke Dashjr) 634f6ec4eb9997d7bd0f8209fad49a4171d42384 contrib: Parse ELF directly for symbol and security checks (Wladimir J. van der Laan) Pull request description: Instead of the ever-messier text parsing of the output of the readelf tool (which is clearly meant for human consumption not to be machine parseable), parse the ELF binaries directly. Add a small dependency-less ELF parser specific to the checks. This is slightly more secure, too, because it removes potential ambiguity due to misparsing and changes in the output format of `elfread`. It also allows for stricter and more specific ELF format checks in the future. This removes the build-time dependency for `readelf`. It passes the test-security-check for me locally, ~~though I haven't checked on all platforms~~. I've checked that this works on the cross-compile output for all ELF platforms supported by Bitcoin Core at the moment, as well as PPC64 LE and BE. Top commit has no ACKs. Tree-SHA512: 7f9241fec83ee512642fecf5afd90546964561efd8c8c0f99826dcf6660604a4db2b7255e1afb1e9bb0211fd06f5dbad18a6175dfc03e39761a40025118e7bfc
2020-12-09contrib: add symbol check test for PEfanquake
2020-12-07Merge #20468: build: warn when generating man pages for binaries built from ↵Wladimir J. van der Laan
a dirty branch 6690adba08006739da0060eb4937126bdfa1181a Warn when binaries are built from a dirty branch. (Tyler Chambers) Pull request description: - Adjusted `--version` flag behavior in bitcoind and bitcoin-wallet to have the same behavior. - Added `--version` flag to bitcoin-tx to match. - Added functionality in gen-manpages.sh to error when attempting to generate man pages for binaries built from a dirty branch. mitigates problem with issue #20412 ACKs for top commit: laanwj: Tested ACK 6690adba08006739da0060eb4937126bdfa1181a Tree-SHA512: b5ca509f1a57f66808c2bebc4b710ca00c6fec7b5ebd7eef58018e28e716f5f2358e36551b8a4df571bf3204baed565a297aeefb93990e7a99add502b97ee1b8
2020-12-03contrib: add MACHO tests to symbol-check testsfanquake
2020-12-03contrib: Add test for ELF symbol-checkWladimir J. van der Laan
Check both failure cases: - Use a glibc symbol from a version that is too new - Use a symbol from a library that is not in the allowlist And also check a conforming binary. Adding a similar check for Windows PE can be done in a separate PR.
2020-11-28Warn when binaries are built from a dirty branch.Tyler Chambers
Adjusted version flag behavior in bitcoin-tx, bitcoin-wallet, and bitcoind to match. Added functionality in gen-manpages.sh to warning when attempting to generate man pages for binaries built from a dirty branch.
2020-11-22contrib: Changes to checks for PowerPC64Luke Dashjr
Changes from #14066.
2020-11-22contrib: Parse ELF directly for symbol and security checksWladimir J. van der Laan
Instead of the ever-messier text parsing of the output of the readelf tool (which is clearly meant for human consumption not to be machine parseable), parse the ELF binaries directly. Add a small dependency-less ELF parser specific to the checks. This is slightly more secure, too, because it removes potential ambiguity due to misparsing and changes in the output format of `elfread`. It also allows for stricter and more specific ELF format checks in the future. This removes the build-time dependency for `readelf`. It passes the test-security-check for me locally, though I haven't checked on all platforms.
2020-11-09swapped "is" for "==" in literal comparisonTyler Chambers
update lint-python.sh to include check F632
2020-08-29scripted-diff: Move previous_release.py to test/get_previous_releases.pyHennadii Stepanov
-BEGIN VERIFY SCRIPT- OLD=contrib/devtools/previous_release.py NEW=test/get_previous_releases.py sed -i "s|$OLD|$NEW|g" $(git grep -l $OLD) git mv $OLD $NEW -END VERIFY SCRIPT-
2020-07-30Merge #18011: Replace current benchmarking framework with nanobenchWladimir J. van der Laan
78c312c983255e15fc274de2368a2ec13ce81cbf Replace current benchmarking framework with nanobench (Martin Ankerl) Pull request description: Replace current benchmarking framework with nanobench This replaces the current benchmarking framework with nanobench [1], an MIT licensed single-header benchmarking library, of which I am the autor. This has in my opinion several advantages, especially on Linux: * fast: Running all benchmarks takes ~6 seconds instead of 4m13s on an Intel i7-8700 CPU @ 3.20GHz. * accurate: I ran e.g. the benchmark for SipHash_32b 10 times and calculate standard deviation / mean = coefficient of variation: * 0.57% CV for old benchmarking framework * 0.20% CV for nanobench So the benchmark results with nanobench seem to vary less than with the old framework. * It automatically determines runtime based on clock precision, no need to specify number of evaluations. * measure instructions, cycles, branches, instructions per cycle, branch misses (only Linux, when performance counters are available) * output in markdown table format. * Warn about unstable environment (frequency scaling, turbo, ...) * For better profiling, it is possible to set the environment variable NANOBENCH_ENDLESS to force endless running of a particular benchmark without the need to recompile. This makes it to e.g. run "perf top" and look at hotspots. Here is an example copy & pasted from the terminal output: | ns/byte | byte/s | err% | ins/byte | cyc/byte | IPC | bra/byte | miss% | total | benchmark |--------------------:|--------------------:|--------:|----------------:|----------------:|-------:|---------------:|--------:|----------:|:---------- | 2.52 | 396,529,415.94 | 0.6% | 25.42 | 8.02 | 3.169 | 0.06 | 0.0% | 0.03 | `bench/crypto_hash.cpp RIPEMD160` | 1.87 | 535,161,444.83 | 0.3% | 21.36 | 5.95 | 3.589 | 0.06 | 0.0% | 0.02 | `bench/crypto_hash.cpp SHA1` | 3.22 | 310,344,174.79 | 1.1% | 36.80 | 10.22 | 3.601 | 0.09 | 0.0% | 0.04 | `bench/crypto_hash.cpp SHA256` | 2.01 | 496,375,796.23 | 0.0% | 18.72 | 6.43 | 2.911 | 0.01 | 1.0% | 0.00 | `bench/crypto_hash.cpp SHA256D64_1024` | 7.23 | 138,263,519.35 | 0.1% | 82.66 | 23.11 | 3.577 | 1.63 | 0.1% | 0.00 | `bench/crypto_hash.cpp SHA256_32b` | 3.04 | 328,780,166.40 | 0.3% | 35.82 | 9.69 | 3.696 | 0.03 | 0.0% | 0.03 | `bench/crypto_hash.cpp SHA512` [1] https://github.com/martinus/nanobench ACKs for top commit: laanwj: ACK 78c312c983255e15fc274de2368a2ec13ce81cbf Tree-SHA512: 9e18770b18b6f95a7d0105a4a5497d31cf4eb5efe6574f4482f6f1b4c88d7e0946b9a4a1e9e8e6ecbf41a3f2d7571240677dcb45af29a6f0584e89b25f32e49e
2020-07-28devtools: Add security check for separate_codeWladimir J. van der Laan
Check that sections are appropriately separated in virtual memory, based on their (expected) permissions. This checks for missing -Wl,-z,separate-code and potentially other problems. Co-authored-by: fanquake <fanquake@gmail.com>
2020-07-21contrib: Clean up previous_releases.pyMarcoFalke
* Replace curl single char options with their verbose counterpart * Stricter check for tarballHash
2020-07-21Merge #19205: script: previous_release.sh rewritten in pythonMarcoFalke
9c34aff39309b8adc99d347e07b6ddb5366498e9 Remove previous_release.sh (Brian Liotti) e1e5960e10a9329d9f55a3967d546ffbdd896030 script: Add previous_release.py (Brian Liotti) Pull request description: Closes #18132 Added functionality: 1) checks file hash before untarring when using the binary download option ACKs for top commit: fjahr: re-ACK 9c34aff39309b8adc99d347e07b6ddb5366498e9 Sjors: tACK 9c34aff39309b8adc99d347e07b6ddb5366498e9 Tree-SHA512: 323f11828736a372a47f048592de8b027ddcd75b38f312dfc73f7b495d1e078bfeb384d9cdf434b3e70f2c6c0ce2da2df48e9a6460ac0e1967c6829a411c52d5
2020-07-21Make test DoS_mapOrphans deterministicDavid Reikher
The RandomOrphan function and the function ecdsa_signature_parse_der_lax in pubkey.cpp were causing non-deterministic test coverage. Force seed in the beginning of the test to make it deterministic. The seed is selected carefully so that all branches of the function ecdsa_signature_parse_der_lax are executed. Prior to this fix, the test was exhibiting non-deterministic coverage since none of the ECDSA signatures that were generated during the test had leading zeroes in either R, S, or both, resulting in some branches of said function not being executed. The seed ensures that both conditions are hit. Removed denialofservice_tests test entry from the list of non-deterministic tests in the coverage script.
2020-07-05Remove previous_release.shBrian Liotti
2020-07-05script: Add previous_release.pyBrian Liotti
closes #18132 added GPG verify for binaries co-authored-by: bboot <bboot@cisco.com>
2020-06-16test: use subprocess.run() in test-security-check.pyfanquake
2020-06-16tests: run test-security-check.py in CIfanquake
2020-06-13Replace current benchmarking framework with nanobenchMartin Ankerl
This replaces the current benchmarking framework with nanobench [1], an MIT licensed single-header benchmarking library, of which I am the autor. This has in my opinion several advantages, especially on Linux: * fast: Running all benchmarks takes ~6 seconds instead of 4m13s on an Intel i7-8700 CPU @ 3.20GHz. * accurate: I ran e.g. the benchmark for SipHash_32b 10 times and calculate standard deviation / mean = coefficient of variation: * 0.57% CV for old benchmarking framework * 0.20% CV for nanobench So the benchmark results with nanobench seem to vary less than with the old framework. * It automatically determines runtime based on clock precision, no need to specify number of evaluations. * measure instructions, cycles, branches, instructions per cycle, branch misses (only Linux, when performance counters are available) * output in markdown table format. * Warn about unstable environment (frequency scaling, turbo, ...) * For better profiling, it is possible to set the environment variable NANOBENCH_ENDLESS to force endless running of a particular benchmark without the need to recompile. This makes it to e.g. run "perf top" and look at hotspots. Here is an example copy & pasted from the terminal output: | ns/byte | byte/s | err% | ins/byte | cyc/byte | IPC | bra/byte | miss% | total | benchmark |--------------------:|--------------------:|--------:|----------------:|----------------:|-------:|---------------:|--------:|----------:|:---------- | 2.52 | 396,529,415.94 | 0.6% | 25.42 | 8.02 | 3.169 | 0.06 | 0.0% | 0.03 | `bench/crypto_hash.cpp RIPEMD160` | 1.87 | 535,161,444.83 | 0.3% | 21.36 | 5.95 | 3.589 | 0.06 | 0.0% | 0.02 | `bench/crypto_hash.cpp SHA1` | 3.22 | 310,344,174.79 | 1.1% | 36.80 | 10.22 | 3.601 | 0.09 | 0.0% | 0.04 | `bench/crypto_hash.cpp SHA256` | 2.01 | 496,375,796.23 | 0.0% | 18.72 | 6.43 | 2.911 | 0.01 | 1.0% | 0.00 | `bench/crypto_hash.cpp SHA256D64_1024` | 7.23 | 138,263,519.35 | 0.1% | 82.66 | 23.11 | 3.577 | 1.63 | 0.1% | 0.00 | `bench/crypto_hash.cpp SHA256_32b` | 3.04 | 328,780,166.40 | 0.3% | 35.82 | 9.69 | 3.696 | 0.03 | 0.0% | 0.03 | `bench/crypto_hash.cpp SHA512` [1] https://github.com/martinus/nanobench * Adds support for asymptotes This adds support to calculate asymptotic complexity of a benchmark. This is similar to #17375, but currently only one asymptote is supported, and I have added support in the benchmark `ComplexMemPool` as an example. Usage is e.g. like this: ``` ./bench_bitcoin -filter=ComplexMemPool -asymptote=25,50,100,200,400,600,800 ``` This runs the benchmark `ComplexMemPool` several times but with different complexityN settings. The benchmark can extract that number and use it accordingly. Here, it's used for `childTxs`. The output is this: | complexityN | ns/op | op/s | err% | ins/op | cyc/op | IPC | total | benchmark |------------:|--------------------:|--------------------:|--------:|----------------:|----------------:|-------:|----------:|:---------- | 25 | 1,064,241.00 | 939.64 | 1.4% | 3,960,279.00 | 2,829,708.00 | 1.400 | 0.01 | `ComplexMemPool` | 50 | 1,579,530.00 | 633.10 | 1.0% | 6,231,810.00 | 4,412,674.00 | 1.412 | 0.02 | `ComplexMemPool` | 100 | 4,022,774.00 | 248.58 | 0.6% | 16,544,406.00 | 11,889,535.00 | 1.392 | 0.04 | `ComplexMemPool` | 200 | 15,390,986.00 | 64.97 | 0.2% | 63,904,254.00 | 47,731,705.00 | 1.339 | 0.17 | `ComplexMemPool` | 400 | 69,394,711.00 | 14.41 | 0.1% | 272,602,461.00 | 219,014,691.00 | 1.245 | 0.76 | `ComplexMemPool` | 600 | 168,977,165.00 | 5.92 | 0.1% | 639,108,082.00 | 535,316,887.00 | 1.194 | 1.86 | `ComplexMemPool` | 800 | 310,109,077.00 | 3.22 | 0.1% |1,149,134,246.00 | 984,620,812.00 | 1.167 | 3.41 | `ComplexMemPool` | coefficient | err% | complexity |--------------:|-------:|------------ | 4.78486e-07 | 4.5% | O(n^2) | 6.38557e-10 | 21.7% | O(n^3) | 3.42338e-05 | 38.0% | O(n log n) | 0.000313914 | 46.9% | O(n) | 0.0129823 | 114.4% | O(log n) | 0.0815055 | 133.8% | O(1) The best fitting curve is O(n^2), so the algorithm seems to scale quadratic with `childTxs` in the range 25 to 800.
generated by cgit v1.2.3 (git 2.26.2) at 2024-11-01 08:44:21 +0000