aboutsummaryrefslogtreecommitdiff
path: root/contrib
AgeCommit message (Collapse)Author
2020-04-10Merge #18295: scripts: add MACHO lazy bindings check to security-check.pyfanquake
5ca90f8b598978437340bb8467f527b9edfb2bbf scripts: add MACHO lazy bindings check to security-check.py (fanquake) Pull request description: This is a slightly belated follow up to #17686 and some discussion with Cory. It's not entirely clear if we should make this change due to the way the macOS dynamic loader appears to work. However I'm opening this for some discussion. Also related to #17768. #### Issue: [`LD64`](https://opensource.apple.com/source/ld64/) doesn't set the [MH_BINDATLOAD](https://opensource.apple.com/source/xnu/xnu-6153.11.26/EXTERNAL_HEADERS/mach-o/loader.h.auto.html) bit in the header of MACHO executables, when building with `-bind_at_load`. This is in contradiction to the [documentation](https://opensource.apple.com/source/ld64/ld64-450.3/doc/man/man1/ld.1.auto.html): ```bash -bind_at_load Sets a bit in the mach header of the resulting binary which tells dyld to bind all symbols when the binary is loaded, rather than lazily. ``` The [`ld` in Apples cctools](https://opensource.apple.com/source/cctools/cctools-927.0.2/ld/layout.c.auto.html) does set the bit, however the [cctools-port](https://github.com/tpoechtrager/cctools-port/) that we use for release builds, bundles `LD64`. However; even if the linker hasn't set that bit, the dynamic loader ([`dyld`](https://opensource.apple.com/source/dyld/)) doesn't seem to ever check for it, and from what I understand, it looks at a different part of the header when determining whether to lazily load symbols. Note that our release binaries are currently working as expected, and no lazy loading occurs. #### Example: Using a small program, we can observe the behaviour of the dynamic loader. Conducted using: ```bash clang++ --version Apple clang version 11.0.0 (clang-1100.0.33.17) Target: x86_64-apple-darwin18.7.0 ld -v @(#)PROGRAM:ld PROJECT:ld64-530 BUILD 18:57:17 Dec 13 2019 LTO support using: LLVM version 11.0.0, (clang-1100.0.33.17) (static support for 23, runtime is 23) TAPI support using: Apple TAPI version 11.0.0 (tapi-1100.0.11) ``` ```cpp #include <iostream> int main() { std::cout << "Hello World!\n"; return 0; } ``` Compile and check the MACHO header: ```bash clang++ test.cpp -o test otool -vh test ... Mach header magic cputype cpusubtype caps filetype ncmds sizeofcmds flags MH_MAGIC_64 X86_64 ALL LIB64 EXECUTE 16 1424 NOUNDEFS DYLDLINK TWOLEVEL WEAK_DEFINES BINDS_TO_WEAK PIE # Run and dump dynamic loader bindings: DYLD_PRINT_BINDINGS=1 DYLD_PRINT_TO_FILE=no_bind.txt ./test Hello World! ``` Recompile with `-bind_at_load`. Note still no `BINDATLOAD` flag: ```bash clang++ test.cpp -o test -Wl,-bind_at_load otool -vh test Mach header magic cputype cpusubtype caps filetype ncmds sizeofcmds flags MH_MAGIC_64 X86_64 ALL LIB64 EXECUTE 16 1424 NOUNDEFS DYLDLINK TWOLEVEL WEAK_DEFINES BINDS_TO_WEAK PIE ... DYLD_PRINT_BINDINGS=1 DYLD_PRINT_TO_FILE=bind.txt ./test Hello World! ``` If we diff the outputs, you can see that `dyld` doesn't perform any lazy bindings when the binary is compiled with `-bind_at_load`, even if the `BINDATLOAD` flag is not set: ```diff @@ -1,11 +1,27 @@ +dyld: bind: test:0x103EDF030 = libc++.1.dylib:__ZNKSt3__16locale9use_facetERNS0_2idE, *0x103EDF030 = 0x7FFF70C9FA58 +dyld: bind: test:0x103EDF038 = libc++.1.dylib:__ZNKSt3__18ios_base6getlocEv, *0x103EDF038 = 0x7FFF70CA12C2 +dyld: bind: test:0x103EDF068 = libc++.1.dylib:__ZNSt3__113basic_ostreamIcNS_11char_traitsIcEEE6sentryC1ERS3_, *0x103EDF068 = 0x7FFF70CA12B6 +dyld: bind: test:0x103EDF070 = libc++.1.dylib:__ZNSt3__113basic_ostreamIcNS_11char_traitsIcEEE6sentryD1Ev, *0x103EDF070 = 0x7FFF70CA1528 +dyld: bind: test:0x103EDF080 = libc++.1.dylib:__ZNSt3__16localeD1Ev, *0x103EDF080 = 0x7FFF70C9FAE6 <trim> -dyld: lazy bind: test:0x10D4AC0C8 = libsystem_platform.dylib:_strlen, *0x10D4AC0C8 = 0x7FFF73C5C6E0 -dyld: lazy bind: test:0x10D4AC068 = libc++.1.dylib:__ZNSt3__113basic_ostreamIcNS_11char_traitsIcEEE6sentryC1ERS3_, *0x10D4AC068 = 0x7FFF70CA12B6 -dyld: lazy bind: test:0x10D4AC038 = libc++.1.dylib:__ZNKSt3__18ios_base6getlocEv, *0x10D4AC038 = 0x7FFF70CA12C2 -dyld: lazy bind: test:0x10D4AC030 = libc++.1.dylib:__ZNKSt3__16locale9use_facetERNS0_2idE, *0x10D4AC030 = 0x7FFF70C9FA58 -dyld: lazy bind: test:0x10D4AC080 = libc++.1.dylib:__ZNSt3__16localeD1Ev, *0x10D4AC080 = 0x7FFF70C9FAE6 -dyld: lazy bind: test:0x10D4AC070 = libc++.1.dylib:__ZNSt3__113basic_ostreamIcNS_11char_traitsIcEEE6sentryD1Ev, *0x10D4AC070 = 0x7FFF70CA1528 ``` Note: `dyld` also has a `DYLD_BIND_AT_LAUNCH=1` environment variable, that when set, will force any lazy bindings to be non-lazy: ```bash dyld: forced lazy bind: test:0x10BEC8068 = libc++.1.dylib:__ZNSt3__113basic_ostream ``` #### Thoughts: After looking at the dyld source, I can't find any checks for `MH_BINDATLOAD`. You can see the flags it does check for, such as MH_PIE or MH_BIND_TO_WEAK [here](https://opensource.apple.com/source/dyld/dyld-732.8/src/ImageLoaderMachO.cpp.auto.html). It seems that the lazy binding of any symbols depends on whether or not [lazy_bind_size](https://opensource.apple.com/source/xnu/xnu-6153.11.26/EXTERNAL_HEADERS/mach-o/loader.h.auto.html) from the `LC_DYLD_INFO_ONLY` load command is > 0. Which was mentioned in [#17686](https://github.com/bitcoin/bitcoin/pull/17686#issue-350216254). #### Changes: This PR is one of [Corys commits](https://github.com/theuni/bitcoin/commit/7b6ba26178d2754568a1308d3d44e038e9ebf450), that I've rebased and modified to make build. I've also included an addition to the `security-check.py` script to check for the flag. However, given the above, I'm not entirely sure this patch is the correct approach. If the linker no-longer inserts it, and the dynamic loader doesn't look for it, there might be little benefit to setting it. Or, maybe this is an oversight from Apple and needs some upstream discussion. Looking for some thoughts / Concept ACK/NACK. One alternate approach we could take is to drop the patch and modify security-check.py to look for `lazy_bind_size` == 0 in the `LC_DYLD_INFO_ONLY` load command, using `otool -l`. ACKs for top commit: theuni: ACK 5ca90f8b598978437340bb8467f527b9edfb2bbf Tree-SHA512: 444022ea9d19ed74dd06dc2ab3857a9c23fbc2f6475364e8552d761b712d684b3a7114d144f20de42328d1a99403b48667ba96885121392affb2e05b834b6e1c
2020-04-06Merge #18506: net: Hardcoded seeds update for 0.20Wladimir J. van der Laan
0eeb0468e7debb1dbe38242769207d22ed52c1df net: Hardcoded seeds update for 0.20 (Wladimir J. van der Laan) Pull request description: Update hardcoded seeds from http://bitcoin.sipa.be/seeds.txt.gz, according to release process. Output from makeseeds.py: ``` IPv4 IPv6 Onion Pass 1364173 244127 2454 Initial 1364173 244127 2454 Skip entries with invalid address 1129552 213117 2345 After removing duplicates 1129548 213117 2345 Skip entries from suspicious hosts 338216 191944 2249 Enforce minimal number of blocks 336851 188993 2189 Require service bit 1 6998 1520 150 Require minimum uptime 5682 1290 89 Require a known and recent user agent 5622 1279 89 Filter out hosts with multiple bitcoin ports 512 146 89 Look up ASNs and limit results per ASN and per net ``` Top commit has no ACKs. Tree-SHA512: ce1c2cda18dd5bd22586a5283a0877f3bd890437cc29dc1d85452ba4a4d28032f591c8b37f3329e8e649556cf83750b6949a068fad76d1773853d93014609da0
2020-04-04scripts: add MACHO lazy bindings check to security-check.pyfanquake
2020-04-03net: Hardcoded seeds update for 0.20Wladimir J. van der Laan
Update hardcoded seeds from seeds_emzy.txt seeds_lukejr.txt seeds_sipa.txt seeds_sjors.txt, according to release process. Output from makeseeds.py: ``` IPv4 IPv6 Onion Pass 1364173 244127 2454 Initial 1364173 244127 2454 Skip entries with invalid address 1129552 213117 2345 After removing duplicates 1129548 213117 2345 Skip entries from suspicious hosts 338216 191944 2249 Enforce minimal number of blocks 336851 188993 2189 Require service bit 1 6998 1520 150 Require minimum uptime 5682 1290 89 Require a known and recent user agent 5622 1279 89 Filter out hosts with multiple bitcoin ports 512 146 89 Look up ASNs and limit results per ASN and per net ```
2020-04-03Merge #18426: scripts: previous_release: improve behaviour on failed downloadfanquake
332f373a9dece71717f75eb06e6a1fc957f2952b [scripts] previous_release: improve failed download error message (Sebastian Falbesoner) Pull request description: Currently, if the earlier release build/fetch script `previous_release.sh` is invoked with the option `-b` (intending to fetch a binary package from `https://bitcoin.org`) and the download fails, the user sees the following confusing output: ``` $ contrib/devtools/previous_release.sh -r -b v0.9.5 [...] gzip: stdin: not in gzip format tar: Child returned status 1 tar: Error is not recoverable: exiting now ``` This implies that the download worked, but the archive is corrupted, when in reality the HTML document containing the delivery fail reason (most likely 404 Not Found) is saved and tried to get unpacked. In contrast to wget, curl is a bit stubborn and needs explicit instructions to react to server errors via the flag `-f` (outputs error message and returns error code, ideal for scripts): https://curl.haxx.se/docs/manpage.html#-f On the PR branch, the output on failed download looks now the following: ``` $ contrib/devtools/previous_release.sh -r -b v0.9.5 [...] curl: (22) The requested URL returned error: 404 Not Found Download failed. ``` ACKs for top commit: fanquake: ACK 332f373a9dece71717f75eb06e6a1fc957f2952b Tree-SHA512: 046c931ad9e78aeb2d13faa4866d46122ed325aa142483547c2b04032d03223ed2411783b00106fcab0cd91b2f78691531ac526ed7bb3ed7547b6e2adbfb2e93
2020-03-26scripts: rename test_64bit_PE to test_PEfanquake
2020-03-26scripts: add MACHO NX check to security-check.pyfanquake
2020-03-26scripts: add MACHO tests to test-security-check.pyfanquake
2020-03-25Merge #18425: releases: Update with new Windows code signing certificateWladimir J. van der Laan
3e0df92bf216e1dce05ca9bf14049f2e42783c30 Update with new Windows code signing certificate (Andrew Chow) Pull request description: The current Windows code signing certificate is about expire (on March 26th 2020). As I have volunteered to take over the Windows code signing duties, I've purchased a new Windows code signing certificate with the same CA and under the same organization (Bitcoin Core Code Signing Association). A signature by the old certificate over the new certificate has been provided to me. This signature can be verified using ``` openssl cms -verify -inform pem -purpose any -content path/to/new/win-codesign.cert -CAfile path/to/old/win-codesign.cert -certfile path/to/old/win-codesign.cert ``` The verification should succeed and the new certificate will be printed out. This can be compared to the contents of `win-codesign.cert`. ``` -----BEGIN PKCS7----- MIIC3AYJKoZIhvcNAQcCoIICzTCCAskCAQExDzANBglghkgBZQMEAgEFADALBgkq hkiG9w0BBwExggKkMIICoAIBATCBkTB8MQswCQYDVQQGEwJHQjEbMBkGA1UECBMS R3JlYXRlciBNYW5jaGVzdGVyMRAwDgYDVQQHEwdTYWxmb3JkMRgwFgYDVQQKEw9T ZWN0aWdvIExpbWl0ZWQxJDAiBgNVBAMTG1NlY3RpZ28gUlNBIENvZGUgU2lnbmlu ZyBDQQIRALWcUnSOxv9FQW3xdaMDO6swDQYJYIZIAWUDBAIBBQCggeQwGAYJKoZI hvcNAQkDMQsGCSqGSIb3DQEHATAcBgkqhkiG9w0BCQUxDxcNMjAwMzI0MjA0ODM3 WjAvBgkqhkiG9w0BCQQxIgQgtLkmnuSQyczDlJSnJeqbi61p3iJ/rpFABrY8JWBO o74weQYJKoZIhvcNAQkPMWwwajALBglghkgBZQMEASowCwYJYIZIAWUDBAEWMAsG CWCGSAFlAwQBAjAKBggqhkiG9w0DBzAOBggqhkiG9w0DAgICAIAwDQYIKoZIhvcN AwICAUAwBwYFKw4DAgcwDQYIKoZIhvcNAwICASgwDQYJKoZIhvcNAQEBBQAEggEA XaCl3Q8HwI9VpLCb9OY9eQh0QOPyl1KWEc3TP3UvwZwR4/gXkfPOKKf19UnS8eRB 48SgUKRMYWoDYfSVUJRMda9BLkbJbQlHG3LFXhSY2alajpPXEHcMto/XPhVAmqzL w6aSNY0Gaorow696JHpetpKqAAlL1r2GjeaPYi2aZyIAifuhay/qwA+ig0SqzGOw UdgFZWMyS5yanq8/WlLCCql6kKOzT4tEqUaleD7R1q8BTcG2+fmhWR8WwJLpIV6y 7GAqt0Cocu8sYpTNBNk8iKHxzZ2hMZKJpH9lHZuiJ/9vSercrvDy2R4/MG+KnBWb OyiFAt2mC51+63RhLOMJfg== -----END PKCS7----- ``` ACKs for top commit: laanwj: ACK 3e0df92bf216e1dce05ca9bf14049f2e42783c30 theuni: ACK 3e0df92bf216e1dce05ca9bf14049f2e42783c30. Tree-SHA512: 4210f4db1e805ab11231fbae49ea197257c6f7e44f1f6219685b63831704984d824ac2f9e0a3b1bd2655953af72636a474f077cb859fb35852551f5a9f8fbde3
2020-03-25Merge #18395: scripts: add PE dylib checking to symbol-check.pyWladimir J. van der Laan
1a0993ae354c36d6f219e67f82ca8236530d6201 scripts: add PE dylib checking to symbol-check.py (fanquake) Pull request description: Uses `objdump -x` and looks for `DLL Name:` lines. i.e: ```bash objdump -x src/qt/bitcoin-qt.exe | grep "DLL Name:" DLL Name: ADVAPI32.dll DLL Name: dwmapi.dll DLL Name: GDI32.dll DLL Name: IMM32.dll DLL Name: IPHLPAPI.DLL DLL Name: KERNEL32.dll DLL Name: msvcrt.dll DLL Name: ole32.dll DLL Name: OLEAUT32.dll DLL Name: SHELL32.dll DLL Name: SHLWAPI.dll DLL Name: USER32.dll DLL Name: UxTheme.dll DLL Name: VERSION.dll DLL Name: WINMM.dll DLL Name: WS2_32.dll ``` ACKs for top commit: dongcarl: Concept ACK 1a0993ae354c36d6f219e67f82ca8236530d6201 hebasto: ACK 1a0993ae354c36d6f219e67f82ca8236530d6201, tested on Linux Mint 19.3: Tree-SHA512: 0099a50e2c616d5239a15cafa9a7c483e9c40244af41549e4738be0f5360f27a2afb956eb50b47cf446b242f4cfc6dc9d111306a056fb83789eefbd71eddabd2
2020-03-25Merge #18331: build: Use git archive as source tarballWladimir J. van der Laan
e4d366788bc2e8dce8e6ca572fce08d913d15d6b build: Drop needless EXTRA_DIST content (Hennadii Stepanov) 6c4da59f5b5b3c40526d38965d4ffa7fd59f2ebc build: Drop SOURCEDIST reordering (Hennadii Stepanov) 5e6b8b391243016cb06e9e107c2e6a13a744b31e build: Use git archive as source tarball (Hennadii Stepanov) Pull request description: This PR: - is an alternative to #17104 - closes #16734 - closes #6753 The idea is clear described by some developers: - [MarcoFalke](https://github.com/bitcoin/bitcoin/pull/17097#issuecomment-540691850): > This whole concept of explicitly listing each and every file manually (or with a fragile wildcard) is an obvious sisyphean task. I'd say all we need to do is run git archive and be done with it forever, see #16734, #6753, #11530 ... - [laanwj](https://github.com/bitcoin/bitcoin/pull/17097#issuecomment-540706025): > I agree, I've never been a fan of it. I don't think we have any files in the git repository we don't want to ship in the source tarball. --- The suggested changes have a downside which is pointed by [**luke-jr**](https://github.com/bitcoin/bitcoin/pull/17104#issuecomment-540828045): > ... but the distfile needs to include autogen-generated files. This means that a user is not able to run `./configure && make` right away. One must run `./autogen.sh` at first. Here are opinions about mandatory use of `./autogen.sh`: - [ryanofsky](https://github.com/bitcoin/bitcoin/issues/16734#issuecomment-534139356): > It's probably ok to require autogen. I think historically configure scripts were supposed to work on obscure unix systems that would just have a generic shell + make tool + c compiler, and not necessarily need gnu packages like m4 which are needed for autogen. - [laanwj](https://github.com/bitcoin/bitcoin/issues/16734#issuecomment-540729483): > I also think it's fine to require autogen. What is one dependency more, if you're building from source. --- ~Also this PR provides Windows users with ZIP archives of the sources. Additionally the commit ID is stored in these ZIP files as a file comment:~ --- Note for reviewers: please verify is `git archive` output deterministic? ACKs for top commit: MarcoFalke: re-ACK e4d366788bc2e8dce8e6ca572fce08d913d15d6b, only change is adding two dots in a the path 🛳 laanwj: ACK e4d366788bc2e8dce8e6ca572fce08d913d15d6b Tree-SHA512: d1153d3ca4a580696019b92be3555ab004d197d9a2146aacff9d3150eb7093b7d40eebd6eea12d861d93ff62d62b68706e04e64dbe5ea796ff6757486e462193
2020-03-25[scripts] previous_release: improve failed download error messageSebastian Falbesoner
before: ------------------------------------------------------------ $ contrib/devtools/previous_release.sh -r -b v0.9.5 [...] gzip: stdin: not in gzip format tar: Child returned status 1 tar: Error is not recoverable: exiting now ------------------------------------------------------------ now: ------------------------------------------------------------ $ contrib/devtools/previous_release.sh -r -b v0.9.5 [...] curl: (22) The requested URL returned error: 404 Not Found Download failed. ------------------------------------------------------------
2020-03-24Update with new Windows code signing certificateAndrew Chow
2020-03-22scripts: add PE dylib checking to symbol-check.pyfanquake
2020-03-15build: Drop needless EXTRA_DIST contentHennadii Stepanov
Some EXTRA_DIST content is needless since a git archive is used as the source tarball.
2020-03-12build: Drop SOURCEDIST reorderingHennadii Stepanov
Making SOURCEDIST deterministic is needless since a git archive is used as the source tarball.
2020-03-12build: Use git archive as source tarballHennadii Stepanov
2020-03-11guix: Remove now-unnecessary gcc make flagCarl Dong
Previously, Guix would produce a gcc which did not know to use the SSP function from glibc, and required a gcc make flag for it to do so, in my attempt to fix it upstream I realized that this is no longer the case. This can be verified by performing a Guix build and doing readelf -s ... | grep __stack_chk to check that symbols are coming from glibc, and doing readelf -d ... | grep NEEDED | grep ssp to see that libssp.so is not being depended on
2020-02-13build: pass -fno-ident in Windows gitian descriptorfanquake
This prevents compilers from emitting compiler name and version number info that can needlessly bloat binaries. Accepted by Clang and GCC. See: https://clang.llvm.org/docs/ClangCommandLineReference.html#cmdoption-clang-qn https://gcc.gnu.org/onlinedocs/gcc/Code-Gen-Options.html#index-fno-ident
2020-02-11[scripts] support release candidates of earlier releasesSjors Provoost
2020-02-11[scripts] build earlier releasesSjors Provoost
2020-02-11Merge #18104: build: Skip i686 build by default in guix and gitianfanquake
fae9084ac5b10f94bdee54853d307838c4254e9c build: Skip i686 build by default in guix and gitian (MarcoFalke) fa55a2554c2661b8f2a759044d5ac85c9979d9ca depends: Remove reference to win32 (MarcoFalke) Pull request description: Closes #17504 Now that we no longer provide downloads for i686 on our website (https://bitcoincore.org/en/download/), there is no need to build them by default. i686 can still be built in depends (tested by ci/travis) and in guix/gitian by setting the appropriate `HOSTS`. ACKs for top commit: practicalswift: ACK fae9084ac5b10f94bdee54853d307838c4254e9c -- patch looks correct dongcarl: ACK fae9084ac5b10f94bdee54853d307838c4254e9c patch looks correct laanwj: Code review ACK fae9084ac5b10f94bdee54853d307838c4254e9c hebasto: ACK fae9084ac5b10f94bdee54853d307838c4254e9c, I have reviewed the code and it looks OK, I agree it can be merged. Tree-SHA512: b000c19a2cd2a596a52028fa298c4022c24cfdfc1bdb3795a90916d0a00a32e4dd22278db93790b6a11724e08ea8451f4f05c77bc40d1664518e11a8c82d6e29
2020-02-10Merge #17398: build: Update leveldb to 1.22+Wladimir J. van der Laan
677fb8e92380d4deb6a3753047c01f7cf7b5af91 test: Add ubsan surpression for crc32c (Wladimir J. van der Laan) 8e68bb1ddeca504bedd40aee8492b5478a88c1e5 build: Disable msvc warning 4722 for leveldb build (Aaron Clauson) be23949765e1b2e050574c6c2a136658a89dee5d build: MSVC changes for leveldb update (Aaron Clauson) 9ebdf047578f0da7e6578d0c51c32f55e84ac157 build: CRC32C build system integration (Wladimir J. van der Laan) 402252a8081e25f22aa1a5c60708714cf1d84ec4 build: Add LCOV exception for crc32c (Wladimir J. van der Laan) 3a037d0067c2c12a1c2c800fb85613a0a2911253 test: Add crc32c exception to various linters and generation scripts (Wladimir J. van der Laan) 84ff1b2076ef91ce688930d0aa0a7f4078ef3e1d test: Add crc32c to subtree check linter (Wladimir J. van der Laan) 7cf13a513409c18d18dff2f6203b3630937b487d doc: Add crc32c subtree to developer notes (Wladimir J. van der Laan) 24d02a9ac00a82d172b171f73554a882df264c80 build: Update build system for new leveldb (Wladimir J. van der Laan) 2e1819311a59fb5cb26e3ca50a510bfe01358350 Squashed 'src/crc32c/' content from commit 224988680f7673cd7c769963d4035cb315aa3388 (Wladimir J. van der Laan) 66480821b36c839ab7615cb9309850015bceadb0 Squashed 'src/leveldb/' changes from f545dfabff4c2e9836efed094dba99a34fbc6b88..f8ae182c1e5176d12e816fb2217ae33a5472fdd7 (Wladimir J. van der Laan) Pull request description: This updates leveldb to currently newest upstream commit https://github.com/bitcoin-core/leveldb/commit/0c40829872a9f00f38e11dc370ff8adb3e19f25b: - CRC32C hardware acceleration is now an external library [crc32c](https://github.com/google/crc32c). This adds acceleration on ARM, and should be faster on x86 because of using prefetch. It also makes it easy to support similar instruction sets on other platforms in the future. - Thread handling uses C++11, instead of platform specific code. - Native windows environment was added. No need to maintain our own hacky one, anymore. - Upstream now builds using CMake. This doesn't mean we need to use that (phew), but internal configuration changed to a a series of checks, instead of OS profiles. This means the blanket error "Cannot build leveldb for $host. Please file a bug report' is removed. All changes: https://github.com/google/leveldb/compare/a53934a3ae1244679f812d998a4f16f2c7f309a6...0c40829872a9f00f38e11dc370ff8adb3e19f25b Pretty much all our changes have been subsumed by upstream, so we figured it was cleaner to start over with a new branch from upstream with the still-relevant patches applied: https://github.com/bitcoin-core/leveldb/tree/bitcoin-fork-new There's quite some testing to be done (see below). See https://github.com/bitcoin-core/leveldb/issues/25 and https://github.com/bitcoin-core/leveldb/pull/26 for more history and context. TODO: - [x] Subtree `crc32c` - [x] Make linters happy about crc32 subtree - [x] Integrate `crc32c` library into build system - [x] MSVC build system ACKs for top commit: sipa: ACK 677fb8e92380d4deb6a3753047c01f7cf7b5af91 Tree-SHA512: 37ee92a750e053e924bc4626b12bb3fd81faa9f8c5ebaa343931fee810c45ba05aa6051fdea82535fa351bf2be7297801b98af9469865fc5ead771650a5d6240
2020-02-09build: Skip i686 build by default in guix and gitianMarcoFalke
2020-02-05Merge #16392: build: macOS toolchain updateWladimir J. van der Laan
7e2104433cd0905ccf94632511b3ca0ce5b0463b build: use macOS 10.14 SDK (fanquake) ca5055a5aa07aba81a87cf12f6f0526a63c423b5 depends: native_cctools 921, ld64 409.12, libtapi 1000.10.8 (fanquake) 1de8c067c74cd171144c8a900a8a20efe3072c43 depends: clang 6.0.1 (fanquake) Pull request description: TLDR: This updates our macOS toolchain to use a newer version of Clang, cctools (including new [dependency on libtapi](https://github.com/tpoechtrager/cctools-port/tree/master#dependencies)), LD64 and the macOS SDK. I've been testing depends builds (`HOST=x86_64-apple-darwin16`) inside a Debian Buster [Docker container](https://github.com/fanquake/core-review/blob/master/docker/debian.dockerfile), and running the resultant `bitcoind` and `bitcoin-qt` binaries on a macOS `10.14.4` system. The `.dmg` generated by a `make deploy` also mounts correctly on the same macOS system. #### Clang Upgraded from `3.7.1` to [`6.0.1`](https://releases.llvm.org/6.0.1/docs/ReleaseNotes.html) #### cctools * cctools `877.8` -> [`921`](https://opensource.apple.com/tarballs/cctools/) * LD64 `253.9` -> [`409.12`](https://opensource.apple.com/source/ld64/) * TAPI [`1000.10.8`](https://opensource.apple.com/tarballs/tapi/) See [tpoechtrager/cctools-port](https://github.com/tpoechtrager/cctools-port/) and [tpoechtrager/apple-libtapi](https://github.com/tpoechtrager/apple-libtapi/). #### macOS SDK Upgraded from building against the macOS `10.11` SDK to the macOS `10.14` SDK. #### TODO - [x] Make the `10.14` SDK available to Travis. Fixes: #16052 Closes: #14797 ACKs for top commit: Sjors: re-ACK 7e2104433cd0905ccf94632511b3ca0ce5b0463b (rebased from 248526e) dongcarl: ACK 7e21044 Tree-SHA512: fd36a33dbfb98c144240f8c69b77343e3f5bc18d8cf7d40fff61f51ad48925ec1872e6daba34c4045b18b4c2c84c22c744ebf4cba11061a0305eed13975ceefe
2020-02-05Merge #17336: scripts: search for first block file for linearize-data with ↵fanquake
some block files pruned 317fb96de9c6257972f1213b4ef2c3fe87dde99f Add search for first blk file with pruned node (Rjected) Pull request description: <!-- *** Please remove the following help text before submitting: *** Pull requests without a rationale and clear improvement may be closed immediately. --> <!-- Please provide clear motivation for your patch and explain how it improves Bitcoin Core user experience or Bitcoin Core developer experience significantly: * Any test improvements or new tests that improve coverage are always welcome. * All other changes should have accompanying unit tests (see `src/test/`) or functional tests (see `test/`). Contributors should note which tests cover modified code. If no tests exist for a region of modified code, new tests should accompany the change. * Bug fixes are most welcome when they come with steps to reproduce or an explanation of the potential issue as well as reasoning for the way the bug was fixed. * Features are welcome, but might be rejected due to design or scope issues. If a feature is based on a lot of dependencies, contributors should first consider building the system outside of Bitcoin Core, if possible. * Refactoring changes are only accepted if they are required for a feature or bug fix or otherwise improve developer experience significantly. For example, most "code style" refactoring changes require a thorough explanation why they are useful, what downsides they have and why they *significantly* improve developer experience or avoid serious programming bugs. Note that code style is often a subjective matter. Unless they are explicitly mentioned to be preferred in the [developer notes](/doc/developer-notes.md), stylistic code changes are usually rejected. --> When bitcoind is running in pruned mode, producing a hashlist with `./linearize-hashes.py linearize.cfg > hashlist.txt` and then executing `linearize-data.py linearize.cfg` will produce: ``` Read 313001 hashes Input file /home/dan/.bitcoin/blocks/blk00000.dat Premature end of block data ``` This happens because `linearize-data` starts by attempting to process `blk00000.dat` regardless of whether or not `blk00000.dat` actually exists - this may not be the case if working with a pruned node. This PR adds a function which finds the first block file that does exist, and calls that function when the `BlockDataCopier` is initialized. This is a refactor of #16431. <!-- Bitcoin Core has a thorough review process and even the most trivial change needs to pass a lot of eyes and requires non-zero or even substantial time effort to review. There is a huge lack of active reviewers on the project, so patches often sit for a long time. --> ACKs for top commit: darosior: ACK 317fb96de9c6257972f1213b4ef2c3fe87dde99f laanwj: Code review ACK 317fb96de9c6257972f1213b4ef2c3fe87dde99f theStack: Code review ACK https://github.com/bitcoin/bitcoin/pull/17336/commits/317fb96de9c6257972f1213b4ef2c3fe87dde99f Tree-SHA512: fc8014282df6cfe7b267e64db8ce7d82b86b758c302fbfea4a3c39b62d93512f5c2e31a0de4e9c5ec18fc0268c917f011257d37b45afaef6033eec90e4aa585f
2020-02-03build: use macOS 10.14 SDKfanquake
Co-Authored-By: Carl Dong <accounts@carldong.me>
2020-01-28test: Add crc32c exception to various linters and generation scriptsWladimir J. van der Laan
2020-01-28Merge #18010: test: rename test suite name "tx_validationcache_tests" to ↵fanquake
match filename b3c4d9bac6910f6c28f6008c5ca7064a315fd2a5 test: rename test suite name "tx_validationcache_tests" to match filename (Sebastian Falbesoner) Pull request description: Quoting `src/test/README.md`, '`Adding test cases`': > "The file naming convention is `<source_filename>_tests.cpp` > and such files should wrap their tests in a test suite > called `<source_filename>_tests`." Currently the unit test source file `txvalidationcache_tests.cpp` contains a unit test suite with the name `tx_validationcache_tests`, which is fixed by this PR. The following shell script shows that this is the only mismatch and for all other unit test source files the test suite names are correct: ``` #!/bin/bash shopt -s globstar for test_full_filename in **/*_tests.cpp; do test_name_file=`basename $test_full_filename .cpp` test_name_suite=`sed -n "s/^.*TEST_SUITE(\(.*_tests\).*$/\1/p" $test_full_filename` if [ $test_name_file != $test_name_suite ]; then echo "TestFilename: $test_name_file != TestSuitname: $test_name_suite" fi done ``` ACKs for top commit: practicalswift: ACK b3c4d9bac6910f6c28f6008c5ca7064a315fd2a5 -- expected naming is better than unexpected naming :) kristapsk: ACK b3c4d9bac6910f6c28f6008c5ca7064a315fd2a5 Tree-SHA512: 29d409b1eb22057ee2cc407508e2580d2bc03f412401df11b8ecf77be5ada6bda8f7d2cb5338c5e079490fa12242c1fd6230a09e47252c1b0d9fe535a828ca4c
2020-01-28Merge #17933: guix: Pin Guix using `guix time-machine`fanquake
88c83636d5a56bd9551577139786bdd3e74852c2 guix: Update documentation for time-machine (Carl Dong) e6050884fdabfa6e51e6afce2041d91e60a5adec guix: Pin Guix using `guix time-machine` (Carl Dong) Pull request description: An alternative to #16519, pinning our version of Guix and eliminating a `guix pull` and changing the default Guix profile of builders. I think this method might be superior, as it: - Eliminates the possibility of future changes to the `guix environment` command line interface breaking our builds - Eliminates the need to set up a separate channel repo It is a more general pinning solution than #16519. ----- The reason why I didn't originally propose this is because `guix time-machine` is a recent addition to Guix, only available since `f675f8dec73d02e319e607559ed2316c299ae8c7` ACKs for top commit: fanquake: ACK 88c83636d5a56bd9551577139786bdd3e74852c2 Tree-SHA512: 85e03b0987ffa86da73e02801e1cd8b7622698d70c4ba4e60561611be1e9717d661c2811a59b3e137b1b8eef2d0ba37c313867d035ebc89c3bd06a23a078064a
2020-01-27test: rename test suite name "tx_validationcache_tests" to match filenameSebastian Falbesoner
Quoting src/test/README.md, 'Adding test cases': "The file naming convention is `<source_filename>_tests.cpp` and such files should wrap their tests in a test suite called `<source_filename>_tests`." Currently the unit test source file txvalidationcache_tests.cpp contains a unit test suite with the name tx_validationcache_tests, which is fixed by this commit. The following shell script shows that this is the only mismatch and for all other unit test source files the test suite names are correct: #!/bin/bash shopt -s globstar for test_full_filename in **/*_tests.cpp; do test_name_file=`basename $test_full_filename .cpp` test_name_suite=`sed -n "s/^.*TEST_SUITE(\(.*_tests\).*$/\1/p" $test_full_filename` if [ $test_name_file != $test_name_suite ]; then echo "TestFilename: $test_name_file != TestSuitname: $test_name_suite" fi done
2020-01-27guix: Update documentation for time-machineCarl Dong
Wait a minute, doc. Are you telling me you built a time machine... Out of a functional package manager?
2020-01-27guix: Pin Guix using `guix time-machine`Carl Dong
2020-01-25Merge #17770: test: bump test timeouts so that functional tests run in valgrindMarcoFalke
2d23082cbe4641175d752a5969f67cdadf1afcea bump test timeouts so that functional tests run in valgrind (Micky Yun Chan) Pull request description: ci/tests: Bump timeouts so all functional tests run on travis in valgrind #17763 Top commit has no ACKs. Tree-SHA512: 5a8c6e2ea02b715facfcb58c761577be15ae58c45a61654beb98c2c2653361196c2eec521bcae4a9a1bab8e409d6807de771ef4c46d3d05996ae47a22d499d54
2020-01-25bump test timeouts so that functional tests run in valgrindMicky Yun Chan
2020-01-22Merge #17863: scripts: Add MACHO dylib checks to symbol-check.pyWladimir J. van der Laan
c491368d8cfddf3a5b6d574f10ed67492fcecbed scripts: add MACHO dylib checking to symbol-check.py (fanquake) 76bf97213f4b153dd3ccf1314088a73c4804601d scripts: fix check-symbols & check-security argument passing (fanquake) Pull request description: Based on #17857. This adds dynamic library checks for MACHO executables to symbol-check.py. The script has been modified to function more like `security-check.py`. The error output is now also slightly different. i.e: ```bash # Linux x86 bitcoin-cli: symbol operator new[](unsigned long) from unsupported version GLIBCXX_3.4 bitcoin-cli: export of symbol vtable for std::basic_ios<char, std::char_traits<char> > not allowed bitcoin-cli: NEEDED library libstdc++.so.6 is not allowed bitcoin-cli: failed IMPORTED_SYMBOLS EXPORTED_SYMBOLS LIBRARY_DEPENDENCIES # RISCV (skips exported symbols checks) bitcoin-tx: symbol operator new[](unsigned long) from unsupported version GLIBCXX_3.4 bitcoin-tx: NEEDED library libstdc++.so.6 is not allowed bitcoin-tx: failed IMPORTED_SYMBOLS LIBRARY_DEPENDENCIES # macOS Checking macOS dynamic libraries... libboost_filesystem.dylib is not in ALLOWED_LIBRARIES! bitcoind: failed DYNAMIC_LIBRARIES ``` Compared to `v0.19.0.1` the macOS allowed dylibs has been slimmed down somewhat: ```diff src/qt/bitcoin-qt: /usr/lib/libSystem.B.dylib -/System/Library/Frameworks/DiskArbitration.framework/Versions/A/DiskArbitration /System/Library/Frameworks/IOKit.framework/Versions/A/IOKit /System/Library/Frameworks/Foundation.framework/Versions/C/Foundation /System/Library/Frameworks/CoreServices.framework/Versions/A/CoreServices /System/Library/Frameworks/AppKit.framework/Versions/C/AppKit /System/Library/Frameworks/ApplicationServices.framework/Versions/A/ApplicationServices /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation -/System/Library/Frameworks/Security.framework/Versions/A/Security -/System/Library/Frameworks/SystemConfiguration.framework/Versions/A/SystemConfiguration /System/Library/Frameworks/CoreGraphics.framework/Versions/A/CoreGraphics -/System/Library/Frameworks/OpenGL.framework/Versions/A/OpenGL -/System/Library/Frameworks/AGL.framework/Versions/A/AGL /System/Library/Frameworks/Carbon.framework/Versions/A/Carbon /usr/lib/libc++.1.dylib -/System/Library/Frameworks/CFNetwork.framework/Versions/A/CFNetwork /System/Library/Frameworks/CoreText.framework/Versions/A/CoreText /System/Library/Frameworks/ImageIO.framework/Versions/A/ImageIO /usr/lib/libobjc.A.dylib ``` ACKs for top commit: laanwj: ACK c491368d8cfddf3a5b6d574f10ed67492fcecbed Tree-SHA512: f8624e4964e80b3e0d34e8d3cc33f3107938f3ef7a01c07828f09b902b5ea31a53c50f9be03576e1896ed832cf2c399e03a7943a4f537a1e1c705f3804aed979
2020-01-22Merge #17483: build: Set gitian arch back to amd64MarcoFalke
fae75306bac4c82dd07a1b85ce5dfb020e052fe8 scripted-diff: Set gitian arch back to amd64 (MarcoFalke) Pull request description: This was required to allow gitian builds on non-amd64 architecture, however, it seems to break the current builds (with lxc), see https://github.com/bitcoin/bitcoin/pull/17409#issuecomment-554099626 Also, the gititan builds wouldn't be deterministic across arches anyway, see #17468 So instead of wasting more time on this, revert the change and hope that guix allows to compile on non-amd64 architectures. Top commit has no ACKs. Tree-SHA512: 801e9a30ae1b0882ef45d5eb3a3cf80f3ace3b99db046069dbd95b6162119e977e3cf3134287d1ac5d09483906206acc71e1ac34d6b74dbc533d46aaf73f5cc2
2020-01-20Merge #17823: scripts: Read suspicious hosts from a file instead of hardcodingWladimir J. van der Laan
e1c582cbaa4c094d204da34c3b1fdd0d4c557519 contrib: makeseeds: Read suspicious hosts from a file instead of hardcoding (Sanjay K) Pull request description: referring to: https://github.com/bitcoin/bitcoin/issues/17020 good first issue: reading SUSPICIOUS_HOSTS from a file. I haven't changed the base hosts that were included in the original source, just made it readable from a file. ACKs for top commit: practicalswift: ACK e1c582cbaa4c094d204da34c3b1fdd0d4c557519 -- diff looks correct Tree-SHA512: 18684abc1c02cf52d63f6f6ecd98df01a9574a7c470524c37e152296504e2e3ffbabd6f3208214b62031512aeb809a6d37446af82c9f480ff14ce4c42c98e7c2
2020-01-16Merge #17691: doc: Add missed copyright headersMarcoFalke
fac86ac7b3ceac2f884412c7a9f4bd5bab5e3916 scripted-diff: Add missed copyright headers (Hennadii Stepanov) 6fde9d5e47fc9a1042b3fb68031eab5bf55e508d script: Update EXLUDE list in copyright_header.py (Hennadii Stepanov) 1998152f15fd2b0e83f5068c375a34feaf73db8c script: Add empty line after C++ copyright (Hennadii Stepanov) 071f2fc204f542c5a287ca8835115a2ee0bf2f50 script: Add ability to insert copyright to *.sh (Hennadii Stepanov) Pull request description: This PR improves `contrib/devtools/copyright_header.py` script and adds copyright headers to the files in `src` and `test` directories with two exceptions: - [`src/reverse_iterator.h`](https://github.com/bitcoin/bitcoin/blob/master/src/reverse_iterator.h) (added to exceptions) - [`src/test/fuzz/FuzzedDataProvider.h`](https://github.com/bitcoin/bitcoin/blob/master/src/test/fuzz/FuzzedDataProvider.h) (added to exceptions) On master 5622d8f3156a293e61d0964c33d4b21d8c9fd5e0: ``` $ ./contrib/devtools/copyright_header.py report . | grep zero 25 with zero copyrights ``` With this PR: ``` $ ./contrib/devtools/copyright_header.py report . | grep zero 2 with zero copyrights ``` ~I am uncertain about our copyright policy with `build_msvc` and `contrib` directories content, so they are out of scope of this PR.~ ACKs for top commit: MarcoFalke: ACK fac86ac7b3ceac2f884412c7a9f4bd5bab5e3916 Tree-SHA512: d7832c4a7a1a3b7806119775b40ec35d7982f49ff0e6199b8cee4c0e0a36e68d51728b6ee9924b1c161df4bc6105bd93391b79d42914357fa522f499cb113fa8
2020-01-15scripted-diff: Bump copyright of files changed in 2020MarcoFalke
-BEGIN VERIFY SCRIPT- ./contrib/devtools/copyright_header.py update ./ -END VERIFY SCRIPT-
2020-01-06contrib: makeseeds: Read suspicious hosts from a file instead of hardcodingSanjay K
2020-01-05Merge #17393: doc: Added regtest config for linearize scriptfanquake
582e66b6e75d58033987a7b0474226cfdd724ce0 doc: Added regtest config for linearize script (Gr0kchain) Pull request description: Updated the example-linearize.cfg file to include support for the regtest chain network config which is used by the ./linearize-data.py Problem: Without the regtest magic, genesis hash and path config, the `linearize-data.py` script cannot generate a bootstrap.dat file. Example: ./linearize-data.py ./linearize.cfg Read 102 hashes Genesis block not found in hashlist Solution: Added netmagic, genesis and input example parameters to file. Resolution 1. Starting bitcoind in regtest mode 2. bitcoin-cli generatetoaddress 101 $(bitcoin-cli getnewaddress) 3. ./linearize-hashes.py ./linearize.cfg > ./hashlist.txt 4. ./linearize-data.py ./linearize.cfg ``` $ ./linearize-data.py ./linearize.cfg Read 102 hashes Input file /Users/gr0kchain/.bitcoin/regtest/blocks/blk00000.dat Output file /Users/gr0kchain/Downloads/bootstrap.dat Done (102 blocks written) ``` ACKs for top commit: fanquake: ACK 582e66b6e75d58033987a7b0474226cfdd724ce0 Tree-SHA512: 699e92e740e68e2e5190ba37538efbbe3e4d4e725ebd6af704a0cf5517683b691754f7ea097bf840845d2b53b793c63258d406e9bd37922db810cf58bed053c3
2020-01-04scripted-diff: Add missed copyright headersHennadii Stepanov
-BEGIN VERIFY SCRIPT- s() { contrib/devtools/copyright_header.py insert "$1"; } s build_msvc/bitcoin_config.h s build_msvc/msvc-autogen.py s build_msvc/testconsensus/testconsensus.cpp s contrib/devtools/circular-dependencies.py s contrib/devtools/gen-manpages.sh s contrib/filter-lcov.py s contrib/gitian-build.py s contrib/install_db4.sh s src/crypto/sha256_avx2.cpp s src/crypto/sha256_sse41.cpp s src/fs.cpp s src/qt/test/addressbooktests.cpp s src/qt/test/addressbooktests.h s src/qt/test/util.cpp s src/qt/test/util.h s src/qt/test/wallettests.cpp s src/qt/test/wallettests.h s src/test/blockchain_tests.cpp s test/functional/combine_logs.py s test/lint/lint-locale-dependence.sh sed -i '1G' test/lint/lint-shebang.sh s test/lint/lint-shebang.sh -END VERIFY SCRIPT-
2020-01-04scripts: add MACHO dylib checking to symbol-check.pyfanquake
2020-01-03script: Update EXLUDE list in copyright_header.pyHennadii Stepanov
2020-01-03script: Add empty line after C++ copyrightHennadii Stepanov
2020-01-03script: Add ability to insert copyright to *.shHennadii Stepanov
2020-01-02scripts: add MACHO NOUNDEFS check to security-check.pyfanquake
2020-01-02scripts: add MACHO PIE check to security-check.pyfanquake
2019-12-30scripted-diff: Bump copyright of files changed in 2019MarcoFalke
-BEGIN VERIFY SCRIPT- ./contrib/devtools/copyright_header.py update ./ -END VERIFY SCRIPT-
generated by cgit v1.2.3 (git 2.26.2) at 2024-07-19 09:38:00 +0000