aboutsummaryrefslogtreecommitdiff
path: root/contrib
AgeCommit message (Collapse)Author
2020-05-06Merge #18741: guix: Make source tarball using git-archivefanquake
bfe1ba2f5b36056e0c41edf8206b93d3d83098df rel-builds: Specify core.abbrev for git-rev-parse (Carl Dong) 27e63e01cce368d67092de8f0c736927d6f6aa69 build: Accomodate makensis v2.x (Carl Dong) 1f2c39a30e0f82046c7aecddfda3eb99cb536816 guix: Remove logical cores requirement (Carl Dong) a4f6ffa71e335d4b2a6bf525b7f416968f9cd9f7 lint: Also enable source statements for non-gitian (Carl Dong) d256f91cb1b0d6ff5170106b99b0266cbe51f5a2 rel-builds: Directly deploy win installer to OUTDIR (Carl Dong) fa791da02f9684e3fd554b687fb692ae6a23d65a nsis: Specify OutFile path only once (Carl Dong) 14701604d0904bc5bbf1c67de08f8ee6d3215523 guix: Expose GIT_COMMON_DIR in container as readonly (Carl Dong) f5a6ac4f48b18f93050d77bcb23f9cf45ec34647 guix: Make source tarball using git-archive (Carl Dong) 395c1137f630dc495ffb2752a23bc1dfd470ee53 gitian: Limit sourced script to just assignments (Carl Dong) Pull request description: Based on: #18556 Related: https://github.com/bitcoin/bitcoin/pull/17595#discussion_r399728721 ACKs for top commit: fanquake: ACK bfe1ba2f5b36056e0c41edf8206b93d3d83098df - I agree with Carl, and am going to merge this. I'd like for Linux Guix builds to be working again, and we can rebase #18818. Tree-SHA512: c87ada7e3de17ca0b692a91029b86573442ded5780fc081c214773f6b374a0cdbeaf6f6898c36669c2e247ee32aa7f82defb1180f8decac52c65f0c140f18674
2020-05-05Merge #18885: contrib: Move optimize-pngs.py script to the maintainer repoMarcoFalke
fa13090d2048c9a1e475eb25de756763f8adddb8 contrib: Remove optimize-pngs.py script, which lives in the maintainer repo (MarcoFalke) Pull request description: Moved to https://github.com/bitcoin-core/bitcoin-maintainer-tools/blob/master/optimize-pngs.py Bitcoin Core should focus on the full node implementation, not on scripts to compress png images. This script is only used when new PNG files are added to the repo. This happens about once every two years. So fetching the script from the other repo should not be a burden, but removing it from this repo is going to cut down on the meta files we need to maintain in the main repo. ACKs for top commit: practicalswift: ACK fa13090d2048c9a1e475eb25de756763f8adddb8 -- `+0 lines, -82 lines` :) promag: ACK fa13090d2048c9a1e475eb25de756763f8adddb8. hebasto: ACK fa13090d2048c9a1e475eb25de756763f8adddb8, verified that script is already [moved](https://github.com/bitcoin-core/bitcoin-maintainer-tools/pull/56). Tree-SHA512: 37d111adae769bcddc6ae88041032d5a2b8b228fec67f555c8333c38de3992f5138b30bea868d7d6d6b7f966a47133e5853134373b149ab23cba3b8b560ecb31
2020-05-05contrib: Remove optimize-pngs.py script, which lives in the maintainer repoMarcoFalke
https://github.com/bitcoin-core/bitcoin-maintainer-tools/blob/master/optimize-pngs.py
2020-05-05valgrind : remove duplicate BCLog::Logger suppressionfanquake
This was added in #17770, but is identical to the supression above.
2020-05-05valgrind: remove rest_blockhash_by_height suppressionfanquake
This should no-longer be necessary after #18785.
2020-05-04rel-builds: Specify core.abbrev for git-rev-parseCarl Dong
Chose 12 because the kernel uses it: https://public-inbox.org/git/CA+55aFy0_pwtFOYS1Tmnxipw9ZkRNCQHmoYyegO00pjMiZQfbg@mail.gmail.com/raw And also because it's a nice number.
2020-05-01guix: Remove logical cores requirementCarl Dong
Thanks MarcoFalke for pushing this to its limits and testing :-)
2020-05-01rel-builds: Directly deploy win installer to OUTDIRCarl Dong
2020-04-28guix: Expose GIT_COMMON_DIR in container as readonlyCarl Dong
When using worktrees or submodules, you'll see a `.git' plain text file at the root of your working tree instead of the usual `.git' directory. This plain text file will point to the real GIT_DIR, under the GIT_COMMON_DIR. From experimentation, the full GIT_COMMON_DIR is required to exist for operations such as git-archive(1), so we expose it as readonly inside the container.
2020-04-28guix: Make source tarball using git-archiveCarl Dong
2020-04-28gitian: Limit sourced script to just assignmentsCarl Dong
Previously, the sourced script would create the source tarball. Now, it only assigns variables and the source-ing script has more flexibility in determining what to do with these variables. See later commit showing how this flexibility is useful in our Guix builds.
2020-04-28Merge #18556: build: Drop make dist in gitian buildsfanquake
2aa48edec0101f8a77a2189244fc62722ff7a123 refactor: Drop unused ${WRAP_DIR}/${HOST} directory (Hennadii Stepanov) 1362be044724bb49d785ca2e296a3b43343c1690 build: Drop make dist in gitian builds (Hennadii Stepanov) Pull request description: After the merge of #18331, the packaged source tarball is created by `git archive`, but the binaries are built from another one which is made by `make dist`. With this PR the only source tarball, created by `git archive`, is used both for binaries building and for packaging to users. Close #16588. Close #18547. As a good side-effect, #18349 becomes redundant. **Change in behavior** The following variables https://github.com/bitcoin/bitcoin/blob/1b151e3ffce7c1a2ee46bf280cc1d96775d1f91e/configure.ac#L2-L6 are no longer used for naming of directories and tarballs. Instead of them the gitian descriptors use a git tag (if available) or a commit hash. --- Also a small refactor commit picked from #18404. ACKs for top commit: dongcarl: ACK 2aa48edec0101f8a77a2189244fc62722ff7a123 MarcoFalke: ACK 2aa48edec0101f8a77a2189244fc62722ff7a123 fanquake: ACK 2aa48edec0101f8a77a2189244fc62722ff7a123 - I've had a quick look over this, and don't want to block merging if this actually gets as closer to finally having this all sorted out. Obviously we've still got #18741, and after speaking to Carl this morning, there will likely be even more changes after that (not Guix specific). Tree-SHA512: d3b16f87e48d1790a3264940c28acd5d881bfd10f3ce94fb0c8a6af76d8039289d01e0cd4972adac49ae24362857251f6c1e5e09e3e9fbf636c10708b4015a7c
2020-04-28Merge #18629: scripts: add PE .reloc section check to security-check.pyfanquake
3e38023af724a76972d39cbccfb0bba4c54a0323 scripts: add PE .reloc section check to security-check.py (fanquake) Pull request description: The `ld` in binutils has historically had a few issues with PE binaries, there's a good summary in this [thread](https://sourceware.org/bugzilla/show_bug.cgi?id=19011). One issue in particular was `ld` stripping the `.reloc` section out of PE binaries, even though it's required for functioning ASLR. This was [reported by a Tor developer in 2014](https://sourceware.org/bugzilla/show_bug.cgi?id=17321) and they have been patching their [own binutils](https://gitweb.torproject.org/builders/tor-browser-build.git/tree/projects/binutils) ever since. However their patch only made it into binutils at the [start of this year](https://sourceware.org/git/gitweb.cgi?p=binutils-gdb.git;a=commit;h=dc9bd8c92af67947db44b3cb428c050259b15cd0). It adds an `--enable-reloc-section` flag, which is turned on by default if you are using `--dynamic-base`. In the mean time this issue has also been worked around by other projects, such as FFmpeg, see [this commit](https://github.com/TheRyuu/FFmpeg/commit/91b668acd6decec0a6f8d20bf56e2644f96adcb9). I have checked our recent supported Windows release binaries, and they do contain a `.reloc` section. From what I understand, we are using all the right compile/linker flags, including `-pie` & `-fPIE`, and have never run into the crashing/entrypoint issues that other projects might have seen. One other thing worth noting here, it how Debian/Ubuntu patch the binutils that they distribute, because that's what we end up using in our gitian builds. In the binutils-mingw-w64 in Bionic (18.04), which we currently use in gitian, PE hardening options/security flags are enabled by default. See the [changelog](https://changelogs.ubuntu.com/changelogs/pool/universe/b/binutils-mingw-w64/binutils-mingw-w64_8ubuntu1/changelog) and the [relevant commit](https://salsa.debian.org/mingw-w64-team/binutils-mingw-w64/-/commit/452b3013b8280cbe35eaeb166a43621b88d5f8b7). However in Focal (20.04), this has now been reversed. PE hardening options are no-longer the default. See the [changelog](https://changelogs.ubuntu.com/changelogs/pool/universe/b/binutils-mingw-w64/binutils-mingw-w64_8.8/changelog) and [relevant commit](https://salsa.debian.org/mingw-w64-team/binutils-mingw-w64/-/commit/7bd8b2fbc242a8c2fc2217f29fd61f94d3babf6f), which cites same .reloc issue mentioned here. Given that we explicitly specify/opt-in to everything that we want to use, the defaults aren't necessarily an issue for us. However I think it highlights the importance of continuing to be explicit about what we want, and not falling-back or relying on upstream. This was also prompted by the possibility of us doing link time garbage collection, see #18579 & #18605. It seemed some sanity checks would be worthwhile in-case the linker goes haywire while garbage collecting. I think Guix is going to bring great benefits when dealing with these kinds of issues. Carl you might have something to say in that regard. ACKs for top commit: dongcarl: ACK 3e38023af724a76972d39cbccfb0bba4c54a0323 Tree-SHA512: af14d63bdb334bde548dd7de3e0946556b7e2598d817b56eb4e75b3f56c705c26aa85dd9783134c4b6a7aeb7cb4de567eed996e94d533d31511f57ed332287da
2020-04-24Merge #18589: Fix naming of macOS SDK and clarify versionfanquake
eb37275a6f972c81caef010b4ee9c5dc88edc759 Fix naming of macOS SDK and clarify version (Andrew Chow) Pull request description: Fixes the `MacOSX10.14.sdk.tar.gz` creation command to have `MacOSX.sdk` be correctly named as `MacOSX10.14.sdk` and for the resulting file to be placed in the current directory. Gitian requires that `tar.gz` contains a folder named `MacOSX10.14.sdk` and the command did not do this originally. Having the file be placed in the current directory is a convenience so builders don't have to go find it. Also clarifies which version of Xcode to download and where it can be downloaded. ACKs for top commit: fanquake: ACK eb37275a6f972c81caef010b4ee9c5dc88edc759 - tested the macOS and Linux SDK extraction. Also noticed something seemingly broken with Apple `tar`, but will open an issue to follow up. Sjors: ACK eb37275 for the macOS instruction Tree-SHA512: d691e14711cf195999291dd6fb7ffe552c86f8b30d2b1a77e88b4db6050dd817ba128b047cf36d29b0bb0d4183e709b7c03aa27f31b64e562ea8cd948434ca55
2020-04-23Fix naming of macOS SDK and clarify versionAndrew Chow
2020-04-23scripts: add PE .reloc section check to security-check.pyfanquake
2020-04-21scripts: add MACHO LAZY_BINDINGS test to test-security-check.pyfanquake
I didn't add the relevant test in #18295.
2020-04-21scripts: add MACHO Canary check to security-check.pyfanquake
2020-04-17Merge #18673: scripted-diff: Sort test includesMarcoFalke
fa4632c41714dfaa699bacc6a947d72668a4deef test: Move boost/stdlib includes last (MarcoFalke) fa488f131fd4f5bab0d01376c5a5013306f1abcd scripted-diff: Bump copyright headers (MarcoFalke) fac5c373006a9e4bcbb56843bb85f1aca4d87599 scripted-diff: Sort test includes (MarcoFalke) Pull request description: When writing tests, often includes need to be added or removed. Currently the list of includes is not sorted, so developers that write tests and have `clang-format` installed will either have an unrelated change (sorting) included in their commit or they will have to manually undo the sort. This pull preempts both issues by just sorting all includes in one commit. Please be aware that this is **NOT** a change to policy to enforce clang-format or any other developer guideline or process. Developers are free to use whatever tool they want, see also #18651. Edit: Also includes a commit to bump the copyright headers, so that the touched files don't need to be touched again for that. ACKs for top commit: practicalswift: ACK fa4632c41714dfaa699bacc6a947d72668a4deef jonatack: ACK fa4632c41714dfaa, light review and sanity checks with gcc build and clang fuzz build Tree-SHA512: 130a8d073a379ba556b1e64104d37c46b671425c0aef0ed725fd60156a95e8dc83fb6f0b5330b2f8152cf5daaf3983b4aca5e75812598f2626c39fd12b88b180
2020-04-17refactor: Drop unused ${WRAP_DIR}/${HOST} directoryHennadii Stepanov
This commit removes the directory that is no longer used since #16667.
2020-04-17build: Drop make dist in gitian buildsHennadii Stepanov
2020-04-16Merge #18598: gitian: Add missing automake package to gitian-win-signer.ymlWladimir J. van der Laan
e44aeefaaed8d698d1b9004b66f85384397b1a75 gitian: Add missing automake package to gitian-win-signer.yml (Andrew Chow) Pull request description: automake is needed to build osslsigncode otherwise autogen.sh fails with the docker virtualization method. ACKs for top commit: hebasto: ACK e44aeefaaed8d698d1b9004b66f85384397b1a75, for `osslsigncode-1.7.1` we did not run `autogen.sh` in the past. fanquake: ACK e44aeefaaed8d698d1b9004b66f85384397b1a75 jonatack: ACK e44aeef Tree-SHA512: a0e615c1b099ee1c469ce41f886f2ece6746234a5a800743a4e8be671e4114fd30e1c35bc0ddcb75778409564129d0fde7ac4e3d70b0f7691f97f729f34c8e0c
2020-04-16scripted-diff: Bump copyright headersMarcoFalke
-BEGIN VERIFY SCRIPT- ./contrib/devtools/copyright_header.py update ./ -END VERIFY SCRIPT-
2020-04-15Merge #18619: gitian: add jonatack gpg key fingerprintfanquake
905e2e85baca388ee1a34c6e2f559b7dd815c253 gitian: add jonatack gpg key fingerprint (Jon Atack) Pull request description: per request https://github.com/bitcoin-core/gitian.sigs/pull/1221#issuecomment-612778063 ACKs for top commit: laanwj: ACK 905e2e85baca388ee1a34c6e2f559b7dd815c253 fanquake: ACK 905e2e85baca388ee1a34c6e2f559b7dd815c253 Tree-SHA512: bddd734f13c53859280db2fa94b47cbb2a8b3f17ed6a6fdda2bf04f7e201e310ae24930e8f4be2f8b65a949659a9d3369704ed70031da9653a66a513fe597b67
2020-04-15Merge #18624: Added my fingerprint Stephan Oeste (Emzy)fanquake
c47adf8df435831c26ca25813fb7272176bd4eb7 Added my fingerprint Stephan Oeste (Emzy) (Stephan Oeste) Pull request description: By request from laanwj added my PGP fingerprint. See: https://github.com/bitcoin-core/gitian.sigs/pull/1220#issuecomment-612778442 ACKs for top commit: Sjors: ACK c47adf8. Fingerprint matches Twitter profile: https://twitter.com/emzy (haven't verified it in any other way) fanquake: ACK c47adf8df435831c26ca25813fb7272176bd4eb7 Tree-SHA512: 3e39ae88f507a12f11fb2d5c779eba79ee2daeddecd0dc3f1fddfa29ce963d0e9af3fa5a10357157812597c10205a6beae31cc70af9471a782da23d8753b7cbd
2020-04-13Added my fingerprint Stephan Oeste (Emzy)Stephan Oeste
By request from added my PGP fingerprint. See: https://github.com/bitcoin-core/gitian.sigs/pull/1220#issuecomment-612778442
2020-04-13gitian: add jonatack gpg key fingerprintJon Atack
2020-04-12build: add linker optimization flags to guixfanquake
Any -O argument will enable optimizations in GNU ld. We can use -O2 here, as this matches our compile flags. Note that this would also enable additional optimizations if using the lld or gold linkers, when compared to -O0.
2020-04-12build: add linker optimization flags to gitian descriptorsfanquake
Any -O argument will enable optimizations in GNU ld. We can use -O2 here, as this matches our compile flags. Note that this would also enable additional optimizations if using the lld or gold linkers, when compared to -O0.
2020-04-12Merge #17595: guix: Enable building for `x86_64-w64-mingw32` targetfanquake
a35e3235891d35daa167116cc70340140e883f06 guix: Appease travis. (Carl Dong) 0b66d22da5f53640e22f05adf880782c613e6d0f guix: Use gcc-9 for mingw-w64 instead of 8 (Carl Dong) ba0b99bdd613ba7f17c6247ece3001e1b44759a3 guix: Don't set MINGW_HAS_SECURE_API CFLAG in depends (Carl Dong) 93439a71eda49fb69f1e82966a23a946733aa6fa guix: Bump to upstream commit with mingw-w64 changes (Carl Dong) 35a96792dda9e78165b1598aeac7b2ab759e7be5 guix: Check mingw symbols, improve SSP fix docs (Carl Dong) 449d8fe25bbe25daacfc67aa89ca32b0a3254c5a guix: Expand on INT trap message (Carl Dong) 3f1f03c67a8e9edf487f08d272adb18b0a3942c8 guix: Spelling fixes (Carl Dong) ff821dd2a1c600488d11e7d9a20e9179ecc9144b guix: Reinstate make-ssp-fixed-gcc (Carl Dong) 360a9e0ad50a36ec79a1a160dbed3966689fd41c guix: Bump time-machine for mingw-w64 patches (Carl Dong) 93e41b7e3b54c17fd1b4c61ee95fc0dc2827e954 guix: Use gcc-8 for mingw-w64 instead of 7 (Carl Dong) ef4f7e4c45c60a69406134122f091c77c6ef740f guix: Set the well-known timezone env var (Carl Dong) acf4b3b3b5accf60a19441a0298ef27001b78e72 guix: Make x86_64-w64-mingw32 builds reproducible (Carl Dong) c4cce00eac691625b78b92f7dba0b7f57def19e5 guix: Remove dead links from README. (Carl Dong) df953a4c9a6143f45864757b706c88b6fa70545a guix: Appease shellcheck. (Carl Dong) 91897c95e191d293eb27d8af15cbeafc5b8f3895 guix: Improve guix-build.sh documentation (Carl Dong) 570d769c6c59b9f6d1a2b95b2ed60432cb33b3ba guix: Build support for Windows (Carl Dong) Pull request description: ~~Based on: https://github.com/bitcoin/bitcoin/pull/16519~~ Based on: #17933 (Time Machines are... shall we say... superior :grin:) This PR allows us to perform Guix builds for the `x86_64-w64-mingw32` target. We do this _without_ splitting up the build script like we do in Gitian by using this newfangled alien technology called `case` statements. (This is WIP and might be changed to `if` statements soon) ACKs for top commit: fanquake: ACK a35e3235891d35daa167116cc70340140e883f06 2/3 Tree-SHA512: c471951c23eb2cda919a71285d8b8f2580cb20f09d5db17b53e13dbd8813e01b3e7a83ea848e4913fd0f2bc12c6c133c5f76b54e65c0d89fed4dfd2e0be19875
2020-04-11gitian: Add missing automake package to gitian-win-signer.ymlAndrew Chow
automake is needed to build osslsigncode otherwise autogen.sh fails.
2020-04-10build: Bump gitian descriptors to 0.21Wladimir J. van der Laan
Per the release process.
2020-04-10Merge #18295: scripts: add MACHO lazy bindings check to security-check.pyfanquake
5ca90f8b598978437340bb8467f527b9edfb2bbf scripts: add MACHO lazy bindings check to security-check.py (fanquake) Pull request description: This is a slightly belated follow up to #17686 and some discussion with Cory. It's not entirely clear if we should make this change due to the way the macOS dynamic loader appears to work. However I'm opening this for some discussion. Also related to #17768. #### Issue: [`LD64`](https://opensource.apple.com/source/ld64/) doesn't set the [MH_BINDATLOAD](https://opensource.apple.com/source/xnu/xnu-6153.11.26/EXTERNAL_HEADERS/mach-o/loader.h.auto.html) bit in the header of MACHO executables, when building with `-bind_at_load`. This is in contradiction to the [documentation](https://opensource.apple.com/source/ld64/ld64-450.3/doc/man/man1/ld.1.auto.html): ```bash -bind_at_load Sets a bit in the mach header of the resulting binary which tells dyld to bind all symbols when the binary is loaded, rather than lazily. ``` The [`ld` in Apples cctools](https://opensource.apple.com/source/cctools/cctools-927.0.2/ld/layout.c.auto.html) does set the bit, however the [cctools-port](https://github.com/tpoechtrager/cctools-port/) that we use for release builds, bundles `LD64`. However; even if the linker hasn't set that bit, the dynamic loader ([`dyld`](https://opensource.apple.com/source/dyld/)) doesn't seem to ever check for it, and from what I understand, it looks at a different part of the header when determining whether to lazily load symbols. Note that our release binaries are currently working as expected, and no lazy loading occurs. #### Example: Using a small program, we can observe the behaviour of the dynamic loader. Conducted using: ```bash clang++ --version Apple clang version 11.0.0 (clang-1100.0.33.17) Target: x86_64-apple-darwin18.7.0 ld -v @(#)PROGRAM:ld PROJECT:ld64-530 BUILD 18:57:17 Dec 13 2019 LTO support using: LLVM version 11.0.0, (clang-1100.0.33.17) (static support for 23, runtime is 23) TAPI support using: Apple TAPI version 11.0.0 (tapi-1100.0.11) ``` ```cpp #include <iostream> int main() { std::cout << "Hello World!\n"; return 0; } ``` Compile and check the MACHO header: ```bash clang++ test.cpp -o test otool -vh test ... Mach header magic cputype cpusubtype caps filetype ncmds sizeofcmds flags MH_MAGIC_64 X86_64 ALL LIB64 EXECUTE 16 1424 NOUNDEFS DYLDLINK TWOLEVEL WEAK_DEFINES BINDS_TO_WEAK PIE # Run and dump dynamic loader bindings: DYLD_PRINT_BINDINGS=1 DYLD_PRINT_TO_FILE=no_bind.txt ./test Hello World! ``` Recompile with `-bind_at_load`. Note still no `BINDATLOAD` flag: ```bash clang++ test.cpp -o test -Wl,-bind_at_load otool -vh test Mach header magic cputype cpusubtype caps filetype ncmds sizeofcmds flags MH_MAGIC_64 X86_64 ALL LIB64 EXECUTE 16 1424 NOUNDEFS DYLDLINK TWOLEVEL WEAK_DEFINES BINDS_TO_WEAK PIE ... DYLD_PRINT_BINDINGS=1 DYLD_PRINT_TO_FILE=bind.txt ./test Hello World! ``` If we diff the outputs, you can see that `dyld` doesn't perform any lazy bindings when the binary is compiled with `-bind_at_load`, even if the `BINDATLOAD` flag is not set: ```diff @@ -1,11 +1,27 @@ +dyld: bind: test:0x103EDF030 = libc++.1.dylib:__ZNKSt3__16locale9use_facetERNS0_2idE, *0x103EDF030 = 0x7FFF70C9FA58 +dyld: bind: test:0x103EDF038 = libc++.1.dylib:__ZNKSt3__18ios_base6getlocEv, *0x103EDF038 = 0x7FFF70CA12C2 +dyld: bind: test:0x103EDF068 = libc++.1.dylib:__ZNSt3__113basic_ostreamIcNS_11char_traitsIcEEE6sentryC1ERS3_, *0x103EDF068 = 0x7FFF70CA12B6 +dyld: bind: test:0x103EDF070 = libc++.1.dylib:__ZNSt3__113basic_ostreamIcNS_11char_traitsIcEEE6sentryD1Ev, *0x103EDF070 = 0x7FFF70CA1528 +dyld: bind: test:0x103EDF080 = libc++.1.dylib:__ZNSt3__16localeD1Ev, *0x103EDF080 = 0x7FFF70C9FAE6 <trim> -dyld: lazy bind: test:0x10D4AC0C8 = libsystem_platform.dylib:_strlen, *0x10D4AC0C8 = 0x7FFF73C5C6E0 -dyld: lazy bind: test:0x10D4AC068 = libc++.1.dylib:__ZNSt3__113basic_ostreamIcNS_11char_traitsIcEEE6sentryC1ERS3_, *0x10D4AC068 = 0x7FFF70CA12B6 -dyld: lazy bind: test:0x10D4AC038 = libc++.1.dylib:__ZNKSt3__18ios_base6getlocEv, *0x10D4AC038 = 0x7FFF70CA12C2 -dyld: lazy bind: test:0x10D4AC030 = libc++.1.dylib:__ZNKSt3__16locale9use_facetERNS0_2idE, *0x10D4AC030 = 0x7FFF70C9FA58 -dyld: lazy bind: test:0x10D4AC080 = libc++.1.dylib:__ZNSt3__16localeD1Ev, *0x10D4AC080 = 0x7FFF70C9FAE6 -dyld: lazy bind: test:0x10D4AC070 = libc++.1.dylib:__ZNSt3__113basic_ostreamIcNS_11char_traitsIcEEE6sentryD1Ev, *0x10D4AC070 = 0x7FFF70CA1528 ``` Note: `dyld` also has a `DYLD_BIND_AT_LAUNCH=1` environment variable, that when set, will force any lazy bindings to be non-lazy: ```bash dyld: forced lazy bind: test:0x10BEC8068 = libc++.1.dylib:__ZNSt3__113basic_ostream ``` #### Thoughts: After looking at the dyld source, I can't find any checks for `MH_BINDATLOAD`. You can see the flags it does check for, such as MH_PIE or MH_BIND_TO_WEAK [here](https://opensource.apple.com/source/dyld/dyld-732.8/src/ImageLoaderMachO.cpp.auto.html). It seems that the lazy binding of any symbols depends on whether or not [lazy_bind_size](https://opensource.apple.com/source/xnu/xnu-6153.11.26/EXTERNAL_HEADERS/mach-o/loader.h.auto.html) from the `LC_DYLD_INFO_ONLY` load command is > 0. Which was mentioned in [#17686](https://github.com/bitcoin/bitcoin/pull/17686#issue-350216254). #### Changes: This PR is one of [Corys commits](https://github.com/theuni/bitcoin/commit/7b6ba26178d2754568a1308d3d44e038e9ebf450), that I've rebased and modified to make build. I've also included an addition to the `security-check.py` script to check for the flag. However, given the above, I'm not entirely sure this patch is the correct approach. If the linker no-longer inserts it, and the dynamic loader doesn't look for it, there might be little benefit to setting it. Or, maybe this is an oversight from Apple and needs some upstream discussion. Looking for some thoughts / Concept ACK/NACK. One alternate approach we could take is to drop the patch and modify security-check.py to look for `lazy_bind_size` == 0 in the `LC_DYLD_INFO_ONLY` load command, using `otool -l`. ACKs for top commit: theuni: ACK 5ca90f8b598978437340bb8467f527b9edfb2bbf Tree-SHA512: 444022ea9d19ed74dd06dc2ab3857a9c23fbc2f6475364e8552d761b712d684b3a7114d144f20de42328d1a99403b48667ba96885121392affb2e05b834b6e1c
2020-04-07guix: Appease travis.Carl Dong
2020-04-07guix: Use gcc-9 for mingw-w64 instead of 8Carl Dong
The libtool unsorted 'find' determinism issue seemed to have been solved in gcc-9's git: d41cd173e23ebea7c758644d6ad6e0fde1c2e3a6 or SVN: r262451 Furthermore, it seems that Ubuntu Focal 20.04 LTS is going to ship with gcc 9 and mingw-w64 7, which will match what we have now. ----- A note on this: Careful observers will see that previously I stated that all released versions of gcc were bootstrapped with a libtool 2.2.7a, meaning that they all had the unsorted 'find' determinism issue first resolved in libtool 2.2.7b. However, I was mistaken, gcc's ltmain.sh CLAIMS it was generated by libtool 2.2.7a, but it was in fact edited manually. It seems that gcc maintains their own versions of ltmain.sh and libtool.m4, and only sometimes backports patches from upstream. Quite confusing.
2020-04-07guix: Don't set MINGW_HAS_SECURE_API CFLAG in dependsCarl Dong
This is no longer needed after 3bef7c22 in the mingw-w64 git repository, which is first included in mingw-w64 v7.0.0. As of the previous bump to our Guix time machine, we now use mingw-w64 v7.0.0.
2020-04-07guix: Bump to upstream commit with mingw-w64 changesCarl Dong
Most of the mingw-w64 toolchain changes have now been upstreamed, we can point to a commit that exists upstream. NOTE: I'm not changing the URL yet until we see that Guix upstream will accept all my patches for macOS. ----- The Guix tree that's referred to by this commit contains the following changes relevant to our mingw-w64 build: b066c25026 Adds a PACKAGES-WITH-*PATCHES procedure which we can use in the future to apply patches to packages if those patches are not considered appropriate to upstream Guix 4719b71572 Adds mingw-w64 (the libc itself) reproducibility patches, taken from debian. 79825bee07 + 401d28e433 + c1c50cb5b0 Add mingw-w64 specific binutils patches, taken from debian. Specifically, the "Make DLL import libraries reproducible" patch made libbitcoinconsensus.dll.a build reproducibly. The followup commits were hotfixes for my mistakes. 0f864175dc Bumps mingw-w64 to v7.0.0. This is the first release that enables secure APIs by default (which we need), and gains _FORTIFY_SOURCE support. This will also be what Ubuntu Focal 20.04 LTS releases with. cdf00cf75d Bumps NSIS to v3.05. This is the first release that includes a fix for a reproducibility bug found by some of the electrum developers. See details here: https://sourceforge.net/p/nsis/bugs/1230/
2020-04-06Merge #18506: net: Hardcoded seeds update for 0.20Wladimir J. van der Laan
0eeb0468e7debb1dbe38242769207d22ed52c1df net: Hardcoded seeds update for 0.20 (Wladimir J. van der Laan) Pull request description: Update hardcoded seeds from http://bitcoin.sipa.be/seeds.txt.gz, according to release process. Output from makeseeds.py: ``` IPv4 IPv6 Onion Pass 1364173 244127 2454 Initial 1364173 244127 2454 Skip entries with invalid address 1129552 213117 2345 After removing duplicates 1129548 213117 2345 Skip entries from suspicious hosts 338216 191944 2249 Enforce minimal number of blocks 336851 188993 2189 Require service bit 1 6998 1520 150 Require minimum uptime 5682 1290 89 Require a known and recent user agent 5622 1279 89 Filter out hosts with multiple bitcoin ports 512 146 89 Look up ASNs and limit results per ASN and per net ``` Top commit has no ACKs. Tree-SHA512: ce1c2cda18dd5bd22586a5283a0877f3bd890437cc29dc1d85452ba4a4d28032f591c8b37f3329e8e649556cf83750b6949a068fad76d1773853d93014609da0
2020-04-04scripts: add MACHO lazy bindings check to security-check.pyfanquake
2020-04-03net: Hardcoded seeds update for 0.20Wladimir J. van der Laan
Update hardcoded seeds from seeds_emzy.txt seeds_lukejr.txt seeds_sipa.txt seeds_sjors.txt, according to release process. Output from makeseeds.py: ``` IPv4 IPv6 Onion Pass 1364173 244127 2454 Initial 1364173 244127 2454 Skip entries with invalid address 1129552 213117 2345 After removing duplicates 1129548 213117 2345 Skip entries from suspicious hosts 338216 191944 2249 Enforce minimal number of blocks 336851 188993 2189 Require service bit 1 6998 1520 150 Require minimum uptime 5682 1290 89 Require a known and recent user agent 5622 1279 89 Filter out hosts with multiple bitcoin ports 512 146 89 Look up ASNs and limit results per ASN and per net ```
2020-04-03Merge #18426: scripts: previous_release: improve behaviour on failed downloadfanquake
332f373a9dece71717f75eb06e6a1fc957f2952b [scripts] previous_release: improve failed download error message (Sebastian Falbesoner) Pull request description: Currently, if the earlier release build/fetch script `previous_release.sh` is invoked with the option `-b` (intending to fetch a binary package from `https://bitcoin.org`) and the download fails, the user sees the following confusing output: ``` $ contrib/devtools/previous_release.sh -r -b v0.9.5 [...] gzip: stdin: not in gzip format tar: Child returned status 1 tar: Error is not recoverable: exiting now ``` This implies that the download worked, but the archive is corrupted, when in reality the HTML document containing the delivery fail reason (most likely 404 Not Found) is saved and tried to get unpacked. In contrast to wget, curl is a bit stubborn and needs explicit instructions to react to server errors via the flag `-f` (outputs error message and returns error code, ideal for scripts): https://curl.haxx.se/docs/manpage.html#-f On the PR branch, the output on failed download looks now the following: ``` $ contrib/devtools/previous_release.sh -r -b v0.9.5 [...] curl: (22) The requested URL returned error: 404 Not Found Download failed. ``` ACKs for top commit: fanquake: ACK 332f373a9dece71717f75eb06e6a1fc957f2952b Tree-SHA512: 046c931ad9e78aeb2d13faa4866d46122ed325aa142483547c2b04032d03223ed2411783b00106fcab0cd91b2f78691531ac526ed7bb3ed7547b6e2adbfb2e93
2020-04-02guix: Check mingw symbols, improve SSP fix docsCarl Dong
2020-04-02guix: Expand on INT trap messageCarl Dong
2020-04-02guix: Spelling fixesCarl Dong
2020-04-02guix: Reinstate make-ssp-fixed-gccCarl Dong
Unfortunately, gcc is still not smart enough to detect whether or not mingw-w64 provides ssp, so let's put it back just for mingw-w64.
2020-04-02guix: Bump time-machine for mingw-w64 patchesCarl Dong
This bump will includes a couple of commits which improve the reproducibility of the mingw-w64 toolchain. Most of which came from debian. They will be upstreamed as upstream Guix release timeline allows.
2020-04-02guix: Use gcc-8 for mingw-w64 instead of 7Carl Dong
We're using mingw-w64 6.0.0, which is paired with gcc-8 in most distros.
2020-04-02guix: Set the well-known timezone env varCarl Dong
2020-04-02guix: Make x86_64-w64-mingw32 builds reproducibleCarl Dong
- Add "--no-insert-timestamp" LDFLAG for x86_64-w64-mingw32 builds "The option --no-insert-timestamp can be used to insert a zero value for the timestamp, this ensuring that binaries produced from identical sources will compare identically." - ld(1) - Set "SetDateSave off" in NSIS script From https://nsis.sourceforge.io/Docs/Chapter4.html#flags "This command sets the file date/time saving flag which is used by the File command to determine whether or not to save the last write date and time of the file, so that it can be restored on installation. Valid flags are 'on' and 'off'. 'on' is the default." - Add commented out NSIS options for reproducibility debugging in NSIS script - Make ZIPs deterministic by reseting file modification times to SOURCE_DATE_EPOCH using touch(1) (Reference: https://reproducible-builds.org/docs/archives/)
2020-04-02guix: Remove dead links from README.Carl Dong