aboutsummaryrefslogtreecommitdiff
path: root/contrib/guix
AgeCommit message (Collapse)Author
2021-07-20Merge bitcoin/bitcoin#22465: guix: Pin kernel-header version, time-machine ↵fanquake
to upstream 1.3.0 commit e6a94d44469f90f4dc88a07a5a8587730811c705 guix: Bump to version-1.3.0 from upstream (Carl Dong) 90fd13b954a364963f58e6cd12962c6f1986f79b guix: Pin kernel header version (Carl Dong) Pull request description: ``` - Use 4.19 for riscv64 (earliest LTS release w/ riscv64 support) - Use 4.9 for all others (second-oldest LTS release, released in combination with glibc glibc 2.24 in Debian stretch) ``` ``` The chosen commit is the HEAD of Guix's version-1.3.0 branch as of July 15th, 2021. Also fix visual indenting. ``` ----- This + the documentation PR should make our Guix system ready for release! ACKs for top commit: MarcoFalke: review ACK e6a94d44469f90f4dc88a07a5a8587730811c705 to change to vanilla guix. Did not review the kernel change. laanwj: ACK e6a94d44469f90f4dc88a07a5a8587730811c705 fanquake: ACK e6a94d44469f90f4dc88a07a5a8587730811c705 Tree-SHA512: a175e4ddb3ee786a39f5e800ce336932ad2f6797a3a28400a6f723875d0f19833fd36cedc41b3580e4604110517211bd9f557be36adf7265fd8e591c434ae032
2021-07-20Merge bitcoin/bitcoin#21711: guix: Add full installation and usage documentationfanquake
fac4814106c796b8786dd90053513cc35142dfe5 doc/release-process: Add torrent creation details (Carl Dong) 5d24cc3d82dad6812f8370c3ccc7c2b5a6c12c11 guix/INSTALL: Guix installs init scripts in libdir (Carl Dong) 5da2ee49d5b44de803b671aedbdd14e5c1d71ea9 guix/INSTALL: Add coreutils/inotify-dir-recreate troubleshooting (Carl Dong) 318c60700b7bbb7ec09a29bf037e7c2787646be6 guix: Adapt release-process.md to new Guix process (Carl Dong) fcab35b2292f9221eaba521740e8b3b2511a8b78 guix-attest: Produce and sign normalized documents (Carl Dong) c2541fd0ca99481a5a792a8f2772925d64fb0491 guix: Overhaul README (Carl Dong) 46ce6ce3782dfbd8f9d26dc2ba0f284755e75f2d tree-wide: Rename gitian-keys to builder-keys (Carl Dong) fc4f8449f34e32b0b9ac9d218d6c3264b02467ba guix: Update various check_tools lists (Carl Dong) 263220a85c1df218431fafbda07c8b23ccc4ce4d guix: Check for a sane services database (Carl Dong) Pull request description: Based on: #21462 Keeping the README in one file so that it's easy to search through. Will add more jumping links later so navigation is easier. Current TODOs: - [x] Shell installer option: prompt user to re-login for `/etc/profile.d` entry to be picked up - [x] Binary tarball option: prompt user to create `/etc/profile.d` entry and re-login - [x] Fanquake docker option: complete section - [x] Arch Linux AUR option: prompt to start `guix-daemon-latest` unit after finishing "optional setup" section - [x] Building from source option: Insert dependency tree diagram that I made - [x] Building from source option: redo sectioning, kind of a mess right now - [x] Optional setup: make clear which parts are only needed if building from source - [x] Workaround 1 for GnuTLS: perhaps mention how to remove Guix build farm's key - [x] Overall (after everything): Make the links work. Note to self: wherever possible, tell user how to check that something is true rather than branching by installation option. ACKs for top commit: fanquake: ACK fac4814106c796b8786dd90053513cc35142dfe5 - going to go ahead and merge this now. It's a lot of documentation, and could probably be nit-picked / improved further, however, that can continue over the next few weeks. I'm sure more (backportable) improvements / clarifications will be made while we progress through RCs towards a new release. Tree-SHA512: dc46c0ecdfc67c7c7743ca26e4a603eb3f54adbf81be2f4c1f4c20577ebb84b5250b9c9ec89c0e9860337ab1c7cff94d7963c603287267deecfe1cd987fa070a
2021-07-19guix/INSTALL: Guix installs init scripts in libdirCarl Dong
2021-07-19guix/INSTALL: Add coreutils/inotify-dir-recreate troubleshootingCarl Dong
2021-07-19guix-attest: Produce and sign normalized documentsCarl Dong
That way we can easily combine the document and detached signature to produce cleartext signature files for upload during the release process. See subsequent commits which modify doc/release-process.md for more details.
2021-07-19guix: Overhaul READMECarl Dong
- Added detailed Guix bootstrap/installation instructions
2021-07-18Merge bitcoin/bitcoin#22410: Avoid GCC 7.1 ABI change warning in guix buildfanquake
1edddf5de41b053049ce0b0bdbc39c2fbb743c40 Avoid GCC 7.1 ABI change warning in guix build (Pieter Wuille) Pull request description: The arm-linux-gnueabihf guix build output is littered with warnings like: ``` /gnu/store/7a96hdqdb2qi8a39f09n84xjy2hr23rs-gcc-cross-arm-linux-gnueabihf-8.4.0/include/c++/bits/stl_vector.h:1085:4: note: parameter passing for argument of type '__gnu_cxx::__normal_iterator<CRecipient*, std::vector<CRecipient> >' changed in GCC 7.1 ``` These are irrelevant for us. Disable them using `-Wno-psabi`. ACKs for top commit: laanwj: ACK 1edddf5de41b053049ce0b0bdbc39c2fbb743c40 hebasto: ACK 1edddf5de41b053049ce0b0bdbc39c2fbb743c40, after thorough reading related materials, I agree this change can be merged. As I mentioned above, I have been compiling my arm-32bit binaries with `-Wno-psabi` flag for two years, and no related flaws were observed. Tree-SHA512: 485c7500547ac5da567ad23847341c18ff832607f5a1002676404cc647e437cf3445b6894ecff5b52929ca52bea946c06bd90eace1997c895e56204e787065e4
2021-07-15guix: Bump to version-1.3.0 from upstreamCarl Dong
The chosen commit is the HEAD of Guix's version-1.3.0 branch as of July 15th, 2021. Also fix visual indenting.
2021-07-15guix: Pin kernel header versionCarl Dong
- Use 4.19 for riscv64 (earliest LTS release w/ riscv64 support) - Use 4.9 for all others (second-oldest LTS release, released in combination with glibc glibc 2.24 in Debian stretch)
2021-07-08guix: Update various check_tools listsCarl Dong
2021-07-08guix: Check for a sane services databaseCarl Dong
On bare systems, it is possible to be lacking a services database. Check for basic entries before attempting a build. See the error message in the diff for more context.
2021-07-07guix: Test security-check sanity before performing themCarl Dong
2021-07-07guix: Patch binutils to add security-related disable flagsCarl Dong
We use these flags in our test-security-check make target, but they are only available because debian patches them in. We can patch them in for our Guix builds so that we can check the sanity of our security/symbol checking suite before running them.
2021-07-06Avoid GCC 7.1 ABI change warning in guix buildPieter Wuille
2021-07-06guix: add additional documentation to patchesfanquake
2021-07-06guix: no-longer pass --enable-glibc-back-compat to Guixfanquake
Now that our Guix builds are performed on glibc 2.24 and 2.27 (RISCV), we no-longer need to pass the --enable-glibc-back-compat option. Replace it with --disable-threadlocal, to prevent the usage of symbols from glibc 2.18. None of the binaries produced required symbols later than 2.17, and 2.27 (RISCV).
2021-07-02guix: Also sort SHA256SUMS.partCarl Dong
2021-07-01guix: Build depends/qt with our platform definitionCarl Dong
Our 'bitcoin-linux-g++' definition better integrates with our depends system than the stock linux-g++-64 definition. This fixes a bug whereby Guix builds on x86_64 for x86_64 did not produce a QMinimalIntegrationPlugin and led to bitcoin-qt not being built.
2021-07-01guix: Rebase toolchain on glibc 2.24 (2.27 for riscv64)Carl Dong
Support for riscv64 in glibc landed in 2.27 so it's unavoidable that we use 2.27. Running a Bitcoin build with toolchains based on 2.24 for platforms other than riscv64 seem to produce binaries which do not have 2.17 symbols. So use 2.24 since it's more recent and maintained by Debian Stretch.
2021-06-18Merge bitcoin/bitcoin#21871: scripts: add checks for minimum required OS ↵fanquake
versions aa80b5759dfa613780a99801641519dd78bb3eca scripts: check macOS SDK version is set (fanquake) c972345bacd0cb01371b3f00941e81dce16278e1 scripts: check minimum required Windows version is set (fanquake) 29615aef52d7f1a29a87a29dfe4d39bf0e9867f3 scripts: check minimum required macOS vesion is set (fanquake) 8732f7b6c92f9dcf37f3ab618e9daab0c52fc781 scripts: LIEF 0.11.5 (fanquake) Pull request description: macOS: We use a compile flag ([-mmacosx-version-min=10.14](https://github.com/bitcoin/bitcoin/blob/master/depends/hosts/darwin.mk#L96)) to set the minimum required version of macOS needed to run our binaries. This adds a sanity check that the version is being set as expected. Clangs Darwin driver should infer the SDK version used during compilation, and forward that through to the linker. Add a check that this has been done, and the expected SDK version is set. Should help prevent issues like #21771 in future. Windows: We use linker flags ([-Wl,--major/minor-subsystem-version](https://github.com/bitcoin/bitcoin/blob/master/configure.ac#L683)) to set the minimum required version of Windows needed to run our binaries. This adds a sanity check that the version is being set as expected. Gitian builds: ```bash # macOS: 8b6fcd61d75001c37b2af3fceb5ae09f5d2fe85e97d361f684214bd91c27954a bitcoin-f015e1c2cac9-osx-unsigned.dmg 3c1e412bc7f5a7a5d0f78e2cd84b7096831414e1304c1307211aa3e135d89bbf bitcoin-f015e1c2cac9-osx-unsigned.tar.gz 50b7b2804e8481f63c69c78e3e8a71c0d811bf2db8895dd6d3edae9c46a738ae bitcoin-f015e1c2cac9-osx64.tar.gz fe6b5c0a550096b76b6727efee30e85b60163a41c83f21868c849fdd9876b675 src/bitcoin-f015e1c2cac9.tar.gz 8a20f21b20673dfc8c23e22b20ae0839bcaf65bf0e02f62381cdf5e7922936f0 bitcoin-core-osx-22-res.yml # Windows: b01fcdc2a5673387050d6c6c4f96f1d350976a121155fde3f76c2af309111f9d bitcoin-f015e1c2cac9-win-unsigned.tar.gz b95bdcbef638804030671d2332d58011f8c4ed4c1db87d6ffd211515c32c9d02 bitcoin-f015e1c2cac9-win64-debug.zip 350bf180252d24a3d40f05e22398fec7bb00e06d812204eb5a421100a8e10638 bitcoin-f015e1c2cac9-win64-setup-unsigned.exe 2730ddabe246d99913c9a779e97edcadb2d55309933d46f1dffd0d23ecf9aae5 bitcoin-f015e1c2cac9-win64.zip fe6b5c0a550096b76b6727efee30e85b60163a41c83f21868c849fdd9876b675 src/bitcoin-f015e1c2cac9.tar.gz aa60d7a753e8cb2d4323cfbbf4d964ad3645e74c918cccd66862888f8646d80f bitcoin-core-win-22-res.yml ``` ACKs for top commit: hebasto: ACK aa80b5759dfa613780a99801641519dd78bb3eca, tested by breaking tests: Tree-SHA512: 10150219910e8131715fbfe20edaa15778387616ef3bfe1a5152c7acd3958fe8f88c74961c3d3641074eb72824680c22764bb1dc01a19e92e946c2d4962a8d2c
2021-06-17Merge bitcoin/bitcoin#22182: guix: Overhaul how guix-{attest,verify} works ↵fanquake
and hierarchy e2c40a4ed5272d72fea997bd936fba28bb753226 guix-attest: Error out if SHA256SUMS is unexpected (Carl Dong) 4cc35daed557f38b080360a89036b2e97a6f78c2 Rewrite guix-{attest,verify} for new hier (Carl Dong) 28a9c9b83924f585b397f0f3b8e9e73780ac0ad6 Make SHA256SUMS fragment right after build (Carl Dong) Pull request description: Based on: #22075 Code reviewers: I recommend reading the new `guix-{attest,verify}` files instead of trying to read the diff The following changes resolve many usability improvements which were pointed out to me: 1. Some maintainers like to extract their "uncodesigned tarball" inside the `output/` directory, resulting in the older `guix-attest` mistakenly attesting to the extracted contents 2. Maintainers whose GPG keys reside on an external smartcard often need to physically interact with the smartcard as a way to approve the signing operation, having one signature per platform means a lot of fidgeting 3. Maintainers wishing to sign on a separate machine now has the option of transferring only a subtree of `output/`, namely `output/*/SHA256SUMS.part`, in order to perform a signature (you may need to specify an `$OUTDIR_BASE` env var) 4. An `all.SHA256SUMS` file should be usable as the base `SHA256SUMS` in bitcoin core torrents and on the release server. For those who sign on an separate machine than the one you do builds on, the following steps will work: 1. `env GUIX_SIGS_REPO=/home/achow101/guix.sigs SIGNER=achow101 NO_SIGN=1 ./contrib/guix/guix-attest` 2. Copy `/home/achow101/guix.sigs/<tag>/achow101` (which does not yet have signatures) to signing machine 3. Sign the `SHA256SUMS` files: ```bash for i in "<path-to-achow101>/*.SHA256SUMS"; do gpg --detach-sign --local-user "<your-key-here>" --armor --output "$i"{.asc,} done ``` 5. Upload `<path-to-achow101>` (now with signatures) to `guix.sigs` ----- After this change, output directories will now include a `SHA256SUMS.part` fragment, created immediately after a successful build: ``` output └── x86_64-w64-mingw32 ├── bitcoin-4e069f7589da-win64-debug.zip ├── bitcoin-4e069f7589da-win64-setup-unsigned.exe ├── bitcoin-4e069f7589da-win64.zip ├── bitcoin-4e069f7589da-win-unsigned.tar.gz └── SHA256SUMS.part ``` These `SHA256SUMS.part` fragments look something like: ``` 3ebd7262b1a0a5bb757fef1f70e7e14033c70f98c059bc4dbfee5d1992b25825 dist-archive/bitcoin-4e069f7589da.tar.gz def2e7d3de5ab3e3f955344e75151df4f33713f9101f5295bd13c9375bdf633b x86_64-w64-mingw32/bitcoin-4e069f7589da-win64-debug.zip 643049fe3ee4a4e83a1739607e67b11b7c9b1a66208a6f35a9ff634ba795500e x86_64-w64-mingw32/bitcoin-4e069f7589da-win64-setup-unsigned.exe a247a1ccec0ccc2e138c648284bd01f6a761f2d8d6d07d91b5b4a6670ec3f288 x86_64-w64-mingw32/bitcoin-4e069f7589da-win-unsigned.tar.gz fab76a836dcc592e39c04fd2396696633fb6eb56e39ecbf6c909bd173ed4280c x86_64-w64-mingw32/bitcoin-4e069f7589da-win64.zip ``` Meaning that they are valid `SHA256SUMS` files when `sha256sum --check`'d at the `guix-build-*/output` directory level When `guix-attest` is invoked, these `SHA256SUMS.part` files are combined and sorted (by `-k2`, `LC_ALL=C`) to create: 1. `noncodesigned.SHA256SUMS` for a manifest of all non-codesigned outputs, and 3. `all.SHA256SUMS` for a manifest of all outputs including non-codesigned outputs Then both files are signed, resulting in the following `guix.sigs` hierarchy: ``` 4e069f7589da/ └── dongcarl ├── all.SHA256SUMS ├── all.SHA256SUMS.asc ├── noncodesigned.SHA256SUMS └── noncodesigned.SHA256SUMS.asc ``` ACKs for top commit: achow101: ACK e2c40a4ed5272d72fea997bd936fba28bb753226 hebasto: ACK e2c40a4ed5272d72fea997bd936fba28bb753226, tested on Linux Mint 20.1 (x86_64) with and w/o `NO_SIGN=1`. Changes in `contrib/guix/libexec/codesign.sh` and `contrib/guix/guix-verify` are reviewed only. Tree-SHA512: 618aacefb0eb6595735a9ab6a98ea6598fce65f9ccf33fa1e7ef93bf140c0f6cfc16e34870c6aa3e4777dd3f004b92a82a994141879870141742df948ec59c1f
2021-06-14guix-attest: Error out if SHA256SUMS is unexpectedCarl Dong
2021-06-11Merge bitcoin/bitcoin#22190: Use latest signapple commitfanquake
683d197970a533690ca1bd4d06d021900e87cb8b Use latest signapple commit (Andrew Chow) Pull request description: Update gitian and guix to use the same latest signapple commit. Also changed guix to use the actual repo. The changes from the fork were incorporated upstream. ACKs for top commit: fanquake: ACK 683d197970a533690ca1bd4d06d021900e87cb8b - sanity checked that the updated package is built: Tree-SHA512: a4981f8bbe33e6c5654632bc9b9f6f2f1e675741a19ac7296205e370f1e64a747101ecb632e0cc82a0134e4c2e9ce47b3f7b4d8c8f75f0f06dd069c078303759
2021-06-10scripts: LIEF 0.11.5fanquake
2021-06-09Rewrite guix-{attest,verify} for new hierCarl Dong
2021-06-09Make SHA256SUMS fragment right after buildCarl Dong
2021-06-09Merge bitcoin/bitcoin#22075: guix: Misc leftover usability improvementsfanquake
108a6be92adc1e80839d90b552e72b8142140f6c guix: Check for disk space availability before building (Carl Dong) d7dec89091ee4a456ff64ad7ce675ae6813668f1 guix: Remove dest if OUTDIR mv fails (Carl Dong) Pull request description: There seems to be some corner cases that can be hit when guix scripts unexpectedly fail in the middle of operation, see: https://gnusha.org/bitcoin-builds/2021-05-24.log - Perform an early disk space check for `guix-build` - Overwrite existing output directory after a successful build (the existing one might be malformed), and cleanup output directory if the `mv` somehow fails ACKs for top commit: laanwj: Tested ACK 108a6be92adc1e80839d90b552e72b8142140f6c achow101: ACK 108a6be92adc1e80839d90b552e72b8142140f6c Tree-SHA512: cf6438317da40bf55714cd2d8cce859b3d435cc66cabefe8d4a53552d7880966acfe84ffe8fadf1c80e368ae6b037992258a6d409df85ffc6ce8bf780e98e2e5
2021-06-08Use latest signapple commitAndrew Chow
Update gitian and guix to use the same latest signapple commit
2021-06-03Merge bitcoin/bitcoin#21654: build, qt: Make Qt rcc output always deterministicfanquake
a58868d201cb6d263aa552815f7f86562c1ca9a5 build: Makes rcc output always deterministic (Hennadii Stepanov) Pull request description: The Qt Resource Compiler ([rcc](https://doc.qt.io/qt-5/rcc.html)) has a command-line option `--format-version` which has the [default value](https://code.qt.io/cgit/qt/qtbase.git/tree/src/tools/rcc/main.cpp?h=5.12.10#n172) 2. The only difference from `--format-version 1` is adding a [last modified timestamp](https://code.qt.io/cgit/qt/qtbase.git/tree/src/tools/rcc/rcc.cpp?h=5.12.10#n207) to the output file ([credits](https://github.com/bitcoin/bitcoin/pull/21654#issuecomment-819198228) to **fanquake**). That, in turn, forces us to use `QT_RCC_SOURCE_DATE_OVERRIDE=1` to get deterministic builds (#13732). This change makes rcc output always deterministic by using `--format-version 1` option that makes usage of the `QT_RCC_SOURCE_DATE_OVERRIDE` needless. --- Also it improves interaction with ccache: On master (f6c44e999b7d1d9a0de5d678ac8f1679aa271f65): ``` $ make && make clean && ccache --zero-stats && make && ccache --show-stats ... cache directory /home/hebasto/.ccache primary config /home/hebasto/.ccache/ccache.conf secondary config (readonly) /etc/ccache.conf stats updated Sun Apr 11 15:45:43 2021 stats zeroed Sun Apr 11 15:45:05 2021 cache hit (direct) 638 cache hit (preprocessed) 0 cache miss 1 cache hit rate 99.84 % called for link 10 cleanups performed 0 files in cache 20023 cache size 13.2 GB max cache size 15.0 GB ``` The missed file is always `qt/libbitcoinqt_a-qrc_bitcoin_locale.o`. With this PR: ``` $ make && make clean && ccache --zero-stats && make && ccache --show-stats ... cache directory /home/hebasto/.ccache primary config /home/hebasto/.ccache/ccache.conf secondary config (readonly) /etc/ccache.conf stats updated Sun Apr 11 15:28:46 2021 stats zeroed Sun Apr 11 15:28:21 2021 cache hit (direct) 639 cache hit (preprocessed) 0 cache miss 0 cache hit rate 100.00 % called for link 10 cleanups performed 0 files in cache 20012 cache size 13.2 GB max cache size 15.0 GB ``` ACKs for top commit: fanquake: ACK a58868d201cb6d263aa552815f7f86562c1ca9a5 Tree-SHA512: 52f4a3267f41883d13025c0de79b6da22e92d60c729e01b986935c6812bbfe7fadc40b742bd715bfdf09df94af6838d4fbbe8208c6123f366108e38c8e1121c5
2021-05-25guix: Check for disk space availability before buildingCarl Dong
2021-05-25guix: Remove dest if OUTDIR mv failsCarl Dong
2021-05-19guix: repro: Sort find output in libtool for gcc-8Carl Dong
Otherwise the resulting .a static libraries (e.g. libstdc++.a) will not be reproducible and end up making the Bitcoin binaries non-reproducible as well. See: https://reproducible-builds.org/docs/archives/#gnu-libtool
2021-05-13guix: Add codesigning functionalityCarl Dong
2021-05-13guix: Package codesigning toolsCarl Dong
2021-05-13guix: Reindent existing manifest.scmCarl Dong
2021-05-12Merge bitcoin/bitcoin#21462: guix: Add guix-{attest,verify} scriptsW. J. van der Laan
d420e5c1c015f58d07aca4d6a805086488f74d03 guix-attest: Avoid incomplete sigdirs with ERR traps (Carl Dong) feda2c8e3180cb983c35976d4440cea23a155b7f guix: Skip attesting to dist-archive (Carl Dong) d522d8006b891eccd7901faf391f9c041ddf8e38 guix: Attest to inputs in inputs.SHA256SUMS (Carl Dong) f9e2960c018103be756a7f8a506816b49d662514 guix: Construct $OUTDIR in ${DISTSRC}/output (Carl Dong) 022abc85fc7e711a900fed8e5071919a151c0a63 guix: Minor quoting fix in libexec/build.sh (Carl Dong) c83c4fa5b78aef33bba36b3a0d273422297bd630 guix-attest: Allow skipping GPG signing with NO_SIGN (Carl Dong) 0e1c2e448c25568f276e4f022128870c76ca216b guix-attest: Use ascii-armor signatures (Carl Dong) b5fd89c4c89136007429688601ce4fa497f5f09e guix-attest: Only use cross-platform flags for find+xargs (Carl Dong) 5926432ba68ba154df6c8eaa74adb18cc0123167 guix: Add guix-verify script (Carl Dong) 30daf76a97c57a5f74c8dad1da282dcc0ff8b3fb guix: Add guix-attest script (Carl Dong) Pull request description: Adds replacements for `gsign` and `gverify`. Personally I'm not a big fan of using the word "sign" as it's been used to refer to both codesigning and GPG signing. ACKs for top commit: laanwj: Code review and tested ACK d420e5c1c015f58d07aca4d6a805086488f74d03 Tree-SHA512: 93d82d201f4596eaea0e3825aa55b013dfb91790e6ccee79893833d37921513d7b4e735f0641103e1e2ea8308abe4cb6218b73160924708802f2e0e3f7f6caf1
2021-05-10build: Makes rcc output always deterministicHennadii Stepanov
The Qt Resource Compiler (rcc) has a command-line option `--format-version` which has the default value 2. The only difference from `--format-version 1` is adding a last modified timestamp to the output file. That, in turn, forces us to use `QT_RCC_SOURCE_DATE_OVERRIDE=1` to get deterministic builds. This change makes rcc output always deterministic by using `--format-version 1` option that makes usage of the `QT_RCC_SOURCE_DATE_OVERRIDE` needless. Also it improves interaction with ccache. Co-authored-by: fanquake <fanquake@gmail.com>
2021-05-05Merge bitcoin/bitcoin#21740: test: add new python linter to check file names ↵W. J. van der Laan
and permissions 46b025e00df40724175735eb5606ac73067cb3b8 test: add new python linter to check file names and permissions (windsok) 6f6bb3ebc7cb8e17a5dfc8ef55aa2d3f2dc6bdea test: fix file permissions on various scripts (windsok) Pull request description: Adds a new python linter test which tests for correct filenames and file permissions in the repository. Replaces the existing tests in the `test/lint/lint-filenames.sh` and `test/lint/lint-shebang.sh` linter tests, as well as adding some new and increased testing. This increased coverage is intended to catch issues such as in #21728 and https://github.com/bitcoin/bitcoin/pull/16807/files#r345547050 Summary of tests: * Checks every file in the repository against an allowed regexp to make sure only lowercase or uppercase alphanumerics (a-zA-Z0-9), underscores (_), hyphens (-), at (@) and dots (.) are used in repository filenames. * Checks only source files (*.cpp, *.h, *.py, *.sh) against a stricter allowed regexp to make sure only lowercase alphanumerics (a-z0-9), underscores (_), hyphens (-) and dots (.) are used in source code filenames. Additionally there is an exception regexp for directories or files which are excepted from matching this regexp (This should replicate the existing `test/lint/lint-filenames.sh` test) * Checks all files in the repository match an allowed executable or non-executable file permission octal. Additionally checks that for executable files, the file contains a shebang line. * Checks that for executable `.py` and `.sh` files, the shebang line used matches an allowable list of shebangs (This should replicate the existing `test/lint/lint-shebang.sh` test) * Checks every file that contains a shebang line to ensure it has an executable permission Additionally updates the permissions on various files to comply with the new tests. Fixes #21729 ACKs for top commit: practicalswift: cr re-ACK 46b025e00df40724175735eb5606ac73067cb3b8: patch still looks correct kiminuo: code review ACK 46b025e00df40724175735eb5606ac73067cb3b8 if `contrib/gitian-descriptors/assign_DISTNAME` permission change is deemed OK. laanwj: Code review ACK 46b025e00df40724175735eb5606ac73067cb3b8 Tree-SHA512: 1c8201a2cee0d9cbce15652b68cec9a6458a8b493fcd5392f98560aca0b1a12e668baab65a47100f116f626dadc3f591deb47f7368468c6a46c6c712c2533455
2021-05-05Merge bitcoin/bitcoin#21664: contrib: use LIEF for macOS and Windows symbol ↵W. J. van der Laan
& security checks 7fc5e865b93af59364e9c8bf75ec68b4decc7e5d test: install lief in CI (fanquake) 955140b3265d3bcb9504c61d73fbfdadfff8a2b2 contrib: consolidate PIE and NX security checks (fanquake) 2aa1631822b2fdbc6cf7a3dcd99adaf4d2745ed4 contrib: use LIEF in PE symbol checks (fanquake) e93ac26b8563576345c13e83c777dd39e7616b1e contrib: use LIEF in macOS symbol checks (fanquake) a632cbcee5ae982f50aba625713b7686aef29168 contrib: use f strings in symbol-check.py (fanquake) 0f5d77c8e4db691733edb455dd9e31dabe933b8d contrib: add PE PIE check to security checks (fanquake) 8e1f40dd9a5135dbdec2c25961fbd0729a42254c contrib: use LIEF for PE security checks (fanquake) a25b2e965c93fe2a46a2f8f1e7bdf5642d453511 contrib: use LIEF for macOS security checks (fanquake) 7e7eae7aa86ab95c44eed601f8c993285a256bbc contrib: use f strings in security-check.py (fanquake) 2e7a9f7ade0c7b31e762c0ddb9e0944a0d9c798e guix: install LIEF in Guix container (fanquake) 465967b5ef4b4f02e9d6783a94eca012d4ebcdab gitian: install LIEF in gitian container (fanquake) Pull request description: This PR is a proof of concept for using [LIEF](https://github.com/lief-project/LIEF) for the PE and MACHO symbol and security checks. It replaces our current approach of manually parsing the output of `objdump` & `otool`. If the consensus is that using LIEF is ok, then I also plan on replacing [pixie.py](https://github.com/bitcoin/bitcoin/blob/master/contrib/devtools/pixie.py), and using LIEF for all checks. LIEF for Linux is also currently blocked (on the next release, unless we want to build master) on one change for RISC-V that I [sent upstream](https://github.com/lief-project/LIEF/pull/562). LIEF is seemingly well maintained, and is the basis for a number of other tools. It also has some very nice documentation; i.e the [Python API for ELF](https://lief.quarkslab.com/doc/latest/api/python/elf.html). It also has many builtins we can take advantage of. i.e [`is_pie`](https://lief.quarkslab.com/doc/latest/api/python/macho.html#lief.MachO.Binary.is_pie), [`has_nx`](https://lief.quarkslab.com/doc/latest/api/python/macho.html#lief.MachO.Binary.has_nx) etc. This means we can [consolidate some of our checks](https://github.com/bitcoin/bitcoin/commit/9c5eeb54848f428109ec24dff55f189a5358e9bc). If/when end up using LIEF for lightning then we can consolidate further, and cleanup these scripts. i.e to not parse the binary inside the checks, but once at the start of the script. Guix builds: ```bash # find guix-build-$(git rev-parse --short=12 HEAD)/output/ -type f -print0 | env LC_ALL=C sort -z | xargs -r0 sha256sum 963a08638c46f9a3d75cd4b0c155d1ca091bbeba27167291adcd3dca03fd4c3d guix-build-f51237d94d98/output/aarch64-linux-gnu/bitcoin-f51237d94d98-aarch64-linux-gnu-debug.tar.gz a3ce927c46b103789a010c41a6ebfafe4548d90ee7d88f2a735c9183b775da5c guix-build-f51237d94d98/output/aarch64-linux-gnu/bitcoin-f51237d94d98-aarch64-linux-gnu.tar.gz 2503ac8901068805d5e7251fd5cfeb7c1f8ba3528bdfcf3aa1e0c40bfd5c1cbc guix-build-f51237d94d98/output/arm-linux-gnueabihf/bitcoin-f51237d94d98-arm-linux-gnueabihf-debug.tar.gz 5798697e58e1788df85aa9e2e4d33fef0456169fcbd2521f13b3b5806ac0d84d guix-build-f51237d94d98/output/arm-linux-gnueabihf/bitcoin-f51237d94d98-arm-linux-gnueabihf.tar.gz 4185adebc6a0abe7241a3cd409a6ab7be031c26f1c4245e30bb5f87eef0925d2 guix-build-f51237d94d98/output/dist-archive/bitcoin-f51237d94d98.tar.gz 9b4b8756c5c84295eb6b61b6b32a07a8d07723fb38aaa8f519b6133935061bda guix-build-f51237d94d98/output/powerpc64-linux-gnu/bitcoin-f51237d94d98-powerpc64-linux-gnu-debug.tar.gz cbd821aa464a9c16f7979dbec1a5e66939e777a567f55f7081499a8d528d42c5 guix-build-f51237d94d98/output/powerpc64-linux-gnu/bitcoin-f51237d94d98-powerpc64-linux-gnu.tar.gz abed530a82e97e3cf621c90a13c0881b0e39ccce2a6f42a3ff80de76e2abc5f7 guix-build-f51237d94d98/output/powerpc64le-linux-gnu/bitcoin-f51237d94d98-powerpc64le-linux-gnu-debug.tar.gz 8b6d2bdd8b58ff1f6072bf8693abe3ce773ff3a7d8d2b7218207e69945b9d31b guix-build-f51237d94d98/output/powerpc64le-linux-gnu/bitcoin-f51237d94d98-powerpc64le-linux-gnu.tar.gz d99cc705032d22ae819975992216899ed960ba25871a05c8789d00b80418511f guix-build-f51237d94d98/output/riscv64-linux-gnu/bitcoin-f51237d94d98-riscv64-linux-gnu-debug.tar.gz 5240ca4f4ef7c62088185224ac319ad9a4a9b40075df10af18d8a6355bca32fb guix-build-f51237d94d98/output/riscv64-linux-gnu/bitcoin-f51237d94d98-riscv64-linux-gnu.tar.gz adc16eaee4b51e8615ce8b3be9f6c018698237df4ad6e0886cf0d4ab6bc9e5c4 guix-build-f51237d94d98/output/x86_64-apple-darwin18/bitcoin-f51237d94d98-osx-unsigned.dmg b188af0572ee682d74cc82c7e6e464115205fc130a457cfe19d42ac9ddd267f8 guix-build-f51237d94d98/output/x86_64-apple-darwin18/bitcoin-f51237d94d98-osx-unsigned.tar.gz e764062fde144e6fb5d6dd776c10fc2daa8d775831f7e43247d17a6c6e060c97 guix-build-f51237d94d98/output/x86_64-apple-darwin18/bitcoin-f51237d94d98-osx64.tar.gz dab3d26ac94c669140f7329d14e57ef02b0fe92b8a8f9d96c32a416adea0da0f guix-build-f51237d94d98/output/x86_64-linux-gnu/bitcoin-f51237d94d98-x86_64-linux-gnu-debug.tar.gz ca59d4379fbe2b9a52deebeaf88508e0eda4215f28d319aff0781289dd159712 guix-build-f51237d94d98/output/x86_64-linux-gnu/bitcoin-f51237d94d98-x86_64-linux-gnu.tar.gz 52b7c35321a85c4f6c95bf0e687574454b71ede9bec1c9cf17f37c578c888a94 guix-build-f51237d94d98/output/x86_64-w64-mingw32/bitcoin-f51237d94d98-win-unsigned.tar.gz a543895a00f8ffb3ba50ca68396d52ad5a18dd8efe38730e0049dd70d283a092 guix-build-f51237d94d98/output/x86_64-w64-mingw32/bitcoin-f51237d94d98-win64-debug.zip aec050d03c65268a986148500f7341cceb8c5f85287e0e3cde8933ce4b4dee32 guix-build-f51237d94d98/output/x86_64-w64-mingw32/bitcoin-f51237d94d98-win64-setup-unsigned.exe 57ba33ed6ee8d3a885e342471359301473e83037d5442895beb686921a4c50e9 guix-build-f51237d94d98/output/x86_64-w64-mingw32/bitcoin-f51237d94d98-win64.zip ``` Gitian builds: ```bash # macOS: 2f066e852bdd30ac46e5ecdf7619d19d408035c318a3edf0f1893ec2e25efb69 bitcoin-41a1b3d1b130-osx-unsigned.dmg 8cf8ac4d21740f490262453c330b5f4a5c5b8139dfc1b322efefce3f3b93d1b2 bitcoin-41a1b3d1b130-osx-unsigned.tar.gz cf1b84efdd9d2588a1ce9513580fb56b38bfafe60e18f8adbeedf03521c6c2b2 bitcoin-41a1b3d1b130-osx64.tar.gz 14995244b0bb3e80e7b79975c9c70fdfb3ee3c04fda3efd5358ce1c4efa3a312 src/bitcoin-41a1b3d1b130.tar.gz 93881069d5e1dc385c08895a7b035a94eb010325afc2776c99b6aafa21096eb8 bitcoin-core-osx-22-res.yml # Windows: 4d56dd7713121684b7eaa448679c65df2fd0aa5319bf8d12fb6cfa9f0b005cf7 bitcoin-f51237d94d98-win-unsigned.tar.gz 4558f4173152b084bcba25aa1a53c605208a70fe20392141b63cefb476528c85 bitcoin-f51237d94d98-win64-debug.zip b63feaca010e86d514cfe38d716e3c8a8b8058e4f969b868aaaeb8a8a3d3dc81 bitcoin-f51237d94d98-win64-setup-unsigned.exe de7d8586cc91ba391fe911853a99d9fd15fc6f9a60f9b91a0447940173aac67a bitcoin-f51237d94d98-win64.zip 4185adebc6a0abe7241a3cd409a6ab7be031c26f1c4245e30bb5f87eef0925d2 src/bitcoin-f51237d94d98.tar.gz 45efaca35b5fad0a04dfd06e44f7c00b990aa91c7bf2faea57e020d3491a6cf0 bitcoin-core-win-22-res.yml # Linux: 055d646c5f8cf4708008374546176012ff758566a2645a3a01e1a33eab1002fe bitcoin-f51237d94d98-aarch64-linux-gnu-debug.tar.gz bfc8b0efc36b0474c88546b12d2723c04b4dc629ae311082025c7e0b8f0d1aa9 bitcoin-f51237d94d98-aarch64-linux-gnu.tar.gz 9dfaa5acfffadad8942b32996458013a155d12ed07be76601f232233627b5cb9 bitcoin-f51237d94d98-arm-linux-gnueabihf-debug.tar.gz 54eb57905ff8513b9f628707b61aa4659c362fb2f6d17e0ee240b4da3674907d bitcoin-f51237d94d98-arm-linux-gnueabihf.tar.gz ad98d876616eff578ad8cfd17dfbabe48ed14200823579687d66694bae3d2fe3 bitcoin-f51237d94d98-powerpc64-linux-gnu-debug.tar.gz fe1b421dd1cb6e04d5dc5d341459dc15fa6e15b80906e5d8e0405cf43495e0f7 bitcoin-f51237d94d98-powerpc64-linux-gnu.tar.gz 9001d95cc7d2722d9d7dd83d9da8e5adf575fddf91b615b76b9bcfece30ecf6f bitcoin-f51237d94d98-powerpc64le-linux-gnu-debug.tar.gz 9e0650ad2aba70c0fd1608a077e95f335dc1bb4a79eab9b0b56ac87427a4fd4f bitcoin-f51237d94d98-powerpc64le-linux-gnu.tar.gz fbfde0134944d3dbd32991455b0a8abdd334853ab8a4c1a1a4c060d9de071c50 bitcoin-f51237d94d98-riscv64-linux-gnu-debug.tar.gz 2fa2cfddce98c44c65305326fc623a7f065129208337503d813a08d51580cb8a bitcoin-f51237d94d98-riscv64-linux-gnu.tar.gz b2d6caeee0e3c350a43165c39876ebed8e588958007af0d06996e341c7060683 bitcoin-f51237d94d98-x86_64-linux-gnu-debug.tar.gz bfdb827e75d43d61462513c9a843620b93c9160d9d246cad13278baaa07f64ea bitcoin-f51237d94d98-x86_64-linux-gnu.tar.gz 4185adebc6a0abe7241a3cd409a6ab7be031c26f1c4245e30bb5f87eef0925d2 src/bitcoin-f51237d94d98.tar.gz 34820a093916fa35b0fd98806a50092f46b20271af7422f43e2a4223ef6f9bb7 bitcoin-core-linux-22-res.yml ``` ACKs for top commit: laanwj: re-ACK 7fc5e865b93af59364e9c8bf75ec68b4decc7e5d Tree-SHA512: 0c30838413448ecfcf55e6273f607fdb01cb1acafa1d2762afad59360fca7d8efa78ec55064f50cba56cb2c9e98741e13665cba8e9b4b8e5b62b8a53f9bf8990
2021-05-04guix: install LIEF in Guix containerfanquake
Co-authored-by: Carl Dong <contact@carldong.me>
2021-05-03guix-attest: Avoid incomplete sigdirs with ERR trapsCarl Dong
Sometimes GPG connects to the wrong agent... or you don't have your smartcard handy...
2021-05-03guix: Consistently use gcc-8 for $HOSTCarl Dong
2021-05-03guix: Skip attesting to dist-archiveCarl Dong
We already attest to the relevant dist-archive in inputs.SHA256SUMS, which is recorded at build-time. We use a SKIPATTEST.TAG file to indicate output directories which do not require attestation (much like the CACHEDIR.TAG specification). Generally, it's better to have build scripts declare properties of directories instead of introducing name-based special cases in attest scripts since build scripts have a more detailed context of what is going on.
2021-05-03guix: Attest to inputs in inputs.SHA256SUMSCarl Dong
At build/codesigning-time, hash build inputs and output the digest to ${OUTDIR}/inputs.SHA256SUMS, which gets included in the final SHA256SUMS constructed by guix-attest. Example final SHA256SUMS: ee832d2a35b7701bff581dea05a536118b118e3ad0a587a2855b6ee8cd6fba20 inputs/bitcoin-78199266af7b.tar.gz ca765e70a0c12866dd63c0be228b675278a26329e5f8f5b5c52fd09200fedf21 bitcoin-78199266af7b-powerpc64le-linux-gnu-debug.tar.gz dae95327d7f2c324e2728c4b73627be6cb2c0d2f2e5bea940d1d5e6463939327 bitcoin-78199266af7b-powerpc64le-linux-gnu.tar.gz
2021-05-03guix: Construct $OUTDIR in ${DISTSRC}/outputCarl Dong
While files are being output to $OUTDIR, it will be under ${DISTSRC}/output, and only when everything is done, will ${DISTSRC}/output be moved to the actual $OUTDIR. This makes it so that a Ctrl-C in the middle of a build is less likely to result in a partially-constructed $OUTDIR. In fact, if I understand correctly, if $OUTDIR and $DISTSRC reside on the same filesystem, the move (rename) is likely atomic. Also, since the "working $OUTDIR" is under ${DISTSRC}/output, it will be cleaned properly by the guix-clean script.
2021-05-03guix: Minor quoting fix in libexec/build.shCarl Dong
2021-05-03guix-attest: Allow skipping GPG signing with NO_SIGNCarl Dong
2021-05-03guix-attest: Use ascii-armor signaturesCarl Dong
2021-05-03guix-attest: Only use cross-platform flags for find+xargsCarl Dong
2021-05-03guix: Add guix-verify scriptCarl Dong
generated by cgit v1.2.3 (git 2.26.2) at 2024-07-25 01:29:19 +0000