aboutsummaryrefslogtreecommitdiff
path: root/contrib/guix/libexec
AgeCommit message (Collapse)Author
2021-06-09Make SHA256SUMS fragment right after buildCarl Dong
2021-06-09Merge bitcoin/bitcoin#22075: guix: Misc leftover usability improvementsfanquake
108a6be92adc1e80839d90b552e72b8142140f6c guix: Check for disk space availability before building (Carl Dong) d7dec89091ee4a456ff64ad7ce675ae6813668f1 guix: Remove dest if OUTDIR mv fails (Carl Dong) Pull request description: There seems to be some corner cases that can be hit when guix scripts unexpectedly fail in the middle of operation, see: https://gnusha.org/bitcoin-builds/2021-05-24.log - Perform an early disk space check for `guix-build` - Overwrite existing output directory after a successful build (the existing one might be malformed), and cleanup output directory if the `mv` somehow fails ACKs for top commit: laanwj: Tested ACK 108a6be92adc1e80839d90b552e72b8142140f6c achow101: ACK 108a6be92adc1e80839d90b552e72b8142140f6c Tree-SHA512: cf6438317da40bf55714cd2d8cce859b3d435cc66cabefe8d4a53552d7880966acfe84ffe8fadf1c80e368ae6b037992258a6d409df85ffc6ce8bf780e98e2e5
2021-06-03Merge bitcoin/bitcoin#21654: build, qt: Make Qt rcc output always deterministicfanquake
a58868d201cb6d263aa552815f7f86562c1ca9a5 build: Makes rcc output always deterministic (Hennadii Stepanov) Pull request description: The Qt Resource Compiler ([rcc](https://doc.qt.io/qt-5/rcc.html)) has a command-line option `--format-version` which has the [default value](https://code.qt.io/cgit/qt/qtbase.git/tree/src/tools/rcc/main.cpp?h=5.12.10#n172) 2. The only difference from `--format-version 1` is adding a [last modified timestamp](https://code.qt.io/cgit/qt/qtbase.git/tree/src/tools/rcc/rcc.cpp?h=5.12.10#n207) to the output file ([credits](https://github.com/bitcoin/bitcoin/pull/21654#issuecomment-819198228) to **fanquake**). That, in turn, forces us to use `QT_RCC_SOURCE_DATE_OVERRIDE=1` to get deterministic builds (#13732). This change makes rcc output always deterministic by using `--format-version 1` option that makes usage of the `QT_RCC_SOURCE_DATE_OVERRIDE` needless. --- Also it improves interaction with ccache: On master (f6c44e999b7d1d9a0de5d678ac8f1679aa271f65): ``` $ make && make clean && ccache --zero-stats && make && ccache --show-stats ... cache directory /home/hebasto/.ccache primary config /home/hebasto/.ccache/ccache.conf secondary config (readonly) /etc/ccache.conf stats updated Sun Apr 11 15:45:43 2021 stats zeroed Sun Apr 11 15:45:05 2021 cache hit (direct) 638 cache hit (preprocessed) 0 cache miss 1 cache hit rate 99.84 % called for link 10 cleanups performed 0 files in cache 20023 cache size 13.2 GB max cache size 15.0 GB ``` The missed file is always `qt/libbitcoinqt_a-qrc_bitcoin_locale.o`. With this PR: ``` $ make && make clean && ccache --zero-stats && make && ccache --show-stats ... cache directory /home/hebasto/.ccache primary config /home/hebasto/.ccache/ccache.conf secondary config (readonly) /etc/ccache.conf stats updated Sun Apr 11 15:28:46 2021 stats zeroed Sun Apr 11 15:28:21 2021 cache hit (direct) 639 cache hit (preprocessed) 0 cache miss 0 cache hit rate 100.00 % called for link 10 cleanups performed 0 files in cache 20012 cache size 13.2 GB max cache size 15.0 GB ``` ACKs for top commit: fanquake: ACK a58868d201cb6d263aa552815f7f86562c1ca9a5 Tree-SHA512: 52f4a3267f41883d13025c0de79b6da22e92d60c729e01b986935c6812bbfe7fadc40b742bd715bfdf09df94af6838d4fbbe8208c6123f366108e38c8e1121c5
2021-05-25guix: Remove dest if OUTDIR mv failsCarl Dong
2021-05-13guix: Add codesigning functionalityCarl Dong
2021-05-12Merge bitcoin/bitcoin#21462: guix: Add guix-{attest,verify} scriptsW. J. van der Laan
d420e5c1c015f58d07aca4d6a805086488f74d03 guix-attest: Avoid incomplete sigdirs with ERR traps (Carl Dong) feda2c8e3180cb983c35976d4440cea23a155b7f guix: Skip attesting to dist-archive (Carl Dong) d522d8006b891eccd7901faf391f9c041ddf8e38 guix: Attest to inputs in inputs.SHA256SUMS (Carl Dong) f9e2960c018103be756a7f8a506816b49d662514 guix: Construct $OUTDIR in ${DISTSRC}/output (Carl Dong) 022abc85fc7e711a900fed8e5071919a151c0a63 guix: Minor quoting fix in libexec/build.sh (Carl Dong) c83c4fa5b78aef33bba36b3a0d273422297bd630 guix-attest: Allow skipping GPG signing with NO_SIGN (Carl Dong) 0e1c2e448c25568f276e4f022128870c76ca216b guix-attest: Use ascii-armor signatures (Carl Dong) b5fd89c4c89136007429688601ce4fa497f5f09e guix-attest: Only use cross-platform flags for find+xargs (Carl Dong) 5926432ba68ba154df6c8eaa74adb18cc0123167 guix: Add guix-verify script (Carl Dong) 30daf76a97c57a5f74c8dad1da282dcc0ff8b3fb guix: Add guix-attest script (Carl Dong) Pull request description: Adds replacements for `gsign` and `gverify`. Personally I'm not a big fan of using the word "sign" as it's been used to refer to both codesigning and GPG signing. ACKs for top commit: laanwj: Code review and tested ACK d420e5c1c015f58d07aca4d6a805086488f74d03 Tree-SHA512: 93d82d201f4596eaea0e3825aa55b013dfb91790e6ccee79893833d37921513d7b4e735f0641103e1e2ea8308abe4cb6218b73160924708802f2e0e3f7f6caf1
2021-05-10build: Makes rcc output always deterministicHennadii Stepanov
The Qt Resource Compiler (rcc) has a command-line option `--format-version` which has the default value 2. The only difference from `--format-version 1` is adding a last modified timestamp to the output file. That, in turn, forces us to use `QT_RCC_SOURCE_DATE_OVERRIDE=1` to get deterministic builds. This change makes rcc output always deterministic by using `--format-version 1` option that makes usage of the `QT_RCC_SOURCE_DATE_OVERRIDE` needless. Also it improves interaction with ccache. Co-authored-by: fanquake <fanquake@gmail.com>
2021-05-05Merge bitcoin/bitcoin#21740: test: add new python linter to check file names ↵W. J. van der Laan
and permissions 46b025e00df40724175735eb5606ac73067cb3b8 test: add new python linter to check file names and permissions (windsok) 6f6bb3ebc7cb8e17a5dfc8ef55aa2d3f2dc6bdea test: fix file permissions on various scripts (windsok) Pull request description: Adds a new python linter test which tests for correct filenames and file permissions in the repository. Replaces the existing tests in the `test/lint/lint-filenames.sh` and `test/lint/lint-shebang.sh` linter tests, as well as adding some new and increased testing. This increased coverage is intended to catch issues such as in #21728 and https://github.com/bitcoin/bitcoin/pull/16807/files#r345547050 Summary of tests: * Checks every file in the repository against an allowed regexp to make sure only lowercase or uppercase alphanumerics (a-zA-Z0-9), underscores (_), hyphens (-), at (@) and dots (.) are used in repository filenames. * Checks only source files (*.cpp, *.h, *.py, *.sh) against a stricter allowed regexp to make sure only lowercase alphanumerics (a-z0-9), underscores (_), hyphens (-) and dots (.) are used in source code filenames. Additionally there is an exception regexp for directories or files which are excepted from matching this regexp (This should replicate the existing `test/lint/lint-filenames.sh` test) * Checks all files in the repository match an allowed executable or non-executable file permission octal. Additionally checks that for executable files, the file contains a shebang line. * Checks that for executable `.py` and `.sh` files, the shebang line used matches an allowable list of shebangs (This should replicate the existing `test/lint/lint-shebang.sh` test) * Checks every file that contains a shebang line to ensure it has an executable permission Additionally updates the permissions on various files to comply with the new tests. Fixes #21729 ACKs for top commit: practicalswift: cr re-ACK 46b025e00df40724175735eb5606ac73067cb3b8: patch still looks correct kiminuo: code review ACK 46b025e00df40724175735eb5606ac73067cb3b8 if `contrib/gitian-descriptors/assign_DISTNAME` permission change is deemed OK. laanwj: Code review ACK 46b025e00df40724175735eb5606ac73067cb3b8 Tree-SHA512: 1c8201a2cee0d9cbce15652b68cec9a6458a8b493fcd5392f98560aca0b1a12e668baab65a47100f116f626dadc3f591deb47f7368468c6a46c6c712c2533455
2021-05-03guix: Consistently use gcc-8 for $HOSTCarl Dong
2021-05-03guix: Skip attesting to dist-archiveCarl Dong
We already attest to the relevant dist-archive in inputs.SHA256SUMS, which is recorded at build-time. We use a SKIPATTEST.TAG file to indicate output directories which do not require attestation (much like the CACHEDIR.TAG specification). Generally, it's better to have build scripts declare properties of directories instead of introducing name-based special cases in attest scripts since build scripts have a more detailed context of what is going on.
2021-05-03guix: Attest to inputs in inputs.SHA256SUMSCarl Dong
At build/codesigning-time, hash build inputs and output the digest to ${OUTDIR}/inputs.SHA256SUMS, which gets included in the final SHA256SUMS constructed by guix-attest. Example final SHA256SUMS: ee832d2a35b7701bff581dea05a536118b118e3ad0a587a2855b6ee8cd6fba20 inputs/bitcoin-78199266af7b.tar.gz ca765e70a0c12866dd63c0be228b675278a26329e5f8f5b5c52fd09200fedf21 bitcoin-78199266af7b-powerpc64le-linux-gnu-debug.tar.gz dae95327d7f2c324e2728c4b73627be6cb2c0d2f2e5bea940d1d5e6463939327 bitcoin-78199266af7b-powerpc64le-linux-gnu.tar.gz
2021-05-03guix: Construct $OUTDIR in ${DISTSRC}/outputCarl Dong
While files are being output to $OUTDIR, it will be under ${DISTSRC}/output, and only when everything is done, will ${DISTSRC}/output be moved to the actual $OUTDIR. This makes it so that a Ctrl-C in the middle of a build is less likely to result in a partially-constructed $OUTDIR. In fact, if I understand correctly, if $OUTDIR and $DISTSRC reside on the same filesystem, the move (rename) is likely atomic. Also, since the "working $OUTDIR" is under ${DISTSRC}/output, it will be cleaned properly by the guix-clean script.
2021-05-03guix: Minor quoting fix in libexec/build.shCarl Dong
2021-04-23test: fix file permissions on various scriptswindsok
Updates permissions on files to comply with the new test added in the following commit
2021-04-11build, qt: No longer need to set QT_RCC_TEST=1 for determinismHennadii Stepanov
Since Qt 5.3.1 hash seeding is disabled for rcc. See commit 5283a6c87beac5a43f612786fefd6e43f2c70bf6.
2021-04-08Merge #21304: guix: Add guix-clean script + establish gc-root for container ↵W. J. van der Laan
profiles 867a5e172a23899a4a70eca4a396c64f1951745e guix: Register garbage collector root for containers (Carl Dong) 8f8b96fb542701b7717683caa3848390b24f77ab guix: Update hint messages to mention guix-clean (Carl Dong) 44f6d4f56b16e1dc5e8a23318b8e7aad0665f178 guix: Record precious directories and add guix-clean (Carl Dong) 84912d4b24382ae022da3a863bd6caa2b8948d94 build: Remove spaces from variable-printing rules (Carl Dong) Pull request description: ``` guix: Record precious directories and add guix-clean Many users have reported problems that stem from having an unclean working tree. To that end, I've written a guix-clean script which should help reset the working tree while respecting user-specified precious directories. Precious directories, such as: - SOURCES_PATH - BASE_CACHE - SDK_PATH - OUTDIR Should be preserved when cleaning the working tree, and are thus recorded in ./contrib/guix/var/precious_dirs. The ./contrib/guix/guix-clean script is able to parse that file and make sure to avoid them when cleaning out the working tree. ``` ACKs for top commit: laanwj: ACK 867a5e172a23899a4a70eca4a396c64f1951745e Tree-SHA512: c498fad781ff5e6406639df2b91b687fc528273fdf266bcdba8f6eec3b3b37ecce544b6da0252f0b9c6717f9d88e844e4c7b72d1877bdbabfc6871ddd0172af5
2021-04-07guix: Register garbage collector root for containersCarl Dong
By registering the container profiles as garbage collector roots, it will prevent `guix gc` from garbage collecting derivations which our container needs and inconvieniencing the user with a rebuild.
2021-04-07doc: Fix typos from codespell lintYerzhan Mazhkenov
2021-04-05guix: Record precious directories and add guix-cleanCarl Dong
Many users have reported problems that stem from having an unclean working tree. To that end, I've written a guix-clean script which should help reset the working tree while respecting user-specified precious directories. Precious directories, such as: - SOURCES_PATH - BASE_CACHE - SDK_PATH - OUTDIR Should be preserved when cleaning the working tree, and are thus recorded in ./contrib/guix/var/precious_dirs. The ./contrib/guix/guix-clean script is able to parse that file and make sure to avoid them when cleaning out the working tree.
2021-04-05guix: Build dmg as a static binaryCarl Dong
This relatively easy change eliminates all runtime dependencies (except for the kernel) for dmg, which is the only native build tool that gets put in our output tarballs. This allows much more flexibility when constructing the codesigning environment, and is much more robust.
2021-04-05guix: Remove codesign_allocate+pagestuff from unsigned tarballCarl Dong
2021-04-05guix: More thoroughly control native toolchainCarl Dong
2021-04-05guix: Adapt guix-build to prelude, restructure hierCarl Dong
2021-04-05guix: Add source-able bash prelude and utilsCarl Dong
2021-04-01guix: Create windeploy inside distsrc-*Carl Dong
./windeploy is a "working directory", and therefore belongs inside distsrc-*. Many people have noticed their Guix builds failing after hours simply because they did not remove windeploy (but did remove the distsrc-* directories).
2021-04-01guix: Use --cores instead of --max-jobsCarl Dong
In Guix, there are two flags for controlling parallelism: Note: When I say "derivation," think "package" --cores=n - controls the number of CPU cores to build each derivation. This is the value passed to `make`'s `--jobs=` flag. - defaults to 0: as many cores as is available --max-jobs=n - controls how many derivations can be built in parallel - defaults to 1 Therefore, if set --max-jobs=$MAX_JOBS and don't set --cores, Guix could theoretically spin up $MAX_JOBS * $(nproc) number of threads, and that's no good. So we could either default to --cores=1, --max-jobs=$MAX_JOBS - Pro: --cores=1 means that `make` will be invoked with `-j1`, avoiding problems with package whose build systems and test suites break when running multi-threaded. - Con: There will be times when only 1 or 2 derivations can be built at a time, because the rest of the dependency graph all depend on those 1 or 2 derivations. During these times, the machine will be severely under-utilized. or --cores=$MAX_JOBS, --max-jobs=1 - Pro: We don't encounter prolonged periods of severe under-utilization mentioned above. - Con: Many packages' build systems and test suites break when running multi-threaded. or --cores=1, --max-jobs=1 and let the user override with $ADDITIONAL_GUIX_COMMON_FLAGS
2021-03-03guix: Update darwin native packages dependenciesHennadii Stepanov
2021-03-01lint: Fix spelling errors in commentsfyquah
2021-02-26Merge #21272: guix: Passthrough SDK_PATH into containerWladimir J. van der Laan
13a9fd11a507fd3398bc2c0a0575bdc81579243f guix: Passthrough SDK_PATH into container (Carl Dong) Pull request description: This is a usability improvement for Guix builders so that they don't have to extract the Xcode tarball into `depends/SDKs` every time. Inspiration: https://github.com/bitcoin/bitcoin/pull/21089#issuecomment-778639698 ACKs for top commit: laanwj: Tested ACK 13a9fd11a507fd3398bc2c0a0575bdc81579243f Tree-SHA512: 63392d537e48a0da9f0ee04a929613b139bef1ac5643187871c9ea5376afd2a3d95df0f5e0950ae0eccd2813b166667be98401e5a248ae9c187fe4e84e54d427
2021-02-23guix: Explicitly set umask in build containerCarl Dong
A difference in system umask value can cause non-determinism when zip archives are produced. Set it to a reasonable default.
2021-02-22guix: Passthrough SDK_PATH into containerCarl Dong
2021-02-19guix: Add support for powerpc64{,le}Carl Dong
The new time-machine commit contains a few small changes that make the powerpc cross-toolchain work.
2021-02-18guix: Jump forwards in time-machine and adaptCarl Dong
The new time-machine commit is Guix v1.2.0 with a yet-unupstreamed patch for NSIS. A few important changes: 1. Guix switched back from using CPATH to C{,PLUS}_INCLUDE_PATH as the way to indicate #include search paths. 2. GCC's library is now split into a separate output, whereas before it was included in the default output. This means that our gcc toolchain packages need to propagate that output. 3. A few package versions were bumped
2021-02-16guix: Passthrough BASE_CACHE into containerCarl Dong
This allows depends-built packages to be cached.
2021-02-08build: Disable --disable-fuzz-binary for guix buildsHennadii Stepanov
2021-01-21guix: Set ZERO_AR_DATE for darwin build determinismCarl Dong
See comments inserted in this commit.
2021-01-21guix: Add support for darwin buildsCarl Dong
2021-01-08guix: Print build params inside/outside of containerCarl Dong
2021-01-08guix: Move DISTSRC determination to guix-build.shCarl Dong
2021-01-08guix: Move OUTDIR determination+creation to guix-build.shCarl Dong
2020-11-05scripted-diff: [build] Ensure source tarball has leading directory nameMarcoFalke
-BEGIN VERIFY SCRIPT- sed -i 's|git archive --|git archive --prefix="${DISTNAME}/" --|g' $(git grep -l 'git archive' ./contrib) sed -i 's|tar -xf "\?${\?GIT_ARCHIVE}\?"\?|tar --strip-components=1 -xf "${GIT_ARCHIVE}"|g' $(git grep -l 'tar -xf' ./contrib) -END VERIFY SCRIPT-
2020-05-12guix: Make V=1 more powerful for debuggingCarl Dong
- Print commands in both unexpanded and expanded forms - Set VERBOSE=1 for CMake
2020-05-01rel-builds: Directly deploy win installer to OUTDIRCarl Dong
2020-04-28guix: Make source tarball using git-archiveCarl Dong
2020-04-12build: add linker optimization flags to guixfanquake
Any -O argument will enable optimizations in GNU ld. We can use -O2 here, as this matches our compile flags. Note that this would also enable additional optimizations if using the lld or gold linkers, when compared to -O0.
2020-04-07guix: Appease travis.Carl Dong
2020-04-07guix: Don't set MINGW_HAS_SECURE_API CFLAG in dependsCarl Dong
This is no longer needed after 3bef7c22 in the mingw-w64 git repository, which is first included in mingw-w64 v7.0.0. As of the previous bump to our Guix time machine, we now use mingw-w64 v7.0.0.
2020-04-02guix: Check mingw symbols, improve SSP fix docsCarl Dong
2020-04-02guix: Set the well-known timezone env varCarl Dong
2020-04-02guix: Make x86_64-w64-mingw32 builds reproducibleCarl Dong
- Add "--no-insert-timestamp" LDFLAG for x86_64-w64-mingw32 builds "The option --no-insert-timestamp can be used to insert a zero value for the timestamp, this ensuring that binaries produced from identical sources will compare identically." - ld(1) - Set "SetDateSave off" in NSIS script From https://nsis.sourceforge.io/Docs/Chapter4.html#flags "This command sets the file date/time saving flag which is used by the File command to determine whether or not to save the last write date and time of the file, so that it can be restored on installation. Valid flags are 'on' and 'off'. 'on' is the default." - Add commented out NSIS options for reproducibility debugging in NSIS script - Make ZIPs deterministic by reseting file modification times to SOURCE_DATE_EPOCH using touch(1) (Reference: https://reproducible-builds.org/docs/archives/)
generated by cgit v1.2.3 (git 2.26.2) at 2024-09-01 17:36:02 +0000