aboutsummaryrefslogtreecommitdiff
path: root/contrib/guix/guix-build
diff options
context:
space:
mode:
authorW. J. van der Laan <laanwj@protonmail.com>2021-04-06 00:59:11 +0200
committerW. J. van der Laan <laanwj@protonmail.com>2021-04-06 01:00:27 +0200
commit0102f80b511307fe5b6d107e6c133b71940838ce (patch)
treea648560d8a10761c29abb7bed7051cb350a0f09d /contrib/guix/guix-build
parent1ea5c7ec786d3d3d8276d2915ff04152e1c68443 (diff)
parent7476b46f1893a4858616d2a8456a7c43238851ed (diff)
downloadbitcoin-0102f80b511307fe5b6d107e6c133b71940838ce.tar.xz
Merge #21375: guix: Misc feedback-based fixes + hier restructuring
7476b46f1893a4858616d2a8456a7c43238851ed guix: Build dmg as a static binary (Carl Dong) 06d6cf6784421290e6235fe8684d5e08ed6f1b62 depends: libdmg-hfsplus: Skip CMake RPATH patching (Carl Dong) 65176ab5730dff34466caaecdd292625ef8294fc guix: Remove codesign_allocate+pagestuff from unsigned tarball (Carl Dong) ca85679eb43b8375a95d82101977829d08fb1e1b guix: Use clang-toolchain instead of clang (Carl Dong) 1aec0eda8fd31a57b0621eea616398017c2ead98 guix: Fallback to local build for substitute-enabled Guix users (Carl Dong) 1742f8e12d163852df09575e03edcd3db73198ee guix: Add early health check for guix-daemon (Carl Dong) c1ae726a13ecfa5e7e9fdc3030a8110b8bb263f8 guix: More thoroughly control native toolchain (Carl Dong) 39741128d3775d198dbee34dc827353bfd18acd8 guix: Supply --link-profile (Carl Dong) d55a1056ee565afed64e42d6f6efb6b0adc5599b guix: Add troubleshooting documentation entries (Carl Dong) 7f401c953f8bb3574cec48561e13ef3b47dedc6e guix: Adapt guix-build to prelude, restructure hier (Carl Dong) 4eccf063b252bfe256cf72d363a24cf0183e926e guix: Remove guix-build.sh filename extension (Carl Dong) 7753357a7bae98ec775c707b9dec4cea1e945802 guix: Add source-able bash prelude and utils (Carl Dong) e5b49a01f5d0f631e7f08f86ca8a2c2b8213319f guix: Create windeploy inside distsrc-* (Carl Dong) 3e9982ab3877eb8fe0a8c0cb3d847ac0913c7336 contrib: Silence git-describe when looking for tag (Carl Dong) d5a71e97853ea9e1b879e8c76bfb01d4bef33172 guix: Use --cores instead of --max-jobs (Carl Dong) Pull request description: This PR addresses a few hiccups encountered by the brave souls who've been experimenting with the Guix scripts: - Resolves confusion between `--cores=` and `--max-jobs=` - `guix`'s `--cores=` actually corresponds to make's `--jobs=`, so let's just control `--cores=` with our overridable env var - `git-describe` will scream `fatal: no tag exactly matches '<hash>'` when looking for a tag, but we don't care, so silence that - `windeploy/unsigned` should be inside `distsrc-*` and created idempotently (sorry I know this one annoyed people) - Add troubleshooting documentation to `README.md` - Add early health check for `guix-daemon` in case user forgot to start a `guix-daemon` - Depending on configuration, a `--fallback` flag may be needed to tell Guix to not fail if substitutes fail but fallback to building locally - `codesign_allocate` and `pagestuff` are now unnecessary for codesigning as we're now using `signapple` A few robustness changes are also included: - We supply the `--link-profile` flag, as some Guix packages may expect the profile to be available under `$HOME/.guix-profile` - We now clear and manually set all toolchain-related env vars (e.g. `C*_INCLUDE_PATH`) ourselves, after patching a Qt::moc bug - We use the native `clang-toolchain` package for darwin builds instead of `clang`, lining up with all our other toolchain packages. Finally, we restructure the guix building hierarchy such that it looks something like: ``` guix-build-<short-hash-or-version-tag> ├── distsrc-<short-hash-or-version-tag>-${HOST} │ ├── contrib │ ├── depends │ ├── src │ └── ... ├── distsrc-<short-hash-or-version-tag>-... └── output ├── dist-archive │ └── bitcoin-<short-hash-or-version-tag>.tar.gz ├── *-linux-* │ ├── bitcoin-<short-hash-or-version-tag>-*-linux-*-debug.tar.gz │ └── bitcoin-<short-hash-or-version-tag>-*-linux-*.tar.gz ├── x86_64-apple-darwin18 │ ├── bitcoin-<short-hash-or-version-tag>-osx64.tar.gz │ ├── bitcoin-<short-hash-or-version-tag>-osx-unsigned.dmg │ └── bitcoin-<short-hash-or-version-tag>-osx-unsigned.tar.gz └── x86_64-w64-mingw32 ├── bitcoin-<short-hash-or-version-tag>-win64-debug.zip ├── bitcoin-<short-hash-or-version-tag>-win64-setup-unsigned.exe ├── bitcoin-<short-hash-or-version-tag>-win64.zip └── bitcoin-<short-hash-or-version-tag>-win-unsigned.tar.gz ``` Separating guix builds by their version identifier (basically namespacing them) allows us to change the layout in the future without worry about potential naming conflicts. ACKs for top commit: sipa: ACK 7476b46f1893a4858616d2a8456a7c43238851ed laanwj: ACK 7476b46f1893a4858616d2a8456a7c43238851ed Tree-SHA512: 0e899aa941aafdf552b2a7e8a08131ee9283180bbef7334439e2461a02aa7235ab7b9ca9c149b80fc5d0a9f4bbd35bc80fcee26197c0836ba8eaf2d86ffa0386
Diffstat (limited to 'contrib/guix/guix-build')
-rwxr-xr-xcontrib/guix/guix-build366
1 files changed, 366 insertions, 0 deletions
diff --git a/contrib/guix/guix-build b/contrib/guix/guix-build
new file mode 100755
index 0000000000..d817fb74ba
--- /dev/null
+++ b/contrib/guix/guix-build
@@ -0,0 +1,366 @@
+#!/usr/bin/env bash
+export LC_ALL=C
+set -e -o pipefail
+
+# Source the common prelude, which:
+# 1. Checks if we're at the top directory of the Bitcoin Core repository
+# 2. Defines a few common functions and variables
+#
+# shellcheck source=libexec/prelude.bash
+source "$(dirname "${BASH_SOURCE[0]}")/libexec/prelude.bash"
+
+
+###################
+## SANITY CHECKS ##
+###################
+
+################
+# Required non-builtin commands should be invokable
+################
+
+check_tools cat mkdir make git guix
+
+################
+# GUIX_BUILD_OPTIONS should be empty
+################
+#
+# GUIX_BUILD_OPTIONS is an environment variable recognized by guix commands that
+# can perform builds. This seems like what we want instead of
+# ADDITIONAL_GUIX_COMMON_FLAGS, but the value of GUIX_BUILD_OPTIONS is actually
+# _appended_ to normal command-line options. Meaning that they will take
+# precedence over the command-specific ADDITIONAL_GUIX_<CMD>_FLAGS.
+#
+# This seems like a poor user experience. Thus we check for GUIX_BUILD_OPTIONS's
+# existence here and direct users of this script to use our (more flexible)
+# custom environment variables.
+if [ -n "$GUIX_BUILD_OPTIONS" ]; then
+cat << EOF
+Error: Environment variable GUIX_BUILD_OPTIONS is not empty:
+ '$GUIX_BUILD_OPTIONS'
+
+Unfortunately this script is incompatible with GUIX_BUILD_OPTIONS, please unset
+GUIX_BUILD_OPTIONS and use ADDITIONAL_GUIX_COMMON_FLAGS to set build options
+across guix commands or ADDITIONAL_GUIX_<CMD>_FLAGS to set build options for a
+specific guix command.
+
+See contrib/guix/README.md for more details.
+EOF
+exit 1
+fi
+
+################
+# The git worktree should not be dirty
+################
+
+if ! git diff-index --quiet HEAD -- && [ -z "$FORCE_DIRTY_WORKTREE" ]; then
+cat << EOF
+ERR: The current git worktree is dirty, which may lead to broken builds.
+
+ Aborting...
+
+Hint: To make your git worktree clean, You may want to:
+ 1. Commit your changes,
+ 2. Stash your changes, or
+ 3. Set the 'FORCE_DIRTY_WORKTREE' environment variable if you insist on
+ using a dirty worktree
+EOF
+exit 1
+fi
+
+mkdir -p "$VERSION_BASE"
+
+################
+# Build directories should not exist
+################
+
+# Default to building for all supported HOSTs (overridable by environment)
+export HOSTS="${HOSTS:-x86_64-linux-gnu arm-linux-gnueabihf aarch64-linux-gnu riscv64-linux-gnu powerpc64-linux-gnu powerpc64le-linux-gnu
+ x86_64-w64-mingw32
+ x86_64-apple-darwin18}"
+
+# Usage: distsrc_for_host HOST
+#
+# HOST: The current platform triple we're building for
+#
+distsrc_for_host() {
+ echo "${DISTSRC_BASE}/distsrc-${VERSION}-${1}"
+}
+
+# Accumulate a list of build directories that already exist...
+hosts_distsrc_exists=""
+for host in $HOSTS; do
+ if [ -e "$(distsrc_for_host "$host")" ]; then
+ hosts_distsrc_exists+=" ${host}"
+ fi
+done
+
+if [ -n "$hosts_distsrc_exists" ]; then
+# ...so that we can print them out nicely in an error message
+cat << EOF
+ERR: Build directories for this commit already exist for the following platform
+ triples you're attempting to build, probably because of previous builds.
+ Please remove, or otherwise deal with them prior to starting another build.
+
+ Aborting...
+
+EOF
+for host in $hosts_distsrc_exists; do
+ echo " ${host} '$(distsrc_for_host "$host")'"
+done
+exit 1
+else
+ mkdir -p "$DISTSRC_BASE"
+fi
+
+################
+# When building for darwin, the macOS SDK should exists
+################
+
+for host in $HOSTS; do
+ case "$host" in
+ *darwin*)
+ OSX_SDK="$(make -C "${PWD}/depends" --no-print-directory HOST="$host" print-OSX_SDK | sed 's@^[^=]\+=[[:space:]]\+@@g')"
+ if [ -e "$OSX_SDK" ]; then
+ echo "Found macOS SDK at '${OSX_SDK}', using..."
+ else
+ echo "macOS SDK does not exist at '${OSX_SDK}', please place the extracted, untarred SDK there to perform darwin builds, exiting..."
+ exit 1
+ fi
+ ;;
+ esac
+done
+
+################
+# Check that we can connect to the guix-daemon
+################
+
+cat << EOF
+Checking that we can connect to the guix-daemon...
+
+Hint: If this hangs, you may want to try turning your guix-daemon off and on
+ again.
+
+EOF
+if ! guix gc --list-failures > /dev/null; then
+cat << EOF
+
+ERR: Failed to connect to the guix-daemon, please ensure that one is running and
+ reachable.
+EOF
+exit 1
+fi
+
+# Developer note: we could use `guix repl` for this check and run:
+#
+# (import (guix store)) (close-connection (open-connection))
+#
+# However, the internal API is likely to change more than the CLI invocation
+
+
+#########
+# SETUP #
+#########
+
+# Determine the maximum number of jobs to run simultaneously (overridable by
+# environment)
+JOBS="${JOBS:-$(nproc)}"
+
+# Usage: host_to_commonname HOST
+#
+# HOST: The current platform triple we're building for
+#
+host_to_commonname() {
+ case "$1" in
+ *darwin*) echo osx ;;
+ *mingw*) echo win ;;
+ *linux*) echo linux ;;
+ *) exit 1 ;;
+ esac
+}
+
+# Download the depends sources now as we won't have internet access in the build
+# container
+for host in $HOSTS; do
+ make -C "${PWD}/depends" -j"$JOBS" download-"$(host_to_commonname "$host")" ${V:+V=1} ${SOURCES_PATH:+SOURCES_PATH="$SOURCES_PATH"}
+done
+
+# Determine the reference time used for determinism (overridable by environment)
+SOURCE_DATE_EPOCH="${SOURCE_DATE_EPOCH:-$(git log --format=%at -1)}"
+
+# Execute "$@" in a pinned, possibly older version of Guix, for reproducibility
+# across time.
+time-machine() {
+ # shellcheck disable=SC2086
+ guix time-machine --url=https://github.com/dongcarl/guix.git \
+ --commit=490e39ff303f4f6873a04bfb8253755bdae1b29c \
+ --cores="$JOBS" \
+ --keep-failed \
+ --fallback \
+ ${SUBSTITUTE_URLS:+--substitute-urls="$SUBSTITUTE_URLS"} \
+ ${ADDITIONAL_GUIX_COMMON_FLAGS} ${ADDITIONAL_GUIX_TIMEMACHINE_FLAGS} \
+ -- "$@"
+}
+
+# Make sure an output directory exists for our builds
+OUTDIR_BASE="${OUTDIR_BASE:-${VERSION_BASE}/output}"
+mkdir -p "$OUTDIR_BASE"
+
+# Usage: outdir_for_host HOST
+#
+# HOST: The current platform triple we're building for
+#
+outdir_for_host() {
+ echo "${OUTDIR_BASE}/${1}"
+}
+
+
+#########
+# BUILD #
+#########
+
+# Function to be called when building for host ${1} and the user interrupts the
+# build
+int_trap() {
+cat << EOF
+** INT received while building ${1}, you may want to clean up the relevant
+ output, deploy, and distsrc-* directories before rebuilding
+
+Hint: To blow everything away, you may want to use:
+
+ $ git clean -xdff --exclude='/depends/SDKs/*'
+
+Specifically, this will remove all files without an entry in the index,
+excluding the SDK directory. Practically speaking, this means that all ignored
+and untracked files and directories will be wiped, allowing you to start anew.
+EOF
+}
+
+# Create SOURCES_PATH, BASE_CACHE, and SDK_PATH if they are non-empty so that we
+# can map them into the container
+[ -z "$SOURCES_PATH" ] || mkdir -p "$SOURCES_PATH"
+[ -z "$BASE_CACHE" ] || mkdir -p "$BASE_CACHE"
+[ -z "$SDK_PATH" ] || mkdir -p "$SDK_PATH"
+
+# Deterministically build Bitcoin Core
+# shellcheck disable=SC2153
+for host in $HOSTS; do
+
+ # Display proper warning when the user interrupts the build
+ trap 'int_trap ${host}' INT
+
+ (
+ # Required for 'contrib/guix/manifest.scm' to output the right manifest
+ # for the particular $HOST we're building for
+ export HOST="$host"
+
+ # shellcheck disable=SC2030
+cat << EOF
+INFO: Building ${VERSION:?not set} for platform triple ${HOST:?not set}:
+ ...using reference timestamp: ${SOURCE_DATE_EPOCH:?not set}
+ ...running at most ${JOBS:?not set} jobs
+ ...from worktree directory: '${PWD}'
+ ...bind-mounted in container to: '/bitcoin'
+ ...in build directory: '$(distsrc_for_host "$HOST")'
+ ...bind-mounted in container to: '$(DISTSRC_BASE=/distsrc-base && distsrc_for_host "$HOST")'
+ ...outdirting in: '$(outdir_for_host "$HOST")'
+ ...bind-mounted in container to: '$(OUTDIR_BASE=/outdir-base && outdir_for_host "$HOST")'
+EOF
+
+ # Run the build script 'contrib/guix/libexec/build.sh' in the build
+ # container specified by 'contrib/guix/manifest.scm'.
+ #
+ # Explanation of `guix environment` flags:
+ #
+ # --container run command within an isolated container
+ #
+ # Running in an isolated container minimizes build-time differences
+ # between machines and improves reproducibility
+ #
+ # --pure unset existing environment variables
+ #
+ # Same rationale as --container
+ #
+ # --no-cwd do not share current working directory with an
+ # isolated container
+ #
+ # When --container is specified, the default behavior is to share
+ # the current working directory with the isolated container at the
+ # same exact path (e.g. mapping '/home/satoshi/bitcoin/' to
+ # '/home/satoshi/bitcoin/'). This means that the $PWD inside the
+ # container becomes a source of irreproducibility. --no-cwd disables
+ # this behaviour.
+ #
+ # --share=SPEC for containers, share writable host file system
+ # according to SPEC
+ #
+ # --share="$PWD"=/bitcoin
+ #
+ # maps our current working directory to /bitcoin
+ # inside the isolated container, which we later cd
+ # into.
+ #
+ # While we don't want to map our current working directory to the
+ # same exact path (as this introduces irreproducibility), we do want
+ # it to be at a _fixed_ path _somewhere_ inside the isolated
+ # container so that we have something to build. '/bitcoin' was
+ # chosen arbitrarily.
+ #
+ # ${SOURCES_PATH:+--share="$SOURCES_PATH"}
+ #
+ # make the downloaded depends sources path available
+ # inside the isolated container
+ #
+ # The isolated container has no network access as it's in a
+ # different network namespace from the main machine, so we have to
+ # make the downloaded depends sources available to it. The sources
+ # should have been downloaded prior to this invocation.
+ #
+ # --keep-failed keep build tree of failed builds
+ #
+ # When builds of the Guix environment itself (not Bitcoin Core)
+ # fail, it is useful for the build tree to be kept for debugging
+ # purposes.
+ #
+ # ${SUBSTITUTE_URLS:+--substitute-urls="$SUBSTITUTE_URLS"}
+ #
+ # fetch substitute from SUBSTITUTE_URLS if they are
+ # authorized
+ #
+ # Depending on the user's security model, it may be desirable to use
+ # substitutes (pre-built packages) from servers that the user trusts.
+ # Please read the README.md in the same directory as this file for
+ # more information.
+ #
+ # shellcheck disable=SC2086,SC2031
+ time-machine environment --manifest="${PWD}/contrib/guix/manifest.scm" \
+ --container \
+ --pure \
+ --no-cwd \
+ --share="$PWD"=/bitcoin \
+ --share="$DISTSRC_BASE"=/distsrc-base \
+ --share="$OUTDIR_BASE"=/outdir-base \
+ --expose="$(git rev-parse --git-common-dir)" \
+ ${SOURCES_PATH:+--share="$SOURCES_PATH"} \
+ ${BASE_CACHE:+--share="$BASE_CACHE"} \
+ ${SDK_PATH:+--share="$SDK_PATH"} \
+ --cores="$JOBS" \
+ --keep-failed \
+ --fallback \
+ --link-profile \
+ ${SUBSTITUTE_URLS:+--substitute-urls="$SUBSTITUTE_URLS"} \
+ ${ADDITIONAL_GUIX_COMMON_FLAGS} ${ADDITIONAL_GUIX_ENVIRONMENT_FLAGS} \
+ -- env HOST="$host" \
+ DISTNAME="$DISTNAME" \
+ JOBS="$JOBS" \
+ SOURCE_DATE_EPOCH="${SOURCE_DATE_EPOCH:?unable to determine value}" \
+ ${V:+V=1} \
+ ${SOURCES_PATH:+SOURCES_PATH="$SOURCES_PATH"} \
+ ${BASE_CACHE:+BASE_CACHE="$BASE_CACHE"} \
+ ${SDK_PATH:+SDK_PATH="$SDK_PATH"} \
+ DISTSRC="$(DISTSRC_BASE=/distsrc-base && distsrc_for_host "$HOST")" \
+ OUTDIR="$(OUTDIR_BASE=/outdir-base && outdir_for_host "$HOST")" \
+ DIST_ARCHIVE_BASE=/outdir-base/dist-archive \
+ bash -c "cd /bitcoin && bash contrib/guix/libexec/build.sh"
+ )
+
+done