aboutsummaryrefslogtreecommitdiff
diff options
context:
space:
mode:
authorfanquake <fanquake@gmail.com>2021-07-20 11:07:18 +0800
committerfanquake <fanquake@gmail.com>2021-07-20 11:09:39 +0800
commite7441a6a458340daabaccfe0a66eb2ec1e9df0b8 (patch)
treee8df2c8fa631dd365b30e43f8f549fd77938acd1
parentd542603c5ac9143b638b256855a63e6665554913 (diff)
parentfac4814106c796b8786dd90053513cc35142dfe5 (diff)
downloadbitcoin-e7441a6a458340daabaccfe0a66eb2ec1e9df0b8.tar.xz
Merge bitcoin/bitcoin#21711: guix: Add full installation and usage documentation
fac4814106c796b8786dd90053513cc35142dfe5 doc/release-process: Add torrent creation details (Carl Dong) 5d24cc3d82dad6812f8370c3ccc7c2b5a6c12c11 guix/INSTALL: Guix installs init scripts in libdir (Carl Dong) 5da2ee49d5b44de803b671aedbdd14e5c1d71ea9 guix/INSTALL: Add coreutils/inotify-dir-recreate troubleshooting (Carl Dong) 318c60700b7bbb7ec09a29bf037e7c2787646be6 guix: Adapt release-process.md to new Guix process (Carl Dong) fcab35b2292f9221eaba521740e8b3b2511a8b78 guix-attest: Produce and sign normalized documents (Carl Dong) c2541fd0ca99481a5a792a8f2772925d64fb0491 guix: Overhaul README (Carl Dong) 46ce6ce3782dfbd8f9d26dc2ba0f284755e75f2d tree-wide: Rename gitian-keys to builder-keys (Carl Dong) fc4f8449f34e32b0b9ac9d218d6c3264b02467ba guix: Update various check_tools lists (Carl Dong) 263220a85c1df218431fafbda07c8b23ccc4ce4d guix: Check for a sane services database (Carl Dong) Pull request description: Based on: #21462 Keeping the README in one file so that it's easy to search through. Will add more jumping links later so navigation is easier. Current TODOs: - [x] Shell installer option: prompt user to re-login for `/etc/profile.d` entry to be picked up - [x] Binary tarball option: prompt user to create `/etc/profile.d` entry and re-login - [x] Fanquake docker option: complete section - [x] Arch Linux AUR option: prompt to start `guix-daemon-latest` unit after finishing "optional setup" section - [x] Building from source option: Insert dependency tree diagram that I made - [x] Building from source option: redo sectioning, kind of a mess right now - [x] Optional setup: make clear which parts are only needed if building from source - [x] Workaround 1 for GnuTLS: perhaps mention how to remove Guix build farm's key - [x] Overall (after everything): Make the links work. Note to self: wherever possible, tell user how to check that something is true rather than branching by installation option. ACKs for top commit: fanquake: ACK fac4814106c796b8786dd90053513cc35142dfe5 - going to go ahead and merge this now. It's a lot of documentation, and could probably be nit-picked / improved further, however, that can continue over the next few weeks. I'm sure more (backportable) improvements / clarifications will be made while we progress through RCs towards a new release. Tree-SHA512: dc46c0ecdfc67c7c7743ca26e4a603eb3f54adbf81be2f4c1f4c20577ebb84b5250b9c9ec89c0e9860337ab1c7cff94d7963c603287267deecfe1cd987fa070a
-rw-r--r--contrib/README.md4
-rw-r--r--contrib/builder-keys/README.md (renamed from contrib/gitian-keys/README.md)14
-rw-r--r--contrib/builder-keys/keys.txt (renamed from contrib/gitian-keys/keys.txt)0
-rw-r--r--contrib/guix/INSTALL.md813
-rw-r--r--contrib/guix/README.md418
-rwxr-xr-xcontrib/guix/guix-attest24
-rwxr-xr-xcontrib/guix/guix-build25
-rw-r--r--doc/release-process.md269
-rwxr-xr-xtest/lint/lint-spelling.sh2
9 files changed, 1250 insertions, 319 deletions
diff --git a/contrib/README.md b/contrib/README.md
index 361975baa4..a2612ab958 100644
--- a/contrib/README.md
+++ b/contrib/README.md
@@ -29,8 +29,8 @@ All other packaging related files can be found in the [bitcoin-core/packaging](h
### [Gitian-descriptors](/contrib/gitian-descriptors) ###
Files used during the gitian build process. For more information about gitian, see the [the Bitcoin Core documentation repository](https://github.com/bitcoin-core/docs).
-### [Gitian-keys](/contrib/gitian-keys)
-PGP keys used for signing Bitcoin Core [Gitian release](/doc/release-process.md) results.
+### [Builder keys](/contrib/builder-keys)
+PGP keys used for signing Bitcoin Core [release](/doc/release-process.md) results.
### [MacDeploy](/contrib/macdeploy) ###
Scripts and notes for Mac builds.
diff --git a/contrib/gitian-keys/README.md b/contrib/builder-keys/README.md
index ffe4fb144b..a7c1d5ae0a 100644
--- a/contrib/gitian-keys/README.md
+++ b/contrib/builder-keys/README.md
@@ -1,10 +1,10 @@
-## PGP keys of Gitian builders and Developers
+## PGP keys of builders and Developers
-The file `keys.txt` contains fingerprints of the public keys of Gitian builders
-and active developers.
+The file `keys.txt` contains fingerprints of the public keys of builders and
+active developers.
The associated keys are mainly used to sign git commits or the build results
-of Gitian builds.
+of Guix builds.
The most recent version of each pgp key can be found on most pgp key servers.
@@ -16,12 +16,12 @@ To fetch the latest version of all pgp keys in your gpg homedir,
gpg --refresh-keys
```
-To fetch keys of Gitian builders and active developers, feed the list of
-fingerprints of the primary keys into gpg:
+To fetch keys of builders and active developers, feed the list of fingerprints
+of the primary keys into gpg:
```sh
while read fingerprint keyholder_name; do gpg --keyserver hkp://subset.pool.sks-keyservers.net --recv-keys ${fingerprint}; done < ./keys.txt
```
-Add your key to the list if you provided Gitian signatures for two major or
+Add your key to the list if you provided Guix attestations for two major or
minor releases of Bitcoin Core.
diff --git a/contrib/gitian-keys/keys.txt b/contrib/builder-keys/keys.txt
index db28cd07a0..db28cd07a0 100644
--- a/contrib/gitian-keys/keys.txt
+++ b/contrib/builder-keys/keys.txt
diff --git a/contrib/guix/INSTALL.md b/contrib/guix/INSTALL.md
new file mode 100644
index 0000000000..86f91cc87b
--- /dev/null
+++ b/contrib/guix/INSTALL.md
@@ -0,0 +1,813 @@
+# Guix Installation and Setup
+
+This only needs to be done once per machine. If you have already completed the
+installation and setup, please proceed to [perform a build](./README.md).
+
+Otherwise, you may choose from one of the following options to install Guix:
+
+1. Using the official **shell installer script** [⤓ skip to section][install-script]
+ - Maintained by Guix developers
+ - Easiest (automatically performs *most* setup)
+ - Works on nearly all Linux distributions
+ - Only installs latest release
+ - Binary installation only, requires high level of trust
+ - Note: The script needs to be run as root, so it should be inspected before it's run
+2. Using the official **binary tarball** [⤓ skip to section][install-bin-tarball]
+ - Maintained by Guix developers
+ - Normal difficulty (full manual setup required)
+ - Works on nearly all Linux distributions
+ - Installs any release
+ - Binary installation only, requires high level of trust
+3. Using fanquake's **Docker image** [↗︎ external instructions][install-fanquake-docker]
+ - Maintained by fanquake
+ - Easy (automatically performs *some* setup)
+ - Works wherever Docker images work
+ - Installs any release
+ - Binary installation only, requires high level of trust
+4. Using a **distribution-maintained package** [⤓ skip to section][install-distro-pkg]
+ - Maintained by distribution's Guix package maintainer
+ - Normal difficulty (manual setup required)
+ - Works only on distributions with Guix packaged, see: https://repology.org/project/guix/versions
+ - Installs a release decided on by package maintainer
+ - Source or binary installation depending on the distribution
+5. Building **from source** [⤓ skip to section][install-source]
+ - Maintained by you
+ - Hard, but rewarding
+ - Can be made to work on most Linux distributions
+ - Installs any commit (more granular)
+ - Source installation, requires lower level of trust
+
+## Options 1 and 2: Using the official shell installer script or binary tarball
+
+The installation instructions for both the official shell installer script and
+the binary tarballs can be found in the GNU Guix Manual's [Binary Installation
+section](https://guix.gnu.org/manual/en/html_node/Binary-Installation.html).
+
+Note that running through the binary tarball installation steps is largely
+equivalent to manually performing what the shell installer script does.
+
+Note that at the time of writing (July 5th, 2021), the shell installer script
+automatically creates an `/etc/profile.d` entry which the binary tarball
+installation instructions do not ask you to create. However, you will likely
+need this entry for better desktop integration. Please see [this
+section](#add-an-etcprofiled-entry) for instructions on how to add a
+`/etc/profile.d/guix.sh` entry.
+
+Regardless of which installation option you chose, the changes to
+`/etc/profile.d` will not take effect until the next shell or desktop session,
+so you should log out and log back in.
+
+## Option 3: Using fanquake's Docker image
+
+Please refer to fanquake's instructions
+[here](https://github.com/fanquake/core-review/tree/master/guix).
+
+Note that the `Dockerfile` is largely equivalent to running through the binary
+tarball installation steps.
+
+## Option 4: Using a distribution-maintained package
+
+Note that this section is based on the distro packaging situation at the time of
+writing (July 2021). Guix is expected to be more widely packaged over time. For
+an up-to-date view on Guix's package status/version across distros, please see:
+https://repology.org/project/guix/versions
+
+### Debian 11 (Bullseye)/Ubuntu 21.04 (Hirsute Hippo)
+
+Guix v1.2.0 is available as a distribution package starting in [Debian
+11](https://packages.debian.org/bullseye/guix) and [Ubuntu
+21.04](https://packages.ubuntu.com/hirsute/guix).
+
+Note that if you intend on using Guix without using any substitutes (more
+details [here][security-model]), v1.2.0 has a known problems when building
+GnuTLS from source. Solutions and workarounds are documented
+[here](#gnutls-test-suite-fail-status-request-revoked).
+
+
+To install:
+```sh
+sudo apt install guix
+```
+
+For up-to-date information on Debian and Ubuntu's release history:
+- [Debian release history](https://www.debian.org/releases/)
+- [Ubuntu release history](https://ubuntu.com/about/release-cycle)
+
+### Arch Linux
+
+Guix is available in the AUR as
+[`guix`](https://aur.archlinux.org/packages/guix/), please follow the
+installation instructions in the Arch Linux Wiki ([live
+link](https://wiki.archlinux.org/index.php/Guix#AUR_Package_Installation),
+[2021/03/30
+permalink](https://wiki.archlinux.org/index.php?title=Guix&oldid=637559#AUR_Package_Installation))
+to install Guix.
+
+At the time of writing (2021/03/30), the `check` phase will fail if the path to
+guix's build directory is longer than 36 characters due to an anachronistic
+character limit on the shebang line. Since the `check` phase happens after the
+`build` phase, which may take quite a long time, it is recommended that users
+either:
+
+1. Skip the `check` phase
+ - For `makepkg`: `makepkg --nocheck ...`
+ - For `yay`: `yay --mflags="--nocheck" ...`
+ - For `paru`: `paru --nocheck ...`
+2. Or, check their build directory's length beforehand
+ - For those building with `makepkg`: `pwd | wc -c`
+
+## Option 5: Building from source
+
+Building Guix from source is a rather involved process but a rewarding one for
+those looking to minimize trust and maximize customizability (e.g. building a
+particular commit of Guix). Previous experience with using autotools-style build
+systems to build packages from source will be helpful. *hic sunt dracones.*
+
+I strongly urge you to at least skim through the entire section once before you
+start issuing commands, as it will save you a lot of unncessary pain and
+anguish.
+
+### Installing common build tools
+
+There are a few basic build tools that are required for most things we'll build,
+so let's install them now:
+
+Text transformation/i18n:
+- `autopoint` (sometimes packaged in `gettext`)
+- `help2man`
+- `po4a`
+- `texinfo`
+
+Build system tools:
+- `g++` w/ C++11 support
+- `libtool`
+- `autoconf`
+- `automake`
+- `pkg-config` (sometimes packaged as `pkgconf`)
+- `make`
+- `cmake`
+
+Miscellaneous:
+- `git`
+- `gnupg`
+- `python3`
+
+### Building and Installing Guix's dependencies
+
+In order to build Guix itself from source, we need to first make sure that the
+necessary dependencies are installed and discoverable. The most up-to-date list
+of Guix's dependencies is kept in the ["Requirements"
+section](https://guix.gnu.org/manual/en/html_node/Requirements.html) of the Guix
+Reference Manual.
+
+Depending on your distribution, most or all of these dependencies may already be
+packaged and installable without manually building and installing.
+
+For reference, the graphic below outlines Guix v1.3.0's dependency graph:
+
+![boostrap map](https://user-images.githubusercontent.com/6399679/125064185-a9a59880-e0b0-11eb-82c1-9b8e5dc9950d.png)
+
+#### Guile
+
+##### Choosing a Guile version and sticking to it
+
+One of the first things you need to decide is which Guile version you want to
+use: Guile v2.2 or Guile v3.0. Unlike the python2 to python3 transition, Guile
+v2.2 and Guile v3.0 are largely compatible, as evidenced by the fact that most
+Guile packages and even [Guix
+itself](https://guix.gnu.org/en/blog/2020/guile-3-and-guix/) support running on
+both.
+
+What is important here is that you **choose one**, and you **remain consistent**
+with your choice throughout **all Guile-related packages**, no matter if they
+are installed via the distribution's package manager or installed from source.
+This is because the files for Guile packages are installed to directories which
+are separated based on the Guile version.
+
+###### Example: Checking that Ubuntu's `guile-git` is compatible with your chosen Guile version
+
+On Ubuntu Focal:
+
+```sh
+$ apt show guile-git
+Package: guile-git
+...
+Depends: guile-2.2, guile-bytestructures, libgit2-dev
+...
+```
+
+As you can see, the package `guile-git` depends on `guile-2.2`, meaning that it
+was likely built for Guile v2.2. This means that if you decided to use Guile
+v3.0 on Ubuntu Focal, you would need to build guile-git from source instead of
+using the distribution package.
+
+On Ubuntu Hirsute:
+
+```sh
+$ apt show guile-git
+Package: guile-git
+...
+Depends: guile-3.0 | guile-2.2, guile-bytestructures (>= 1.0.7-3~), libgit2-dev (>= 1.0)
+...
+```
+
+In this case, `guile-git` depends on either `guile-3.0` or `guile-2.2`, meaning
+that it would work no matter what Guile version you decided to use.
+
+###### Corner case: Multiple versions of Guile on one system
+
+It is recommended to only install one version of Guile, so that build systems do
+not get confused about which Guile to use.
+
+However, if you insist on having both Guile v2.2 and Guile v3.0 installed on
+your system, then you need to **consistently** specify one of
+`GUILE_EFFECTIVE_VERSION=3.0` or `GUILE_EFFECTIVE_VERSION=2.2` to all
+`./configure` invocations for Guix and its dependencies.
+
+##### Installing Guile
+
+Guile is most likely already packaged for your distribution, so after you have
+[chosen a Guile version](#choosing-a-guile-version-and-sticking-to-it), install
+it via your distribution's package manager.
+
+If your distribution splits packages into `-dev`-suffixed and
+non-`-dev`-suffixed sub-packages (as is the case for Debian-derived
+distributions), please make sure to install both. For example, to install Guile
+v2.2 on Debian/Ubuntu:
+
+```sh
+apt install guile-2.2 guile-2.2-dev
+```
+
+#### Mixing distribution packages and source-built packages
+
+At the time of writing, most distributions have _some_ of Guix's dependencies
+packaged, but not all. This means that you may want to install the distribution
+package for some dependencies, and manually build-from-source for others.
+
+Distribution packages usually install to `/usr`, which is different from the
+default `./configure` prefix of source-built packages: `/usr/local`.
+
+This means that if you mix-and-match distribution packages and source-built
+packages and do not specify exactly `--prefix=/usr` to `./configure` for
+source-built packages, you will need to augment the `GUILE_LOAD_PATH` and
+`GUILE_LOAD_COMPILED_PATH` environment variables so that Guile will look
+under the right prefix and find your source-built packages.
+
+For example, if you are using Guile v2.2, and have Guile packages in the
+`/usr/local` prefix, either add the following lines to your `.profile` or
+`.bash_profile` so that the environment variable is properly set for all future
+shell logins, or paste the lines into a POSIX-style shell to temporarily modify
+the environment variables of your current shell session.
+
+```sh
+# Help Guile v2.2.x find packages in /usr/local
+export GUILE_LOAD_PATH="/usr/local/share/guile/site/2.2${GUILE_LOAD_PATH:+:}$GUILE_LOAD_PATH"
+export GUILE_LOAD_COMPILED_PATH="/usr/local/lib/guile/2.2/site-ccache${GUILE_LOAD_COMPILED_PATH:+:}$GUILE_COMPILED_LOAD_PATH"
+```
+
+Note that these environment variables are used to check for packages during
+`./configure`, so they should be set as soon as possible should you want to use
+a prefix other than `/usr`.
+
+<!-- ##### Example: Consistently using Guile 3.0 on Ubuntu -->
+
+<!-- For example, on Ubuntu, if you choose to use Guile 3.0 and install the -->
+<!-- `guile-3.0` package, you want to make sure that if you also want to install the -->
+<!-- `guile-git` package with `apt` that said `guile-git` package was built for Guile -->
+<!-- v3.0. This can be checked by invoking the following: -->
+
+<!-- ``` -->
+<!-- apt update -->
+<!-- apt show guile-git -->
+<!-- ``` -->
+
+#### Building and installing source-built packages
+
+***IMPORTANT**: A few dependencies have non-obvious quirks/erratas which are documented in the
+sub-sections immediately below. Please read these sections before proceeding to
+build and install these packages.*
+
+Although you should always refer to the README or INSTALL files for the most
+accurate information, most of these dependencies use autoconf-style build
+systems (check if there's a `configure.ac` file), and will likely do the right
+thing with the following:
+
+Clone the repository and check out the latest release:
+```sh
+git clone <git-repo-of-dependency>/<dependency>.git
+cd <dependency>
+git tag -l # check for the latest release
+git checkout <latest-release>
+```
+
+For autoconf-based build systems (if `./autogen.sh` or `configure.ac` exists at
+the root of the repository):
+
+```sh
+./autogen.sh || autoreconf -vfi
+./configure --prefix=<prefix>
+make
+sudo make install
+```
+
+For CMake-based build systems (if `CMakeLists.txt` exists at the root of the
+repository):
+
+```sh
+mkdir build && cd build
+cmake .. -DCMAKE_INSTALL_PREFIX=<prefix>
+sudo cmake --build . --target install
+```
+
+If you choose not to specify exactly `--prefix=/usr` to `./configure`, please
+make sure you've carefully read the [previous section] on mixing distribution
+packages and source-built packages.
+
+##### Binding packages require `-dev`-suffixed packages
+
+Relevant for:
+- Everyone
+
+When building bindings, the `-dev`-suffixed version of the original package
+needs to be installed. For example, building `Guile-zlib` on Debian-derived
+distributions requires that `zlib1g-dev` is installed.
+
+When using bindings, the `-dev`-suffixed version of the original package still
+needs to be installed. This is particularly problematic when distribution
+packages are mispackaged like `guile-sqlite3` is in Ubuntu Focal such that
+installing `guile-sqlite3` does not automatically install `libsqlite3-dev` as a
+dependency.
+
+Below is a list of relevant Guile bindings and their corresponding `-dev`
+packages in Debian at the time of writing.
+
+| Guile binding package | -dev Debian package |
+|-----------------------|---------------------|
+| guile-gcrypt | libgcrypt-dev |
+| guile-git | libgit2-dev |
+| guile-lzlib | liblz-dev |
+| guile-ssh | libssh-dev |
+| guile-sqlite3 | libsqlite3-dev |
+| guile-zlib | zlib1g-dev |
+
+##### `guile-git` actually depends on `libgit2 >= 1.1`
+
+Relevant for:
+- Those building `guile-git` from source against `libgit2 < 1.1`
+- Those installing `guile-git` from their distribution where `guile-git` is
+ built against `libgit2 < 1.1`
+
+As of v0.4.0, `guile-git` claims to only require `libgit2 >= 0.28.0`, however,
+it actually requires `libgit2 >= 1.1`, otherwise, it will be confused by a
+reference of `origin/keyring`: instead of interpreting the reference as "the
+'keyring' branch of the 'origin' remote", the reference is interpreted as "the
+branch literally named 'origin/keyring'"
+
+This is especially notable because Ubuntu Focal packages `libgit2 v0.28.4`, and
+`guile-git` is built against it.
+
+Should you be in this situation, you need to build both `libgit2 v1.1.x` and
+`guile-git` from source.
+
+Source: http://logs.guix.gnu.org/guix/2020-11-12.log#232527
+
+##### `{scheme,guile}-bytestructures` v1.0.8 and v1.0.9 are broken for Guile v2.2
+
+Relevant for:
+- Those building `{scheme,guile}-bytestructures` from source against Guile v2.2
+
+Commit
+[707eea3](https://github.com/TaylanUB/scheme-bytestructures/commit/707eea3a85e1e375e86702229ebf73d496377669)
+introduced a regression for Guile v2.2 and was first included in v1.0.8, this
+was later corrected in commit
+[ec9a721](https://github.com/TaylanUB/scheme-bytestructures/commit/ec9a721957c17bcda13148f8faa5f06934431ff7)
+and included in v1.1.0.
+
+TL;DR If you decided to use Guile v2.2, do not use `{scheme,guile}-bytestructures` v1.0.8 or v1.0.9.
+
+### Building and Installing Guix itself
+
+Start by cloning Guix:
+
+```
+git clone https://git.savannah.gnu.org/git/guix.git
+cd guix
+```
+
+You will likely want to build the latest release, however, if the latest release
+when you're reading this is still 1.2.0 then you may want to use 95aca29 instead
+to avoid a problem in the GnuTLS test suite.
+
+```
+git branch -a -l 'origin/version-*' # check for the latest release
+git checkout <latest-release>
+```
+
+Bootstrap the build system:
+```
+./bootstrap
+```
+
+Configure with the recommended `--localstatedir` flag:
+```
+./configure --localstatedir=/var
+```
+
+Note: If you intend to hack on Guix in the future, you will need to supply the
+same `--localstatedir=` flag for all future Guix `./configure` invocations. See
+the last paragraph of this
+[section](https://guix.gnu.org/manual/en/html_node/Requirements.html) for more
+details.
+
+Build Guix (this will take a while):
+```
+make -j$(nproc)
+```
+
+Install Guix:
+
+```
+sudo make install
+```
+
+### Post-"build from source" Setup
+
+#### Creating and starting a `guix-daemon-original` service with a fixed `argv[0]`
+
+At this point, guix will be installed to `${bindir}`, which is likely
+`/usr/local/bin` if you did not override directory variables at
+`./configure`-time. More information on standard Automake directory variables
+can be found
+[here](https://www.gnu.org/software/automake/manual/html_node/Standard-Directory-Variables.html).
+
+However, the Guix init scripts and service configurations for Upstart, systemd,
+SysV, and OpenRC are installed (in `${libdir}`) to launch
+`${localstatedir}/guix/profiles/per-user/root/current-guix/bin/guix-daemon`,
+which does not yet exist, and will only exist after [`root` performs their first
+`guix pull`](#guix-pull-as-root).
+
+We need to create a `-original` version of these init scripts that's pointed to
+the binaries we just built and `make install`'ed in `${bindir}` (normally,
+`/usr/local/bin`).
+
+Example for `systemd`, run as `root`:
+
+```sh
+# Create guix-daemon-original.service by modifying guix-daemon.service
+libdir=# set according to your PREFIX (default is /usr/local/lib)
+bindir="$(dirname $(command -v guix-daemon))"
+sed -E -e "s|/\S*/guix/profiles/per-user/root/current-guix/bin/guix-daemon|${bindir}/guix-daemon|" "${libdir}"/systemd/system/guix-daemon.service > /etc/systemd/system/guix-daemon-original.service
+chmod 664 /etc/systemd/system/guix-daemon-original.service
+
+# Make systemd recognize the new service
+systemctl daemon-reload
+
+# Make sure that the non-working guix-daemon.service is stopped and disabled
+systemctl stop guix-daemon
+systemctl disable guix-daemon
+
+# Make sure that the working guix-daemon-original.service is started and enabled
+systemctl enable guix-daemon-original
+systemctl start guix-daemon-original
+```
+
+#### Creating `guix-daemon` users / groups
+
+Please see the [relevant
+section](https://guix.gnu.org/manual/en/html_node/Build-Environment-Setup.html)
+in the Guix Reference Manual for more details.
+
+## Optional setup
+
+At this point, you are set up to [use Guix to build Bitcoin
+Core](./README.md#usage). However, if you want to polish your setup a bit and
+make it "what Guix intended", then read the next few subsections.
+
+### Add an `/etc/profile.d` entry
+
+This section definitely does not apply to you if you installed Guix using:
+1. The shell installer script
+2. fanquake's Docker image
+3. Debian's `guix` package
+
+#### Background
+
+Although Guix knows how to update itself and its packages, it does so in a
+non-invasive way (it does not modify `/usr/local/bin/guix`).
+
+Instead, it does the following:
+
+- After a `guix pull`, it updates
+ `/var/guix/profiles/per-user/$USER/current-guix`, and creates a symlink
+ targeting this directory at `$HOME/.config/guix/current`
+
+- After a `guix install`, it updates
+ `/var/guix/profiles/per-user/$USER/guix-profile`, and creates a symlink
+ targeting this directory at `$HOME/.guix-profile`
+
+Therefore, in order for these operations to affect your shell/desktop sessions
+(and for the principle of least astonishment to hold), their corresponding
+directories have to be added to well-known environment variables like `$PATH`,
+`$INFOPATH`, `$XDG_DATA_DIRS`, etc.
+
+In other words, if `$HOME/.config/guix/current/bin` does not exist in your
+`$PATH`, a `guix pull` will have no effect on what `guix` you are using. Same
+goes for `$HOME/.guix-profile/bin`, `guix install`, and installed packages.
+
+Helpfully, after a `guix pull` or `guix install`, a message will be printed like
+so:
+
+```
+hint: Consider setting the necessary environment variables by running:
+
+ GUIX_PROFILE="$HOME/.guix-profile"
+ . "$GUIX_PROFILE/etc/profile"
+
+Alternately, see `guix package --search-paths -p "$HOME/.guix-profile"'.
+```
+
+However, this is somewhat tedious to do for both `guix pull` and `guix install`
+for each user on the system that wants to properly use `guix`. I recommend that
+you instead add an entry to `/etc/profile.d` instead. This is done by default
+when installing the Debian package later than 1.2.0-4 and when using the shell
+script installer.
+
+#### Instructions
+
+Create `/etc/profile.d/guix.sh` with the following content:
+```sh
+# _GUIX_PROFILE: `guix pull` profile
+_GUIX_PROFILE="$HOME/.config/guix/current"
+if [ -L $_GUIX_PROFILE ]; then
+ export PATH="$_GUIX_PROFILE/bin${PATH:+:}$PATH"
+ # Export INFOPATH so that the updated info pages can be found
+ # and read by both /usr/bin/info and/or $GUIX_PROFILE/bin/info
+ # When INFOPATH is unset, add a trailing colon so that Emacs
+ # searches 'Info-default-directory-list'.
+ export INFOPATH="$_GUIX_PROFILE/share/info:$INFOPATH"
+fi
+
+# GUIX_PROFILE: User's default profile
+GUIX_PROFILE="$HOME/.guix-profile"
+[ -L $GUIX_PROFILE ] || return
+GUIX_LOCPATH="$GUIX_PROFILE/lib/locale"
+export GUIX_PROFILE GUIX_LOCPATH
+
+[ -f "$GUIX_PROFILE/etc/profile" ] && . "$GUIX_PROFILE/etc/profile"
+
+# set XDG_DATA_DIRS to include Guix installations
+export XDG_DATA_DIRS="$GUIX_PROFILE/share:${XDG_DATA_DIRS:-/usr/local/share/:/usr/share/}"
+```
+
+Please note that this will not take effect until the next shell or desktop
+session (log out and log back in).
+
+### `guix pull` as root
+
+Before you do this, you need to read the section on [choosing your security
+model][security-model] and adjust `guix` and `guix-daemon` flags according to
+your choice, as invoking `guix pull` may pull substitutes from substitute
+servers (which you may not want).
+
+As mentioned in a previous section, Guix expects
+`${localstatedir}/guix/profiles/per-user/root/current-guix` to be populated with
+`root`'s Guix profile, `guix pull`-ed and built by some former version of Guix.
+However, this is not the case when we build from source. Therefore, we need to
+perform a `guix pull` as `root`:
+
+```sh
+sudo --login guix pull --branch=version-<latest-release-version>
+# or
+sudo --login guix pull --commit=<particular-commit>
+```
+
+`guix pull` is quite a long process (especially if you're using
+`--no-substitute`). If you encounter build problems, please refer to the
+[troubleshooting section](#troubleshooting).
+
+Note that running a bare `guix pull` with no commit or branch specified will
+pull the latest commit on Guix's master branch, which is likely fine, but not
+recommended.
+
+If you installed Guix from source, you may get an error like the following:
+```sh
+error: while creating symlink '/root/.config/guix/current' No such file or directory
+```
+To resolve this, simply:
+```
+sudo mkdir -p /root/.config/guix
+```
+Then try the `guix pull` command again.
+
+After the `guix pull` finishes successfully,
+`${localstatedir}/guix/profiles/per-user/root/current-guix` should be populated.
+
+#### Using the newly-pulled `guix` by restarting the daemon
+
+Depending on how you installed Guix, you should now make sure that your init
+scripts and service configurations point to the newly-pulled `guix-daemon`.
+
+##### If you built Guix from source
+
+If you followed the instructions for [fixing argv\[0\]][fix-argv0], you can now
+do the following:
+
+```sh
+systemctl stop guix-daemon-original
+systemctl disable guix-daemon-original
+
+systemctl enable guix-daemon
+systemctl start guix-daemon
+```
+
+##### If you installed Guix via the Debian/Ubuntu distribution packages
+
+You will need to create a `guix-daemon-latest` service which points to the new
+`guix` rather than a pinned one.
+
+```sh
+# Create guix-daemon-latest.service by modifying guix-daemon.service
+sed -E -e "s|/usr/bin/guix-daemon|/var/guix/profiles/per-user/root/current-guix/bin/guix-daemon|" /etc/systemd/system/guix-daemon.service > /lib/systemd/system/guix-daemon-latest.service
+chmod 664 /lib/systemd/system/guix-daemon-latest.service
+
+# Make systemd recognize the new service
+systemctl daemon-reload
+
+# Make sure that the old guix-daemon.service is stopped and disabled
+systemctl stop guix-daemon
+systemctl disable guix-daemon
+
+# Make sure that the new guix-daemon-latest.service is started and enabled
+systemctl enable guix-daemon-latest
+systemctl start guix-daemon-latest
+```
+
+##### If you installed Guix via lantw44's Arch Linux AUR package
+
+At the time of writing (July 5th, 2021) the systemd unit for "updated Guix" is
+`guix-daemon-latest.service`, therefore, you should do the following:
+
+```sh
+systemctl stop guix-daemon
+systemctl disable guix-daemon
+
+systemctl enable guix-daemon-latest
+systemctl start guix-daemon-latest
+```
+
+##### Otherwise...
+
+Simply do:
+
+```sh
+systemctl restart guix-daemon
+```
+
+### Checking everything
+
+If you followed all the steps above to make your Guix setup "prim and proper,"
+you can check that you did everything properly by running through this
+checklist.
+
+1. `/etc/profile.d/guix.sh` should exist and be sourced at each shell login
+
+2. `guix describe` should not print `guix describe: error: failed to determine
+ origin`, but rather something like:
+
+ ```
+ Generation 38 Feb 22 2021 16:39:31 (current)
+ guix f350df4
+ repository URL: https://git.savannah.gnu.org/git/guix.git
+ branch: version-1.2.0
+ commit: f350df405fbcd5b9e27e6b6aa500da7f101f41e7
+ ```
+
+3. `guix-daemon` should be running from `${localstatedir}/guix/profiles/per-user/root/current-guix`
+
+# Troubleshooting
+
+## Derivation failed to build
+
+When you see a build failure like below:
+
+```
+building /gnu/store/...-foo-3.6.12.drv...
+/ 'check' phasenote: keeping build directory `/tmp/guix-build-foo-3.6.12.drv-0'
+builder for `/gnu/store/...-foo-3.6.12.drv' failed with exit code 1
+build of /gnu/store/...-foo-3.6.12.drv failed
+View build log at '/var/log/guix/drvs/../...-foo-3.6.12.drv.bz2'.
+cannot build derivation `/gnu/store/...-qux-7.69.1.drv': 1 dependencies couldn't be built
+cannot build derivation `/gnu/store/...-bar-3.16.5.drv': 1 dependencies couldn't be built
+cannot build derivation `/gnu/store/...-baz-2.0.5.drv': 1 dependencies couldn't be built
+guix time-machine: error: build of `/gnu/store/...-baz-2.0.5.drv' failed
+```
+
+It means that `guix` failed to build a package named `foo`, which was a
+dependency of `qux`, `bar`, and `baz`. Importantly, note that the last "failed"
+line is not necessarily the root cause, the first "failed" line is.
+
+Most of the time, the build failure is due to a spurious test failure or the
+package's build system/test suite breaking when running multi-threaded. To
+rebuild _just_ this derivation in a single-threaded fashion (please don't forget
+to add other `guix` flags like `--no-substitutes` as appropriate):
+
+```sh
+$ guix build --cores=1 /gnu/store/...-foo-3.6.12.drv
+```
+
+If the single-threaded rebuild did not succeed, you may need to dig deeper.
+You may view `foo`'s build logs in `less` like so (please replace paths with the
+path you see in the build failure output):
+
+```sh
+$ bzcat /var/log/guix/drvs/../...-foo-3.6.12.drv.bz2 | less
+```
+
+`foo`'s build directory is also preserved and available at
+`/tmp/guix-build-foo-3.6.12.drv-0`. However, if you fail to build `foo` multiple
+times, it may be `/tmp/...drv-1` or `/tmp/...drv-2`. Always consult the build
+failure output for the most accurate, up-to-date information.
+
+### python(-minimal): [Errno 84] Invalid or incomplete multibyte or wide character
+
+This error occurs when your `$TMPDIR` (default: /tmp) exists on a filesystem
+which rejects characters not present in the UTF-8 character code set. An example
+is ZFS with the utf8only=on option set.
+
+More information: https://bugs.python.org/issue37584
+
+### GnuTLS: test-suite FAIL: status-request-revoked
+
+*The derivation is likely identified by: `/gnu/store/vhphki5sg9xkdhh2pbc8gi6vhpfzryf0-gnutls-3.6.12.drv`*
+
+This unfortunate error is most common for non-substitute builders who installed
+Guix v1.2.0. The problem stems from the fact that one of GnuTLS's tests uses a
+hardcoded certificate which expired on 2020-10-24.
+
+What's more unfortunate is that this GnuTLS derivation is somewhat special in
+Guix's dependency graph and is not affected by the package transformation flags
+like `--without-tests=`.
+
+The easiest solution for those encountering this problem is to install a newer
+version of Guix. However, there are ways to work around this issue:
+
+#### Workaround 1: Using substitutes for this single derivation
+
+If you've authorized the official Guix build farm's key (more info
+[here](./README.md#step-1-authorize-the-signing-keys)), then you can use
+substitutes just for this single derivation by invoking the following:
+
+```sh
+guix build --substitute-urls="https://ci.guix.gnu.org" /gnu/store/vhphki5sg9xkdhh2pbc8gi6vhpfzryf0-gnutls-3.6.12.drv
+```
+
+See [this section](./README.md#removing-authorized-keys) for instructions on how
+to remove authorized keys if you don't want to keep the build farm's key
+authorized.
+
+#### Workaround 2: Temporarily setting the system clock back
+
+This workaround was described [here](https://issues.guix.gnu.org/44559#5).
+
+Basically:
+1. Turn off networking
+2. Turn off NTP
+3. Set system time to 2020-10-01
+4. guix build --no-substitutes /gnu/store/vhphki5sg9xkdhh2pbc8gi6vhpfzryf0-gnutls-3.6.12.drv
+5. Set system time back to accurate current time
+6. Turn NTP back on
+7. Turn networking back on
+
+### coreutils: FAIL: tests/tail-2/inotify-dir-recreate
+
+The inotify-dir-create test fails on "remote" filesystems such as overlayfs
+(Docker's default filesystem) due to the filesystem being mistakenly recognized
+as non-remote.
+
+A relatively easy workaround to this is to make sure that a somewhat traditional
+filesystem is mounted at `/tmp` (where `guix-daemon` performs its builds). For
+Docker users, this might mean [using a volume][docker/volumes], [binding
+mounting][docker/bind-mnt] from host, or (for those with enough RAM and swap)
+[mounting a tmpfs][docker/tmpfs] using the `--tmpfs` flag.
+
+Please see the following links for more details:
+
+- An upstream coreutils bug has been filed: [debbugs#47940](https://debbugs.gnu.org/cgi/bugreport.cgi?bug=47940)
+- A Guix bug detailing the underlying problem has been filed: [guix-issues#47935](https://issues.guix.gnu.org/47935)
+- A commit to skip this test in Guix has been merged into the core-updates branch:
+[savannah/guix@6ba1058](https://git.savannah.gnu.org/cgit/guix.git/commit/?id=6ba1058df0c4ce5611c2367531ae5c3cdc729ab4)
+
+
+[install-script]: #options-1-and-2-using-the-official-shell-installer-script-or-binary-tarball
+[install-bin-tarball]: #options-1-and-2-using-the-official-shell-installer-script-or-binary-tarball
+[install-fanquake-docker]: #option-3-using-fanquakes-docker-image
+[install-distro-pkg]: #option-4-using-a-distribution-maintained-package
+[install-source]: #option-5-building-from-source
+
+[fix-argv0]: #creating-and-starting-a-guix-daemon-original-service-with-a-fixed-argv0
+[security-model]: ./README.md#choosing-your-security-model
+
+[docker/volumes]: https://docs.docker.com/storage/volumes/
+[docker/bind-mnt]: https://docs.docker.com/storage/bind-mounts/
+[docker/tmpfs]: https://docs.docker.com/storage/tmpfs/
diff --git a/contrib/guix/README.md b/contrib/guix/README.md
index e604b370e3..4680368a6f 100644
--- a/contrib/guix/README.md
+++ b/contrib/guix/README.md
@@ -9,83 +9,171 @@ downloads.
We achieve bootstrappability by using Guix as a functional package manager.
-## Requirements
+# Requirements
Conservatively, a x86_64 machine with:
- 16GB of free disk space on the partition that /gnu/store will reside in
-- 8GB of free disk space per platform triple you're planning on building (see
- the `HOSTS` environment variable description)
+- 8GB of free disk space **per platform triple** you're planning on building
+ (see the `HOSTS` [environment variable description][env-vars-list])
-## Setup
+# Installation and Setup
-### Installing Guix
+If you don't have Guix installed and set up, please follow the instructions in
+[INSTALL.md](./INSTALL.md)
-If you're just testing this out, you can use the
-[Dockerfile][fanquake/guix-docker] for convenience. It automatically speeds up
-your builds by [using substitutes](#speeding-up-builds-with-substitute-servers).
-If you don't want this behaviour, refer to the [next
-section](#choosing-your-security-model).
+# Usage
-Otherwise, follow the [Guix installation guide][guix/bin-install].
+If you haven't considered your security model yet, please read [the relevant
+section](#choosing-your-security-model) before proceeding to perform a build.
-> Note: For those who like to keep their filesystems clean, Guix is designed to
-> be very standalone and _will not_ conflict with your system's package
-> manager/existing setup. It _only_ touches `/var/guix`, `/gnu`, and
-> `~/.config/guix`.
+## Making the Xcode SDK available for macOS cross-compilation
-### Choosing your security model
+In order to perform a build for macOS (which is included in the default set of
+platform triples to build), you'll need to extract the macOS SDK tarball using
+tools found in the [`macdeploy` directory](../macdeploy/README.md).
-Guix allows us to achieve better binary security by using our CPU time to build
-everything from scratch. However, it doesn't sacrifice user choice in pursuit of
-this: users can decide whether or not to bootstrap and to use substitutes
-(pre-built packages).
+You can then either point to the SDK using the `SDK_PATH` environment variable:
-After installation, you may want to consider [adding substitute
-servers](#speeding-up-builds-with-substitute-servers) from which to download
-pre-built packages to speed up your build if that fits your security model (say,
-if you're just testing that this works). Substitute servers are set up by
-default if you're using the [Dockerfile][fanquake/guix-docker].
+```sh
+# Extract the SDK tarball to /path/to/parent/dir/of/extracted/SDK/Xcode-<foo>-<bar>-extracted-SDK-with-libcxx-headers
+tar -C /path/to/parent/dir/of/extracted/SDK -xaf /path/to/Xcode-<foo>-<bar>-extracted-SDK-with-libcxx-headers.tar.gz
-If you prefer not to use any substitutes, make sure to supply `--no-substitutes`
-like in the following snippet. The first build will take a while, but the
-resulting packages will be cached for future builds.
+# Indicate where to locate the SDK tarball
+export SDK_PATH=/path/to/parent/dir/of/extracted/SDK
+```
+
+or extract it into `depends/SDKs`:
```sh
-export ADDITIONAL_GUIX_COMMON_FLAGS='--no-substitutes'
+mkdir -p depends/SDKs
+tar -C depends/SDKs -xaf /path/to/SDK/tarball
```
-Likewise, to perform a bootstrapped build (takes even longer):
+## Building
+
+*The author highly recommends at least reading over the [common usage patterns
+and examples](#common-guix-build-invocation-patterns-and-examples) section below
+before starting a build. For a full list of customization options, see the
+[recognized environment variables][env-vars-list] section.*
+
+To build Bitcoin Core reproducibly with all default options, invoke the
+following from the top of a clean repository:
```sh
-export ADDITIONAL_GUIX_COMMON_FLAGS='--no-substitutes' ADDITIONAL_GUIX_ENVIRONMENT_FLAGS='--bootstrap'
+./contrib/guix/guix-build
```
-### Using a version of Guix with `guix time-machine` capabilities
+## Codesigning build outputs
-> Note: This entire section can be skipped if you are already using a version of
-> Guix that has [the `guix time-machine` command][guix/time-machine].
+The `guix-codesign` command attaches codesignatures (produced by codesigners) to
+existing non-codesigned outputs. Please see the [release process
+documentation](/doc/release-process.md) for more context.
-Once Guix is installed, if it doesn't have the `guix time-machine` command, pull
-the latest `guix`.
+It respects many of the same environment variable flags as `guix-build`, with 2
+crucial differences:
+
+1. Since only Windows and macOS build outputs require codesigning, the `HOSTS`
+ environment variable will have a sane default value of `x86_64-w64-mingw32
+ x86_64-apple-darwin18` instead of all the platforms.
+2. The `guix-codesign` command ***requires*** a `DETACHED_SIGS_REPO` flag.
+ * _**DETACHED_SIGS_REPO**_
+
+ Set the directory where detached codesignatures can be found for the current
+ Bitcoin Core version being built.
+
+ _REQUIRED environment variable_
+
+An invocation with all default options would look like:
+
+```
+env DETACHED_SIGS_REPO=<path/to/bitcoin-detached-sigs> ./contrib/guix-codesign
+```
+
+## Cleaning intermediate work directories
+
+By default, `guix-build` leaves all intermediate files or "work directories"
+(e.g. `depends/work`, `guix-build-*/distsrc-*`) intact at the end of a build so
+that they are available to the user (to aid in debugging, etc.). However, these
+directories usually take up a large amount of disk space. Therefore, a
+`guix-clean` convenience script is provided which cleans the current `git`
+worktree to save disk space:
+
+```
+./contrib/guix/guix-clean
+```
+
+
+## Attesting to build outputs
+
+Much like how Gitian build outputs are attested to in a `gitian.sigs`
+repository, Guix build outputs are attested to in the [`guix.sigs`
+repository](https://github.com/bitcoin-core/guix.sigs).
+
+After you've cloned the `guix.sigs` repository, to attest to the current
+worktree's commit/tag:
+
+```
+env GUIX_SIGS_REPO=<path/to/guix.sigs> SIGNER=<gpg-key-name> ./contrib/guix/guix-attest
+```
+
+See `./contrib/guix/guix-attest --help` for more information on the various ways
+`guix-attest` can be invoked.
+
+## Verifying build output attestations
+
+After at least one other signer has uploaded their signatures to the `guix.sigs`
+repository:
+
+```
+git -C <path/to/guix.sigs> pull
+env GUIX_SIGS_REPO=<path/to/guix.sigs> ./contrib/guix/guix-verify
+```
+
+
+## Common `guix-build` invocation patterns and examples
+
+### Keeping caches and SDKs outside of the worktree
+
+If you perform a lot of builds and have a bunch of worktrees, you may find it
+more efficient to keep the depends tree's download cache, build cache, and SDKs
+outside of the worktrees to avoid duplicate downloads and unnecessary builds. To
+help with this situation, the `guix-build` script honours the `SOURCES_PATH`,
+`BASE_CACHE`, and `SDK_PATH` environment variables and will pass them on to the
+depends tree so that you can do something like:
```sh
-guix pull --max-jobs=4 # change number of jobs accordingly
+env SOURCES_PATH="$HOME/depends-SOURCES_PATH" BASE_CACHE="$HOME/depends-BASE_CACHE" SDK_PATH="$HOME/macOS-SDKs" ./contrib/guix/guix-build
```
-Make sure that you are using your current profile. (You are prompted to do this
-at the end of the `guix pull`)
+Note that the paths that these environment variables point to **must be
+directories**, and **NOT symlinks to directories**.
+
+See the [recognized environment variables][env-vars-list] section for more
+details.
+
+### Building a subset of platform triples
-```bash
-export PATH="${HOME}/.config/guix/current/bin${PATH:+:}$PATH"
+Sometimes you only want to build a subset of the supported platform triples, in
+which case you can override the default list by setting the space-separated
+`HOSTS` environment variable:
+
+```sh
+env HOSTS='x86_64-w64-mingw32 x86_64-apple-darwin18' ./contrib/guix/guix-build
```
+See the [recognized environment variables][env-vars-list] section for more
+details.
+
### Controlling the number of threads used by `guix` build commands
+Depending on your system's RAM capacity, you may want to decrease the number of
+threads used to decrease RAM usage or vice versa.
+
By default, the scripts under `./contrib/guix` will invoke all `guix` build
commands with `--cores="$JOBS"`. Note that `$JOBS` defaults to `$(nproc)` if not
-specified. However, astute manual readers will also notice that there is a
-`--max-jobs=` flag (which defaults to 1 if unspecified).
+specified. However, astute manual readers will also notice that `guix` build
+commands also accept a `--max-jobs=` flag (which defaults to 1 if unspecified).
Here is the difference between `--cores=` and `--max-jobs=`:
@@ -124,30 +212,18 @@ packages when the dependency graph allows for it, you may want to try:
export JOBS=1 ADDITIONAL_GUIX_COMMON_FLAGS='--max-jobs=8'
```
-## Usage
-
-### As a Tool for Deterministic Builds
-
-From the top of a clean Bitcoin Core repository:
-
-```sh
-./contrib/guix/guix-build
-```
-
-After the build finishes successfully (check the status code please), compare
-hashes:
-
-```sh
-find output/ -type f -print0 | sort -z | xargs -r0 sha256sum
-```
+See the [recognized environment variables][env-vars-list] section for more
+details.
-#### Recognized environment variables
+## Recognized environment variables
* _**HOSTS**_
Override the space-separated list of platform triples for which to perform a
- bootstrappable build. _(defaults to "x86\_64-linux-gnu arm-linux-gnueabihf
- aarch64-linux-gnu riscv64-linux-gnu powerpc64-linux-gnu powerpc64le-linux-gnu
+ bootstrappable build.
+
+ _(defaults to "x86\_64-linux-gnu arm-linux-gnueabihf aarch64-linux-gnu
+ riscv64-linux-gnu powerpc64-linux-gnu powerpc64le-linux-gnu
x86\_64-w64-mingw32 x86\_64-apple-darwin18")_
* _**SOURCES_PATH**_
@@ -156,18 +232,27 @@ find output/ -type f -print0 | sort -z | xargs -r0 sha256sum
depends tree. Setting this to the same directory across multiple builds of the
depends tree can eliminate unnecessary redownloading of package sources.
+ The path that this environment variable points to **must be a directory**, and
+ **NOT a symlink to a directory**.
+
* _**BASE_CACHE**_
Set the depends tree cache for built packages. This is passed through to the
depends tree. Setting this to the same directory across multiple builds of the
depends tree can eliminate unnecessary building of packages.
+ The path that this environment variable points to **must be a directory**, and
+ **NOT a symlink to a directory**.
+
* _**SDK_PATH**_
Set the path where _extracted_ SDKs can be found. This is passed through to
the depends tree. Note that this is should be set to the _parent_ directory of
- the actual SDK (e.g. SDK_PATH=$HOME/Downloads/macOS-SDKs instead of
- $HOME/Downloads/macOS-SDKs/Xcode-12.1-12A7403-extracted-SDK-with-libcxx-headers).
+ the actual SDK (e.g. `SDK_PATH=$HOME/Downloads/macOS-SDKs` instead of
+ `$HOME/Downloads/macOS-SDKs/Xcode-12.1-12A7403-extracted-SDK-with-libcxx-headers`).
+
+ The path that this environment variable points to **must be a directory**, and
+ **NOT a symlink to a directory**.
* _**JOBS**_
@@ -178,13 +263,17 @@ find output/ -type f -print0 | sort -z | xargs -r0 sha256sum
- `make` as in `make --jobs="$JOBS"`
- `xargs` as in `xargs -P"$JOBS"`
+ See [here](#controlling-the-number-of-threads-used-by-guix-build-commands) for
+ more details.
+
_(defaults to the value of `nproc` outside the container)_
* _**SOURCE_DATE_EPOCH**_
Override the reference UNIX timestamp used for bit-for-bit reproducibility,
- the variable name conforms to [standard][r12e/source-date-epoch]. _(defaults
- to the output of `$(git log --format=%at -1)`)_
+ the variable name conforms to [standard][r12e/source-date-epoch].
+
+ _(defaults to the output of `$(git log --format=%at -1)`)_
* _**V**_
@@ -200,8 +289,7 @@ find output/ -type f -print0 | sort -z | xargs -r0 sha256sum
A whitespace-delimited list of URLs from which to download pre-built packages.
A URL is only used if its signing key is authorized (refer to the [substitute
- servers section](#speeding-up-builds-with-substitute-servers) for more
- details).
+ servers section](#option-1-building-with-substitutes) for more details).
* _**ADDITIONAL_GUIX_COMMON_FLAGS**_
@@ -216,28 +304,65 @@ find output/ -type f -print0 | sort -z | xargs -r0 sha256sum
Additional flags to be passed to the invocation of `guix environment` inside
`guix time-machine`.
-## Tips and Tricks
-
-### Speeding up builds with substitute servers
+# Choosing your security model
-_This whole section is automatically done in the convenience
-[Dockerfiles][fanquake/guix-docker]_
+No matter how you installed Guix, you need to decide on your security model for
+building packages with Guix.
-For those who are used to life in the fast _(and trustful)_ lane, you can
-specify [substitute servers][guix/substitutes] from which to download pre-built
-packages.
+Guix allows us to achieve better binary security by using our CPU time to build
+everything from scratch. However, it doesn't sacrifice user choice in pursuit of
+this: users can decide whether or not to use **substitutes** (pre-built
+packages).
+
+## Option 1: Building with substitutes
+
+### Step 1: Authorize the signing keys
+
+Depending on the installation procedure you followed, you may have already
+authorized the Guix build farm key. In particular, the official shell installer
+script asks you if you want the key installed, and the debian distribution
+package authorized the key during installation.
+
+You can check the current list of authorized keys at `/etc/guix/acl`.
+
+At the time of writing, a `/etc/guix/acl` with just the Guix build farm key
+authorized looks something like:
+
+```lisp
+(acl
+ (entry
+ (public-key
+ (ecc
+ (curve Ed25519)
+ (q #8D156F295D24B0D9A86FA5741A840FF2D24F60F7B6C4134814AD55625971B394#)
+ )
+ )
+ (tag
+ (guix import)
+ )
+ )
+ )
+```
-> For those who only want to use substitutes from the official Guix build farm
-> and have authorized the build farm's signing key during Guix's installation,
-> you don't need to do anything.
+If you've determined that the official Guix build farm key hasn't been
+authorized, and you would like to authorize it, run the following as root:
-#### Step 1: Authorize the signing keys
+```
+guix archive --authorize < /var/guix/profiles/per-user/root/current-guix/share/guix/ci.guix.gnu.org.pub
+```
-For the official Guix build farm at https://ci.guix.gnu.org, run as root:
+If
+`/var/guix/profiles/per-user/root/current-guix/share/guix/ci.guix.gnu.org.pub`
+doesn't exist, try:
+```sh
+guix archive --authorize < <PREFIX>/share/guix/ci.guix.gnu.org.pub
```
-guix archive --authorize < ~root/.config/guix/current/share/guix/ci.guix.gnu.org.pub
-```
+
+Where `<PREFIX>` is likely:
+- `/usr` if you installed from a distribution package
+- `/usr/local` if you installed Guix from source and didn't supply any
+ prefix-modifying flags to Guix's `./configure`
For dongcarl's substitute server at https://guix.carldong.io, run as root:
@@ -245,90 +370,103 @@ For dongcarl's substitute server at https://guix.carldong.io, run as root:
wget -qO- 'https://guix.carldong.io/signing-key.pub' | guix archive --authorize
```
-#### Step 2: Specify the substitute servers
+#### Removing authorized keys
-The official Guix build farm at https://ci.guix.gnu.org is automatically used
-unless the `--no-substitutes` flag is supplied.
+To remove previously authorized keys, simply edit `/etc/guix/acl` and remove the
+`(entry (public-key ...))` entry.
-This can be overridden for all `guix` invocations by passing the
-`--substitute-urls` option to your invocation of `guix-daemon`. This can also be
-overridden on a call-by-call basis by passing the same `--substitute-urls`
-option to client tools such at `guix environment`.
+### Step 2: Specify the substitute servers
-To use dongcarl's substitute server for Bitcoin Core builds after having
-[authorized his signing key](#authorize-the-signing-keys):
+Once its key is authorized, the official Guix build farm at
+https://ci.guix.gnu.org is automatically used unless the `--no-substitutes` flag
+is supplied. This default list of substitute servers is overridable both on a
+`guix-daemon` level and when you invoke `guix` commands. See examples below for
+the various ways of adding dongcarl's substitute server after having [authorized
+his signing key](#authorize-the-signing-keys).
-```
-export SUBSTITUTE_URLS='https://guix.carldong.io https://ci.guix.gnu.org'
+Change the **default list** of substitute servers by starting `guix-daemon` with
+the `--substitute-urls` option (you will likely need to edit your init script):
+
+```sh
+guix-daemon <cmd> --substitute-urls='https://guix.carldong.io https://ci.guix.gnu.org'
```
-## Troubleshooting
+Override the default list of substitute servers by passing the
+`--substitute-urls` option for invocations of `guix` commands:
-### Derivation failed to build
+```sh
+guix <cmd> --substitute-urls='https://guix.carldong.io https://ci.guix.gnu.org'
+```
-When you see a build failure like below:
+For scripts under `./contrib/guix`, set the `SUBSTITUTE_URLS` environment
+variable:
-```
-building /gnu/store/...-foo-3.6.12.drv...
-/ 'check' phasenote: keeping build directory `/tmp/guix-build-foo-3.6.12.drv-0'
-builder for `/gnu/store/...-foo-3.6.12.drv' failed with exit code 1
-build of /gnu/store/...-foo-3.6.12.drv failed
-View build log at '/var/log/guix/drvs/../...-foo-3.6.12.drv.bz2'.
-cannot build derivation `/gnu/store/...-qux-7.69.1.drv': 1 dependencies couldn't be built
-cannot build derivation `/gnu/store/...-bar-3.16.5.drv': 1 dependencies couldn't be built
-cannot build derivation `/gnu/store/...-baz-2.0.5.drv': 1 dependencies couldn't be built
-guix time-machine: error: build of `/gnu/store/...-baz-2.0.5.drv' failed
+```sh
+export SUBSTITUTE_URLS='https://guix.carldong.io https://ci.guix.gnu.org'
```
-It means that `guix` failed to build a package named `foo`, which was a
-dependency of `qux`, `bar`, and `baz`. Importantly, note that the last "failed"
-line is not necessarily the root cause, the first "failed" line is.
+## Option 2: Disabling substitutes on an ad-hoc basis
-Most of the time, the build failure is due to a spurious test failure or the
-package's build system/test suite breaking when running multi-threaded. To
-rebuild _just_ this derivation in a single-threaded fashion:
+If you prefer not to use any substitutes, make sure to supply `--no-substitutes`
+like in the following snippet. The first build will take a while, but the
+resulting packages will be cached for future builds.
+For direct invocations of `guix`:
```sh
-$ guix build --cores=1 /gnu/store/...-foo-3.6.12.drv
+guix <cmd> --no-substitutes
```
-If the single-threaded rebuild did not succeed, you may need to dig deeper.
-You may view `foo`'s build logs in `less` like so (please replace paths with the
-path you see in the build failure output):
-
+For the scripts under `./contrib/guix/`:
```sh
-$ bzcat /var/log/guix/drvs/../...-foo-3.6.12.drv.bz2 | less
+export ADDITIONAL_GUIX_COMMON_FLAGS='--no-substitutes'
```
-`foo`'s build directory is also preserved and available at
-`/tmp/guix-build-foo-3.6.12.drv-0`. However, if you fail to build `foo` multiple
-times, it may be `/tmp/...drv-1` or `/tmp/...drv-2`. Always consult the build
-failure output for the most accurate, up-to-date information.
+## Option 3: Disabling substitutes by default
+
+`guix-daemon` accepts a `--no-substitutes` flag, which will make sure that,
+unless otherwise overridden by a command line invocation, no substitutes will be
+used.
-#### python(-minimal): [Errno 84] Invalid or incomplete multibyte or wide character
+If you start `guix-daemon` using an init script, you can edit said script to
+supply this flag.
-This error occurs when your `$TMPDIR` (default: /tmp) exists on a filesystem
-which rejects characters not present in the UTF-8 character code set. An example
-is ZFS with the utf8only=on option set.
-More information: https://bugs.python.org/issue37584
+# Purging/Uninstalling Guix
-## FAQ
+In the extraordinarily rare case where you messed up your Guix installation in
+an irreversible way, you may want to completely purge Guix from your system and
+start over.
-### How can I trust the binary installation?
+1. Uninstall Guix itself according to the way you installed it. (e.g. `sudo apt
+ purge guix` for Ubuntu packaging, `sudo make uninstall` for
+ built-from-source).
+2. Remove all build users and groups
-As mentioned at the bottom of [this manual page][guix/bin-install]:
+ You may check for relevant users and groups using:
-> The binary installation tarballs can be (re)produced and verified simply by
-> running the following command in the Guix source tree:
->
-> make guix-binary.x86_64-linux.tar.xz
+ ```
+ getent passwd | grep guix
+ getent group | grep guix
+ ```
-### Is Guix packaged in my operating system?
+ Then, you may remove users and groups using:
-Guix is shipped starting with [Debian Bullseye][debian/guix-bullseye] and
-[Ubuntu 21.04 "Hirsute Hippo"][ubuntu/guix-hirsute]. Other operating systems
-are working on packaging Guix as well.
+ ```
+ sudo userdel <user>
+ sudo groupdel <group>
+ ```
+
+3. Remove all possible Guix-related directories
+ - `/var/guix/`
+ - `/var/log/guix/`
+ - `/gnu/`
+ - `/etc/guix/`
+ - `/home/*/.config/guix/`
+ - `/home/*/.cache/guix/`
+ - `/home/*/.guix-profile/`
+ - `/root/.config/guix/`
+ - `/root/.cache/guix/`
+ - `/root/.guix-profile/`
[b17e]: http://bootstrappable.org/
[r12e/source-date-epoch]: https://reproducible-builds.org/docs/source-date-epoch/
@@ -343,3 +481,5 @@ are working on packaging Guix as well.
[debian/guix-bullseye]: https://packages.debian.org/bullseye/guix
[ubuntu/guix-hirsute]: https://packages.ubuntu.com/hirsute/guix
[fanquake/guix-docker]: https://github.com/fanquake/core-review/tree/master/guix
+
+[env-vars-list]: #recognized-environment-variables
diff --git a/contrib/guix/guix-attest b/contrib/guix/guix-attest
index c8cf73d400..51d589c1de 100755
--- a/contrib/guix/guix-attest
+++ b/contrib/guix/guix-attest
@@ -18,7 +18,7 @@ source "$(dirname "${BASH_SOURCE[0]}")/libexec/prelude.bash"
# Required non-builtin commands should be invokable
################
-check_tools cat env basename mkdir xargs find
+check_tools cat env basename mkdir diff sort
if [ -z "$NO_SIGN" ]; then
check_tools gpg
fi
@@ -159,6 +159,20 @@ Hint: You may wish to remove the existing attestations and their signatures by
EOF
}
+# Given a document with unix line endings (just <LF>) in stdin, make all lines
+# end in <CR><LF> and make sure there's no trailing <LF> at the end of the file.
+#
+# This is necessary as cleartext signatures are calculated on text after their
+# line endings are canonicalized.
+#
+# For more information:
+# 1. https://security.stackexchange.com/a/104261
+# 2. https://datatracker.ietf.org/doc/html/rfc4880#section-7.1
+#
+rfc4880_normalize_document() {
+ sed 's/$/\r/' | head -c -2
+}
+
echo "Attesting to build outputs for version: '${VERSION}'"
echo ""
@@ -174,7 +188,8 @@ mkdir -p "$outsigdir"
cat "${noncodesigned_fragments[@]}" \
| sort -u \
| sort -k2 \
- > "$temp_noncodesigned"
+ | rfc4880_normalize_document \
+ > "$temp_noncodesigned"
if [ -e noncodesigned.SHA256SUMS ]; then
# The SHA256SUMS already exists, make sure it's exactly what we
# expect, error out if not
@@ -201,7 +216,9 @@ mkdir -p "$outsigdir"
cat "${sha256sum_fragments[@]}" \
| sort -u \
| sort -k2 \
- > "$temp_codesigned"
+ | sed 's/$/\r/' \
+ | rfc4880_normalize_document \
+ > "$temp_codesigned"
if [ -e codesigned.SHA256SUMS ]; then
# The SHA256SUMS already exists, make sure it's exactly what we
# expect, error out if not
@@ -226,6 +243,7 @@ mkdir -p "$outsigdir"
for i in *.SHA256SUMS; do
if [ ! -e "$i".asc ]; then
gpg --detach-sign \
+ --digest-algo sha256 \
--local-user "$gpg_key_name" \
--armor \
--output "$i".asc "$i"
diff --git a/contrib/guix/guix-build b/contrib/guix/guix-build
index 29d6701b25..de53530e8f 100755
--- a/contrib/guix/guix-build
+++ b/contrib/guix/guix-build
@@ -18,7 +18,7 @@ source "$(dirname "${BASH_SOURCE[0]}")/libexec/prelude.bash"
# Required non-builtin commands should be invocable
################
-check_tools cat mkdir make git guix
+check_tools cat mkdir make getent curl git guix
################
# GUIX_BUILD_OPTIONS should be empty
@@ -186,6 +186,29 @@ fi
#
# However, the internal API is likely to change more than the CLI invocation
+################
+# Services database must have basic entries
+################
+
+if ! getent services http https ftp; then
+cat << EOF
+ERR: Your system's C library can not find service database entries for at least
+ one of the following services: http, https, ftp.
+
+Hint: Most likely, /etc/services does not exist yet (common for docker images
+ and minimal distros), or you don't have permissions to access it.
+
+ If /etc/services does not exist yet, you may want to install the
+ appropriate package for your distro which provides it.
+
+ On Debian/Ubuntu: netbase
+ On Arch Linux: iana-etc
+
+ For more information, see: getent(1), services(5)
+
+EOF
+
+fi
#########
# SETUP #
diff --git a/doc/release-process.md b/doc/release-process.md
index 75a574ee31..0ac67b9146 100644
--- a/doc/release-process.md
+++ b/doc/release-process.md
@@ -44,10 +44,6 @@ Release Process
- Make an announcement that translators can start translating for the new version. You can use one of the [previous announcements](https://www.transifex.com/bitcoin/bitcoin/announcements/) as a template.
- Change the auto-update URL for the resource to `master`, e.g. `https://raw.githubusercontent.com/bitcoin/bitcoin/master/src/qt/locale/bitcoin_en.xlf`. (Do this only after the previous steps, to prevent an auto-update from interfering.)
-#### After branch-off (on master)
-
-- Update the version of `contrib/gitian-descriptors/*.yml`.
-
#### After branch-off (on the major release branch)
- Update the versions.
@@ -72,14 +68,14 @@ This will perform a few last-minute consistency checks in the build system files
### First time / New builders
-If you're using the automated script (found in [contrib/gitian-build.py](/contrib/gitian-build.py)), then at this point you should run it with the "--setup" command. Otherwise ignore this.
+Install Guix using one of the installation methods detailed in
+[contrib/guix/INSTALL.md](/contrib/guix/INSTALL.md).
Check out the source code in the following directory hierarchy.
cd /path/to/your/toplevel/build
- git clone https://github.com/bitcoin-core/gitian.sigs.git
+ git clone https://github.com/bitcoin-core/guix.sigs.git
git clone https://github.com/bitcoin-core/bitcoin-detached-sigs.git
- git clone https://github.com/devrandom/gitian-builder.git
git clone https://github.com/bitcoin/bitcoin.git
### Write the release notes
@@ -94,110 +90,56 @@ Generate list of authors:
git log --format='- %aN' v(current version, e.g. 0.20.0)..v(new version, e.g. 0.20.1) | sort -fiu
-### Setup and perform Gitian builds
-
-If you're using the automated script (found in [contrib/gitian-build.py](/contrib/gitian-build.py)), then at this point you should run it with the "--build" command. Otherwise ignore this.
-
-Setup Gitian descriptors:
-
- pushd ./bitcoin
- export SIGNER="(your Gitian key, ie bluematt, sipa, etc)"
- export VERSION=(new version, e.g. 0.20.0)
- git fetch
- git checkout v${VERSION}
- popd
-
-Ensure your gitian.sigs are up-to-date if you wish to gverify your builds against other Gitian signatures.
-
- pushd ./gitian.sigs
- git pull
- popd
-
-Ensure gitian-builder is up-to-date:
-
- pushd ./gitian-builder
- git pull
- popd
-
-### Fetch and create inputs: (first time, or when dependency versions change)
-
- pushd ./gitian-builder
- mkdir -p inputs
- wget -O inputs/osslsigncode-2.0.tar.gz https://github.com/mtrojnar/osslsigncode/archive/2.0.tar.gz
- echo '5a60e0a4b3e0b4d655317b2f12a810211c50242138322b16e7e01c6fbb89d92f inputs/osslsigncode-2.0.tar.gz' | sha256sum -c
- popd
-
-Create the macOS SDK tarball, see the [macdeploy instructions](/contrib/macdeploy/README.md#deterministic-macos-dmg-notes) for details, and copy it into the inputs directory.
-
-### Optional: Seed the Gitian sources cache and offline git repositories
-
-NOTE: Gitian is sometimes unable to download files. If you have errors, try the step below.
-
-By default, Gitian will fetch source files as needed. To cache them ahead of time, make sure you have checked out the tag you want to build in bitcoin, then:
-
- pushd ./gitian-builder
- make -C ../bitcoin/depends download SOURCES_PATH=`pwd`/cache/common
- popd
-
-Only missing files will be fetched, so this is safe to re-run for each build.
-
-NOTE: Offline builds must use the --url flag to ensure Gitian fetches only from local URLs. For example:
-
- pushd ./gitian-builder
- ./bin/gbuild --url bitcoin=/path/to/bitcoin,signature=/path/to/sigs {rest of arguments}
- popd
+### Setup and perform Guix builds
-The gbuild invocations below <b>DO NOT DO THIS</b> by default.
+Checkout the Bitcoin Core version you'd like to build:
-### Build and sign Bitcoin Core for Linux, Windows, and macOS:
+```sh
+pushd ./bitcoin
+SIGNER='(your builder key, ie bluematt, sipa, etc)'
+VERSION='(new version without v-prefix, e.g. 0.20.0)'
+git fetch "v${VERSION}"
+git checkout "v${VERSION}"
+popd
+```
- pushd ./gitian-builder
- ./bin/gbuild --num-make 2 --memory 3000 --commit bitcoin=v${VERSION} ../bitcoin/contrib/gitian-descriptors/gitian-linux.yml
- ./bin/gsign --signer "$SIGNER" --release ${VERSION}-linux --destination ../gitian.sigs/ ../bitcoin/contrib/gitian-descriptors/gitian-linux.yml
- mv build/out/bitcoin-*.tar.gz build/out/src/bitcoin-*.tar.gz ../
+Ensure your guix.sigs are up-to-date if you wish to `guix-verify` your builds
+against other `guix-attest` signatures.
- ./bin/gbuild --num-make 2 --memory 3000 --commit bitcoin=v${VERSION} ../bitcoin/contrib/gitian-descriptors/gitian-win.yml
- ./bin/gsign --signer "$SIGNER" --release ${VERSION}-win-unsigned --destination ../gitian.sigs/ ../bitcoin/contrib/gitian-descriptors/gitian-win.yml
- mv build/out/bitcoin-*-win-unsigned.tar.gz inputs/bitcoin-win-unsigned.tar.gz
- mv build/out/bitcoin-*.zip build/out/bitcoin-*.exe ../
+```sh
+git -C ./guix.sigs pull
+```
- ./bin/gbuild --num-make 2 --memory 3000 --commit bitcoin=v${VERSION} ../bitcoin/contrib/gitian-descriptors/gitian-osx.yml
- ./bin/gsign --signer "$SIGNER" --release ${VERSION}-osx-unsigned --destination ../gitian.sigs/ ../bitcoin/contrib/gitian-descriptors/gitian-osx.yml
- mv build/out/bitcoin-*-osx-unsigned.tar.gz inputs/bitcoin-osx-unsigned.tar.gz
- mv build/out/bitcoin-*.tar.gz build/out/bitcoin-*.dmg ../
- popd
+### Create the macOS SDK tarball: (first time, or when SDK version changes)
-Build output expected:
+Create the macOS SDK tarball, see the [macdeploy
+instructions](/contrib/macdeploy/README.md#deterministic-macos-dmg-notes) for
+details.
- 1. source tarball (`bitcoin-${VERSION}.tar.gz`)
- 2. linux 32-bit and 64-bit dist tarballs (`bitcoin-${VERSION}-linux[32|64].tar.gz`)
- 3. windows 32-bit and 64-bit unsigned installers and dist zips (`bitcoin-${VERSION}-win[32|64]-setup-unsigned.exe`, `bitcoin-${VERSION}-win[32|64].zip`)
- 4. macOS unsigned installer and dist tarball (`bitcoin-${VERSION}-osx-unsigned.dmg`, `bitcoin-${VERSION}-osx64.tar.gz`)
- 5. Gitian signatures (in `gitian.sigs/${VERSION}-<linux|{win,osx}-unsigned>/(your Gitian key)/`)
+### Build and attest to build outputs:
-### Verify other gitian builders signatures to your own. (Optional)
+Follow the relevant Guix README.md sections:
+- [Performing a build](/contrib/guix/README.md#performing-a-build)
+- [Attesting to build outputs](/contrib/guix/README.md#attesting-to-build-outputs)
-Add other gitian builders keys to your gpg keyring, and/or refresh keys: See `../bitcoin/contrib/gitian-keys/README.md`.
+### Verify other builders' signatures to your own. (Optional)
-Verify the signatures
+Add other builders keys to your gpg keyring, and/or refresh keys: See `../bitcoin/contrib/builder-keys/README.md`.
- pushd ./gitian-builder
- ./bin/gverify -v -d ../gitian.sigs/ -r ${VERSION}-linux ../bitcoin/contrib/gitian-descriptors/gitian-linux.yml
- ./bin/gverify -v -d ../gitian.sigs/ -r ${VERSION}-win-unsigned ../bitcoin/contrib/gitian-descriptors/gitian-win.yml
- ./bin/gverify -v -d ../gitian.sigs/ -r ${VERSION}-osx-unsigned ../bitcoin/contrib/gitian-descriptors/gitian-osx.yml
- popd
+Follow the relevant Guix README.md sections:
+- [Verifying build output attestations](/contrib/guix/README.md#verifying-build-output-attestations)
### Next steps:
-Commit your signature to gitian.sigs:
+Commit your signature to guix.sigs:
- pushd gitian.sigs
- git add ${VERSION}-linux/"${SIGNER}"
- git add ${VERSION}-win-unsigned/"${SIGNER}"
- git add ${VERSION}-osx-unsigned/"${SIGNER}"
- git commit -m "Add ${VERSION} unsigned sigs for ${SIGNER}"
- git push # Assuming you can push to the gitian.sigs tree
- popd
+```sh
+pushd ./guix.sigs
+git add "${VERSION}/${SIGNER}"/noncodesigned.SHA256SUMS{,.asc}
+git commit -m "Add ${VERSION} unsigned sigs for ${SIGNER}"
+git push # Assuming you can push to the guix.sigs tree
+popd
+```
Codesigner only: Create Windows/macOS detached signatures:
- Only one person handles codesigning. Everyone else should skip to the next step.
@@ -209,7 +151,7 @@ Codesigner only: Sign the macOS binary:
tar xf bitcoin-osx-unsigned.tar.gz
./detached-sig-create.sh -s "Key ID"
Enter the keychain password and authorize the signature
- Move signature-osx.tar.gz back to the gitian host
+ Move signature-osx.tar.gz back to the guix-build host
Codesigner only: Sign the windows binaries:
@@ -220,93 +162,88 @@ Codesigner only: Sign the windows binaries:
Codesigner only: Commit the detached codesign payloads:
- cd ~/bitcoin-detached-sigs
- checkout the appropriate branch for this release series
- rm -rf *
- tar xf signature-osx.tar.gz
- tar xf signature-win.tar.gz
- git add -A
- git commit -m "point to ${VERSION}"
- git tag -s v${VERSION} HEAD
- git push the current branch and new tag
+```sh
+pushd ./bitcoin-detached-sigs
+# checkout the appropriate branch for this release series
+rm -rf ./*
+tar xf signature-osx.tar.gz
+tar xf signature-win.tar.gz
+git add -A
+git commit -m "point to ${VERSION}"
+git tag -s "v${VERSION}" HEAD
+git push the current branch and new tag
+popd
+```
Non-codesigners: wait for Windows/macOS detached signatures:
- Once the Windows/macOS builds each have 3 matching signatures, they will be signed with their respective release keys.
- Detached signatures will then be committed to the [bitcoin-detached-sigs](https://github.com/bitcoin-core/bitcoin-detached-sigs) repository, which can be combined with the unsigned apps to create signed binaries.
-Create (and optionally verify) the signed macOS binary:
+Create (and optionally verify) the codesigned outputs:
- pushd ./gitian-builder
- ./bin/gbuild -i --commit signature=v${VERSION} ../bitcoin/contrib/gitian-descriptors/gitian-osx-signer.yml
- ./bin/gsign --signer "$SIGNER" --release ${VERSION}-osx-signed --destination ../gitian.sigs/ ../bitcoin/contrib/gitian-descriptors/gitian-osx-signer.yml
- ./bin/gverify -v -d ../gitian.sigs/ -r ${VERSION}-osx-signed ../bitcoin/contrib/gitian-descriptors/gitian-osx-signer.yml
- mv build/out/bitcoin-osx-signed.dmg ../bitcoin-${VERSION}-osx.dmg
- popd
+- [Codesigning](/contrib/guix/README.md#codesigning)
-Create (and optionally verify) the signed Windows binaries:
+Commit your signature for the signed macOS/Windows binaries:
- pushd ./gitian-builder
- ./bin/gbuild -i --commit signature=v${VERSION} ../bitcoin/contrib/gitian-descriptors/gitian-win-signer.yml
- ./bin/gsign --signer "$SIGNER" --release ${VERSION}-win-signed --destination ../gitian.sigs/ ../bitcoin/contrib/gitian-descriptors/gitian-win-signer.yml
- ./bin/gverify -v -d ../gitian.sigs/ -r ${VERSION}-win-signed ../bitcoin/contrib/gitian-descriptors/gitian-win-signer.yml
- mv build/out/bitcoin-*win64-setup.exe ../bitcoin-${VERSION}-win64-setup.exe
- popd
+```sh
+pushd ./guix.sigs
+git add "${VERSION}/${SIGNER}"/all.SHA256SUMS{,.asc}
+git commit -m "Add ${SIGNER} ${VERSION} signed binaries signatures"
+git push # Assuming you can push to the guix.sigs tree
+popd
+```
-Commit your signature for the signed macOS/Windows binaries:
+### After 3 or more people have guix-built and their results match:
- pushd gitian.sigs
- git add ${VERSION}-osx-signed/"${SIGNER}"
- git add ${VERSION}-win-signed/"${SIGNER}"
- git commit -m "Add ${SIGNER} ${VERSION} signed binaries signatures"
- git push # Assuming you can push to the gitian.sigs tree
- popd
+Combine `all.SHA256SUMS` and `all.SHA256SUMS.asc` into a clear-signed
+`SHA256SUMS.asc` message:
-### After 3 or more people have gitian-built and their results match:
+```sh
+echo -e "-----BEGIN PGP SIGNED MESSAGE-----\nHash: SHA256\n\n$(cat all.SHA256SUMS)\n$(cat filename.txt.asc)" > SHA256SUMS.asc
+```
-- Create `SHA256SUMS.asc` for the builds, and GPG-sign it:
+Here's an equivalent, more readable command if you're confident that you won't
+mess up whitespaces when copy-pasting:
```bash
-sha256sum * > SHA256SUMS
-```
+cat << EOF > SHA256SUMS.asc
+-----BEGIN PGP SIGNED MESSAGE-----
+Hash: SHA256
-The list of files should be:
+$(cat all.SHA256SUMS)
+$(cat all.SHA256SUMS.asc)
+EOF
```
-bitcoin-${VERSION}-aarch64-linux-gnu.tar.gz
-bitcoin-${VERSION}-arm-linux-gnueabihf.tar.gz
-bitcoin-${VERSION}-riscv64-linux-gnu.tar.gz
-bitcoin-${VERSION}-x86_64-linux-gnu.tar.gz
-bitcoin-${VERSION}-osx64.tar.gz
-bitcoin-${VERSION}-osx.dmg
-bitcoin-${VERSION}.tar.gz
-bitcoin-${VERSION}-win64-setup.exe
-bitcoin-${VERSION}-win64.zip
-```
-The `*-debug*` files generated by the gitian build contain debug symbols
-for troubleshooting by developers. It is assumed that anyone that is interested
-in debugging can run gitian to generate the files for themselves. To avoid
-end-user confusion about which file to pick, as well as save storage
-space *do not upload these to the bitcoincore.org server, nor put them in the torrent*.
-- GPG-sign it, delete the unsigned file:
-```
-gpg --digest-algo sha256 --clearsign SHA256SUMS # outputs SHA256SUMS.asc
-rm SHA256SUMS
-```
-(the digest algorithm is forced to sha256 to avoid confusion of the `Hash:` header that GPG adds with the SHA256 used for the files)
-Note: check that SHA256SUMS itself doesn't end up in SHA256SUMS, which is a spurious/nonsensical entry.
+- Upload to the bitcoincore.org server (`/var/www/bin/bitcoin-core-${VERSION}`):
+ 1. The contents of `./bitcoin/guix-build-${VERSION}/output`, except for
+ `*-debug*` files.
-- Upload zips and installers, as well as `SHA256SUMS.asc` from last step, to the bitcoincore.org server
- into `/var/www/bin/bitcoin-core-${VERSION}`
+ The `*-debug*` files generated by the guix build contain debug symbols
+ for troubleshooting by developers. It is assumed that anyone that is
+ interested in debugging can run guix to generate the files for
+ themselves. To avoid end-user confusion about which file to pick, as well
+ as save storage space *do not upload these to the bitcoincore.org server,
+ nor put them in the torrent*.
-- A `.torrent` will appear in the directory after a few minutes. Optionally help seed this torrent. To get the `magnet:` URI use:
-```bash
-transmission-show -m <torrent file>
-```
-Insert the magnet URI into the announcement sent to mailing lists. This permits
-people without access to `bitcoincore.org` to download the binary distribution.
-Also put it into the `optional_magnetlink:` slot in the YAML file for
-bitcoincore.org.
+ 2. The combined clear-signed message you just created `SHA256SUMS.asc`
+
+- Create a torrent of the `/var/www/bin/bitcoin-core-${VERSION}` directory such
+ that at the top level there is only one file: the `bitcoin-core-${VERSION}`
+ directory containing everything else. Name the torrent
+ `bitcoin-${VERSION}.torrent` (note that there is no `-core-` in this name).
+
+ Optionally help seed this torrent. To get the `magnet:` URI use:
+
+ ```sh
+ transmission-show -m <torrent file>
+ ```
+
+ Insert the magnet URI into the announcement sent to mailing lists. This permits
+ people without access to `bitcoincore.org` to download the binary distribution.
+ Also put it into the `optional_magnetlink:` slot in the YAML file for
+ bitcoincore.org.
- Update other repositories and websites for new version
@@ -344,14 +281,14 @@ bitcoincore.org.
- https://code.launchpad.net/~bitcoin-core/bitcoin-core-snap/+git/packaging/+ref/0.xx (Click "Create snap package")
- Name it "bitcoin-core-snap-0.xx"
- Leave owner and series as-is
- - Select architectures that are compiled via gitian
+ - Select architectures that are compiled via guix
- Leave "automatically build when branch changes" unticked
- Tick "automatically upload to store"
- Put "bitcoin-core" in the registered store package name field
- Tick the "edge" box
- Put "0.xx" in the track field
- Click "create snap package"
- - Click "Request builds" for every new release on this branch (after updating the snapcraft.yml in the branch to reflect the latest gitian results)
+ - Click "Request builds" for every new release on this branch (after updating the snapcraft.yml in the branch to reflect the latest guix results)
- Promote release on https://snapcraft.io/bitcoin-core/releases if it passes sanity checks
- This repo
diff --git a/test/lint/lint-spelling.sh b/test/lint/lint-spelling.sh
index 238fa63c45..111091b7f8 100755
--- a/test/lint/lint-spelling.sh
+++ b/test/lint/lint-spelling.sh
@@ -15,6 +15,6 @@ if ! command -v codespell > /dev/null; then
fi
IGNORE_WORDS_FILE=test/lint/lint-spelling.ignore-words.txt
-if ! codespell --check-filenames --disable-colors --quiet-level=7 --ignore-words=${IGNORE_WORDS_FILE} $(git ls-files -- ":(exclude)build-aux/m4/" ":(exclude)contrib/seeds/*.txt" ":(exclude)depends/" ":(exclude)doc/release-notes/" ":(exclude)src/leveldb/" ":(exclude)src/crc32c/" ":(exclude)src/qt/locale/" ":(exclude)src/qt/*.qrc" ":(exclude)src/secp256k1/" ":(exclude)src/univalue/" ":(exclude)contrib/gitian-keys/keys.txt" ":(exclude)contrib/guix/patches"); then
+if ! codespell --check-filenames --disable-colors --quiet-level=7 --ignore-words=${IGNORE_WORDS_FILE} $(git ls-files -- ":(exclude)build-aux/m4/" ":(exclude)contrib/seeds/*.txt" ":(exclude)depends/" ":(exclude)doc/release-notes/" ":(exclude)src/leveldb/" ":(exclude)src/crc32c/" ":(exclude)src/qt/locale/" ":(exclude)src/qt/*.qrc" ":(exclude)src/secp256k1/" ":(exclude)src/univalue/" ":(exclude)contrib/builder-keys/keys.txt" ":(exclude)contrib/guix/patches"); then
echo "^ Warning: codespell identified likely spelling errors. Any false positives? Add them to the list of ignored words in ${IGNORE_WORDS_FILE}"
fi