From 318c60700b7bbb7ec09a29bf037e7c2787646be6 Mon Sep 17 00:00:00 2001 From: Carl Dong Date: Fri, 2 Jul 2021 19:21:05 -0400 Subject: guix: Adapt release-process.md to new Guix process Also, clean up release-process.md --- doc/release-process.md | 263 ++++++++++++++++++------------------------------- 1 file changed, 98 insertions(+), 165 deletions(-) diff --git a/doc/release-process.md b/doc/release-process.md index 7164367759..0cf1d1f3ce 100644 --- a/doc/release-process.md +++ b/doc/release-process.md @@ -38,10 +38,6 @@ Release Process that causes rejection of blocks in the past history. - Clear the release notes and move them to the wiki (see "Write the release notes" below). -#### After branch-off (on master) - -- Update the version of `contrib/gitian-descriptors/*.yml`. - #### After branch-off (on the major release branch) - Update the versions. @@ -64,14 +60,14 @@ This will perform a few last-minute consistency checks in the build system files ### First time / New builders -If you're using the automated script (found in [contrib/gitian-build.py](/contrib/gitian-build.py)), then at this point you should run it with the "--setup" command. Otherwise ignore this. +Install Guix using one of the installation methods detailed in +[contrib/guix/INSTALL.md](/contrib/guix/INSTALL.md). Check out the source code in the following directory hierarchy. cd /path/to/your/toplevel/build - git clone https://github.com/bitcoin-core/gitian.sigs.git + git clone https://github.com/bitcoin-core/guix.sigs.git git clone https://github.com/bitcoin-core/bitcoin-detached-sigs.git - git clone https://github.com/devrandom/gitian-builder.git git clone https://github.com/bitcoin/bitcoin.git ### Write the release notes @@ -86,110 +82,56 @@ Generate list of authors: git log --format='- %aN' v(current version, e.g. 0.20.0)..v(new version, e.g. 0.20.1) | sort -fiu -### Setup and perform Gitian builds - -If you're using the automated script (found in [contrib/gitian-build.py](/contrib/gitian-build.py)), then at this point you should run it with the "--build" command. Otherwise ignore this. - -Setup Gitian descriptors: - - pushd ./bitcoin - export SIGNER="(your Gitian key, ie bluematt, sipa, etc)" - export VERSION=(new version, e.g. 0.20.0) - git fetch - git checkout v${VERSION} - popd - -Ensure your gitian.sigs are up-to-date if you wish to gverify your builds against other Gitian signatures. - - pushd ./gitian.sigs - git pull - popd - -Ensure gitian-builder is up-to-date: - - pushd ./gitian-builder - git pull - popd - -### Fetch and create inputs: (first time, or when dependency versions change) - - pushd ./gitian-builder - mkdir -p inputs - wget -O inputs/osslsigncode-2.0.tar.gz https://github.com/mtrojnar/osslsigncode/archive/2.0.tar.gz - echo '5a60e0a4b3e0b4d655317b2f12a810211c50242138322b16e7e01c6fbb89d92f inputs/osslsigncode-2.0.tar.gz' | sha256sum -c - popd - -Create the macOS SDK tarball, see the [macdeploy instructions](/contrib/macdeploy/README.md#deterministic-macos-dmg-notes) for details, and copy it into the inputs directory. - -### Optional: Seed the Gitian sources cache and offline git repositories - -NOTE: Gitian is sometimes unable to download files. If you have errors, try the step below. - -By default, Gitian will fetch source files as needed. To cache them ahead of time, make sure you have checked out the tag you want to build in bitcoin, then: - - pushd ./gitian-builder - make -C ../bitcoin/depends download SOURCES_PATH=`pwd`/cache/common - popd - -Only missing files will be fetched, so this is safe to re-run for each build. - -NOTE: Offline builds must use the --url flag to ensure Gitian fetches only from local URLs. For example: +### Setup and perform Guix builds - pushd ./gitian-builder - ./bin/gbuild --url bitcoin=/path/to/bitcoin,signature=/path/to/sigs {rest of arguments} - popd +Checkout the Bitcoin Core version you'd like to build: -The gbuild invocations below DO NOT DO THIS by default. +```sh +pushd ./bitcoin +SIGNER='(your builder key, ie bluematt, sipa, etc)' +VERSION='(new version without v-prefix, e.g. 0.20.0)' +git fetch "v${VERSION}" +git checkout "v${VERSION}" +popd +``` -### Build and sign Bitcoin Core for Linux, Windows, and macOS: +Ensure your guix.sigs are up-to-date if you wish to `guix-verify` your builds +against other `guix-attest` signatures. - pushd ./gitian-builder - ./bin/gbuild --num-make 2 --memory 3000 --commit bitcoin=v${VERSION} ../bitcoin/contrib/gitian-descriptors/gitian-linux.yml - ./bin/gsign --signer "$SIGNER" --release ${VERSION}-linux --destination ../gitian.sigs/ ../bitcoin/contrib/gitian-descriptors/gitian-linux.yml - mv build/out/bitcoin-*.tar.gz build/out/src/bitcoin-*.tar.gz ../ +```sh +git -C ./guix.sigs pull +``` - ./bin/gbuild --num-make 2 --memory 3000 --commit bitcoin=v${VERSION} ../bitcoin/contrib/gitian-descriptors/gitian-win.yml - ./bin/gsign --signer "$SIGNER" --release ${VERSION}-win-unsigned --destination ../gitian.sigs/ ../bitcoin/contrib/gitian-descriptors/gitian-win.yml - mv build/out/bitcoin-*-win-unsigned.tar.gz inputs/bitcoin-win-unsigned.tar.gz - mv build/out/bitcoin-*.zip build/out/bitcoin-*.exe ../ +### Create the macOS SDK tarball: (first time, or when SDK version changes) - ./bin/gbuild --num-make 2 --memory 3000 --commit bitcoin=v${VERSION} ../bitcoin/contrib/gitian-descriptors/gitian-osx.yml - ./bin/gsign --signer "$SIGNER" --release ${VERSION}-osx-unsigned --destination ../gitian.sigs/ ../bitcoin/contrib/gitian-descriptors/gitian-osx.yml - mv build/out/bitcoin-*-osx-unsigned.tar.gz inputs/bitcoin-osx-unsigned.tar.gz - mv build/out/bitcoin-*.tar.gz build/out/bitcoin-*.dmg ../ - popd +Create the macOS SDK tarball, see the [macdeploy +instructions](/contrib/macdeploy/README.md#deterministic-macos-dmg-notes) for +details. -Build output expected: +### Build and attest to build outputs: - 1. source tarball (`bitcoin-${VERSION}.tar.gz`) - 2. linux 32-bit and 64-bit dist tarballs (`bitcoin-${VERSION}-linux[32|64].tar.gz`) - 3. windows 32-bit and 64-bit unsigned installers and dist zips (`bitcoin-${VERSION}-win[32|64]-setup-unsigned.exe`, `bitcoin-${VERSION}-win[32|64].zip`) - 4. macOS unsigned installer and dist tarball (`bitcoin-${VERSION}-osx-unsigned.dmg`, `bitcoin-${VERSION}-osx64.tar.gz`) - 5. Gitian signatures (in `gitian.sigs/${VERSION}-/(your Gitian key)/`) +Follow the relevant Guix README.md sections: +- [Performing a build](/contrib/guix/README.md#performing-a-build) +- [Attesting to build outputs](/contrib/guix/README.md#attesting-to-build-outputs) -### Verify other gitian builders signatures to your own. (Optional) +### Verify other builders' signatures to your own. (Optional) Add other builders keys to your gpg keyring, and/or refresh keys: See `../bitcoin/contrib/builder-keys/README.md`. -Verify the signatures - - pushd ./gitian-builder - ./bin/gverify -v -d ../gitian.sigs/ -r ${VERSION}-linux ../bitcoin/contrib/gitian-descriptors/gitian-linux.yml - ./bin/gverify -v -d ../gitian.sigs/ -r ${VERSION}-win-unsigned ../bitcoin/contrib/gitian-descriptors/gitian-win.yml - ./bin/gverify -v -d ../gitian.sigs/ -r ${VERSION}-osx-unsigned ../bitcoin/contrib/gitian-descriptors/gitian-osx.yml - popd +Follow the relevant Guix README.md sections: +- [Verifying build output attestations](/contrib/guix/README.md#verifying-build-output-attestations) ### Next steps: -Commit your signature to gitian.sigs: +Commit your signature to guix.sigs: - pushd gitian.sigs - git add ${VERSION}-linux/"${SIGNER}" - git add ${VERSION}-win-unsigned/"${SIGNER}" - git add ${VERSION}-osx-unsigned/"${SIGNER}" - git commit -m "Add ${VERSION} unsigned sigs for ${SIGNER}" - git push # Assuming you can push to the gitian.sigs tree - popd +```sh +pushd ./guix.sigs +git add "${VERSION}/${SIGNER}"/noncodesigned.SHA256SUMS{,.asc} +git commit -m "Add ${VERSION} unsigned sigs for ${SIGNER}" +git push # Assuming you can push to the guix.sigs tree +popd +``` Codesigner only: Create Windows/macOS detached signatures: - Only one person handles codesigning. Everyone else should skip to the next step. @@ -201,7 +143,7 @@ Codesigner only: Sign the macOS binary: tar xf bitcoin-osx-unsigned.tar.gz ./detached-sig-create.sh -s "Key ID" Enter the keychain password and authorize the signature - Move signature-osx.tar.gz back to the gitian host + Move signature-osx.tar.gz back to the guix-build host Codesigner only: Sign the windows binaries: @@ -212,93 +154,84 @@ Codesigner only: Sign the windows binaries: Codesigner only: Commit the detached codesign payloads: - cd ~/bitcoin-detached-sigs - checkout the appropriate branch for this release series - rm -rf * - tar xf signature-osx.tar.gz - tar xf signature-win.tar.gz - git add -A - git commit -m "point to ${VERSION}" - git tag -s v${VERSION} HEAD - git push the current branch and new tag +```sh +pushd ./bitcoin-detached-sigs +# checkout the appropriate branch for this release series +rm -rf ./* +tar xf signature-osx.tar.gz +tar xf signature-win.tar.gz +git add -A +git commit -m "point to ${VERSION}" +git tag -s "v${VERSION}" HEAD +git push the current branch and new tag +popd +``` Non-codesigners: wait for Windows/macOS detached signatures: - Once the Windows/macOS builds each have 3 matching signatures, they will be signed with their respective release keys. - Detached signatures will then be committed to the [bitcoin-detached-sigs](https://github.com/bitcoin-core/bitcoin-detached-sigs) repository, which can be combined with the unsigned apps to create signed binaries. -Create (and optionally verify) the signed macOS binary: +Create (and optionally verify) the codesigned outputs: - pushd ./gitian-builder - ./bin/gbuild -i --commit signature=v${VERSION} ../bitcoin/contrib/gitian-descriptors/gitian-osx-signer.yml - ./bin/gsign --signer "$SIGNER" --release ${VERSION}-osx-signed --destination ../gitian.sigs/ ../bitcoin/contrib/gitian-descriptors/gitian-osx-signer.yml - ./bin/gverify -v -d ../gitian.sigs/ -r ${VERSION}-osx-signed ../bitcoin/contrib/gitian-descriptors/gitian-osx-signer.yml - mv build/out/bitcoin-osx-signed.dmg ../bitcoin-${VERSION}-osx.dmg - popd +- [Codesigning](/contrib/guix/README.md#codesigning) -Create (and optionally verify) the signed Windows binaries: +Commit your signature for the signed macOS/Windows binaries: - pushd ./gitian-builder - ./bin/gbuild -i --commit signature=v${VERSION} ../bitcoin/contrib/gitian-descriptors/gitian-win-signer.yml - ./bin/gsign --signer "$SIGNER" --release ${VERSION}-win-signed --destination ../gitian.sigs/ ../bitcoin/contrib/gitian-descriptors/gitian-win-signer.yml - ./bin/gverify -v -d ../gitian.sigs/ -r ${VERSION}-win-signed ../bitcoin/contrib/gitian-descriptors/gitian-win-signer.yml - mv build/out/bitcoin-*win64-setup.exe ../bitcoin-${VERSION}-win64-setup.exe - popd +```sh +pushd ./guix.sigs +git add "${VERSION}/${SIGNER}"/all.SHA256SUMS{,.asc} +git commit -m "Add ${SIGNER} ${VERSION} signed binaries signatures" +git push # Assuming you can push to the guix.sigs tree +popd +``` -Commit your signature for the signed macOS/Windows binaries: +### After 3 or more people have guix-built and their results match: - pushd gitian.sigs - git add ${VERSION}-osx-signed/"${SIGNER}" - git add ${VERSION}-win-signed/"${SIGNER}" - git commit -m "Add ${SIGNER} ${VERSION} signed binaries signatures" - git push # Assuming you can push to the gitian.sigs tree - popd +Combine `all.SHA256SUMS` and `all.SHA256SUMS.asc` into a clear-signed +`SHA256SUMS.asc` message: -### After 3 or more people have gitian-built and their results match: +```sh +echo -e "-----BEGIN PGP SIGNED MESSAGE-----\nHash: SHA256\n\n$(cat all.SHA256SUMS)\n$(cat filename.txt.asc)" > SHA256SUMS.asc +``` -- Create `SHA256SUMS.asc` for the builds, and GPG-sign it: +Here's an equivalent, more readable command if you're confident that you won't +mess up whitespaces when copy-pasting: ```bash -sha256sum * > SHA256SUMS -``` +cat << EOF > SHA256SUMS.asc +-----BEGIN PGP SIGNED MESSAGE----- +Hash: SHA256 -The list of files should be: -``` -bitcoin-${VERSION}-aarch64-linux-gnu.tar.gz -bitcoin-${VERSION}-arm-linux-gnueabihf.tar.gz -bitcoin-${VERSION}-riscv64-linux-gnu.tar.gz -bitcoin-${VERSION}-x86_64-linux-gnu.tar.gz -bitcoin-${VERSION}-osx64.tar.gz -bitcoin-${VERSION}-osx.dmg -bitcoin-${VERSION}.tar.gz -bitcoin-${VERSION}-win64-setup.exe -bitcoin-${VERSION}-win64.zip +$(cat all.SHA256SUMS) +$(cat all.SHA256SUMS.asc) +EOF ``` -The `*-debug*` files generated by the gitian build contain debug symbols -for troubleshooting by developers. It is assumed that anyone that is interested -in debugging can run gitian to generate the files for themselves. To avoid -end-user confusion about which file to pick, as well as save storage -space *do not upload these to the bitcoincore.org server, nor put them in the torrent*. -- GPG-sign it, delete the unsigned file: -``` -gpg --digest-algo sha256 --clearsign SHA256SUMS # outputs SHA256SUMS.asc -rm SHA256SUMS -``` -(the digest algorithm is forced to sha256 to avoid confusion of the `Hash:` header that GPG adds with the SHA256 used for the files) -Note: check that SHA256SUMS itself doesn't end up in SHA256SUMS, which is a spurious/nonsensical entry. +- Upload to the bitcoincore.org server (`/var/www/bin/bitcoin-core-${VERSION}`): + 1. The contents of `./bitcoin/guix-build-${VERSION}/output`, except for + `*-debug*` files. -- Upload zips and installers, as well as `SHA256SUMS.asc` from last step, to the bitcoincore.org server - into `/var/www/bin/bitcoin-core-${VERSION}` + The `*-debug*` files generated by the guix build contain debug symbols + for troubleshooting by developers. It is assumed that anyone that is + interested in debugging can run guix to generate the files for + themselves. To avoid end-user confusion about which file to pick, as well + as save storage space *do not upload these to the bitcoincore.org server, + nor put them in the torrent*. -- A `.torrent` will appear in the directory after a few minutes. Optionally help seed this torrent. To get the `magnet:` URI use: -```bash -transmission-show -m -``` -Insert the magnet URI into the announcement sent to mailing lists. This permits -people without access to `bitcoincore.org` to download the binary distribution. -Also put it into the `optional_magnetlink:` slot in the YAML file for -bitcoincore.org. + 2. The combined clear-signed message you just created `SHA256SUMS.asc` + +- A `.torrent` will appear in the directory after a few minutes. Optionally help + seed this torrent. To get the `magnet:` URI use: + + ```sh + transmission-show -m + ``` + + Insert the magnet URI into the announcement sent to mailing lists. This permits + people without access to `bitcoincore.org` to download the binary distribution. + Also put it into the `optional_magnetlink:` slot in the YAML file for + bitcoincore.org. - Update other repositories and websites for new version @@ -336,14 +269,14 @@ bitcoincore.org. - https://code.launchpad.net/~bitcoin-core/bitcoin-core-snap/+git/packaging/+ref/0.xx (Click "Create snap package") - Name it "bitcoin-core-snap-0.xx" - Leave owner and series as-is - - Select architectures that are compiled via gitian + - Select architectures that are compiled via guix - Leave "automatically build when branch changes" unticked - Tick "automatically upload to store" - Put "bitcoin-core" in the registered store package name field - Tick the "edge" box - Put "0.xx" in the track field - Click "create snap package" - - Click "Request builds" for every new release on this branch (after updating the snapcraft.yml in the branch to reflect the latest gitian results) + - Click "Request builds" for every new release on this branch (after updating the snapcraft.yml in the branch to reflect the latest guix results) - Promote release on https://snapcraft.io/bitcoin-core/releases if it passes sanity checks - This repo -- cgit v1.2.3