diff options
Diffstat (limited to 'doc/gitian-building.md')
-rw-r--r-- | doc/gitian-building.md | 424 |
1 files changed, 424 insertions, 0 deletions
diff --git a/doc/gitian-building.md b/doc/gitian-building.md new file mode 100644 index 0000000000..23059a0506 --- /dev/null +++ b/doc/gitian-building.md @@ -0,0 +1,424 @@ +Gitian building +================ + +*Setup instructions for a gitian build of Bitcoin using a Debian VM or physical system.* + +Gitian is the deterministic build process that is used to build the Bitcoin +Core executables [1]. It provides a way to be reasonably sure that the +executables are really built from source on github. It also makes sure that +the same, tested dependencies are used and statically built into the executable. + +Multiple developers build the source code by following a specific descriptor +("recipe"), cryptographically sign the result, and upload the resulting signature. +These results are compared and only if they match, the build is accepted and uploaded +to bitcoin.org. + +More independent gitian builders are needed, which is why I wrote this +guide. It is preferred to follow these steps yourself instead of using someone else's +VM image to avoid 'contaminating' the build. + +[1] For all platforms except for MacOSX, at this point. Work for deterministic +builds for Mac is under way here: https://github.com/theuni/osx-cross-depends . + +Table of Contents +------------------ + +- [Create a new VirtualBox VM](#create-a-new-virtualbox-vm) +- [Connecting to the VM](#connecting-to-the-vm) +- [Setting up Debian for gitian building](#setting-up-debian-for-gitian-building) +- [Installing gitian](#installing-gitian) +- [Setting up gitian images](#setting-up-gitian-images) +- [Getting and building the inputs](#getting-and-building-the-inputs) +- [Building Bitcoin](#building-bitcoin) +- [Building an alternative repository](#building-an-alternative-repository) +- [Signing externally](#signing-externally) +- [Uploading signatures](#uploading-signatures) + +Create a new VirtualBox VM +--------------------------- + +The first step is to create a new Virtual Machine, which will be explained in +this section. This VM will be used to do the Gitian builds. In this guide it +will be explained how to set up the environment, and how to get the builds +started. + +Debian Linux was chosen as the host distribution because it has a lightweight install (in +contrast to Ubuntu) and is readily available. We here show the steps for +VirtualBox [1], but any kind of virtualization can be used. You can also install +on actual hardware instead of using a VM, in this case you can skip this section. + +In the VirtualBox GUI click "Create" and choose the following parameters in the wizard: + +![](gitian-building/create_vm_page1.png) + +- Type: Linux, Debian (64 bit) + +![](gitian-building/create_vm_memsize.png) + +- Memory Size: at least 1024MB, anything lower will really slow the build down + +![](gitian-building/create_vm_hard_drive.png) + +- Hard Drive: Create a virtual hard drive now + +![](gitian-building/create_vm_hard_drive_file_type.png) + +- Hard Drive file type: Use the default, VDI (VirtualBox Disk Image) + +![](gitian-building/create_vm_storage_physical_hard_drive.png) + +- Storage on Physical hard drive: Dynamically Allocated + +![](gitian-building/create_vm_file_location_size.png) + +- Disk size: at least 40GB; as low as 20GB *may* be possible, but better to err on the safe side +- Push the `Create` button + +Get the [Debian 7.4 net installer](http://cdimage.debian.org/debian-cd/7.4.0/amd64/iso-cd/debian-7.4.0-amd64-netinst.iso). +This DVD image can be validated using a SHA256 hashing tool, for example on +Unixy OSes by entering the following in a terminal: + + echo "b712a141bc60269db217d3b3e456179bd6b181645f90e4aac9c42ed63de492e9 /home/orion/Downloads/debian-7.4.0-amd64-netinst.iso" | sha256sum -c + # (must return OK) + +After creating the VM, we need to configure it. + +- Click the `Settings` button, then go to the `Network` tab. Adapter 1 should be attacked to `NAT`. + +![](gitian-building/network_settings.png) + +- Click `Advanced`, then `Port Forwarding`. We want to set up a port through where we can reach the VM to get files in and out. +- Create a new rule by clicking the plus icon. + +![](gitian-building/port_forwarding_rules.png) + +- Set up the new rule the following way: + - Name: `SSH` + - Protocol: `TCP` + - Leave Host IP empty + - Host Port: `22222` + - Leave Guest IP empty + - Guest Port: `22` + +- Click `Ok` twice to save. + +Then start the VM. On the first launch you will be asked for a CD or DVD image. Choose the downloaded iso. + +![](gitian-building/select_startup_disk.png) + +[1] https://www.virtualbox.org/ + +Installing Debian +------------------ + +In this section it will be explained how to install Debian on the newly created VM. + +- Choose the non-graphical installer. We do not need the graphical environment, it will only increase installation time and disk usage. + +![](gitian-building/debian_install_1_boot_menu.png) + +**Note**: Navigation in the Debian installer: To keep a setting at the default +and proceed, just press `Enter`. To select a different button, press `Tab`. + +- Choose locale and keyboard settings (doesn't matter, you can just go with the defaults or select your own information) + +![](gitian-building/debian_install_2_select_a_language.png) +![](gitian-building/debian_install_3_select_location.png) +![](gitian-building/debian_install_4_configure_keyboard.png) + +- The VM will detect network settings using DHCP, this should all proceed automatically +- Configure the network: + - System name `debian`. + - Leave domain name empty. + +![](gitian-building/debian_install_5_configure_the_network.png) + +- Choose a root password and enter it twice (and remember it for later) + +![](gitian-building/debian_install_6a_set_up_root_password.png) + +- Name the new user `debian` (the full name doesn't matter, you can leave it empty) + +![](gitian-building/debian_install_7_set_up_user_fullname.png) +![](gitian-building/debian_install_8_set_up_username.png) + +- Choose a user password and enter it twice (and remember it for later) + +![](gitian-building/debian_install_9_user_password.png) + +- The installer will set up the clock using a time server, this process should be automatic +- Set up the clock: choose a time zone (depends on the locale settings that you picked earlier; specifics don't matter) + +![](gitian-building/debian_install_10_configure_clock.png) + +- Disk setup + - Partitioning method: Guided - Use the entire disk + +![](gitian-building/debian_install_11_partition_disks.png) + + - Select disk to partition: SCSI1 (0,0,0) + +![](gitian-building/debian_install_12_choose_disk.png) + + - Partitioning scheme: All files in one partition + +![](gitian-building/debian_install_13_partition_scheme.png) + + - Finish partitioning and write changes to disk -> *Yes* (`Tab`, `Enter` to select the `Yes` button) + +![](gitian-building/debian_install_14_finish.png) +![](gitian-building/debian_install_15_write_changes.png) + +- The base system will be installed, this will take a minute or so +- Choose a mirror (any will do) + +![](gitian-building/debian_install_16_choose_a_mirror.png) + +- Enter proxy information (unless you are on an intranet, you can leave this empty) + +![](gitian-building/debian_install_18_proxy_settings.png) + +- Wait a bit while 'Select and install software' runs +- Participate in popularity contest -> *No* +- Choose software to install. We need just the base system. + +![](gitian-building/debian_install_19_software_selection.png) + +- Make sure only 'SSH server' and 'Standard System Utilities' are checked +- Uncheck 'Debian Desktop Environment' and 'Print Server' + +![](gitian-building/debian_install_20_install_grub.png) + +- Install the GRUB boot loader to the master boot record? -> Yes + +![](gitian-building/debian_install_21_finish_installation.png) + +- Installation Complete -> *Continue* +- After installation, the VM will reboot and you will have a working Debian VM. Congratulations! + +Connecting to the VM +---------------------- + +After the VM has booted you can connect to it using SSH, and files can be copied from and to the VM using a SFTP utility. +Connect to `localhost`, port `22222` (or the port configured when installing the VM). +On Windows you can use putty[1] and WinSCP[2]. + +For example to connect as `root` from a Linux command prompt use + + $ ssh root@localhost -p 22222 + The authenticity of host '[localhost]:22222 ([127.0.0.1]:22222)' can't be established. + ECDSA key fingerprint is 8e:71:f9:5b:62:46:de:44:01:da:fb:5f:34:b5:f2:18. + Are you sure you want to continue connecting (yes/no)? yes + Warning: Permanently added '[localhost]:22222' (ECDSA) to the list of known hosts. + root@localhost's password: (enter root password configured during install) + Linux debian 3.2.0-4-amd64 #1 SMP Debian 3.2.54-2 x86_64 + root@debian:~# + +Replace `root` with `debian` to log in as user. + +[1] http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html +[2] http://winscp.net/eng/index.php + +Setting up Debian for gitian building +-------------------------------------- + +In this section we will be setting up the Debian installation for Gitian building. + +First we need to log in as `root` to set up dependencies and make sure that our +user can use the sudo command. Type/paste the following in the terminal: + + +```bash +apt-get install git ruby sudo apt-cacher-ng qemu-utils debootstrap lxc python-cheetah parted kpartx bridge-utils +adduser debian sudo +``` + +When you get a colorful screen with a question about the 'LXC directory', just +go with the default (`/var/lib/lxc`). + +Then set up LXC and the rest with the following is a complex jumble of settings and workarounds: + +```bash +# the version of lxc-start in Debian 7.4 needs to run as root, so make sure +# that the build script can exectute it without providing a password +echo "%sudo ALL=NOPASSWD: /usr/bin/lxc-start" > /etc/sudoers.d/gitian-lxc +# add cgroup for LXC +echo "cgroup /sys/fs/cgroup cgroup defaults 0 0" >> /etc/fstab +# make /etc/rc.local script that sets up bridge between guest and host +echo '#!/bin/sh -e' > /etc/rc.local +echo 'brctl addbr br0' >> /etc/rc.local +echo 'ifconfig br0 10.0.3.2/24 up' >> /etc/rc.local +echo 'exit 0' >> /etc/rc.local +# make sure that USE_LXC is always set when logging in as debian +echo 'export USE_LXC=1' >> /home/debian/.profile +reboot +``` + +At the end the VM is rebooted to make sure that the changes take effect. + +**Note**: If you're following this guide on a physical system instead of a VirtualBox VM you could use `10.0.2.2` instead +of `10.0.3.2` in the above `ifconfig` line. This avoids having to patch gitian-builder in next section. + +Installing gitian +------------------ + +Re-login as the user `debian` that was created during installation. +The rest of the steps in this guide will be performed as that user. + +There is no `python-vm-builder` package in Debian, so we need to install it from source ourselves, + +```bash +wget http://archive.ubuntu.com/ubuntu/pool/universe/v/vm-builder/vm-builder_0.12.4+bzr489.orig.tar.gz +echo "ec12e0070a007989561bfee5862c89a32c301992dd2771c4d5078ef1b3014f03 vm-builder_0.12.4+bzr489.orig.tar.gz" | sha256sum -c +# (verification -- must return OK) +tar -zxvf vm-builder_0.12.4+bzr489.orig.tar.gz +cd vm-builder-0.12.4+bzr489 +sudo python setup.py install +cd .. +``` + +**Note**: When sudo asks for a password, enter the password for the user *debian* not for *root*. + +Clone the git repositories for bitcoin and gitian, + +```bash +git clone https://github.com/devrandom/gitian-builder.git +git clone https://github.com/bitcoin/bitcoin +``` + +We need to change the guest IP range for the gitian builder because otherwise it will +collide with VirtualBox its NAT IP range. Gitian does not have a way yet to configure +this, so we need to patch the IPs using `sed`. This is not nice but it will +have to do for now... (a [pull request +(#52)](https://github.com/devrandom/gitian-builder/pull/52) to make this +configurable without patching has been submitted): + +```bash +sed -i 's/10.0.2.2/10.0.3.2/g' gitian-builder/target-bin/bootstrap-fixup +sed -i 's/10.0.2.5/10.0.3.5/g' gitian-builder/etc/lxc.config.in +``` + +*note* After you update the gitian-builder repository, you may need to repeat these manual changes. + +Setting up gitian images +------------------------- + +Gitian needs virtual images of the operating system to build in. +Currently this is Ubuntu Precise for both x86 architectures. +These images will be copied and used every time that a build is started to +make sure that the build is deterministic. +Creating the images will take a while, but only has to be done once. + +Execute the following as user `debian`: + +```bash +cd gitian-builder +bin/make-base-vm --lxc --arch i386 --suite precise +bin/make-base-vm --lxc --arch amd64 --suite precise +``` + +There will be a lot of warnings printed during build of the images. These can be ignored. + +**Note**: When sudo asks for a password, enter the password for the user *debian* not for *root*. + +Getting and building the inputs +-------------------------------- + +In [doc/release-process.md](release-process.md) in the bitcoin repository under 'Fetch and build inputs'. +you will find a list of `wget` commands that can be executed to get the dependencies. + +I needed to add `--no-check-certificate` to the OpenSSL wget line to make it work. +Likely this is because the ca-certificates in Debian 7.4 is fairly old. This does not create a +security issue as the gitian descriptors check integrity of the input archives and refuse to work +if any one is corrupted. + +After downloading the archives, execute the `gbuild` commends to build the dependencies. +This can take a long time, but only has to be done when the dependencies change, for example +to upgrade the used version. + +**Note**: Do not forget to copy the result from `build/out` to `inputs` after every gbuild command! This will save +you a lot of time. + +At any time you can check the package installation and build progress with + +```bash +tail -f var/install.log +tail -f var/build.log +``` + +Building Bitcoin +---------------- + +To build Bitcoin (for Linux and/or Windows) just follow the steps under 'perform +gitian builds' in [doc/release-process.md](release-process.md) in the bitcoin repository. + +Output from `gbuild` will look something like + + Initialized empty Git repository in /home/debian/gitian-builder/inputs/bitcoin/.git/ + remote: Reusing existing pack: 35606, done. + remote: Total 35606 (delta 0), reused 0 (delta 0) + Receiving objects: 100% (35606/35606), 26.52 MiB | 4.28 MiB/s, done. + Resolving deltas: 100% (25724/25724), done. + From https://github.com/bitcoin/bitcoin + ... (new tags, new branch etc) + --- Building for precise i386 --- + Stopping target if it is up + Making a new image copy + stdin: is not a tty + Starting target + Checking if target is up + Preparing build environment + Updating apt-get repository (log in var/install.log) + Installing additional packages (log in var/install.log) + Grabbing package manifest + stdin: is not a tty + Creating build script (var/build-script) + lxc-start: Connection refused - inotify event with no name (mask 32768) + Running build script (log in var/build.log) + +As when building the dependencies, the progress of package installation and building +can be inspected in `var/install.log` and `var/build.log`. + +Building an alternative repository +----------------------------------- + +If you want to do a test build of a pull on github it can be useful to point +the gitian builder at an alternative repository, using the same descriptors +and inputs. + +For example: +```bash +URL=https://github.com/laanwj/bitcoin.git +COMMIT=2014_03_windows_unicode_path +./bin/gbuild --commit bitcoin=${COMMIT} --url bitcoin=${URL} ../bitcoin/contrib/gitian-descriptors/gitian-linux.yml +./bin/gbuild --commit bitcoin=${COMMIT} --url bitcoin=${URL} ../bitcoin/contrib/gitian-descriptors/gitian-win.yml +``` + +Signing externally +------------------- + +If you want to do the PGP signing on another device that's possible too; just define `SIGNER` as mentioned +and follow the steps in the build process as normally. + + gpg: skipped "laanwj": secret key not available + +When you execute `gsign` you will get an error from GPG, which can be ignored. Copy the resulting `.assert` files +in `gitian.sigs` to your signing machine and do + +```bash + gpg --detach-sign ${VERSION}/${SIGNER}/bitcoin-build.assert + gpg --detach-sign ${VERSION}-win/${SIGNER}/bitcoin-build.assert +``` + +This will create the `.sig` files that can be committed together with the `.assert` files to assert your +gitian build. + +Uploading signatures +--------------------- + +After building and signing you can push your signatures (both the `.assert` and +`.assert.sig` files) to the +[bitcoin/gitian.sigs](https://github.com/bitcoin/gitian.sigs/) repository, or +if not possible create a pull request. You can also mail the files to me +(laanwj@gmail.com) and I'll commit them. + |