diff options
Diffstat (limited to 'doc')
| -rw-r--r-- | doc/build-osx.md | 2 | ||||
| -rw-r--r-- | doc/gitian-building.md | 60 | ||||
| -rw-r--r-- | doc/release-process.md | 67 | ||||
| -rw-r--r-- | doc/zmq.md | 98 |
4 files changed, 205 insertions, 22 deletions
diff --git a/doc/build-osx.md b/doc/build-osx.md index 8fad8b5b00..201fe9522b 100644 --- a/doc/build-osx.md +++ b/doc/build-osx.md @@ -70,7 +70,7 @@ Download Qt Creator from http://www.qt.io/download/. Download the "community edi 6. Confirm the "summary page" 7. In the "Projects" tab select "Manage Kits..." 8. Select the default "Desktop" kit and select "Clang (x86 64bit in /usr/bin)" as compiler -9. Select LLDB as debugger (you might need to set the path to your installtion) +9. Select LLDB as debugger (you might need to set the path to your installation) 10. Start debugging with Qt Creator Creating a release build diff --git a/doc/gitian-building.md b/doc/gitian-building.md index 169727adc0..b434ae8a5b 100644 --- a/doc/gitian-building.md +++ b/doc/gitian-building.md @@ -330,10 +330,11 @@ There will be a lot of warnings printed during the build of the image. These can Getting and building the inputs -------------------------------- -Follow the instructions in [doc/release-process.md](release-process.md#fetch-and-build-inputs-first-time-or-when-dependency-versions-change) -in the bitcoin repository to install sources which require manual intervention. Also follow -the next step: 'Seed the Gitian sources cache', which will fetch all the necessary source -files to allow gitian to work offline. +Follow the instructions in [doc/release-process.md](release-process.md#fetch-and-build-inputs-first-time-or-when-dependency-versions-change) +in the bitcoin repository under 'Fetch and build inputs' to install sources which require +manual intervention. Also optionally follow the next step: 'Seed the Gitian sources cache +and offline git repositories' which will fetch the remaining files required for building +offline. Building Bitcoin ---------------- @@ -391,6 +392,57 @@ COMMIT=2014_03_windows_unicode_path ./bin/gbuild --commit bitcoin=${COMMIT} --url bitcoin=${URL} ../bitcoin/contrib/gitian-descriptors/gitian-osx.yml ``` +Building fully offline +----------------------- + +For building fully offline including attaching signatures to unsigned builds, the detached-sigs repository +and the bitcoin git repository with the desired tag must both be available locally, and then gbuild must be +told where to find them. It also requires an apt-cacher-ng which is fully-populated but set to offline mode, or +manually disabling gitian-builder's use of apt-get to update the VM build environment. + +To configure apt-cacher-ng as an offline cacher, you will need to first populate its cache with the relevant +files. You must additionally patch target-bin/bootstrap-fixup to set its apt sources to something other than +plain archive.ubuntu.com: us.archive.ubuntu.com works. + +So, if you use LXC: + +```bash +export PATH="$PATH":/path/to/gitian-builder/libexec +export USE_LXC=1 +cd /path/to/gitian-builder +./libexec/make-clean-vm --suite precise --arch amd64 + +LXC_ARCH=amd64 LXC_SUITE=precise on-target -u root apt-get update +LXC_ARCH=amd64 LXC_SUITE=precise on-target -u root \ + -e DEBIAN_FRONTEND=noninteractive apt-get --no-install-recommends -y install \ + $( sed -ne '/^packages:/,/[^-] .*/ {/^- .*/{s/"//g;s/- //;p}}' ../bitcoin/contrib/gitian-descriptors/*|sort|uniq ) +LXC_ARCH=amd64 LXC_SUITE=precise on-target -u root apt-get -q -y purge grub +LXC_ARCH=amd64 LXC_SUITE=precise on-target -u root -e DEBIAN_FRONTEND=noninteractive apt-get -y dist-upgrade +``` + +And then set offline mode for apt-cacher-ng: + +``` +/etc/apt-cacher-ng/acng.conf +[...] +Offlinemode: 1 +[...] + +service apt-cacher-ng restart +``` + +Then when building, override the remote URLs that gbuild would otherwise pull from the gitian descriptors:: +```bash + +cd /some/root/path/ +git clone https://github.com/bitcoin/bitcoin-detached-sigs.git + +BTCPATH=/some/root/path/bitcoin.git +SIGPATH=/some/root/path/bitcoin-detached-sigs.git + +./bin/gbuild --url bitcoin=${BTCPATH},signature=${SIGPATH} ../bitcoin/contrib/gitian-descriptors/gitian-win-signer.yml +``` + Signing externally ------------------- diff --git a/doc/release-process.md b/doc/release-process.md index 5ecb9334f5..1bfdb8fabd 100644 --- a/doc/release-process.md +++ b/doc/release-process.md @@ -6,39 +6,54 @@ Release Process * * * -###update (commit) version in sources +###first time only or for new builders, check out the source in the following directory hierarchy + cd /path/to/your/toplevel/build + git clone https://github.com/bitcoin/gitian.sigs.git + git clone https://github.com/devrandom/gitian-builder.git + git clone https://github.com/bitcoin/bitcoin.git + +###for bitcoin maintainers/release engineers, update (commit) version in sources + + pushd ./bitcoin contrib/verifysfbinaries/verify.sh doc/README* share/setup.nsi src/clientversion.h (change CLIENT_VERSION_IS_RELEASE to true) -###tag version in git +###for bitcoin maintainers/release engineers, tag version in git git tag -s v(new version, e.g. 0.8.0) -###write release notes. git shortlog helps a lot, for example: +###for bitcoin maintainers/release engineers, write release notes. git shortlog helps a lot, for example: git shortlog --no-merges v(current version, e.g. 0.7.2)..v(new version, e.g. 0.8.0) + popd * * * -###update gitian - - In order to take advantage of the new caching features in gitian, be sure to update to a recent version (`e9741525c` or later is recommended) +###update gitian, gitian.sigs, checkout bitcoin version, and perform gitian builds -###perform gitian builds - - From a directory containing the bitcoin source, gitian-builder and gitian.sigs + To ensure your gitian descriptors are accurate for direct reference for gbuild, below, run the following from a directory containing the bitcoin source: + pushd ./bitcoin export SIGNER=(your gitian key, ie bluematt, sipa, etc) export VERSION=(new version, e.g. 0.8.0) - pushd ./bitcoin git checkout v${VERSION} popd + + Ensure your gitian.sigs are up-to-date if you wish to gverify your builds against other gitian signatures: + + pushd ./gitian.sigs + git pull + popd + + Ensure your gitian-builder sources are up-to-date to take advantage of the new caching features of gitian (`e9741525c` or later is recommended) + pushd ./gitian-builder + git pull -###fetch and build inputs: (first time, or when dependency versions change) +###fetch and create inputs: (first time, or when dependency versions change) mkdir -p inputs wget -P inputs https://bitcoincore.org/cfields/osslsigncode-Backports-to-1.7.1.patch @@ -52,28 +67,44 @@ Release Process tar -C /Volumes/Xcode/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/ -czf MacOSX10.9.sdk.tar.gz MacOSX10.9.sdk -###Optional: Seed the Gitian sources cache +###Optional: Seed the Gitian sources cache and offline git repositories - By default, gitian will fetch source files as needed. For offline builds, they can be fetched ahead of time: +By default, gitian will fetch source files as needed. To cache them ahead of time: make -C ../bitcoin/depends download SOURCES_PATH=`pwd`/cache/common - Only missing files will be fetched, so this is safe to re-run for each build. +Only missing files will be fetched, so this is safe to re-run for each build. + +Clone the detached-sigs repository: + + popd + git clone https://github.com/bitcoin/bitcoin-detached-sigs.git + pushd ./bitcoin-builder + +NOTE: Offline builds must use the --url flag to ensure gitian fetches only from local URLs. +For example: ./bin/bguild --url bitcoin=/path/to/bitcoin,signature=/path/to/sigs {rest of arguments} +The following gbuild invocations DO NOT DO THIS by default. -###Build Bitcoin Core for Linux, Windows, and OS X: +###Build (and optionally verify) Bitcoin Core for Linux, Windows, and OS X: ./bin/gbuild --commit bitcoin=v${VERSION} ../bitcoin/contrib/gitian-descriptors/gitian-linux.yml ./bin/gsign --signer $SIGNER --release ${VERSION}-linux --destination ../gitian.sigs/ ../bitcoin/contrib/gitian-descriptors/gitian-linux.yml + ./bin/gverify -v -d ../gitian.sigs/ -r ${VERSION}-linux ../bitcoin/contrib/gitian-descriptors/gitian-linux.yml mv build/out/bitcoin-*.tar.gz build/out/src/bitcoin-*.tar.gz ../ + ./bin/gbuild --commit bitcoin=v${VERSION} ../bitcoin/contrib/gitian-descriptors/gitian-win.yml ./bin/gsign --signer $SIGNER --release ${VERSION}-win-unsigned --destination ../gitian.sigs/ ../bitcoin/contrib/gitian-descriptors/gitian-win.yml + ./bin/gverify -v -d ../gitian.sigs/ -r ${VERSION}-win-unsigned ../bitcoin/contrib/gitian-descriptors/gitian-win.yml mv build/out/bitcoin-*-win-unsigned.tar.gz inputs/bitcoin-win-unsigned.tar.gz mv build/out/bitcoin-*.zip build/out/bitcoin-*.exe ../ + ./bin/gbuild --commit bitcoin=v${VERSION} ../bitcoin/contrib/gitian-descriptors/gitian-osx.yml ./bin/gsign --signer $SIGNER --release ${VERSION}-osx-unsigned --destination ../gitian.sigs/ ../bitcoin/contrib/gitian-descriptors/gitian-osx.yml + ./bin/gverify -v -d ../gitian.sigs/ -r ${VERSION}-osx-unsigned ../bitcoin/contrib/gitian-descriptors/gitian-osx.yml mv build/out/bitcoin-*-osx-unsigned.tar.gz inputs/bitcoin-osx-unsigned.tar.gz mv build/out/bitcoin-*.tar.gz build/out/bitcoin-*.dmg ../ popd + Build output expected: 1. source tarball (bitcoin-${VERSION}.tar.gz) @@ -98,19 +129,21 @@ Commit your signature to gitian.sigs: Once the Windows/OSX builds each have 3 matching signatures, they will be signed with their respective release keys. Detached signatures will then be committed to the bitcoin-detached-sigs repository, which can be combined with the unsigned apps to create signed binaries. - Create the signed OSX binary: + Create (and optionally verify) the signed OSX binary: pushd ./gitian-builder ./bin/gbuild -i --commit signature=v${VERSION} ../bitcoin/contrib/gitian-descriptors/gitian-osx-signer.yml ./bin/gsign --signer $SIGNER --release ${VERSION}-osx-signed --destination ../gitian.sigs/ ../bitcoin/contrib/gitian-descriptors/gitian-osx-signer.yml + ./bin/gverify -v -d ../gitian.sigs/ -r ${VERSION}-osx-signed ../bitcoin/contrib/gitian-descriptors/gitian-osx-signer.yml mv build/out/bitcoin-osx-signed.dmg ../bitcoin-${VERSION}-osx.dmg popd - Create the signed Windows binaries: + Create (and optionally verify) the signed Windows binaries: pushd ./gitian-builder ./bin/gbuild -i --commit signature=v${VERSION} ../bitcoin/contrib/gitian-descriptors/gitian-win-signer.yml ./bin/gsign --signer $SIGNER --release ${VERSION}-win-signed --destination ../gitian.sigs/ ../bitcoin/contrib/gitian-descriptors/gitian-win-signer.yml + ./bin/gverify -v -d ../gitian.sigs/ -r ${VERSION}-win-signed ../bitcoin/contrib/gitian-descriptors/gitian-win-signer.yml mv build/out/bitcoin-*win64-setup.exe ../bitcoin-${VERSION}-win64-setup.exe mv build/out/bitcoin-*win32-setup.exe ../bitcoin-${VERSION}-win32-setup.exe popd diff --git a/doc/zmq.md b/doc/zmq.md new file mode 100644 index 0000000000..fd04f6d9f0 --- /dev/null +++ b/doc/zmq.md @@ -0,0 +1,98 @@ +# Block and Transaction Broadcasting With ZeroMQ + +[ZeroMQ](http://zeromq.org/) is a lightweight wrapper around TCP +connections, inter-process communications, and shared-memory, +providing various message-oriented semantics such as publish/subcribe, +request/reply, and push/pull. + +The Bitcoin Core daemon can be configured to act as a trusted "border +router", implementing the bitcoin wire protocol and relay, making +consensus decisions, maintaining the local blockchain database, +broadcasting locally generated transactions into the network, and +providing a queryable RPC interface to interact on a polled basis for +requesting blockchain related data. However, there exists only a +limited service to notify external software of events like the arrival +of new blocks or transactions. + +The ZeroMQ facility implements a notification interface through a +set of specific notifiers. Currently there are notifiers that publish +blocks and transactions. This read-only facility requires only the +connection of a corresponding ZeroMQ subscriber port in receiving +software; it is not authenticated nor is there any two-way protocol +involvement. Therefore, subscribers should validate the received data +since it may be out of date, incomplete or even invalid. + +ZeroMQ sockets are self-connecting and self-healing; that is, connects +made between two endpoints will be automatically restored after an +outage, and either end may be freely started or stopped in any order. + +Because ZeroMQ is message oriented, subscribers receive transactions +and blocks all-at-once and do not need to implement any sort of +buffering or reassembly. + +## Prerequisites + +The ZeroMQ feature in Bitcoin Core uses only a very small part of the +ZeroMQ C API, and is thus compatible with any version of ZeroMQ +from 2.1 onward, including all versions in the 3.x and 4.x release +series. Typically, it is packaged by distributions as something like +*libzmq-dev*. + +The C++ wrapper for ZeroMQ is *not* needed. + +## Enabling + +By default, the ZeroMQ port functionality is enabled. Two steps are +required to enable--compiling in the ZeroMQ code, and configuring +runtime operation on the command-line or configuration file. + + $ ./configure --enable-zmq (other options) + +This will produce a binary that is capable of providing the ZeroMQ +facility, but will not do so until also configured properly. + +## Usage + +Currently, the following notifications are supported: + + -zmqpubhashtx=address + -zmqpubhashblock=address + -zmqpubrawblock=address + -zmqpubrawtx=address + +The socket type is PUB and the address must be a valid ZeroMQ +socket address. The same address can be used in more than one notification. + +For instance: + + $ bitcoind -zmqpubhashtx=tcp://127.0.0.1:28332 -zmqpubrawtx=ipc:///tmp/bitcoind.tx.raw + +Each PUB notification has a topic and body, where the header +corresponds to the notification type. For instance, for the notification +`-zmqpubhashtx` the topic is `hashtx` (no null terminator) and the body is the +hexadecimal transaction hash (32 bytes). + +These options can also be provided in bitcoin.conf. + +ZeroMQ endpoint specifiers for TCP (and others) are documented in the +[ZeroMQ API](http://api.zeromq.org). + +Client side, then, the ZeroMQ subscriber socket must have the +ZMQ_SUBSCRIBE option set to one or either of these prefixes (for instance, just `hash`); without +doing so will result in no messages arriving. Please see `contrib/zmq/zmq_sub.py` +for a working example. + +## Remarks + +From the perspective of bitcoind, the ZeroMQ socket is write-only; PUB +sockets don't even have a read function. Thus, there is no state +introduced into bitcoind directly. Furthermore, no information is +broadcast that wasn't already received from the public P2P network. + +No authentication or authorization is done on connecting clients; it +is assumed that the ZeroMQ port is exposed only to trusted entities, +using other means such as firewalling. + +Note that when the block chain tip changes, a reorganisation may occur and just +the tip will be notified. It is up to the subscriber to retrieve the chain +from the last known block to the new tip. |