diff options
Diffstat (limited to 'doc')
| -rw-r--r-- | doc/README_osx.md | 97 | ||||
| -rw-r--r-- | doc/build-osx.md | 100 | ||||
| -rw-r--r-- | doc/fuzzing.md | 16 | ||||
| -rw-r--r-- | doc/init.md | 18 | ||||
| -rw-r--r-- | doc/release-notes/release-notes-pr12255.md | 17 | ||||
| -rw-r--r-- | doc/release-process.md | 2 | ||||
| -rw-r--r-- | doc/translation_process.md | 20 |
7 files changed, 158 insertions, 112 deletions
diff --git a/doc/README_osx.md b/doc/README_osx.md deleted file mode 100644 index 739e22d634..0000000000 --- a/doc/README_osx.md +++ /dev/null @@ -1,97 +0,0 @@ -Deterministic macOS DMG Notes. - -Working macOS DMGs are created in Linux by combining a recent clang, -the Apple binutils (ld, ar, etc) and DMG authoring tools. - -Apple uses clang extensively for development and has upstreamed the necessary -functionality so that a vanilla clang can take advantage. It supports the use -of -F, -target, -mmacosx-version-min, and --sysroot, which are all necessary -when building for macOS. - -Apple's version of binutils (called cctools) contains lots of functionality -missing in the FSF's binutils. In addition to extra linker options for -frameworks and sysroots, several other tools are needed as well such as -install_name_tool, lipo, and nmedit. These do not build under linux, so they -have been patched to do so. The work here was used as a starting point: -[mingwandroid/toolchain4](https://github.com/mingwandroid/toolchain4). - -In order to build a working toolchain, the following source packages are needed -from Apple: cctools, dyld, and ld64. - -These tools inject timestamps by default, which produce non-deterministic -binaries. The ZERO_AR_DATE environment variable is used to disable that. - -This version of cctools has been patched to use the current version of clang's -headers and its libLTO.so rather than those from llvmgcc, as it was -originally done in toolchain4. - -To complicate things further, all builds must target an Apple SDK. These SDKs -are free to download, but not redistributable. -To obtain it, register for a developer account, then download the [Xcode 7.3.1 dmg](https://developer.apple.com/devcenter/download.action?path=/Developer_Tools/Xcode_7.3.1/Xcode_7.3.1.dmg). - -This file is several gigabytes in size, but only a single directory inside is -needed: -``` -Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk -``` - -Unfortunately, the usual linux tools (7zip, hpmount, loopback mount) are incapable of opening this file. -To create a tarball suitable for Gitian input, there are two options: - -Using macOS, you can mount the dmg, and then create it with: -``` - $ hdiutil attach Xcode_7.3.1.dmg - $ tar -C /Volumes/Xcode/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/ -czf MacOSX10.11.sdk.tar.gz MacOSX10.11.sdk -``` - -Alternatively, you can use 7zip and SleuthKit to extract the files one by one. -The script contrib/macdeploy/extract-osx-sdk.sh automates this. First ensure -the dmg file is in the current directory, and then run the script. You may wish -to delete the intermediate 5.hfs file and MacOSX10.11.sdk (the directory) when -you've confirmed the extraction succeeded. - -```bash -apt-get install p7zip-full sleuthkit -contrib/macdeploy/extract-osx-sdk.sh -rm -rf 5.hfs MacOSX10.11.sdk -``` - -The Gitian descriptors build 2 sets of files: Linux tools, then Apple binaries -which are created using these tools. The build process has been designed to -avoid including the SDK's files in Gitian's outputs. All interim tarballs are -fully deterministic and may be freely redistributed. - -genisoimage is used to create the initial DMG. It is not deterministic as-is, -so it has been patched. A system genisoimage will work fine, but it will not -be deterministic because the file-order will change between invocations. -The patch can be seen here: [theuni/osx-cross-depends](https://raw.githubusercontent.com/theuni/osx-cross-depends/master/patches/cdrtools/genisoimage.diff). -No effort was made to fix this cleanly, so it likely leaks memory badly. But -it's only used for a single invocation, so that's no real concern. - -genisoimage cannot compress DMGs, so afterwards, the 'dmg' tool from the -libdmg-hfsplus project is used to compress it. There are several bugs in this -tool and its maintainer has seemingly abandoned the project. It has been forked -and is available (with fixes) here: [theuni/libdmg-hfsplus](https://github.com/theuni/libdmg-hfsplus). - -The 'dmg' tool has the ability to create DMGs from scratch as well, but this -functionality is broken. Only the compression feature is currently used. -Ideally, the creation could be fixed and genisoimage would no longer be necessary. - -Background images and other features can be added to DMG files by inserting a -.DS_Store before creation. This is generated by the script -contrib/macdeploy/custom_dsstore.py. - -As of OS X 10.9 Mavericks, using an Apple-blessed key to sign binaries is a -requirement in order to satisfy the new Gatekeeper requirements. Because this -private key cannot be shared, we'll have to be a bit creative in order for the -build process to remain somewhat deterministic. Here's how it works: - -- Builders use Gitian to create an unsigned release. This outputs an unsigned - dmg which users may choose to bless and run. It also outputs an unsigned app - structure in the form of a tarball, which also contains all of the tools - that have been previously (deterministically) built in order to create a - final dmg. -- The Apple keyholder uses this unsigned app to create a detached signature, - using the script that is also included there. Detached signatures are available from this [repository](https://github.com/bitcoin-core/bitcoin-detached-sigs). -- Builders feed the unsigned app + detached signature back into Gitian. It - uses the pre-built tools to recombine the pieces into a deterministic dmg. diff --git a/doc/build-osx.md b/doc/build-osx.md index c9a59bab83..119896dc67 100644 --- a/doc/build-osx.md +++ b/doc/build-osx.md @@ -105,3 +105,103 @@ Notes * Tested on OS X 10.10 Yosemite through macOS 10.13 High Sierra on 64-bit Intel processors only. * Building with downloaded Qt binaries is not officially supported. See the notes in [#7714](https://github.com/bitcoin/bitcoin/issues/7714) + +Deterministic macOS DMG Notes +----------------------------- + +Working macOS DMGs are created in Linux by combining a recent clang, +the Apple binutils (ld, ar, etc) and DMG authoring tools. + +Apple uses clang extensively for development and has upstreamed the necessary +functionality so that a vanilla clang can take advantage. It supports the use +of -F, -target, -mmacosx-version-min, and --sysroot, which are all necessary +when building for macOS. + +Apple's version of binutils (called cctools) contains lots of functionality +missing in the FSF's binutils. In addition to extra linker options for +frameworks and sysroots, several other tools are needed as well such as +install_name_tool, lipo, and nmedit. These do not build under linux, so they +have been patched to do so. The work here was used as a starting point: +[mingwandroid/toolchain4](https://github.com/mingwandroid/toolchain4). + +In order to build a working toolchain, the following source packages are needed +from Apple: cctools, dyld, and ld64. + +These tools inject timestamps by default, which produce non-deterministic +binaries. The ZERO_AR_DATE environment variable is used to disable that. + +This version of cctools has been patched to use the current version of clang's +headers and its libLTO.so rather than those from llvmgcc, as it was +originally done in toolchain4. + +To complicate things further, all builds must target an Apple SDK. These SDKs +are free to download, but not redistributable. +To obtain it, register for a developer account, then download the [Xcode 7.3.1 dmg](https://developer.apple.com/devcenter/download.action?path=/Developer_Tools/Xcode_7.3.1/Xcode_7.3.1.dmg). + +This file is several gigabytes in size, but only a single directory inside is +needed: +``` +Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk +``` + +Unfortunately, the usual linux tools (7zip, hpmount, loopback mount) are incapable of opening this file. +To create a tarball suitable for Gitian input, there are two options: + +Using macOS, you can mount the dmg, and then create it with: +``` + $ hdiutil attach Xcode_7.3.1.dmg + $ tar -C /Volumes/Xcode/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/ -czf MacOSX10.11.sdk.tar.gz MacOSX10.11.sdk +``` + +Alternatively, you can use 7zip and SleuthKit to extract the files one by one. +The script contrib/macdeploy/extract-osx-sdk.sh automates this. First ensure +the dmg file is in the current directory, and then run the script. You may wish +to delete the intermediate 5.hfs file and MacOSX10.11.sdk (the directory) when +you've confirmed the extraction succeeded. + +```bash +apt-get install p7zip-full sleuthkit +contrib/macdeploy/extract-osx-sdk.sh +rm -rf 5.hfs MacOSX10.11.sdk +``` + +The Gitian descriptors build 2 sets of files: Linux tools, then Apple binaries +which are created using these tools. The build process has been designed to +avoid including the SDK's files in Gitian's outputs. All interim tarballs are +fully deterministic and may be freely redistributed. + +genisoimage is used to create the initial DMG. It is not deterministic as-is, +so it has been patched. A system genisoimage will work fine, but it will not +be deterministic because the file-order will change between invocations. +The patch can be seen here: [theuni/osx-cross-depends](https://raw.githubusercontent.com/theuni/osx-cross-depends/master/patches/cdrtools/genisoimage.diff). +No effort was made to fix this cleanly, so it likely leaks memory badly. But +it's only used for a single invocation, so that's no real concern. + +genisoimage cannot compress DMGs, so afterwards, the 'dmg' tool from the +libdmg-hfsplus project is used to compress it. There are several bugs in this +tool and its maintainer has seemingly abandoned the project. It has been forked +and is available (with fixes) here: [theuni/libdmg-hfsplus](https://github.com/theuni/libdmg-hfsplus). + +The 'dmg' tool has the ability to create DMGs from scratch as well, but this +functionality is broken. Only the compression feature is currently used. +Ideally, the creation could be fixed and genisoimage would no longer be necessary. + +Background images and other features can be added to DMG files by inserting a +.DS_Store before creation. This is generated by the script +contrib/macdeploy/custom_dsstore.py. + +As of OS X 10.9 Mavericks, using an Apple-blessed key to sign binaries is a +requirement in order to satisfy the new Gatekeeper requirements. Because this +private key cannot be shared, we'll have to be a bit creative in order for the +build process to remain somewhat deterministic. Here's how it works: + +- Builders use Gitian to create an unsigned release. This outputs an unsigned + dmg which users may choose to bless and run. It also outputs an unsigned app + structure in the form of a tarball, which also contains all of the tools + that have been previously (deterministically) built in order to create a + final dmg. +- The Apple keyholder uses this unsigned app to create a detached signature, + using the script that is also included there. Detached signatures are available from this [repository](https://github.com/bitcoin-core/bitcoin-detached-sigs). +- Builders feed the unsigned app + detached signature back into Gitian. It + uses the pre-built tools to recombine the pieces into a deterministic dmg. + diff --git a/doc/fuzzing.md b/doc/fuzzing.md index 23317e938e..08b73d3b3c 100644 --- a/doc/fuzzing.md +++ b/doc/fuzzing.md @@ -1,9 +1,9 @@ Fuzz-testing Bitcoin Core ========================== -A special test harness `test_bitcoin_fuzzy` is provided to provide an easy -entry point for fuzzers and the like. In this document we'll describe how to -use it with AFL and libFuzzer. +A special test harness in `src/test/fuzz/` is provided for each fuzz target to +provide an easy entry point for fuzzers and the like. In this document we'll +describe how to use it with AFL and libFuzzer. ## AFL @@ -23,10 +23,10 @@ export AFLPATH=$PWD To build Bitcoin Core using AFL instrumentation (this assumes that the `AFLPATH` was set as above): ``` -./configure --disable-ccache --disable-shared --enable-tests CC=${AFLPATH}/afl-gcc CXX=${AFLPATH}/afl-g++ +./configure --disable-ccache --disable-shared --enable-tests --enable-fuzz CC=${AFLPATH}/afl-gcc CXX=${AFLPATH}/afl-g++ export AFL_HARDEN=1 cd src/ -make test/test_bitcoin_fuzzy +make ``` We disable ccache because we don't want to pollute the ccache with instrumented objects, and similarly don't want to use non-instrumented cached objects linked @@ -35,7 +35,7 @@ in. The fuzzing can be sped up significantly (~200x) by using `afl-clang-fast` and `afl-clang-fast++` in place of `afl-gcc` and `afl-g++` when compiling. When compiling using `afl-clang-fast`/`afl-clang-fast++` the resulting -`test_bitcoin_fuzzy` binary will be instrumented in such a way that the AFL +binary will be instrumented in such a way that the AFL features "persistent mode" and "deferred forkserver" can be used. See https://github.com/mcarpenter/afl/tree/master/llvm_mode for details. @@ -63,7 +63,7 @@ Extract these (or other starting inputs) into the `inputs` directory before star To start the actual fuzzing use: ``` -$AFLPATH/afl-fuzz -i ${AFLIN} -o ${AFLOUT} -m52 -- test/test_bitcoin_fuzzy +$AFLPATH/afl-fuzz -i ${AFLIN} -o ${AFLOUT} -m52 -- test/fuzz/fuzz_target_foo ``` You may have to change a few kernel parameters to test optimally - `afl-fuzz` @@ -77,7 +77,7 @@ found in the `compiler-rt` runtime libraries package). To build the `test/test_bitcoin_fuzzy` executable run ``` -./configure --disable-ccache --with-sanitizers=fuzzer,address CC=clang CXX=clang++ +./configure --disable-ccache --enable-fuzz --with-sanitizers=fuzzer,address CC=clang CXX=clang++ make ``` diff --git a/doc/init.md b/doc/init.md index 5778b09d05..a6c9bb94d8 100644 --- a/doc/init.md +++ b/doc/init.md @@ -56,7 +56,7 @@ All three configurations assume several paths that might need to be adjusted. Binary: `/usr/bin/bitcoind` Configuration file: `/etc/bitcoin/bitcoin.conf` Data directory: `/var/lib/bitcoind` -PID file: `/var/run/bitcoind/bitcoind.pid` (OpenRC and Upstart) or `/var/lib/bitcoind/bitcoind.pid` (systemd) +PID file: `/var/run/bitcoind/bitcoind.pid` (OpenRC and Upstart) or `/run/bitcoind/bitcoind.pid` (systemd) Lock file: `/var/lock/subsys/bitcoind` (CentOS) The configuration file, PID directory (if applicable) and data directory @@ -65,6 +65,22 @@ reasons to make the configuration file and data directory only readable by the bitcoin user and group. Access to bitcoin-cli and other bitcoind rpc clients can then be controlled by group membership. +NOTE: When using the systemd .service file, the creation of the aforementioned +directories and the setting of their permissions is automatically handled by +systemd. Directories are given a permission of 710, giving the bitcoin group +access to files under it _if_ the files themselves give permission to the +bitcoin group to do so (e.g. when `-sysperms` is specified). This does not allow +for the listing of files under the directory. + +NOTE: It is not currently possible to override `datadir` in +`/etc/bitcoin/bitcoin.conf` with the current systemd, OpenRC, and Upstart init +files out-of-the-box. This is because the command line options specified in the +init files take precedence over the configurations in +`/etc/bitcoin/bitcoin.conf`. However, some init systems have their own +configuration mechanisms that would allow for overriding the command line +options specified in the init files (e.g. setting `BITCOIND_DATADIR` for +OpenRC). + ### macOS Binary: `/usr/local/bin/bitcoind` diff --git a/doc/release-notes/release-notes-pr12255.md b/doc/release-notes/release-notes-pr12255.md new file mode 100644 index 0000000000..5ac8b44283 --- /dev/null +++ b/doc/release-notes/release-notes-pr12255.md @@ -0,0 +1,17 @@ +systemd init file +========= + +The systemd init file (`contrib/init/bitcoind.service`) has been changed to use +`/var/lib/bitcoind` as the data directory instead of `~bitcoin/.bitcoin`. This +change makes Bitcoin Core more consistent with other services, and makes the +systemd init config more consistent with existing Upstart and OpenRC configs. + +The configuration, PID, and data directories are now completely managed by +systemd, which will take care of their creation, permissions, etc. See +[`systemd.exec (5)`](https://www.freedesktop.org/software/systemd/man/systemd.exec.html#RuntimeDirectory=) +for more details. + +When using the provided init files under `contrib/init`, overriding the +`datadir` option in `/etc/bitcoin/bitcoin.conf` will have no effect. This is +because the command line arguments specified in the init files take precedence +over the options specified in `/etc/bitcoin/bitcoin.conf`. diff --git a/doc/release-process.md b/doc/release-process.md index 9fcd5e2298..d20a3dc6b3 100644 --- a/doc/release-process.md +++ b/doc/release-process.md @@ -92,7 +92,7 @@ Ensure gitian-builder is up-to-date: echo 'f9a8cdb38b9c309326764ebc937cba1523a3a751a7ab05df3ecc99d18ae466c9 inputs/osslsigncode-1.7.1.tar.gz' | sha256sum -c popd -Create the macOS SDK tarball, see the [macOS readme](README_osx.md) for details, and copy it into the inputs directory. +Create the macOS SDK tarball, see the [macOS build instructions](build-osx.md#deterministic-macos-dmg-notes) for details, and copy it into the inputs directory. ### Optional: Seed the Gitian sources cache and offline git repositories diff --git a/doc/translation_process.md b/doc/translation_process.md index 19f145e9bf..9692832842 100644 --- a/doc/translation_process.md +++ b/doc/translation_process.md @@ -68,11 +68,21 @@ The Transifex Bitcoin project config file is included as part of the repo. It ca To assist in updating translations, we have created a script to help. 1. `python contrib/devtools/update-translations.py` -2. Update `src/qt/bitcoin_locale.qrc` manually or via - `ls src/qt/locale/*ts|xargs -n1 basename|sed 's/\(bitcoin_\(.*\)\).ts/<file alias="\2">locale\/\1.qm<\/file>/'` -3. Update `src/Makefile.qt.include` manually or via - `ls src/qt/locale/*ts|xargs -n1 basename|sed 's/\(bitcoin_\(.*\)\).ts/ qt\/locale\/\1.ts \\/'` -4. `git add` new translations from `src/qt/locale/` +2. `git add` new translations from `src/qt/locale/` +3. Update `src/qt/bitcoin_locale.qrc` manually or via +```bash +git ls-files src/qt/locale/*ts|xargs -n1 basename|sed 's/\(bitcoin_\(.*\)\).ts/<file alias="\2">locale\/\1.qm<\/file>/' +``` +4. Update `src/Makefile.qt.include` manually or via +```bash +git ls-files src/qt/locale/*ts|xargs -n1 basename|sed 's/\(bitcoin_\(.*\)\).ts/ qt\/locale\/\1.ts \\/' +``` +5. Update `build_msvc/libbitcoin_qt/libbitcoin_qt.vcxproj` or via +```bash +git ls-files src/qt/locale/*ts|xargs -n1 basename | + sed 's/@/%40/' | + sed 's/\(bitcoin_\(.*\)\).ts/ <None Include="..\\..\\src\\qt\\locale\\\1.ts">\n <DeploymentContent>true<\/DeploymentContent>\n <\/None>/' +``` **Do not directly download translations** one by one from the Transifex website, as we do a few post-processing steps before committing the translations. |