From e5a680dc6a69cc5c35f2debeba15d4886fa4f789 Mon Sep 17 00:00:00 2001 From: fanquake Date: Sun, 12 Jun 2016 12:59:02 +0800 Subject: [Doc] Update OS X build notes for 10.11 SDK --- doc/README_osx.md | 82 ++++++++++++++++++++++++++++++++++++++++++++++++++ doc/README_osx.txt | 82 -------------------------------------------------- doc/build-osx.md | 8 +++-- doc/release-process.md | 16 +++------- 4 files changed, 92 insertions(+), 96 deletions(-) create mode 100644 doc/README_osx.md delete mode 100644 doc/README_osx.txt (limited to 'doc') diff --git a/doc/README_osx.md b/doc/README_osx.md new file mode 100644 index 0000000000..aed3cd97e1 --- /dev/null +++ b/doc/README_osx.md @@ -0,0 +1,82 @@ +Deterministic OS X Dmg Notes. + +Working OS X DMGs are created in Linux by combining a recent clang, +the Apple binutils (ld, ar, etc) and DMG authoring tools. + +Apple uses clang extensively for development and has upstreamed the necessary +functionality so that a vanilla clang can take advantage. It supports the use +of -F, -target, -mmacosx-version-min, and --sysroot, which are all necessary +when building for OS X. + +Apple's version of binutils (called cctools) contains lots of functionality +missing in the FSF's binutils. In addition to extra linker options for +frameworks and sysroots, several other tools are needed as well such as +install_name_tool, lipo, and nmedit. These do not build under linux, so they +have been patched to do so. The work here was used as a starting point: +[mingwandroid/toolchain4](https://github.com/mingwandroid/toolchain4). + +In order to build a working toolchain, the following source packages are needed +from Apple: cctools, dyld, and ld64. + +These tools inject timestamps by default, which produce non-deterministic +binaries. The ZERO_AR_DATE environment variable is used to disable that. + +This version of cctools has been patched to use the current version of clang's +headers and and its libLTO.so rather than those from llvmgcc, as it was +originally done in toolchain4. + +To complicate things further, all builds must target an Apple SDK. These SDKs +are free to download, but not redistributable. +To obtain it, register for a developer account, then download the [Xcode 7.3.1 dmg](https://developer.apple.com/devcenter/download.action?path=/Developer_Tools/Xcode_7.3.1/Xcode_7.3.1.dmg). + +This file is several gigabytes in size, but only a single directory inside is +needed: +``` +Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk +``` + +Unfortunately, the usual linux tools (7zip, hpmount, loopback mount) are incapable of opening this file. +To create a tarball suitable for Gitian input, mount the dmg in OS X, then create it with: +``` + $ tar -C /Volumes/Xcode/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/ -czf MacOSX10.11.sdk.tar.gz MacOSX10.11.sdk +``` + +The Gitian descriptors build 2 sets of files: Linux tools, then Apple binaries +which are created using these tools. The build process has been designed to +avoid including the SDK's files in Gitian's outputs. All interim tarballs are +fully deterministic and may be freely redistributed. + +genisoimage is used to create the initial DMG. It is not deterministic as-is, +so it has been patched. A system genisoimage will work fine, but it will not +be deterministic because the file-order will change between invocations. +The patch can be seen here: [theuni/osx-cross-depends](https://raw.githubusercontent.com/theuni/osx-cross-depends/master/patches/cdrtools/genisoimage.diff). +No effort was made to fix this cleanly, so it likely leaks memory badly. But +it's only used for a single invocation, so that's no real concern. + +genisoimage cannot compress DMGs, so afterwards, the 'dmg' tool from the +libdmg-hfsplus project is used to compress it. There are several bugs in this +tool and its maintainer has seemingly abandoned the project. It has been forked +and is available (with fixes) here: [theuni/libdmg-hfsplus](https://github.com/theuni/libdmg-hfsplus). + +The 'dmg' tool has the ability to create DMGs from scratch as well, but this +functionality is broken. Only the compression feature is currently used. +Ideally, the creation could be fixed and genisoimage would no longer be necessary. + +Background images and other features can be added to DMG files by inserting a +.DS_Store before creation. This is generated by the script +contrib/macdeploy/custom_dsstore.py. + +As of OS X Mavericks (10.9), using an Apple-blessed key to sign binaries is a +requirement in order to satisfy the new Gatekeeper requirements. Because this +private key cannot be shared, we'll have to be a bit creative in order for the +build process to remain somewhat deterministic. Here's how it works: + +- Builders use Gitian to create an unsigned release. This outputs an unsigned + dmg which users may choose to bless and run. It also outputs an unsigned app + structure in the form of a tarball, which also contains all of the tools + that have been previously (deterministically) built in order to create a + final dmg. +- The Apple keyholder uses this unsigned app to create a detached signature, + using the script that is also included there. Detached signatures are available from this [repository](https://github.com/bitcoin-core/bitcoin-detached-sigs). +- Builders feed the unsigned app + detached signature back into Gitian. It + uses the pre-built tools to recombine the pieces into a deterministic dmg. diff --git a/doc/README_osx.txt b/doc/README_osx.txt deleted file mode 100644 index c13efaa145..0000000000 --- a/doc/README_osx.txt +++ /dev/null @@ -1,82 +0,0 @@ -Deterministic OS X Dmg Notes. - -Working OS X DMGs are created in Linux by combining a recent clang, -the Apple's binutils (ld, ar, etc), and DMG authoring tools. - -Apple uses clang extensively for development and has upstreamed the necessary -functionality so that a vanilla clang can take advantage. It supports the use -of -F, -target, -mmacosx-version-min, and --sysroot, which are all necessary -when building for OS X. A pre-compiled version of 3.2 is used because it was not -available in the Precise repositories at the time this work was started. In the -future, it can be switched to use system packages instead. - -Apple's version of binutils (called cctools) contains lots of functionality -missing in the FSF's binutils. In addition to extra linker options for -frameworks and sysroots, several other tools are needed as well such as -install_name_tool, lipo, and nmedit. These do not build under linux, so they -have been patched to do so. The work here was used as a starting point: -https://github.com/mingwandroid/toolchain4 - -In order to build a working toolchain, the following source packages are needed -from Apple: cctools, dyld, and ld64. - -These tools inject timestamps by default, which produce non-deterministic -binaries. The ZERO_AR_DATE environment variable is used to disable that. - -This version of cctools has been patched to use the current version of clang's -headers and and its libLTO.so rather than those from llvmgcc, as it was -originally done in toolchain4. - -To complicate things further, all builds must target an Apple SDK. These SDKs -are free to download, but not redistributable. -To obtain it, register for a developer account, then download the Xcode 6.1.1 dmg: -https://developer.apple.com/devcenter/download.action?path=/Developer_Tools/xcode_6.1.1/xcode_6.1.1.dmg - -This file is several gigabytes in size, but only a single directory inside is -needed: Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.9.sdk - -Unfortunately, the usual linux tools (7zip, hpmount, loopback mount) are incapable of opening this file. -To create a tarball suitable for Gitian input, mount the dmg in OS X, then create it with: - $ tar -C /Volumes/Xcode/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/ -czf MacOSX10.9.sdk.tar.gz MacOSX10.9.sdk - - -The Gitian descriptors build 2 sets of files: Linux tools, then Apple binaries -which are created using these tools. The build process has been designed to -avoid including the SDK's files in Gitian's outputs. All interim tarballs are -fully deterministic and may be freely redistributed. - -genisoimage is used to create the initial DMG. It is not deterministic as-is, -so it has been patched. A system genisoimage will work fine, but it will not -be deterministic because the file-order will change between invocations. -The patch can be seen here: -https://raw.githubusercontent.com/theuni/osx-cross-depends/master/patches/cdrtools/genisoimage.diff -No effort was made to fix this cleanly, so it likely leaks memory badly. But -it's only used for a single invocation, so that's no real concern. - -genisoimage cannot compress DMGs, so afterwards, the 'dmg' tool from the -libdmg-hfsplus project is used to compress it. There are several bugs in this -tool and its maintainer has seemingly abandoned the project. It has been forked -and is available (with fixes) here: https://github.com/theuni/libdmg-hfsplus . - -The 'dmg' tool has the ability to create DMGs from scratch as well, but this -functionality is broken. Only the compression feature is currently used. -Ideally, the creation could be fixed and genisoimage would no longer be necessary. - -Background images and other features can be added to DMG files by inserting a -.DS_Store before creation. This is generated by the script -contrib/macdeploy/custom_dsstore.py. - -As of OS X Mavericks (10.9), using an Apple-blessed key to sign binaries is a -requirement in order to satisfy the new Gatekeeper requirements. Because this -private key cannot be shared, we'll have to be a bit creative in order for the -build process to remain somewhat deterministic. Here's how it works: - -- Builders use Gitian to create an unsigned release. This outputs an unsigned - dmg which users may choose to bless and run. It also outputs an unsigned app - structure in the form of a tarball, which also contains all of the tools - that have been previously (deterministically) built in order to create a - final dmg. -- The Apple keyholder uses this unsigned app to create a detached signature, - using the script that is also included there. -- Builders feed the unsigned app + detached signature back into Gitian. It - uses the pre-built tools to recombine the pieces into a deterministic dmg. diff --git a/doc/build-osx.md b/doc/build-osx.md index 89d7816c9a..c9eb4225ab 100644 --- a/doc/build-osx.md +++ b/doc/build-osx.md @@ -5,11 +5,13 @@ The built-in one is located in `/Applications/Utilities/Terminal.app`. Preparation ----------- -Download and install [Xcode](https://developer.apple.com/xcode/download). +Install the OS X command line tools: -Once installed, run `xcode-select --install` to install the OS X command line tools. +`xcode-select --install` -Install [Homebrew](http://brew.sh). +When the popup appears, click `Install`. + +Then install [Homebrew](http://brew.sh). Dependencies ---------------------- diff --git a/doc/release-process.md b/doc/release-process.md index 3bfcc38177..f5d0d836d1 100644 --- a/doc/release-process.md +++ b/doc/release-process.md @@ -3,7 +3,7 @@ Release Process Before every release candidate: -* Update translations (ping wumpus on IRC) see [translation_process.md](https://github.com/bitcoin/bitcoin/blob/master/doc/translation_process.md#syncing-with-transifex) +* Update translations (ping wumpus on IRC) see [translation_process.md](https://github.com/bitcoin/bitcoin/blob/master/doc/translation_process.md#synchronising-translations). Before every minor and major release: @@ -13,7 +13,7 @@ Before every minor and major release: Before every major release: -* Update hardcoded [seeds](/contrib/seeds/README.md), see [this pull request](https://github.com/bitcoin/bitcoin/pull/7415) for an example. +* Update hardcoded [seeds](/contrib/seeds/README.md), see [this pull request](https://github.com/bitcoin/bitcoin/pull/7415) for an example. ### First time / New builders @@ -75,7 +75,7 @@ Ensure your gitian.sigs are up-to-date if you wish to gverify your builds agains git pull popd -Ensure gitian-builder is up-to-date to take advantage of new caching features (`e9741525c` or later is recommended). +Ensure gitian-builder is up-to-date: pushd ./gitian-builder git pull @@ -89,13 +89,7 @@ Ensure gitian-builder is up-to-date to take advantage of new caching features (` wget -P inputs http://downloads.sourceforge.net/project/osslsigncode/osslsigncode/osslsigncode-1.7.1.tar.gz popd -Register and download the Apple SDK: see [OS X readme](README_osx.txt) for details. - -https://developer.apple.com/devcenter/download.action?path=/Developer_Tools/xcode_6.1.1/xcode_6.1.1.dmg - -Using a Mac, create a tarball for the 10.9 SDK and copy it to the inputs directory: - - tar -C /Volumes/Xcode/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/ -czf MacOSX10.9.sdk.tar.gz MacOSX10.9.sdk +Create the OS X SDK tarball, see the [OS X readme](README_osx.md) for details, and copy it into the inputs directory. ### Optional: Seed the Gitian sources cache and offline git repositories @@ -239,7 +233,7 @@ transmission-show -m Insert the magnet URI into the announcement sent to mailing lists. This permits people without access to `bitcoin.org` to download the binary distribution. Also put it into the `optional_magnetlink:` slot in the YAML file for -bitcoin.org (see below for bitcoin.org update instructions). +bitcoin.org (see below for bitcoin.org update instructions). - Update bitcoin.org version -- cgit v1.2.3