aboutsummaryrefslogtreecommitdiff
diff options
context:
space:
mode:
authorMartin Erlandsson <merland@users.noreply.github.com>2019-01-17 12:26:55 +0100
committerMartin Erlandsson <merland@users.noreply.github.com>2019-01-21 20:24:33 +0100
commitf24ed6d39f963e7f1c0b4dbd9b784132ef975f2a (patch)
tree20b05ef7ccb4d959356ccbb72d8c25e86f63bfa1
parent391a27376b30492f55b398fc220ee96cf3769a32 (diff)
downloadbitcoin-f24ed6d39f963e7f1c0b4dbd9b784132ef975f2a.tar.xz
Delete README_osx.md and move its contents into build-osx.md
-rw-r--r--doc/README_osx.md97
-rw-r--r--doc/build-osx.md100
-rw-r--r--doc/release-process.md2
3 files changed, 101 insertions, 98 deletions
diff --git a/doc/README_osx.md b/doc/README_osx.md
deleted file mode 100644
index 739e22d634..0000000000
--- a/doc/README_osx.md
+++ /dev/null
@@ -1,97 +0,0 @@
-Deterministic macOS DMG Notes.
-
-Working macOS DMGs are created in Linux by combining a recent clang,
-the Apple binutils (ld, ar, etc) and DMG authoring tools.
-
-Apple uses clang extensively for development and has upstreamed the necessary
-functionality so that a vanilla clang can take advantage. It supports the use
-of -F, -target, -mmacosx-version-min, and --sysroot, which are all necessary
-when building for macOS.
-
-Apple's version of binutils (called cctools) contains lots of functionality
-missing in the FSF's binutils. In addition to extra linker options for
-frameworks and sysroots, several other tools are needed as well such as
-install_name_tool, lipo, and nmedit. These do not build under linux, so they
-have been patched to do so. The work here was used as a starting point:
-[mingwandroid/toolchain4](https://github.com/mingwandroid/toolchain4).
-
-In order to build a working toolchain, the following source packages are needed
-from Apple: cctools, dyld, and ld64.
-
-These tools inject timestamps by default, which produce non-deterministic
-binaries. The ZERO_AR_DATE environment variable is used to disable that.
-
-This version of cctools has been patched to use the current version of clang's
-headers and its libLTO.so rather than those from llvmgcc, as it was
-originally done in toolchain4.
-
-To complicate things further, all builds must target an Apple SDK. These SDKs
-are free to download, but not redistributable.
-To obtain it, register for a developer account, then download the [Xcode 7.3.1 dmg](https://developer.apple.com/devcenter/download.action?path=/Developer_Tools/Xcode_7.3.1/Xcode_7.3.1.dmg).
-
-This file is several gigabytes in size, but only a single directory inside is
-needed:
-```
-Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk
-```
-
-Unfortunately, the usual linux tools (7zip, hpmount, loopback mount) are incapable of opening this file.
-To create a tarball suitable for Gitian input, there are two options:
-
-Using macOS, you can mount the dmg, and then create it with:
-```
- $ hdiutil attach Xcode_7.3.1.dmg
- $ tar -C /Volumes/Xcode/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/ -czf MacOSX10.11.sdk.tar.gz MacOSX10.11.sdk
-```
-
-Alternatively, you can use 7zip and SleuthKit to extract the files one by one.
-The script contrib/macdeploy/extract-osx-sdk.sh automates this. First ensure
-the dmg file is in the current directory, and then run the script. You may wish
-to delete the intermediate 5.hfs file and MacOSX10.11.sdk (the directory) when
-you've confirmed the extraction succeeded.
-
-```bash
-apt-get install p7zip-full sleuthkit
-contrib/macdeploy/extract-osx-sdk.sh
-rm -rf 5.hfs MacOSX10.11.sdk
-```
-
-The Gitian descriptors build 2 sets of files: Linux tools, then Apple binaries
-which are created using these tools. The build process has been designed to
-avoid including the SDK's files in Gitian's outputs. All interim tarballs are
-fully deterministic and may be freely redistributed.
-
-genisoimage is used to create the initial DMG. It is not deterministic as-is,
-so it has been patched. A system genisoimage will work fine, but it will not
-be deterministic because the file-order will change between invocations.
-The patch can be seen here: [theuni/osx-cross-depends](https://raw.githubusercontent.com/theuni/osx-cross-depends/master/patches/cdrtools/genisoimage.diff).
-No effort was made to fix this cleanly, so it likely leaks memory badly. But
-it's only used for a single invocation, so that's no real concern.
-
-genisoimage cannot compress DMGs, so afterwards, the 'dmg' tool from the
-libdmg-hfsplus project is used to compress it. There are several bugs in this
-tool and its maintainer has seemingly abandoned the project. It has been forked
-and is available (with fixes) here: [theuni/libdmg-hfsplus](https://github.com/theuni/libdmg-hfsplus).
-
-The 'dmg' tool has the ability to create DMGs from scratch as well, but this
-functionality is broken. Only the compression feature is currently used.
-Ideally, the creation could be fixed and genisoimage would no longer be necessary.
-
-Background images and other features can be added to DMG files by inserting a
-.DS_Store before creation. This is generated by the script
-contrib/macdeploy/custom_dsstore.py.
-
-As of OS X 10.9 Mavericks, using an Apple-blessed key to sign binaries is a
-requirement in order to satisfy the new Gatekeeper requirements. Because this
-private key cannot be shared, we'll have to be a bit creative in order for the
-build process to remain somewhat deterministic. Here's how it works:
-
-- Builders use Gitian to create an unsigned release. This outputs an unsigned
- dmg which users may choose to bless and run. It also outputs an unsigned app
- structure in the form of a tarball, which also contains all of the tools
- that have been previously (deterministically) built in order to create a
- final dmg.
-- The Apple keyholder uses this unsigned app to create a detached signature,
- using the script that is also included there. Detached signatures are available from this [repository](https://github.com/bitcoin-core/bitcoin-detached-sigs).
-- Builders feed the unsigned app + detached signature back into Gitian. It
- uses the pre-built tools to recombine the pieces into a deterministic dmg.
diff --git a/doc/build-osx.md b/doc/build-osx.md
index c9a59bab83..119896dc67 100644
--- a/doc/build-osx.md
+++ b/doc/build-osx.md
@@ -105,3 +105,103 @@ Notes
* Tested on OS X 10.10 Yosemite through macOS 10.13 High Sierra on 64-bit Intel processors only.
* Building with downloaded Qt binaries is not officially supported. See the notes in [#7714](https://github.com/bitcoin/bitcoin/issues/7714)
+
+Deterministic macOS DMG Notes
+-----------------------------
+
+Working macOS DMGs are created in Linux by combining a recent clang,
+the Apple binutils (ld, ar, etc) and DMG authoring tools.
+
+Apple uses clang extensively for development and has upstreamed the necessary
+functionality so that a vanilla clang can take advantage. It supports the use
+of -F, -target, -mmacosx-version-min, and --sysroot, which are all necessary
+when building for macOS.
+
+Apple's version of binutils (called cctools) contains lots of functionality
+missing in the FSF's binutils. In addition to extra linker options for
+frameworks and sysroots, several other tools are needed as well such as
+install_name_tool, lipo, and nmedit. These do not build under linux, so they
+have been patched to do so. The work here was used as a starting point:
+[mingwandroid/toolchain4](https://github.com/mingwandroid/toolchain4).
+
+In order to build a working toolchain, the following source packages are needed
+from Apple: cctools, dyld, and ld64.
+
+These tools inject timestamps by default, which produce non-deterministic
+binaries. The ZERO_AR_DATE environment variable is used to disable that.
+
+This version of cctools has been patched to use the current version of clang's
+headers and its libLTO.so rather than those from llvmgcc, as it was
+originally done in toolchain4.
+
+To complicate things further, all builds must target an Apple SDK. These SDKs
+are free to download, but not redistributable.
+To obtain it, register for a developer account, then download the [Xcode 7.3.1 dmg](https://developer.apple.com/devcenter/download.action?path=/Developer_Tools/Xcode_7.3.1/Xcode_7.3.1.dmg).
+
+This file is several gigabytes in size, but only a single directory inside is
+needed:
+```
+Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk
+```
+
+Unfortunately, the usual linux tools (7zip, hpmount, loopback mount) are incapable of opening this file.
+To create a tarball suitable for Gitian input, there are two options:
+
+Using macOS, you can mount the dmg, and then create it with:
+```
+ $ hdiutil attach Xcode_7.3.1.dmg
+ $ tar -C /Volumes/Xcode/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/ -czf MacOSX10.11.sdk.tar.gz MacOSX10.11.sdk
+```
+
+Alternatively, you can use 7zip and SleuthKit to extract the files one by one.
+The script contrib/macdeploy/extract-osx-sdk.sh automates this. First ensure
+the dmg file is in the current directory, and then run the script. You may wish
+to delete the intermediate 5.hfs file and MacOSX10.11.sdk (the directory) when
+you've confirmed the extraction succeeded.
+
+```bash
+apt-get install p7zip-full sleuthkit
+contrib/macdeploy/extract-osx-sdk.sh
+rm -rf 5.hfs MacOSX10.11.sdk
+```
+
+The Gitian descriptors build 2 sets of files: Linux tools, then Apple binaries
+which are created using these tools. The build process has been designed to
+avoid including the SDK's files in Gitian's outputs. All interim tarballs are
+fully deterministic and may be freely redistributed.
+
+genisoimage is used to create the initial DMG. It is not deterministic as-is,
+so it has been patched. A system genisoimage will work fine, but it will not
+be deterministic because the file-order will change between invocations.
+The patch can be seen here: [theuni/osx-cross-depends](https://raw.githubusercontent.com/theuni/osx-cross-depends/master/patches/cdrtools/genisoimage.diff).
+No effort was made to fix this cleanly, so it likely leaks memory badly. But
+it's only used for a single invocation, so that's no real concern.
+
+genisoimage cannot compress DMGs, so afterwards, the 'dmg' tool from the
+libdmg-hfsplus project is used to compress it. There are several bugs in this
+tool and its maintainer has seemingly abandoned the project. It has been forked
+and is available (with fixes) here: [theuni/libdmg-hfsplus](https://github.com/theuni/libdmg-hfsplus).
+
+The 'dmg' tool has the ability to create DMGs from scratch as well, but this
+functionality is broken. Only the compression feature is currently used.
+Ideally, the creation could be fixed and genisoimage would no longer be necessary.
+
+Background images and other features can be added to DMG files by inserting a
+.DS_Store before creation. This is generated by the script
+contrib/macdeploy/custom_dsstore.py.
+
+As of OS X 10.9 Mavericks, using an Apple-blessed key to sign binaries is a
+requirement in order to satisfy the new Gatekeeper requirements. Because this
+private key cannot be shared, we'll have to be a bit creative in order for the
+build process to remain somewhat deterministic. Here's how it works:
+
+- Builders use Gitian to create an unsigned release. This outputs an unsigned
+ dmg which users may choose to bless and run. It also outputs an unsigned app
+ structure in the form of a tarball, which also contains all of the tools
+ that have been previously (deterministically) built in order to create a
+ final dmg.
+- The Apple keyholder uses this unsigned app to create a detached signature,
+ using the script that is also included there. Detached signatures are available from this [repository](https://github.com/bitcoin-core/bitcoin-detached-sigs).
+- Builders feed the unsigned app + detached signature back into Gitian. It
+ uses the pre-built tools to recombine the pieces into a deterministic dmg.
+
diff --git a/doc/release-process.md b/doc/release-process.md
index 9fcd5e2298..d20a3dc6b3 100644
--- a/doc/release-process.md
+++ b/doc/release-process.md
@@ -92,7 +92,7 @@ Ensure gitian-builder is up-to-date:
echo 'f9a8cdb38b9c309326764ebc937cba1523a3a751a7ab05df3ecc99d18ae466c9 inputs/osslsigncode-1.7.1.tar.gz' | sha256sum -c
popd
-Create the macOS SDK tarball, see the [macOS readme](README_osx.md) for details, and copy it into the inputs directory.
+Create the macOS SDK tarball, see the [macOS build instructions](build-osx.md#deterministic-macos-dmg-notes) for details, and copy it into the inputs directory.
### Optional: Seed the Gitian sources cache and offline git repositories