From cc3db874b5cdff4e31d11bacbf4db6986ff48805 Mon Sep 17 00:00:00 2001 From: Shawn Wilkinson Date: Wed, 22 Apr 2015 00:06:39 -0400 Subject: doc: Documentation in Markdown for Depends Dir Documentation more readable when viewed on Github. Some extra changes by @laanwj: - Make README.usage the default README. This is more convenient from a user perspective. Link to other documentation in this default README - Add list of popular targets for cross compilation, change default to Win64 instead of Win32 --- depends/README | 55 ------------------ depends/README.md | 56 ++++++++++++++++++ depends/README.packages | 130 ------------------------------------------ depends/README.usage | 34 ----------- depends/description.md | 53 +++++++++++++++++ depends/packages.md | 147 ++++++++++++++++++++++++++++++++++++++++++++++++ 6 files changed, 256 insertions(+), 219 deletions(-) delete mode 100644 depends/README create mode 100644 depends/README.md delete mode 100644 depends/README.packages delete mode 100644 depends/README.usage create mode 100644 depends/description.md create mode 100644 depends/packages.md diff --git a/depends/README b/depends/README deleted file mode 100644 index 55e7222697..0000000000 --- a/depends/README +++ /dev/null @@ -1,55 +0,0 @@ -This is a system of building and caching dependencies necessary for building -Bitcoin. - -There are several features that make it different from most similar systems: - -- It is designed to be builder and host agnostic - -In theory, binaries for any target OS/architecture can be created, from a -builder running any OS/architecture. In practice, build-side tools must be -specified when the defaults don't fit, and packages must be amended to work -on new hosts. For now, a build architecture of x86_64 is assumed, either on -Linux or OSX. - -- No reliance on timestamps - -File presence is used to determine what needs to be built. This makes the -results distributable and easily digestable by automated builders. - -- Each build only has its specified dependencies available at build-time. - -For each build, the sysroot is wiped and the (recursive) dependencies are -installed. This makes each build deterministic, since there will never be any -unknown files available to cause side-effects. - -- Each package is cached and only rebuilt as needed. - -Before building, a unique build-id is generated for each package. This id -consists of a hash of all files used to build the package (Makefiles, packages, -etc), and as well as a hash of the same data for each recursive dependency. If -any portion of a package's build recipe changes, it will be rebuilt as well as -any other package that depends on it. If any of the main makefiles (Makefile, -funcs.mk, etc) are changed, all packages will be rebuilt. After building, the -results are cached into a tarball that can be re-used and distributed. - -- Package build results are (relatively) deterministic. - -Each package is configured and patched so that it will yield the same -build-results with each consequent build, within a reasonable set of -constraints. Some things like timestamp insertion are unavoidable, and are -beyond the scope of this system. Additionally, the toolchain itself must be -capable of deterministic results. When revisions are properly bumped, a cached -build should represent an exact single payload. - -- Sources are fetched and verified automatically - -Each package must define its source location and checksum. The build will fail -if the fetched source does not match. Sources may be pre-seeded and/or cached -as desired. - -- Self-cleaning - -Build and staging dirs are wiped after use, and any previous version of a -cached result is removed following a successful build. Automated builders -should be able to build each revision and store the results with no further -intervention. diff --git a/depends/README.md b/depends/README.md new file mode 100644 index 0000000000..2dc0b9e47e --- /dev/null +++ b/depends/README.md @@ -0,0 +1,56 @@ +### Usage + +To build dependencies for the current arch+OS: + + make + +To build for another arch/OS: + + make HOST=host-platform-triplet + +For example: + + make HOST=x86_64-w64-mingw32 -j4 + +A prefix will be generated that's suitable for plugging into Bitcoin's +configure. In the above example, a dir named i686-w64-mingw32 will be +created. To use it for Bitcoin: + + ./configure --prefix=`pwd`/depends/x86_64-w64-mingw32 + +Common `host-platform-triplets` for cross compilation are: + +- `i686-w64-mingw32` for Win32 +- `x86_64-w64-mingw32` for Win64 +- `x86_64-apple-darwin11` for MacOSX +- `arm-linux-gnueabihf` for Linux ARM + +No other options are needed, the paths are automatically configured. + +Dependency Options: +The following can be set when running make: make FOO=bar + + SOURCES_PATH: downloaded sources will be placed here + BASE_CACHE: built packages will be placed here + SDK_PATH: Path where sdk's can be found (used by OSX) + FALLBACK_DOWNLOAD_PATH: If a source file can't be fetched, try here before giving up + NO_QT: Don't download/build/cache qt and its dependencies + NO_WALLET: Don't download/build/cache libs needed to enable the wallet + NO_UPNP: Don't download/build/cache packages needed for enabling upnp + DEBUG: disable some optimizations and enable more runtime checking + +If some packages are not built, for example `make NO_WALLET=1`, the appropriate +options will be passed to bitcoin's configure. In this case, `--disable-wallet`. + +Additional targets: + + download: run 'make download' to fetch all sources without building them + download-osx: run 'make download-osx' to fetch all sources needed for osx builds + download-win: run 'make download-win' to fetch all sources needed for win builds + download-linux: run 'make download-linux' to fetch all sources needed for linux builds + +### Other documentation + +- [description.md](description.md): General description of the depends system +- [packages.md](packages.md): Steps for adding packages + diff --git a/depends/README.packages b/depends/README.packages deleted file mode 100644 index 6d87402ebc..0000000000 --- a/depends/README.packages +++ /dev/null @@ -1,130 +0,0 @@ -Each recipe consists of 3 main parts: defining identifiers, setting build -variables, and defining build commands. - -The package "mylib" will be used here as an example - -General tips: -mylib_foo is written as $(package)_foo in order to make recipes more similar. - -Identifiers: -Each package is required to define at least these variables: - $(package)_version: - Version of the upstream library or program. If there is no version, a - placeholder such as 1.0 can be used. - $(package)_download_path: - Location of the upstream source, without the file-name. Usually http or - ftp. - $(package)_file_name: - The upstream source filename available at the download path. - $(package)_sha256_hash: - The sha256 hash of the upstream file - -These variables are optional: - $(package)_build_subdir: - cd to this dir before running configure/build/stage commands. - $(package)_download_file: - The file-name of the upstream source if it differs from how it should be - stored locally. This can be used to avoid storing file-names with strange - characters. - $(package)_dependencies: - Names of any other packages that this one depends on. - $(package)_patches: - Filenames of any patches needed to build the package - $(package)_extra_sources: - Any extra files that will be fetched via $(package)_fetch_cmds. These are - specified so that they can be fetched and verified via 'make download'. - -Build Variables: -After defining the main identifiers, build variables may be added or customized -before running the build commands. They should be added to a function called -$(package)_set_vars. For example: - -define $(package)_set_vars -... -endef - -Most variables can be prefixed with the host, architecture, or both, to make -the modifications specific to that case. For example: - - Universal: $(package)_cc=gcc - Linux only: $(package)_linux_cc=gcc - x86_64 only: $(package)_x86_64_cc = gcc - x86_64 linux only: $(package)_x86_64_linux_cc = gcc - -These variables may be set to override or append their default values. - $(package)_cc - $(package)_cxx - $(package)_objc - $(package)_objcxx - $(package)_ar - $(package)_ranlib - $(package)_libtool - $(package)_nm - $(package)_cflags - $(package)_cxxflags - $(package)_ldflags - $(package)_cppflags - $(package)_config_env - $(package)_build_env - $(package)_stage_env - $(package)_build_opts - $(package)_config_opts - -The *_env variables are used to add environment variables to the respective -commands. - -Many variables respect a debug/release suffix as well, in order to use them for -only the appropriate build config. For example: - $(package)_cflags_release = -O3 - $(package)_cflags_i686_debug = -g - $(package)_config_opts_release = --disable-debug - -These will be used in addition to the options that do not specify -debug/release. All builds are considered to be release unless DEBUG=1 is set by -the user. - -Other variables may be defined as needed. - -Build commands: - - For each build, a unique build dir and staging dir are created. For example, - work/build/mylib/1.0-1adac830f6e and work/staging/mylib/1.0-1adac830f6e. - - The following build commands are available for each recipe: - - $(package)_fetch_cmds: - Runs from: build dir - Fetch the source file. If undefined, it will be fetched and verified - against its hash. - $(package)_extract_cmds: - Runs from: build dir - Verify the source file against its hash and extract it. If undefined, the - source is assumed to be a tarball. - $(package)_preprocess_cmds: - Runs from: build dir/$(package)_build_subdir - Preprocess the source as necessary. If undefined, does nothing. - $(package)_config_cmds: - Runs from: build dir/$(package)_build_subdir - Configure the source. If undefined, does nothing. - $(package)_build_cmds: - Runs from: build dir/$(package)_build_subdir - Build the source. If undefined, does nothing. - $(package)_stage_cmds: - Runs from: build dir/$(package)_build_subdir - Stage the build results. If undefined, does nothing. - - The following variables are available for each recipe: - $(1)_staging_dir: package's destination sysroot path - $(1)_staging_prefix_dir: prefix path inside of the package's staging dir - $(1)_extract_dir: path to the package's extracted sources - $(1)_build_dir: path where configure/build/stage commands will be run - $(1)_patch_dir: path where the package's patches (if any) are found - -Notes on build commands: - -For packages built with autotools, $($(package)_autoconf) can be used in the -configure step to (usually) correctly configure automatically. Any -$($(package)_config_opts) will be appended. - -Most autotools projects can be properly staged using: - $(MAKE) DESTDIR=$($(package)_staging_dir) install diff --git a/depends/README.usage b/depends/README.usage deleted file mode 100644 index 24e1231d82..0000000000 --- a/depends/README.usage +++ /dev/null @@ -1,34 +0,0 @@ -To build dependencies for the current arch+OS: - make -To build for another arch/OS: - make HOST=host-platform-triplet && make HOST=host-platform-triplet - (For example: make HOST=i686-w64-mingw32 -j4) - -A prefix will be generated that's suitable for plugging into Bitcoin's -configure. In the above example, a dir named i686-w64-mingw32 will be -created. To use it for Bitcoin: - -./configure --prefix=`pwd`/depends/i686-w64-mingw32 - -No other options are needed, the paths are automatically configured. - -Dependency Options: -The following can be set when running make: make FOO=bar - -SOURCES_PATH: downloaded sources will be placed here -BASE_CACHE: built packages will be placed here -SDK_PATH: Path where sdk's can be found (used by OSX) -FALLBACK_DOWNLOAD_PATH: If a source file can't be fetched, try here before giving up -NO_QT: Don't download/build/cache qt and its dependencies -NO_WALLET: Don't download/build/cache libs needed to enable the wallet -NO_UPNP: Don't download/build/cache packages needed for enabling upnp -DEBUG: disable some optimizations and enable more runtime checking - -If some packages are not built, for example 'make NO_WALLET=1', the appropriate -options will be passed to bitcoin's configure. In this case, --disable-wallet. - -Additional targets: -download: run 'make download' to fetch all sources without building them -download-osx: run 'make download-osx' to fetch all sources needed for osx builds -download-win: run 'make download-win' to fetch all sources needed for win builds -download-linux: run 'make download-linux' to fetch all sources needed for linux builds diff --git a/depends/description.md b/depends/description.md new file mode 100644 index 0000000000..74f9ef3f20 --- /dev/null +++ b/depends/description.md @@ -0,0 +1,53 @@ +This is a system of building and caching dependencies necessary for building Bitcoin. +There are several features that make it different from most similar systems: + +### It is designed to be builder and host agnostic + +In theory, binaries for any target OS/architecture can be created, from a +builder running any OS/architecture. In practice, build-side tools must be +specified when the defaults don't fit, and packages must be amended to work +on new hosts. For now, a build architecture of x86_64 is assumed, either on +Linux or OSX. + +### No reliance on timestamps + +File presence is used to determine what needs to be built. This makes the +results distributable and easily digestable by automated builders. + +### Each build only has its specified dependencies available at build-time. + +For each build, the sysroot is wiped and the (recursive) dependencies are +installed. This makes each build deterministic, since there will never be any +unknown files available to cause side-effects. + +### Each package is cached and only rebuilt as needed. + +Before building, a unique build-id is generated for each package. This id +consists of a hash of all files used to build the package (Makefiles, packages, +etc), and as well as a hash of the same data for each recursive dependency. If +any portion of a package's build recipe changes, it will be rebuilt as well as +any other package that depends on it. If any of the main makefiles (Makefile, +funcs.mk, etc) are changed, all packages will be rebuilt. After building, the +results are cached into a tarball that can be re-used and distributed. + +### Package build results are (relatively) deterministic. + +Each package is configured and patched so that it will yield the same +build-results with each consequent build, within a reasonable set of +constraints. Some things like timestamp insertion are unavoidable, and are +beyond the scope of this system. Additionally, the toolchain itself must be +capable of deterministic results. When revisions are properly bumped, a cached +build should represent an exact single payload. + +### Sources are fetched and verified automatically + +Each package must define its source location and checksum. The build will fail +if the fetched source does not match. Sources may be pre-seeded and/or cached +as desired. + +### Self-cleaning + +Build and staging dirs are wiped after use, and any previous version of a +cached result is removed following a successful build. Automated builders +should be able to build each revision and store the results with no further +intervention. diff --git a/depends/packages.md b/depends/packages.md new file mode 100644 index 0000000000..7c80362509 --- /dev/null +++ b/depends/packages.md @@ -0,0 +1,147 @@ +Each recipe consists of 3 main parts: defining identifiers, setting build +variables, and defining build commands. + +The package "mylib" will be used here as an example + +General tips: +- mylib_foo is written as $(package)_foo in order to make recipes more similar. + +## Identifiers +Each package is required to define at least these variables: + + $(package)_version: + Version of the upstream library or program. If there is no version, a + placeholder such as 1.0 can be used. + + $(package)_download_path: + Location of the upstream source, without the file-name. Usually http or + ftp. + + $(package)_file_name: + The upstream source filename available at the download path. + + $(package)_sha256_hash: + The sha256 hash of the upstream file + +These variables are optional: + + $(package)_build_subdir: + cd to this dir before running configure/build/stage commands. + + $(package)_download_file: + The file-name of the upstream source if it differs from how it should be + stored locally. This can be used to avoid storing file-names with strange + characters. + + $(package)_dependencies: + Names of any other packages that this one depends on. + + $(package)_patches: + Filenames of any patches needed to build the package + + $(package)_extra_sources: + Any extra files that will be fetched via $(package)_fetch_cmds. These are + specified so that they can be fetched and verified via 'make download'. + + +## Build Variables: +After defining the main identifiers, build variables may be added or customized +before running the build commands. They should be added to a function called +$(package)_set_vars. For example: + + define $(package)_set_vars + ... + endef + +Most variables can be prefixed with the host, architecture, or both, to make +the modifications specific to that case. For example: + + Universal: $(package)_cc=gcc + Linux only: $(package)_linux_cc=gcc + x86_64 only: $(package)_x86_64_cc = gcc + x86_64 linux only: $(package)_x86_64_linux_cc = gcc + +These variables may be set to override or append their default values. + + $(package)_cc + $(package)_cxx + $(package)_objc + $(package)_objcxx + $(package)_ar + $(package)_ranlib + $(package)_libtool + $(package)_nm + $(package)_cflags + $(package)_cxxflags + $(package)_ldflags + $(package)_cppflags + $(package)_config_env + $(package)_build_env + $(package)_stage_env + $(package)_build_opts + $(package)_config_opts + +The *_env variables are used to add environment variables to the respective +commands. + +Many variables respect a debug/release suffix as well, in order to use them for +only the appropriate build config. For example: + + $(package)_cflags_release = -O3 + $(package)_cflags_i686_debug = -g + $(package)_config_opts_release = --disable-debug + +These will be used in addition to the options that do not specify +debug/release. All builds are considered to be release unless DEBUG=1 is set by +the user. Other variables may be defined as needed. + +## Build commands: + + For each build, a unique build dir and staging dir are created. For example, + `work/build/mylib/1.0-1adac830f6e` and `work/staging/mylib/1.0-1adac830f6e`. + + The following build commands are available for each recipe: + + $(package)_fetch_cmds: + Runs from: build dir + Fetch the source file. If undefined, it will be fetched and verified + against its hash. + + $(package)_extract_cmds: + Runs from: build dir + Verify the source file against its hash and extract it. If undefined, the + source is assumed to be a tarball. + + $(package)_preprocess_cmds: + Runs from: build dir/$(package)_build_subdir + Preprocess the source as necessary. If undefined, does nothing. + + $(package)_config_cmds: + Runs from: build dir/$(package)_build_subdir + Configure the source. If undefined, does nothing. + + $(package)_build_cmds: + Runs from: build dir/$(package)_build_subdir + Build the source. If undefined, does nothing. + + $(package)_stage_cmds: + Runs from: build dir/$(package)_build_subdir + Stage the build results. If undefined, does nothing. + + The following variables are available for each recipe: + + $(1)_staging_dir: package's destination sysroot path + $(1)_staging_prefix_dir: prefix path inside of the package's staging dir + $(1)_extract_dir: path to the package's extracted sources + $(1)_build_dir: path where configure/build/stage commands will be run + $(1)_patch_dir: path where the package's patches (if any) are found + +Notes on build commands: + +For packages built with autotools, $($(package)_autoconf) can be used in the +configure step to (usually) correctly configure automatically. Any +$($(package)_config_opts) will be appended. + +Most autotools projects can be properly staged using: + + $(MAKE) DESTDIR=$($(package)_staging_dir) install -- cgit v1.2.3