aboutsummaryrefslogtreecommitdiff
path: root/contrib/guix/README.md
diff options
context:
space:
mode:
authorCarl Dong <contact@carldong.me>2021-02-22 17:04:19 -0500
committerCarl Dong <contact@carldong.me>2021-04-01 16:53:17 -0400
commitd5a71e97853ea9e1b879e8c76bfb01d4bef33172 (patch)
tree2305ac3d3962c7bdef5cfa1afe53e76fce38f456 /contrib/guix/README.md
parent66daf4cb3b47e8218623936d23ad504fd189e70b (diff)
downloadbitcoin-d5a71e97853ea9e1b879e8c76bfb01d4bef33172.tar.xz
guix: Use --cores instead of --max-jobs
In Guix, there are two flags for controlling parallelism: Note: When I say "derivation," think "package" --cores=n - controls the number of CPU cores to build each derivation. This is the value passed to `make`'s `--jobs=` flag. - defaults to 0: as many cores as is available --max-jobs=n - controls how many derivations can be built in parallel - defaults to 1 Therefore, if set --max-jobs=$MAX_JOBS and don't set --cores, Guix could theoretically spin up $MAX_JOBS * $(nproc) number of threads, and that's no good. So we could either default to --cores=1, --max-jobs=$MAX_JOBS - Pro: --cores=1 means that `make` will be invoked with `-j1`, avoiding problems with package whose build systems and test suites break when running multi-threaded. - Con: There will be times when only 1 or 2 derivations can be built at a time, because the rest of the dependency graph all depend on those 1 or 2 derivations. During these times, the machine will be severely under-utilized. or --cores=$MAX_JOBS, --max-jobs=1 - Pro: We don't encounter prolonged periods of severe under-utilization mentioned above. - Con: Many packages' build systems and test suites break when running multi-threaded. or --cores=1, --max-jobs=1 and let the user override with $ADDITIONAL_GUIX_COMMON_FLAGS
Diffstat (limited to 'contrib/guix/README.md')
-rw-r--r--contrib/guix/README.md58
1 files changed, 53 insertions, 5 deletions
diff --git a/contrib/guix/README.md b/contrib/guix/README.md
index 1122ec9ba5..133f250ad2 100644
--- a/contrib/guix/README.md
+++ b/contrib/guix/README.md
@@ -80,6 +80,50 @@ at the end of the `guix pull`)
export PATH="${HOME}/.config/guix/current/bin${PATH:+:}$PATH"
```
+### Controlling the number of threads used by `guix` build commands
+
+By default, the scripts under `./contrib/guix` will invoke all `guix` build
+commands with `--cores="$JOBS"`. Note that `$JOBS` defaults to `$(nproc)` if not
+specified. However, astute manual readers will also notice that there is a
+`--max-jobs=` flag (which defaults to 1 if unspecified).
+
+Here is the difference between `--cores=` and `--max-jobs=`:
+
+> Note: When I say "derivation," think "package"
+
+`--cores=`
+
+ - controls the number of CPU cores to build each derivation. This is the value
+ passed to `make`'s `--jobs=` flag.
+
+`--max-jobs=`
+
+ - controls how many derivations can be built in parallel
+ - defaults to 1
+
+Therefore, the default is for `guix` build commands to build one derivation at a
+time, utilizing `$JOBS` threads.
+
+Specifying the `$JOBS` environment variable will only modify `--cores=`, but you
+can also modify the value for `--max-jobs=` by specifying
+`$ADDITIONAL_GUIX_COMMON_FLAGS`. For example, if you have a LOT of memory, you
+may want to set:
+
+```sh
+export ADDITIONAL_GUIX_COMMON_FLAGS='--max-jobs=8'
+```
+
+Which allows for a maximum of 8 derivations to be built at the same time, each
+utilizing `$JOBS` threads.
+
+Or, if you'd like to avoid spurious build failures caused by issues with
+parallelism within a single package, but would still like to build multiple
+packages when the dependency graph allows for it, you may want to try:
+
+```sh
+export JOBS=1 ADDITIONAL_GUIX_COMMON_FLAGS='--max-jobs=8'
+```
+
## Usage
### As a Tool for Deterministic Builds
@@ -125,12 +169,16 @@ find output/ -type f -print0 | sort -z | xargs -r0 sha256sum
the actual SDK (e.g. SDK_PATH=$HOME/Downloads/macOS-SDKs instead of
$HOME/Downloads/macOS-SDKs/Xcode-11.3.1-11C505-extracted-SDK-with-libcxx-headers).
-* _**MAX_JOBS**_
+* _**JOBS**_
+
+ Override the number of jobs to run simultaneously, you might want to do so on
+ a memory-limited machine. This may be passed to:
+
+ - `guix` build commands as in `guix environment --cores="$JOBS"`
+ - `make` as in `make --jobs="$JOBS"`
+ - `xargs` as in `xargs -P"$JOBS"`
- Override the maximum number of jobs to run simultaneously, you might want to
- do so on a memory-limited machine. This may be passed to `make` as in `make
- --jobs="$MAX_JOBS"` or `xargs` as in `xargs -P"$MAX_JOBS"`. _(defaults to the
- value of `nproc` outside the container)_
+ _(defaults to the value of `nproc` outside the container)_
* _**SOURCE_DATE_EPOCH**_