diff options
| author | merge-script <fanquake@gmail.com> | 2024-10-22 15:46:46 +0100 |
|---|---|---|
| committer | merge-script <fanquake@gmail.com> | 2024-10-22 15:46:46 +0100 |
| commit | ffe4261cb0669b1e1a926638e0498ae5b63f3599 (patch) | |
| tree | d5d2ed00914d6a38079763fa1136774f1d5e682a /ci/test/02_run_container.sh | |
| parent | 28ce159bc327e6dfec34077ff2e379b23a95db65 (diff) | |
| parent | fa71bedf8609f06618aa85342ea6f5c4d2c5fea0 (diff) | |
fa71bedf8609f06618aa85342ea6f5c4d2c5fea0 ci: Approximate MAKEJOBS in image build phase (MarcoFalke)
Pull request description:
The `MAKEJOBS` env var is the default in image builds, which is fine, because it is only relevant when building msan (or iwyu) and only differs when setting MAKEJOBS to something other than `nproc` (currently used as an approximation).
So the normal workflow of `MAKEJOBS="-j$(nproc)" FILE_ENV="./ci/test/00_setup_env_native_msan.sh" ./ci/test_run_all.sh` already works today.
However, `MAKEJOBS="-j1" FILE_ENV="./ci/test/00_setup_env_native_msan.sh" ./ci/test_run_all.sh` does not.
This is hard to fix, because making the env var a build arg means that changing it (and only it) requires a new (expensive and redundant) build.
So add an option `HAVE_CGROUP_CPUSET`, which can be set to approximate `MAKEJOBS` a bit. Can be tested via:
`HAVE_CGROUP_CPUSET=yo MAKEJOBS="-j_something" FILE_ENV="./ci/test/00_setup_env_native_msan.sh" ./ci/test_run_all.sh`
ACKs for top commit:
fanquake:
ACK fa71bedf8609f06618aa85342ea6f5c4d2c5fea0
Tree-SHA512: 43ef194c71d726f4cfa3fe08a5894c7872150f37da7e4fa0c2d89e4572bc63acadb5dae3286a5e5cc14a8ce3e1ebcc14571f1a3541e8db2d18d2f7503764a2f3
Diffstat (limited to 'ci/test/02_run_container.sh')
| -rwxr-xr-x | ci/test/02_run_container.sh | 10 |
1 files changed, 10 insertions, 0 deletions
diff --git a/ci/test/02_run_container.sh b/ci/test/02_run_container.sh index 1727f9296b..9069fba156 100755 --- a/ci/test/02_run_container.sh +++ b/ci/test/02_run_container.sh @@ -15,12 +15,22 @@ if [ -z "$DANGER_RUN_CI_ON_HOST" ]; then python3 -c 'import os; [print(f"{key}={value}") for key, value in os.environ.items() if "\n" not in value and "HOME" != key and "PATH" != key and "USER" != key]' | tee "/tmp/env-$USER-$CONTAINER_NAME" # System-dependent env vars must be kept as is. So read them from the container. docker run --rm "${CI_IMAGE_NAME_TAG}" bash -c "env | grep --extended-regexp '^(HOME|PATH|USER)='" | tee --append "/tmp/env-$USER-$CONTAINER_NAME" + + # Env vars during the build can not be changed. For example, a modified + # $MAKEJOBS is ignored in the build process. Use --cpuset-cpus as an + # approximation to respect $MAKEJOBS somewhat, if cpuset is available. + MAYBE_CPUSET="" + if [ "$HAVE_CGROUP_CPUSET" ]; then + MAYBE_CPUSET="--cpuset-cpus=$( python3 -c "import random;P=$( nproc );M=min(P,int('$MAKEJOBS'.lstrip('-j')));print(','.join(map(str,sorted(random.sample(range(P),M)))))" )" + fi echo "Creating $CI_IMAGE_NAME_TAG container to run in" + # shellcheck disable=SC2086 DOCKER_BUILDKIT=1 docker build \ --file "${BASE_READ_ONLY_DIR}/ci/test_imagefile" \ --build-arg "CI_IMAGE_NAME_TAG=${CI_IMAGE_NAME_TAG}" \ --build-arg "FILE_ENV=${FILE_ENV}" \ + $MAYBE_CPUSET \ --label="${CI_IMAGE_LABEL}" \ --tag="${CONTAINER_NAME}" \ "${BASE_READ_ONLY_DIR}" |