Merge #20560: fuzz: Link all targets once

fa13e1b0c52738492310b6b421d8e38cb04da5b1 build: Add option --enable-danger-fuzz-link-all (MarcoFalke)
44444ba759480237172d83f42374c5c29c76eda0 fuzz: Link all targets once (MarcoFalke)

Pull request description:

  Currently the linker is invoked more than 150 times when compiling with `--enable-fuzz`. This is problematic for several reasons:

  * It wastes disk space north of 20 GB, as all libraries and sanitizers are linked more than 150 times
  * It wastes CPU time, as the link step can practically not be cached (similar to ccache for object files)
  * It makes it a blocker to compile the fuzz tests by default for non-fuzz builds #19388, for the aforementioned reasons
  * The build file is several thousand lines of code, without doing anything meaningful except listing each fuzz target in a highly verbose manner
  * It makes writing new fuzz tests unnecessarily hard, as build system knowledge is required; Compare that to boost unit tests, which can be added by simply editing an existing cpp file
  * It encourages fuzz tests that re-use the `buffer` or assume the `buffer` to be concatenations of seeds, which increases complexity of seeds and complexity for the fuzz engine to explore; Thus reducing the effectiveness of the affected fuzz targets

  Fixes #20088

ACKs for top commit:
  practicalswift:
    Tested ACK fa13e1b0c52738492310b6b421d8e38cb04da5b1
  sipa:
    ACK fa13e1b0c52738492310b6b421d8e38cb04da5b1. Reviewed the code changes, and tested the 3 different test_runner.py modes (run once, merge, generate). I also tested building with the new --enable-danger-fuzz-link-all

Tree-SHA512: 962ab33269ebd51810924c51266ecc62edd6ddf2fcd9a8c359ed906766f58c3f73c223f8d3cc49f2c60f0053f65e8bdd86ce9c19e673f8c2b3cd676e913f2642

1 files changed, 28 insertions, 0 deletions

diff --git a/src/test/fuzz/danger_link_all.sh b/src/test/fuzz/danger_link_all.sh
new file mode 100755
index 0000000000..2ddd00c658
--- /dev/null
+++ b/src/test/fuzz/danger_link_all.sh
@@ -0,0 +1,28 @@
+#!/usr/bin/env bash
+# Copyright (c) 2020 The Bitcoin Core developers
+# Distributed under the MIT software license, see the accompanying
+# file COPYING or http://www.opensource.org/licenses/mit-license.php.
+
+export LC_ALL=C.UTF-8
+
+set -e
+
+ROOT_DIR="$(git rev-parse --show-toplevel)"
+
+# Run only once (break make recursion)
+if [ -d "${ROOT_DIR}/lock_fuzz_link_all" ]; then
+  exit
+fi
+mkdir "${ROOT_DIR}/lock_fuzz_link_all"
+
+echo "Linking each fuzz target separately."
+for FUZZING_HARNESS in $(PRINT_ALL_FUZZ_TARGETS_AND_ABORT=1 "${ROOT_DIR}/src/test/fuzz/fuzz" | sort -u); do
+    echo "Building src/test/fuzz/${FUZZING_HARNESS} ..."
+    git checkout -- "${ROOT_DIR}/src/test/fuzz/fuzz.cpp"
+    sed -i "s/std::getenv(\"FUZZ\")/\"${FUZZING_HARNESS}\"/g" "${ROOT_DIR}/src/test/fuzz/fuzz.cpp"
+    make
+    mv "${ROOT_DIR}/src/test/fuzz/fuzz" "${ROOT_DIR}/src/test/fuzz/${FUZZING_HARNESS}"
+done
+git checkout -- "${ROOT_DIR}/src/test/fuzz/fuzz.cpp"
+rmdir "${ROOT_DIR}/lock_fuzz_link_all"
+echo "Successfully built all fuzz targets."