Merge bitcoin/bitcoin#24263: doc: Fix gen-manpages, rewrite in Python

87f54060ffffdb56c97594efdca378bace5323df doc: Swap gen-manpages and update RC steps in release process (laanwj)
42c202893b879d1bda54624d44c90b28143fc167 doc: Fix gen-manpages, rewrite in Python (laanwj)

Pull request description:

  Rewrite the manual page generation script in Python.

  This:
  - solves '-' stripping issue (fixes #22681)
  - makes that a copyright footer is generated correctly again

  Also change the release process to swap gen-manpages and update RC steps, so that the pages will have the correct rc and/or final version.

ACKs for top commit:
  dongcarl:
    Code Review ACK 87f54060ffffdb56c97594efdca378bace5323df
  fanquake:
    ACK 87f54060ffffdb56c97594efdca378bace5323df - tested generating and opening the man pages locally, but didn't run through the release process. Will propose some changes to address consolidating the help / version output.

Tree-SHA512: 39254721ca84e4f223a321c554f2e08c36428b15019a0f9fa3eff408b4c6f1e1d74941143f4d2927427afa3ad7a7e6f999d6ec660132d817809b640a87ae9f7d

4 files changed, 77 insertions, 56 deletions

diff --git a/contrib/devtools/README.md b/contrib/devtools/README.md
index afbad096c4..79b0134adc 100644
--- a/contrib/devtools/README.md
+++ b/contrib/devtools/README.md
@@ -76,7 +76,7 @@ year rather than two hyphenated years.
 If the file already has a copyright for `The Bitcoin Core developers`, the
 script will exit.
 
-gen-manpages.sh
+gen-manpages.py
 ===============
 
 A small script to automatically create manpages in ../../doc/man by running the release binaries with the -help option.
@@ -87,7 +87,7 @@ repostitory. To use this tool with out-of-tree builds set `BUILDDIR`. For
 example:
 
 ```bash
-BUILDDIR=$PWD/build contrib/devtools/gen-manpages.sh
+BUILDDIR=$PWD/build contrib/devtools/gen-manpages.py
 ```
 
 security-check.py and test-security-check.py
diff --git a/contrib/devtools/gen-manpages.py b/contrib/devtools/gen-manpages.py
new file mode 100755
index 0000000000..05f613da38
--- /dev/null
+++ b/contrib/devtools/gen-manpages.py
@@ -0,0 +1,73 @@
+#!/usr/bin/env python3
+# Copyright (c) 2022 The Bitcoin Core developers
+# Distributed under the MIT software license, see the accompanying
+# file COPYING or http://www.opensource.org/licenses/mit-license.php.
+import os
+import subprocess
+import sys
+import tempfile
+
+BINARIES = [
+'src/bitcoind',
+'src/bitcoin-cli',
+'src/bitcoin-tx',
+'src/bitcoin-wallet',
+'src/bitcoin-util',
+'src/qt/bitcoin-qt',
+]
+
+# Paths to external utilities.
+git = os.getenv('GIT', 'git')
+help2man = os.getenv('HELP2MAN', 'help2man')
+
+# If not otherwise specified, get top directory from git.
+topdir = os.getenv('TOPDIR')
+if not topdir:
+    r = subprocess.run([git, 'rev-parse', '--show-toplevel'], stdout=subprocess.PIPE, check=True, universal_newlines=True)
+    topdir = r.stdout.rstrip()
+
+# Get input and output directories.
+builddir = os.getenv('BUILDDIR', topdir)
+mandir = os.getenv('MANDIR', os.path.join(topdir, 'doc/man'))
+
+# Verify that all the required binaries are usable, and extract copyright
+# message in a first pass.
+copyright = None
+versions = []
+for relpath in BINARIES:
+    abspath = os.path.join(builddir, relpath)
+    try:
+        r = subprocess.run([abspath, '--version'], stdout=subprocess.PIPE, universal_newlines=True)
+    except IOError:
+        print(f'{abspath} not found or not an executable', file=sys.stderr)
+        sys.exit(1)
+    # take first line (which must contain version)
+    verstr = r.stdout.split('\n')[0]
+    # last word of line is the actual version e.g. v22.99.0-5c6b3d5b3508
+    verstr = verstr.split()[-1]
+    assert verstr.startswith('v')
+
+    # Only bitcoin-qt prints the copyright message on --version, so store it specifically.
+    if relpath == 'src/qt/bitcoin-qt':
+        copyright = r.stdout.split('\n')[1:]
+
+    versions.append((abspath, verstr))
+
+if any(verstr.endswith('-dirty') for (_, verstr) in versions):
+    print("WARNING: Binaries were built from a dirty tree.")
+    print('man pages generated from dirty binaries should NOT be committed.')
+    print('To properly generate man pages, please commit your changes (or discard them), rebuild, then run this script again.')
+    print()
+
+with tempfile.NamedTemporaryFile('w', suffix='.h2m') as footer:
+    # Create copyright footer, and write it to a temporary include file.
+    assert copyright
+    footer.write('[COPYRIGHT]\n')
+    footer.write('\n'.join(copyright).strip())
+    footer.flush()
+
+    # Call the binaries through help2man to produce a manual page for each of them.
+    for (abspath, verstr) in versions:
+        outname = os.path.join(mandir, os.path.basename(abspath) + '.1')
+        print(f'Generating {outname}…')
+        subprocess.run([help2man, '-N', '--version-string=' + verstr, '--include=' + footer.name, '-o', outname, abspath], check=True)
diff --git a/contrib/devtools/gen-manpages.sh b/contrib/devtools/gen-manpages.sh
deleted file mode 100755
index 8da6ff1204..0000000000
--- a/contrib/devtools/gen-manpages.sh
+++ /dev/null
@@ -1,53 +0,0 @@
-#!/usr/bin/env bash
-# Copyright (c) 2016-2021 The Bitcoin Core developers
-# Distributed under the MIT software license, see the accompanying
-# file COPYING or http://www.opensource.org/licenses/mit-license.php.
-
-export LC_ALL=C
-TOPDIR=${TOPDIR:-$(git rev-parse --show-toplevel)}
-BUILDDIR=${BUILDDIR:-$TOPDIR}
-
-BINDIR=${BINDIR:-$BUILDDIR/src}
-MANDIR=${MANDIR:-$TOPDIR/doc/man}
-
-BITCOIND=${BITCOIND:-$BINDIR/bitcoind}
-BITCOINCLI=${BITCOINCLI:-$BINDIR/bitcoin-cli}
-BITCOINTX=${BITCOINTX:-$BINDIR/bitcoin-tx}
-WALLET_TOOL=${WALLET_TOOL:-$BINDIR/bitcoin-wallet}
-BITCOINUTIL=${BITCOINQT:-$BINDIR/bitcoin-util}
-BITCOINQT=${BITCOINQT:-$BINDIR/qt/bitcoin-qt}
-
-[ ! -x "$BITCOIND" ] && echo "$BITCOIND not found or not executable." && exit 1
-
-# Don't allow man pages to be generated for binaries built from a dirty tree
-DIRTY=""
-for cmd in $BITCOIND $BITCOINCLI $BITCOINTX $WALLET_TOOL $BITCOINUTIL $BITCOINQT; do
-  VERSION_OUTPUT=$($cmd --version)
-  if [[ $VERSION_OUTPUT == *"dirty"* ]]; then
-    DIRTY="${DIRTY}${cmd}\n"
-  fi
-done
-if [ -n "$DIRTY" ]
-then
-  echo -e "WARNING: the following binaries were built from a dirty tree:\n"
-  echo -e "$DIRTY"
-  echo "man pages generated from dirty binaries should NOT be committed."
-  echo "To properly generate man pages, please commit your changes to the above binaries, rebuild them, then run this script again."
-fi
-
-# The autodetected version git tag can screw up manpage output a little bit
-read -r -a BTCVER <<< "$($BITCOINCLI --version | head -n1 | awk -F'[ -]' '{ print $6, $7 }')"
-
-# Create a footer file with copyright content.
-# This gets autodetected fine for bitcoind if --version-string is not set,
-# but has different outcomes for bitcoin-qt and bitcoin-cli.
-echo "[COPYRIGHT]" > footer.h2m
-$BITCOIND --version | sed -n '1!p' >> footer.h2m
-
-for cmd in $BITCOIND $BITCOINCLI $BITCOINTX $WALLET_TOOL $BITCOINUTIL $BITCOINQT; do
-  cmdname="${cmd##*/}"
-  help2man -N --version-string="${BTCVER[0]}" --include=footer.h2m -o "${MANDIR}/${cmdname}.1" "${cmd}"
-  sed -i "s/\\\-${BTCVER[1]}//g" "${MANDIR}/${cmdname}.1"
-done
-
-rm -f footer.h2m
diff --git a/doc/release-process.md b/doc/release-process.md
index c5dba46238..5a74f72b6e 100644
--- a/doc/release-process.md
+++ b/doc/release-process.md
@@ -6,13 +6,14 @@ Release Process
 ### Before every release candidate
 
 * Update translations see [translation_process.md](https://github.com/bitcoin/bitcoin/blob/master/doc/translation_process.md#synchronising-translations).
-* Update manpages, see [gen-manpages.sh](https://github.com/bitcoin/bitcoin/blob/master/contrib/devtools/README.md#gen-manpagessh).
 * Update release candidate version in `configure.ac` (`CLIENT_VERSION_RC`).
+* Update manpages (after rebuilding the binaries), see [gen-manpages.py](https://github.com/bitcoin/bitcoin/blob/master/contrib/devtools/README.md#gen-manpagespy).
 
 ### Before every major and minor release
 
 * Update [bips.md](bips.md) to account for changes since the last release (don't forget to bump the version number on the first line).
 * Update version in `configure.ac` (don't forget to set `CLIENT_VERSION_RC` to `0`).
+* Update manpages (see previous section)
 * Write release notes (see "Write the release notes" below).
 
 ### Before every major release