Merge #16546: External signer support - Wallet Box edition

f75e0c1edde39a91cc353b0102638e232def9476 doc: add external-signer.md (Sjors Provoost)
d4b0107d68a91ed4d1a5c78c8ca76251329d3f3c rpc: send: support external signer (Sjors Provoost)
245b4457cf9265190a05529a0a97e1cb258cca8a rpc: signerdisplayaddress (Sjors Provoost)
7ebc7c0215979c53b92a436acc8b5b607b8d735a wallet: ExternalSigner: add GetDescriptors method (Sjors Provoost)
fc5da520f5c72287f59823b8a6d748dda49c574a wallet: add GetExternalSigner() (Sjors Provoost)
259f52cc33817a00b91ec9c7d078c07b88db7ab4 test: external_signer wallet flag is immutable (Sjors Provoost)
2655197e1c2dea9536c32afe1482ced4a1f481e9 rpc: add external_signer option to createwallet (Sjors Provoost)
2700f09c4130af6167ce71f46960e92ca800e205 rpc: signer: add enumeratesigners to list external signers (Sjors Provoost)
07b7c940a7da138d55a484ef83fee19ebf58a867 rpc: add external signer RPC files (Sjors Provoost)
8ce7767071779a0170364e6426bd393ed71bf281 wallet: add ExternalSignerScriptPubKeyMan (Sjors Provoost)
157ea7c614950d61bfe405310e2aaabcee31f7a3 wallet: add external_signer flag (Sjors Provoost)
f3e6ce78fba2b31173fe7b606aa9edb5b615bff3 test: add external signer test (Sjors Provoost)
8cf543f96dcd6fdfac1367b9e2b1d7d51be8bb76 wallet: add -signer argument for external signer command (Sjors Provoost)
f7eb7ecc6750ab267a979d9268ce5b5d151c26de test: framework: add skip_if_no_external_signer (Sjors Provoost)
87a97941f667483bbf2ab00929e03a2199cb8a62 configure: add --enable-external-signer (Sjors Provoost)

Pull request description:

  Big picture overview in [this gist](https://gist.github.com/Sjors/29d06728c685e6182828c1ce9b74483d).

  This PR lets `bitcoind` call an arbitrary command `-signer=<cmd>`, e.g. a hardware wallet driver,  where it can fetch public keys, ask to display an address, and sign a transaction (using PSBT under the hood).

  It's design to work with https://github.com/bitcoin-core/HWI, which supports multiple hardware wallets. Any command with the same arguments and return values will work. It simplifies the manual procedure described [here](https://github.com/bitcoin-core/HWI/blob/master/docs/bitcoin-core-usage.md).

  Usage is documented in [doc/external-signer.md](
  https://github.com/Sjors/bitcoin/blob/2019/08/hww-box2/doc/external-signer.md), which also describes what protocol a different signer binary should conform to.

  Use `--enable-external-signer` to opt in, requires Boost::Process:

  ```
  Options used to compile and link:
    with wallet     = yes
    with gui / qt   = no
    external signer = yes
  ```

  It adds the following RPC methods:
  * `enumeratesigners`: asks <cmd> for a list of signers (e.g. devices) and their master key fingerprint
  * `signerdisplayaddress <address>`:  asks <cmd> to display an address

  It enhances the following RPC methods:
  * `createwallet`: takes an additional `external_signer` argument and fetches keys from device
  * `send`: automatically sends transaction to device and waits

  Usage TL&DR:
  * clone HWI repo somewhere and launch `bitcoind -signer=../HWI/hwi.py`
  * check if you can see your hardware device: `bitcoin-cli enumeratesigners`
  * create wallet and auto import keys `bitcoin-cli createwallet "hww" true true "" true true true`
  * display address on device: `bitcoin-cli signerdisplayaddress ...`
  * to spend, use `send` RPC and approve transaction on device

  Prerequisites:
  - [x] #21127 load wallet flags before everything else
  - [x] #21182 remove mostly pointless BOOST_PROCESS macro

  Potentially useful followups:
  - GUI support: bitcoin-core/gui#4
  - bumpfee support
  - (automatically) verify (a subset of) keys on the device after import, through message signing

ACKs for top commit:
  laanwj:
    re-ACK f75e0c1edde39a91cc353b0102638e232def9476

Tree-SHA512: 7db8afd54762295c1424c3f01d8c587ec256a72f34bd5256e04b21832dabd5dc212be8ab975ae3b67de75259fd569a561491945750492f417111dc7b6641e77f

2 files changed, 172 insertions, 1 deletions

diff --git a/doc/Doxyfile.in b/doc/Doxyfile.in
index 2f79168212..21bf587eaf 100644
--- a/doc/Doxyfile.in
+++ b/doc/Doxyfile.in
@@ -2073,7 +2073,7 @@ INCLUDE_FILE_PATTERNS  =
 # recursively expanded use the := operator instead of the = operator.
 # This tag requires that the tag ENABLE_PREPROCESSING is set to YES.
 
-PREDEFINED             = HAVE_BOOST_PROCESS
+PREDEFINED             = ENABLE_EXTERNAL_SIGNER
 
 # If the MACRO_EXPANSION and EXPAND_ONLY_PREDEF tags are set to YES then this
 # tag can be used to specify a list of macro names that should be expanded. The
diff --git a/doc/external-signer.md b/doc/external-signer.md
new file mode 100644
index 0000000000..053752ee2f
--- /dev/null
+++ b/doc/external-signer.md
@@ -0,0 +1,171 @@
+# Support for signing transactions outside of Bitcoin Core
+
+Bitcoin Core can be launched with `-signer=<cmd>` where `<cmd>` is an external tool which can sign transactions and perform other functions. For example, it can be used to communicate with a hardware wallet.
+
+## Example usage
+
+The following example is based on the [HWI](https://github.com/bitcoin-core/HWI) tool. Although this tool is hosted under the Bitcoin Core GitHub organization and maintained by Bitcoin Core developers, it should be used with caution. It is considered experimental and has far less review than Bitcoin Core itself. Be particularly careful when running tools such as these on a computer with private keys on it.
+
+When using a hardware wallet, consult the manufacturer website for (alternative) software they recommend. As long as their software conforms to the standard below, it should be able to work with Bitcoin Core.
+
+Start Bitcoin Core:
+
+```sh
+$ bitcoind -signer=../HWI/hwi.py
+```
+
+### Device setup
+
+Follow the hardware manufacturers instructions for the initial device setup, as well as their instructions for creating a backup. Alternatively, for some devices, you can use the `setup`, `restore` and `backup` commands provided by [HWI](https://github.com/bitcoin-core/HWI).
+
+### Create wallet and import keys
+
+Get a list of signing devices / services:
+
+```
+$ bitcoin-cli enumeratesigners
+{
+  "signers": [
+    {
+      "fingerprint": "c8df832a"
+    }
+]
+```
+
+The master key fingerprint is used to identify a device.
+
+Create a wallet, this automatically imports the public keys:
+
+```sh
+$ bitcoin-cli createwallet "hww" true true "" true true true
+```
+
+### Verify an address
+
+Display an address on the device:
+
+```sh
+$ bitcoin-cli -rpcwallet=<wallet> getnewaddress
+$ bitcoin-cli -rpcwallet=<wallet> signerdisplayaddress <address>
+```
+
+Replace `<address>` with the result of `getnewaddress`.
+
+### Spending
+
+Under the hood this uses a [Partially Signed Bitcoin Transaction](psbt.md).
+
+```sh
+$ bitcoin-cli -rpcwallet=<wallet> sendtoaddress <address> <amount>
+```
+
+This prompts your hardware wallet to sign, and fail if it's not connected. If successful
+it automatically broadcasts the transaction.
+
+```sh
+{"complete": true, "txid": <txid>}
+```
+
+## Signer API
+
+In order to be compatible with Bitcoin Core any signer command should conform to the specification below. This specification is subject to change. Ideally a BIP should propose a standard so that other wallets can also make use of it.
+
+Prerequisite knowledge:
+* [Output Descriptors](descriptors.md)
+* Partially Signed Bitcoin Transaction ([PSBT](psbt.md))
+
+### `enumerate` (required)
+
+Usage:
+```
+$ <cmd> enumerate
+[
+    {
+        "fingerprint": "00000000"
+    }
+]
+```
+
+The command MUST return an (empty) array with at least a `fingerprint` field.
+
+A future extension could add an optional return field with device capabilities. Perhaps a descriptor with wildcards. For example: `["pkh("44'/0'/$'/{0,1}/*"), sh(wpkh("49'/0'/$'/{0,1}/*")), wpkh("84'/0'/$'/{0,1}/*")]`. This would indicate the device supports legacy, wrapped SegWit and native SegWit. In addition it restricts the derivation paths that can used for those, to maintain compatibility with other wallet software. It also indicates the device, or the driver, doesn't support multisig.
+
+A future extension could add an optional return field `reachable`, in case `<cmd>` knows a signer exists but can't currently reach it.
+
+### `signtransaction` (required)
+
+Usage:
+```
+$ <cmd> --fingerprint=<fingerprint> (--testnet) signtransaction <psbt>
+base64_encode_signed_psbt
+```
+
+The command returns a psbt with any signatures.
+
+The `psbt` SHOULD include bip32 derivations. The command SHOULD fail if none of the bip32 derivations match a key owned by the device.
+
+The command SHOULD fail if the user cancels.
+
+The command MAY complain if `--testnet` is set, but any of the BIP32 derivation paths contain a coin type other than `1h` (and vice versa).
+
+### `getdescriptors` (optional)
+
+Usage:
+
+```
+$ <cmd> --fingerprint=<fingerprint> (--testnet) getdescriptors <account>
+<xpub>
+```
+
+Returns descriptors supported by the device. Example:
+
+```
+$ <cmd> --fingerprint=00000000 --testnet getdescriptors
+{
+  "receive": [
+    "pkh([00000000/44h/0h/0h]xpub6C.../0/*)#fn95jwmg",
+    "sh(wpkh([00000000/49h/0h/0h]xpub6B..../0/*))#j4r9hntt",
+    "wpkh([00000000/84h/0h/0h]xpub6C.../0/*)#qw72dxa9"
+  ],
+  "internal": [
+    "pkh([00000000/44h/0h/0h]xpub6C.../1/*)#c8q40mts",
+    "sh(wpkh([00000000/49h/0h/0h]xpub6B..../1/*))#85dn0v75",
+    "wpkh([00000000/84h/0h/0h]xpub6C..../1/*)#36mtsnda"
+  ]
+}
+```
+
+### `displayaddress` (optional)
+
+Usage:
+```
+<cmd> --fingerprint=<fingerprint> (--testnet) displayaddress --desc descriptor
+```
+
+Example, display the first native SegWit receive address on Testnet:
+
+```
+<cmd> --fingerprint=00000000 --testnet displayaddress --desc "wpkh([00000000/84h/1h/0h]tpubDDUZ..../0/0)"
+```
+
+The command MUST be able to figure out the address type from the descriptor.
+
+If <descriptor> contains a master key fingerprint, the command MUST fail if it does not match the fingerprint known by the device.
+
+If <descriptor> contains an xpub, the command MUST fail if it does not match the xpub known by the device.
+
+The command MAY complain if `--testnet` is set, but the BIP32 coin type is not `1h` (and vice versa).
+
+## How Bitcoin Core uses the Signer API
+
+The `enumeratesigners` RPC simply calls `<cmd> enumerate`.
+
+The `createwallet` RPC calls:
+
+* `<cmd> --fingerprint=00000000 getdescriptors 0`
+
+It then imports descriptors for all support address types, in a BIP44/49/84 compatible manner.
+
+The `displayaddress` RPC reuses some code from `getaddressinfo` on the provided address and obtains the inferred descriptor. It then calls `<cmd> --fingerprint=00000000 displayaddress --desc=<descriptor>`.
+
+`sendtoaddress` and `sendmany` check `inputs->bip32_derivs` to see if any inputs have the same `master_fingerprint` as the signer. If so, it calls `<cmd> --fingerprint=00000000 signtransaction <psbt>`. It waits for the device to return a (partially) signed psbt, tries to finalize it and broadcasts the transation.