+- [Assumeutxo design](assumeutxo.md)

+# assumeutxo

+

+Assumeutxo is a feature that allows fast bootstrapping of a validating bitcoind

+instance with a very similar security model to assumevalid.

+

+The RPC commands `dumptxoutset` and `loadtxoutset` are used to respectively generate

+and load UTXO snapshots. The utility script `./contrib/devtools/utxo_snapshot.sh` may

+be of use.

+

+## General background

+

+- [assumeutxo proposal](https://github.com/jamesob/assumeutxo-docs/tree/2019-04-proposal/proposal)

+- [Github issue](https://github.com/bitcoin/bitcoin/issues/15605)

+- [draft PR](https://github.com/bitcoin/bitcoin/pull/15606)

+

+## Design notes

+

+- A new block index `nStatus` flag is introduced, `BLOCK_ASSUMED_VALID`, to mark block

+ index entries that are required to be assumed-valid by a chainstate created

+ from a UTXO snapshot. This flag is mostly used as a way to modify certain

+ CheckBlockIndex() logic to account for index entries that are pending validation by a

+ chainstate running asynchronously in the background. We also use this flag to control

+ which index entries are added to setBlockIndexCandidates during LoadBlockIndex().

+

+- Indexing implementations via BaseIndex can no longer assume that indexation happens

+ sequentially, since background validation chainstates can submit BlockConnected

+ events out of order with the active chain.

+

+- The concept of UTXO snapshots is treated as an implementation detail that lives

+ behind the ChainstateManager interface. The external presentation of the changes

+ required to facilitate the use of UTXO snapshots is the understanding that there are

+ now certain regions of the chain that can be temporarily assumed to be valid (using

+ the nStatus flag mentioned above). In certain cases, e.g. wallet rescanning, this is

+ very similar to dealing with a pruned chain.

+

+ Logic outside ChainstateManager should try not to know about snapshots, instead

+ preferring to work in terms of more general states like assumed-valid.

+

+

+## Chainstate phases

+

+Chainstate within the system goes through a number of phases when UTXO snapshots are

+used, as managed by `ChainstateManager`. At various points there can be multiple

+`CChainState` objects in existence to facilitate both maintaining the network tip and

+performing historical validation of the assumed-valid chain.

+

+It is worth noting that though there are multiple separate chainstates, those

+chainstates share use of a common block index (i.e. they hold the same `BlockManager`

+reference).

+

+The subheadings below outline the phases and the corresponding changes to chainstate

+data.

+

+### "Normal" operation via initial block download

+

+`ChainstateManager` manages a single CChainState object, for which

+`m_snapshot_blockhash` is null. This chainstate is (maybe obviously)

+considered active. This is the "traditional" mode of operation for bitcoind.

+

+| | |

+| ---------- | ----------- |

+| number of chainstates | 1 |

+| active chainstate | ibd |

+

+### User loads a UTXO snapshot via `loadtxoutset` RPC

+

+`ChainstateManager` initializes a new chainstate (see `ActivateSnapshot()`) to load the

+snapshot contents into. During snapshot load and validation (see

+`PopulateAndValidateSnapshot()`), the new chainstate is not considered active and the

+original chainstate remains in use as active.

+

+| | |

+| ---------- | ----------- |

+| number of chainstates | 2 |

+| active chainstate | ibd |

+

+Once the snapshot chainstate is loaded and validated, it is promoted to active

+chainstate and a sync to tip begins. A new chainstate directory is created in the

+datadir for the snapshot chainstate called

+`chainstate_[SHA256 blockhash of snapshot base block]`.

+

+| | |

+| ---------- | ----------- |

+| number of chainstates | 2 |

+| active chainstate | snapshot |

+

+The snapshot begins to sync to tip from its base block, technically in parallel with

+the original chainstate, but it is given priority during block download and is

+allocated most of the cache (see `MaybeRebalanceCaches()` and usages) as our chief

+consideration is getting to network tip.

+

+**Failure consideration:** if shutdown happens at any point during this phase, both

+chainstates will be detected during the next init and the process will resume.

+

+### Snapshot chainstate hits network tip

+

+Once the snapshot chainstate leaves IBD, caches are rebalanced

+(via `MaybeRebalanceCaches()` in `ActivateBestChain()`) and more cache is given

+to the background chainstate, which is responsible for doing full validation of the

+assumed-valid parts of the chain.

+

+**Note:** at this point, ValidationInterface callbacks will be coming in from both

+chainstates. Considerations here must be made for indexing, which may no longer be happening

+sequentially.

+

+### Background chainstate hits snapshot base block

+

+Once the tip of the background chainstate hits the base block of the snapshot

+chainstate, we stop use of the background chainstate by setting `m_stop_use` (not yet

+committed - see #15606), in `CompleteSnapshotValidation()`, which is checked in

+`ActivateBestChain()`). We hash the background chainstate's UTXO set contents and

+ensure it matches the compiled value in `CMainParams::m_assumeutxo_data`.

+

+The background chainstate data lingers on disk until shutdown, when in

+`ChainstateManager::Reset()`, the background chainstate is cleaned up with

+`ValidatedSnapshotShutdownCleanup()`, which renames the `chainstate_[hash]` datadir as

+`chainstate`.

+

+| | |

+| ---------- | ----------- |

+| number of chainstates | 2 (ibd has `m_stop_use=true`) |

+| active chainstate | snapshot |

+

+**Failure consideration:** if bitcoind unexpectedly halts after `m_stop_use` is set on

+the background chainstate but before `CompleteSnapshotValidation()` can finish, the

+need to complete snapshot validation will be detected on subsequent init by

+`ChainstateManager::CheckForUncleanShutdown()`.

+

+### Bitcoind restarts sometime after snapshot validation has completed

+

+When bitcoind initializes again, what began as the snapshot chainstate is now

+indistinguishable from a chainstate that has been built from the traditional IBD

+process, and will be initialized as such.

+

+| | |

+| ---------- | ----------- |

+| number of chainstates | 1 |

+| active chainstate | ibd |

+* [`BIP 380`](https://github.com/bitcoin/bips/blob/master/bip-0380.mediawiki)

+ [`381`](https://github.com/bitcoin/bips/blob/master/bip-0381.mediawiki)

+ [`382`](https://github.com/bitcoin/bips/blob/master/bip-0382.mediawiki)

+ [`383`](https://github.com/bitcoin/bips/blob/master/bip-0383.mediawiki)

+ [`384`](https://github.com/bitcoin/bips/blob/master/bip-0384.mediawiki)

+ [`385`](https://github.com/bitcoin/bips/blob/master/bip-0385.mediawiki):

+ Output Script Descriptors, and most of Script Expressions are implemented as of **v0.17.0** ([PR 13697](https://github.com/bitcoin/bitcoin/pull/13697)).

+* [`BIP 386`](https://github.com/bitcoin/bips/blob/master/bip-0386.mediawiki): tr() Output Script Descriptors are implemented as of **v22.0** ([PR 22051](https://github.com/bitcoin/bitcoin/pull/22051)).

+## Dependencies

+

+Before proceeding with an Android build one needs to get the [Android SDK](https://developer.android.com/studio) and use the "SDK Manager" tool to download the NDK and one or more "Platform packages" (these are Android versions and have a corresponding API level).

+

+The minimum supported Android NDK version is [r21](https://github.com/android/ndk/wiki/Changelog-r21).

+

+In order to build `ANDROID_API_LEVEL` (API level corresponding to the Android version targeted, e.g. Android 9.0 Pie is 28 and its "Platform package" needs to be available) and `ANDROID_TOOLCHAIN_BIN` (path to toolchain binaries depending on the platform the build is being performed on) need to be set.

+

+API levels from 24 to 29 have been tested to work.

+

+If the build includes Qt, environment variables `ANDROID_SDK` and `ANDROID_NDK` need to be set as well but can otherwise be omitted.

+This is an example command for a default build with no disabled dependencies:

+

+ ANDROID_SDK=/home/user/Android/Sdk ANDROID_NDK=/home/user/Android/Sdk/ndk-bundle make HOST=aarch64-linux-android ANDROID_API_LEVEL=28 ANDROID_TOOLCHAIN_BIN=/home/user/Android/Sdk/ndk-bundle/toolchains/llvm/prebuilt/linux-x86_64/bin

+

+ libdb4.8 | Berkeley DB | Wallet storage (only needed when legacy wallet enabled)

+ libqrencode | QR codes in GUI | QR code generation (only needed when GUI enabled)

+ libzmq3 | ZMQ notification | ZMQ notifications (requires ZMQ version >= 4.0.0)

+ sqlite3 | SQLite DB | Wallet storage (only needed when descriptor wallet enabled)

+ systemtap | Tracing (USDT) | Statically defined tracepoints

+Berkeley DB is required for the legacy wallet. Ubuntu and Debian have their own `libdb-dev` and `libdb++-dev` packages,

+but these will install Berkeley DB 5.1 or later. This will break binary wallet compatibility with the distributed

+executables, which are based on BerkeleyDB 4.8. If you do not care about wallet compatibility, pass

+`--with-incompatible-bdb` to configure. Otherwise, you can build Berkeley DB [yourself](#berkeley-db).

+

+SQLite is required for the descriptor wallet:

+

+ sudo dnf install sqlite-devel

+

+Berkeley DB is required for the legacy wallet:

+pass `--with-incompatible-bdb` to configure. Otherwise, you can build Berkeley DB [yourself](#berkeley-db).

+

+The legacy wallet uses Berkeley DB. To ensure backwards compatibility it is

+recommended to use Berkeley DB 4.8. If you have to build it yourself, you can

+use [the installation script included in contrib/](/contrib/install_db4.sh)

+`--with-incompatible-bdb` according to the [PKGBUILD](https://github.com/archlinux/svntogit-community/blob/packages/bitcoin/trunk/PKGBUILD).

+| glibc | | [2.18](https://www.gnu.org/software/libc/) | | | | |

+#### Basic multisig example

+

+For a good example of a basic M-of-N multisig between multiple participants using descriptor

+wallets and PSBTs, as well as a signing flow, see [this functional test](/test/functional/wallet_multisig_descriptor_psbt.py).

+

+Disclaimers: It is important to note that this example serves as a quick-start and is kept basic for readability. A downside of the approach

+outlined here is that each participant must maintain (and backup) two separate wallets: a signer and the corresponding multisig.

+It should also be noted that privacy best-practices are not "by default" here - participants should take care to only use the signer to sign

+transactions related to the multisig. Lastly, it is not recommended to use anything other than a Bitcoin Core descriptor wallet to serve as your

+signer(s). Other wallets, whether hardware or software, likely impose additional checks and safeguards to prevent users from signing transactions that

+could lead to loss of funds, or are deemed security hazards. Conforming to various 3rd-party checks and verifications is not in the scope of this example.

+

+The basic steps are:

+

+ 1. Every participant generates an xpub. The most straightforward way is to create a new descriptor wallet which we will refer to as

+ the participant's signer wallet. Avoid reusing this wallet for any purpose other than signing transactions from the

+ corresponding multisig we are about to create. Hint: extract the wallet's xpubs using `listdescriptors` and pick the one from the

+ `pkh` descriptor since it's least likely to be accidentally reused (legacy addresses)

+ 2. Create a watch-only descriptor wallet (blank, private keys disabled). Now the multisig is created by importing the two descriptors:

+ `wsh(sortedmulti(<M>,XPUB1/0/*,XPUB2/0/*,…,XPUBN/0/*))` and `wsh(sortedmulti(<M>,XPUB1/1/*,XPUB2/1/*,…,XPUBN/1/*))`

+ (one descriptor w/ `0` for receiving addresses and another w/ `1` for change). Every participant does this

+ 3. A receiving address is generated for the multisig. As a check to ensure step 2 was done correctly, every participant

+ should verify they get the same addresses

+ 4. Funds are sent to the resulting address

+ 5. A sending transaction from the multisig is created using `walletcreatefundedpsbt` (anyone can initiate this). It is simple to do

+ this in the GUI by going to the `Send` tab in the multisig wallet and creating an unsigned transaction (PSBT)

+ 6. At least `M` participants check the PSBT with their multisig using `decodepsbt` to verify the transaction is OK before signing it.

+ 7. (If OK) the participant signs the PSBT with their signer wallet using `walletprocesspsbt`. It is simple to do this in the GUI by

+ loading the PSBT from file and signing it

+ 8. The signed PSBTs are collected with `combinepsbt`, finalized w/ `finalizepsbt`, and then the resulting transaction is broadcasted

+ to the network. Note that any wallet (eg one of the signers or multisig) is capable of doing this.

+ 9. Checks that balances are correct after the transaction has been included in a block

+

+You may prefer a daisy chained signing flow where each participant signs the PSBT one after another until

+the PSBT has been signed `M` times and is "complete." For the most part, the steps above remain the same, except (6, 7)

+change slightly from signing the original PSBT in parallel to signing it in series. `combinepsbt` is not necessary with

+this signing flow and the last (`m`th) signer can just broadcast the PSBT after signing. Note that a parallel signing flow may be

+preferable in cases where there are more signers. This signing flow is also included in the test / Python example.

+[The test](/test/functional/wallet_multisig_descriptor_psbt.py) is meant to be documentation as much as it is a functional test, so

+it is kept as simple and readable as possible.

+

+- src/minisketch

+ - Upstream at https://github.com/sipa/minisketch ; maintained by Core contributors.

+

+- Use `fs::path::u8string()` and `fs::u8path()` functions when converting path

+ to JSON strings, not `fs::PathToString` and `fs::PathFromString`

+

+ - *Rationale*: JSON strings are Unicode strings, not byte strings, and

+ RFC8259 requires JSON to be encoded as UTF-8.

+

+## Overview of Bitcoin Core fuzzing

+

+[Google](https://github.com/google/fuzzing/) has a good overview of fuzzing in general, with contributions from key architects of some of the most-used fuzzers. [This paper](https://agroce.github.io/bitcoin_report.pdf) includes an external overview of the status of Bitcoin Core fuzzing, as of summer 2021. [John Regehr](https://blog.regehr.org/archives/1687) provides good advice on writing code that assists fuzzers in finding bugs, which is useful for developers to keep in mind.

+

+For a quick start see [Basic M-of-N multisig example using descriptor wallets and PSBTs](./descriptors.md#basic-multisig-example).

+If you are using legacy wallets feel free to continue with the example provided here.

+

+Notable changes

+===============

+

+Updated RPCs

+------------

+

+- `upgradewallet` will now automatically flush the keypool if upgrading

+from a non-HD wallet to an HD wallet, to immediately start using the

+newly-generated HD keys.

+- a new RPC `newkeypool` has been added, which will flush (entirely

+clear and refill) the keypool.

+Fee estimation changes

+----------------------

+

+- Fee estimation now takes the feerate of replacement (RBF) transactions into

+ account. (#22539)

+

+- The `getblock` RPC command now supports verbose level 3 containing transaction inputs

+ `prevout` information. The existing `/rest/block/` REST endpoint is modified to contain

+ this information too. Every `vin` field will contain an additional `prevout` subfield

+ describing the spent output. `prevout` contains the following keys:

+ - `generated` - true if the spent coins was a coinbase.

+ - `height`

+ - `value`

+ - `scriptPubKey`

+* On the new release branch in [`configure.ac`](../configure.ac)(see [this commit](https://github.com/bitcoin/bitcoin/commit/742f7dd)):

+ ../bitcoin-maintainer-tools/make-tag.py v(new version, e.g. 23.0)

+ - Delete post-EOL [release branches](https://github.com/bitcoin/bitcoin/branches/all) and create a tag `v${branch_name}-final`.

+

+ - Delete ["Needs backport" labels](https://github.com/bitcoin/bitcoin/labels?q=backport) for non-existing branches.

+

+1. Block Header Hash as `pointer to unsigned chars` (i.e. 32 bytes in little-endian)