diff options
Diffstat (limited to 'doc')
32 files changed, 877 insertions, 261 deletions
diff --git a/doc/README.md b/doc/README.md index 446684b482..7b6dacaf4f 100644 --- a/doc/README.md +++ b/doc/README.md @@ -59,7 +59,6 @@ The Bitcoin repo's [root README](/README.md) contains relevant information on th - [Translation Strings Policy](translation_strings_policy.md) - [JSON-RPC Interface](JSON-RPC-interface.md) - [Unauthenticated REST Interface](REST-interface.md) -- [Shared Libraries](shared-libraries.md) - [BIPS](bips.md) - [Dnsseed Policy](dnsseed-policy.md) - [Benchmarking](benchmarking.md) diff --git a/doc/bips.md b/doc/bips.md index 952d289daa..8309ee7e92 100644 --- a/doc/bips.md +++ b/doc/bips.md @@ -49,7 +49,7 @@ BIPs that are implemented by Bitcoin Core: * [`BIP 173`](https://github.com/bitcoin/bips/blob/master/bip-0173.mediawiki): Bech32 addresses for native Segregated Witness outputs are supported as of **v0.16.0** ([PR 11167](https://github.com/bitcoin/bitcoin/pull/11167)). Bech32 addresses are generated by default as of **v0.20.0** ([PR 16884](https://github.com/bitcoin/bitcoin/pull/16884)). * [`BIP 174`](https://github.com/bitcoin/bips/blob/master/bip-0174.mediawiki): RPCs to operate on Partially Signed Bitcoin Transactions (PSBT) are present as of **v0.17.0** ([PR 13557](https://github.com/bitcoin/bitcoin/pull/13557)). * [`BIP 176`](https://github.com/bitcoin/bips/blob/master/bip-0176.mediawiki): Bits Denomination [QT only] is supported as of **v0.16.0** ([PR 12035](https://github.com/bitcoin/bitcoin/pull/12035)). -* [`BIP 324`](https://github.com/bitcoin/bips/blob/master/bip-0324.mediawiki): The v2 transport protocol specified by BIP324 and the associated `NODE_P2P_V2` service bit are supported as of **v26.0**, but off by default ([PR 28331](https://github.com/bitcoin/bitcoin/pull/28331)). +* [`BIP 324`](https://github.com/bitcoin/bips/blob/master/bip-0324.mediawiki): The v2 transport protocol specified by BIP324 and the associated `NODE_P2P_V2` service bit are supported as of **v26.0**, but off by default ([PR 28331](https://github.com/bitcoin/bitcoin/pull/28331)). On by default as of **v27.0** ([PR 29347](https://github.com/bitcoin/bitcoin/pull/29347)). * [`BIP 325`](https://github.com/bitcoin/bips/blob/master/bip-0325.mediawiki): Signet test network is supported as of **v0.21.0** ([PR 18267](https://github.com/bitcoin/bitcoin/pull/18267)). * [`BIP 339`](https://github.com/bitcoin/bips/blob/master/bip-0339.mediawiki): Relay of transactions by wtxid is supported as of **v0.21.0** ([PR 18044](https://github.com/bitcoin/bitcoin/pull/18044)). * [`BIP 340`](https://github.com/bitcoin/bips/blob/master/bip-0340.mediawiki) diff --git a/doc/build-freebsd.md b/doc/build-freebsd.md index aa10e4a891..bf86a0ee4b 100644 --- a/doc/build-freebsd.md +++ b/doc/build-freebsd.md @@ -1,6 +1,6 @@ # FreeBSD Build Guide -**Updated for FreeBSD [12.3](https://www.freebsd.org/releases/12.3R/announce/)** +**Updated for FreeBSD [14.0](https://www.freebsd.org/releases/14.0R/announce/)** This guide describes how to build bitcoind, command-line utilities, and GUI on FreeBSD. @@ -64,10 +64,11 @@ sh/bash: export BDB_PREFIX=[path displayed above] #### GUI Dependencies ###### Qt5 -Bitcoin Core includes a GUI built with the cross-platform Qt Framework. To compile the GUI, we need to install `qt5`. Skip if you don't intend to use the GUI. +Bitcoin Core includes a GUI built with the cross-platform Qt Framework. To compile the GUI, we need to install the necessary parts of Qt. Skip if you don't intend to use the GUI. ```bash -pkg install qt5 +pkg install qt5-buildtools qt5-core qt5-gui qt5-linguisttools qt5-testlib qt5-widgets ``` + ###### libqrencode The GUI can encode addresses in a QR Code. To build in QR support for the GUI, install `libqrencode`. Skip if not using the GUI or don't want QR code functionality. diff --git a/doc/build-openbsd.md b/doc/build-openbsd.md index 96ee714341..264d5f2d08 100644 --- a/doc/build-openbsd.md +++ b/doc/build-openbsd.md @@ -1,6 +1,6 @@ # OpenBSD Build Guide -**Updated for OpenBSD [7.3](https://www.openbsd.org/73.html)** +**Updated for OpenBSD [7.5](https://www.openbsd.org/75.html)** This guide describes how to build bitcoind, command-line utilities, and GUI on OpenBSD. @@ -43,6 +43,8 @@ BerkeleyDB is only required to support legacy wallets. It is recommended to use Berkeley DB 4.8. You cannot use the BerkeleyDB library from ports. However you can build it yourself, [using depends](/depends). +Refer to [depends/README.md](/depends/README.md) for detailed instructions. + ```bash gmake -C depends NO_BOOST=1 NO_LIBEVENT=1 NO_QT=1 NO_SQLITE=1 NO_NATPMP=1 NO_UPNP=1 NO_ZMQ=1 NO_USDT=1 ... @@ -61,7 +63,7 @@ export BDB_PREFIX="/path/to/bitcoin/depends/x86_64-unknown-openbsd" Bitcoin Core includes a GUI built with the cross-platform Qt Framework. To compile the GUI, Qt 5 is required. ```bash -pkg_add qt5 +pkg_add qtbase qttools ``` ## Building Bitcoin Core diff --git a/doc/build-osx.md b/doc/build-osx.md index 5c3dc1ac7f..20c92ab7a4 100644 --- a/doc/build-osx.md +++ b/doc/build-osx.md @@ -51,6 +51,20 @@ To install, run the following from your terminal: brew install automake libtool boost pkg-config libevent ``` +For macOS 11 (Big Sur) and 12 (Monterey) you need to install a more recent version of llvm. + +``` bash +brew install llvm +``` + +And append the following to the configure commands below: + +``` bash +CC=$(brew --prefix llvm)/bin/clang CXX=$(brew --prefix llvm)/bin/clang++ +``` + +Try `llvm@17` if compilation fails with the default version of llvm. + ### 4. Clone Bitcoin repository `git` should already be installed by default on your system. diff --git a/doc/build-unix.md b/doc/build-unix.md index bf367fc421..de54fb4eeb 100644 --- a/doc/build-unix.md +++ b/doc/build-unix.md @@ -30,7 +30,7 @@ tuned to conserve memory with additional CXXFLAGS: Alternatively, or in addition, debugging information can be skipped for compilation. The default compile flags are `-g -O2`, and can be changed with: - ./configure CXXFLAGS="-O2" + ./configure CXXFLAGS="-g0" Finally, clang (often less resource hungry) can be used instead of gcc, which is used by default: @@ -81,7 +81,7 @@ To build without GUI pass `--without-gui`. To build with Qt 5 you need the following: - sudo apt-get install libqt5gui5 libqt5core5a libqt5dbus5 qttools5-dev qttools5-dev-tools + sudo apt-get install qtbase5-dev qttools5-dev qttools5-dev-tools Additionally, to support Wayland protocol for modern desktop environments: diff --git a/doc/dependencies.md b/doc/dependencies.md index ba8643be79..2b9d9128d3 100644 --- a/doc/dependencies.md +++ b/doc/dependencies.md @@ -8,8 +8,8 @@ You can find installation instructions in the `build-*.md` file for your platfor | --- | --- | | [Autoconf](https://www.gnu.org/software/autoconf/) | [2.69](https://github.com/bitcoin/bitcoin/pull/17769) | | [Automake](https://www.gnu.org/software/automake/) | [1.13](https://github.com/bitcoin/bitcoin/pull/18290) | -| [Clang](https://clang.llvm.org) | [13.0](https://github.com/bitcoin/bitcoin/pull/28210) | -| [GCC](https://gcc.gnu.org) | [10.1](https://github.com/bitcoin/bitcoin/pull/28348) | +| [Clang](https://clang.llvm.org) | [15.0](https://github.com/bitcoin/bitcoin/pull/29165) | +| [GCC](https://gcc.gnu.org) | [11.1](https://github.com/bitcoin/bitcoin/pull/29091) | | [Python](https://www.python.org) (scripts, tests) | [3.9](https://github.com/bitcoin/bitcoin/pull/28211) | | [systemtap](https://sourceware.org/systemtap/) ([tracing](tracing.md))| N/A | @@ -30,13 +30,13 @@ You can find installation instructions in the `build-*.md` file for your platfor | [Fontconfig](../depends/packages/fontconfig.mk) | [link](https://www.freedesktop.org/wiki/Software/fontconfig/) | [2.12.6](https://github.com/bitcoin/bitcoin/pull/23495) | 2.6 | Yes | | [FreeType](../depends/packages/freetype.mk) | [link](https://freetype.org) | [2.11.0](https://github.com/bitcoin/bitcoin/commit/01544dd78ccc0b0474571da854e27adef97137fb) | 2.3.0 | Yes | | [qrencode](../depends/packages/qrencode.mk) | [link](https://fukuchi.org/works/qrencode/) | [4.1.1](https://github.com/bitcoin/bitcoin/pull/27312) | | No | -| [Qt](../depends/packages/qt.mk) | [link](https://download.qt.io/official_releases/qt/) | [5.15.11](https://github.com/bitcoin/bitcoin/pull/28769) | [5.11.3](https://github.com/bitcoin/bitcoin/pull/24132) | No | +| [Qt](../depends/packages/qt.mk) | [link](https://download.qt.io/official_releases/qt/) | [5.15.13](https://github.com/bitcoin/bitcoin/pull/29732) | [5.11.3](https://github.com/bitcoin/bitcoin/pull/24132) | No | ### Networking | Dependency | Releases | Version used | Minimum required | Runtime | | --- | --- | --- | --- | --- | -| [libnatpmp](../depends/packages/libnatpmp.mk) | [link](https://github.com/miniupnp/libnatpmp/) | commit [07004b9...](https://github.com/bitcoin/bitcoin/pull/25917) | | No | -| [MiniUPnPc](../depends/packages/miniupnpc.mk) | [link](https://miniupnp.tuxfamily.org/) | [2.2.2](https://github.com/bitcoin/bitcoin/pull/20421) | 2.1 | No | +| [libnatpmp](../depends/packages/libnatpmp.mk) | [link](https://github.com/miniupnp/libnatpmp/) | commit [f2433be...](https://github.com/bitcoin/bitcoin/pull/29708) | | No | +| [MiniUPnPc](../depends/packages/miniupnpc.mk) | [link](https://miniupnp.tuxfamily.org/) | [2.2.7](https://github.com/bitcoin/bitcoin/pull/29707) | 2.1 | No | ### Notifications | Dependency | Releases | Version used | Minimum required | Runtime | diff --git a/doc/descriptors.md b/doc/descriptors.md index 1baf652f30..3b94ec03e4 100644 --- a/doc/descriptors.md +++ b/doc/descriptors.md @@ -243,7 +243,18 @@ Often it is useful to communicate a description of scripts along with the necessary private keys. For this reason, anywhere a public key or xpub is supported, a private key in WIF format or xprv may be provided instead. This is useful when private keys are necessary for hardened derivation -steps, or for dumping wallet descriptors including private key material. +steps, for signing transactions, or for dumping wallet descriptors +including private key material. + +For example, after importing the following 2-of-3 multisig descriptor +into a wallet, one could use `signrawtransactionwithwallet` +to sign a transaction with the first key: +``` +sh(multi(2,xprv.../84'/0'/0'/0/0,xpub1...,xpub2...)) +``` +Note how the first key is an xprv private key with a specific derivation path, +while the other two are public keys. + ### Compatibility with old wallets diff --git a/doc/design/assumeutxo.md b/doc/design/assumeutxo.md index 66962a629d..a4980729d0 100644 --- a/doc/design/assumeutxo.md +++ b/doc/design/assumeutxo.md @@ -16,12 +16,12 @@ load it. A pruned node can load a snapshot. To save space, it's possible to delete the snapshot file as soon as `loadtxoutset` finishes. -The minimum `-dbcache` setting is 550 MiB, but this functionality ignores that +The minimum `-prune` setting is 550 MiB, but this functionality ignores that minimum and uses at least 1100 MiB. As the background sync continues there will be temporarily two chainstate directories, each multiple gigabytes in size (likely growing larger than the -the downloaded snapshot). +downloaded snapshot). ### Indexes @@ -51,18 +51,12 @@ The utility script ## Design notes -- A new block index `nStatus` flag is introduced, `BLOCK_ASSUMED_VALID`, to mark block - index entries that are required to be assumed-valid by a chainstate created - from a UTXO snapshot. This flag is used as a way to modify certain - CheckBlockIndex() logic to account for index entries that are pending validation by a - chainstate running asynchronously in the background. - - The concept of UTXO snapshots is treated as an implementation detail that lives behind the ChainstateManager interface. The external presentation of the changes required to facilitate the use of UTXO snapshots is the understanding that there are - now certain regions of the chain that can be temporarily assumed to be valid (using - the nStatus flag mentioned above). In certain cases, e.g. wallet rescanning, this is - very similar to dealing with a pruned chain. + now certain regions of the chain that can be temporarily assumed to be valid. + In certain cases, e.g. wallet rescanning, this is very similar to dealing with + a pruned chain. Logic outside ChainstateManager should try not to know about snapshots, instead preferring to work in terms of more general states like assumed-valid. @@ -145,7 +139,7 @@ sequentially. Once the tip of the background chainstate hits the base block of the snapshot chainstate, we stop use of the background chainstate by setting `m_disabled`, in -`CompleteSnapshotValidation()`, which is checked in `ActivateBestChain()`). We hash the +`MaybeCompleteSnapshotValidation()`, which is checked in `ActivateBestChain()`). We hash the background chainstate's UTXO set contents and ensure it matches the compiled value in `CMainParams::m_assumeutxo_data`. diff --git a/doc/design/libraries.md b/doc/design/libraries.md index 7cda64e713..251c52199d 100644 --- a/doc/design/libraries.md +++ b/doc/design/libraries.md @@ -4,10 +4,9 @@ |--------------------------|-------------| | *libbitcoin_cli* | RPC client functionality used by *bitcoin-cli* executable | | *libbitcoin_common* | Home for common functionality shared by different executables and libraries. Similar to *libbitcoin_util*, but higher-level (see [Dependencies](#dependencies)). | -| *libbitcoin_consensus* | Stable, backwards-compatible consensus functionality used by *libbitcoin_node* and *libbitcoin_wallet* and also exposed as a [shared library](../shared-libraries.md). | -| *libbitcoinconsensus* | Shared library build of static *libbitcoin_consensus* library | -| *libbitcoin_kernel* | Consensus engine and support library used for validation by *libbitcoin_node* and also exposed as a [shared library](../shared-libraries.md). | -| *libbitcoinqt* | GUI functionality used by *bitcoin-qt* and *bitcoin-gui* executables | +| *libbitcoin_consensus* | Stable, backwards-compatible consensus functionality used by *libbitcoin_node* and *libbitcoin_wallet*. | +| *libbitcoin_kernel* | Consensus engine and support library used for validation by *libbitcoin_node*. | +| *libbitcoinqt* | GUI functionality used by *bitcoin-qt* and *bitcoin-gui* executables. | | *libbitcoin_ipc* | IPC functionality used by *bitcoin-node*, *bitcoin-wallet*, *bitcoin-gui* executables to communicate when [`--enable-multiprocess`](multiprocess.md) is used. | | *libbitcoin_node* | P2P and RPC server functionality used by *bitcoind* and *bitcoin-qt* executables. | | *libbitcoin_util* | Home for common functionality shared by different executables and libraries. Similar to *libbitcoin_common*, but lower-level (see [Dependencies](#dependencies)). | @@ -17,7 +16,7 @@ ## Conventions -- Most libraries are internal libraries and have APIs which are completely unstable! There are few or no restrictions on backwards compatibility or rules about external dependencies. Exceptions are *libbitcoin_consensus* and *libbitcoin_kernel* which have external interfaces documented at [../shared-libraries.md](../shared-libraries.md). +- Most libraries are internal libraries and have APIs which are completely unstable! There are few or no restrictions on backwards compatibility or rules about external dependencies. An exception is *libbitcoin_kernel*, which, at some future point, will have a documented external interface. - Generally each library should have a corresponding source directory and namespace. Source code organization is a work in progress, so it is true that some namespaces are applied inconsistently, and if you look at [`libbitcoin_*_SOURCES`](../../src/Makefile.am) lists you can see that many libraries pull in files from outside their source directory. But when working with libraries, it is good to follow a consistent pattern like: @@ -96,7 +95,7 @@ class bitcoin-qt,bitcoind,bitcoin-cli,bitcoin-wallet bold - The only thing that should depend on *libbitcoin_kernel* internally should be *libbitcoin_node*. GUI and wallet libraries *libbitcoinqt* and *libbitcoin_wallet* in particular should not depend on *libbitcoin_kernel* and the unneeded functionality it would pull in, like block validation. To the extent that GUI and wallet code need scripting and signing functionality, they should be get able it from *libbitcoin_consensus*, *libbitcoin_common*, and *libbitcoin_util*, instead of *libbitcoin_kernel*. -- GUI, node, and wallet code internal implementations should all be independent of each other, and the *libbitcoinqt*, *libbitcoin_node*, *libbitcoin_wallet* libraries should never reference each other's symbols. They should only call each other through [`src/interfaces/`](`../../src/interfaces/`) abstract interfaces. +- GUI, node, and wallet code internal implementations should all be independent of each other, and the *libbitcoinqt*, *libbitcoin_node*, *libbitcoin_wallet* libraries should never reference each other's symbols. They should only call each other through [`src/interfaces/`](../../src/interfaces/) abstract interfaces. ## Work in progress diff --git a/doc/design/multiprocess.md b/doc/design/multiprocess.md index 45681d12de..49410a4213 100644 --- a/doc/design/multiprocess.md +++ b/doc/design/multiprocess.md @@ -1,72 +1,264 @@ -# Multiprocess Bitcoin +# Multiprocess Bitcoin Design Document -On unix systems, the `--enable-multiprocess` build option can be passed to `./configure` to build new `bitcoin-node`, `bitcoin-wallet`, and `bitcoin-gui` executables alongside existing `bitcoind` and `bitcoin-qt` executables. +Guide to the design and architecture of the Bitcoin Core multiprocess feature -`bitcoin-node` is a drop-in replacement for `bitcoind`, and `bitcoin-gui` is a drop-in replacement for `bitcoin-qt`, and there are no differences in use or external behavior between the new and old executables. But internally (after [#10102](https://github.com/bitcoin/bitcoin/pull/10102)), `bitcoin-gui` will spawn a `bitcoin-node` process to run P2P and RPC code, communicating with it across a socket pair, and `bitcoin-node` will spawn `bitcoin-wallet` to run wallet code, also communicating over a socket pair. This will let node, wallet, and GUI code run in separate address spaces for better isolation, and allow future improvements like being able to start and stop components independently on different machines and environments. +_This document describes the design of the multiprocess feature. For usage information, see the top-level [multiprocess.md](../multiprocess.md) file._ -## Next steps +## Table of contents -Specific next steps after [#10102](https://github.com/bitcoin/bitcoin/pull/10102) will be: +- [Introduction](#introduction) +- [Current Architecture](#current-architecture) +- [Proposed Architecture](#proposed-architecture) +- [Component Overview: Navigating the IPC Framework](#component-overview-navigating-the-ipc-framework) +- [Design Considerations](#design-considerations) + - [Selection of Cap’n Proto](#selection-of-capn-proto) + - [Hiding IPC](#hiding-ipc) + - [Interface Definition Maintenance](#interface-definition-maintenance) + - [Interface Stability](#interface-stability) +- [Security Considerations](#security-considerations) +- [Example Use Cases and Flows](#example-use-cases-and-flows) + - [Retrieving a Block Hash](#retrieving-a-block-hash) +- [Future Enhancements](#future-enhancements) +- [Conclusion](#conclusion) +- [Appendices](#appendices) + - [Glossary of Terms](#glossary-of-terms) + - [References](#references) +- [Acknowledgements](#acknowledgements) -- [ ] Adding `-ipcbind` and `-ipcconnect` options to `bitcoin-node`, `bitcoin-wallet`, and `bitcoin-gui` executables so they can listen and connect to TCP ports and unix socket paths. This will allow separate processes to be started and stopped any time and connect to each other. -- [ ] Adding `-server` and `-rpcbind` options to the `bitcoin-wallet` executable so wallet processes can handle RPC requests directly without going through the node. -- [ ] Supporting windows, not just unix systems. The existing socket code is already cross-platform, so the only windows-specific code that needs to be written is code spawning a process and passing a socket descriptor. This can be implemented with `CreateProcess` and `WSADuplicateSocket`. Example: https://memset.wordpress.com/2010/10/13/win32-api-passing-socket-with-ipc-method/. -- [ ] Adding sandbox features, restricting subprocess access to resources and data. See [https://eklitzke.org/multiprocess-bitcoin](https://eklitzke.org/multiprocess-bitcoin). +## Introduction -## Debugging +The Bitcoin Core software has historically employed a monolithic architecture. The existing design has integrated functionality like P2P network operations, wallet management, and a GUI into a single executable. While effective, it has limitations in flexibility, security, and scalability. This project introduces changes that transition Bitcoin Core to a more modular architecture. It aims to enhance security, improve usability, and facilitate maintenance and development of the software in the long run. -The `-debug=ipc` command line option can be used to see requests and responses between processes. +## Current Architecture -## Installation +The current system features two primary executables: `bitcoind` and `bitcoin-qt`. `bitcoind` combines a Bitcoin P2P node with an integrated JSON-RPC server, wallet, and indexes. `bitcoin-qt` extends this by incorporating a Qt-based GUI. This monolithic structure, although robust, presents challenges such as limited operational flexibility and increased security risks due to the tight integration of components. -The multiprocess feature requires [Cap'n Proto](https://capnproto.org/) and [libmultiprocess](https://github.com/chaincodelabs/libmultiprocess) as dependencies. A simple way to get starting using it without installing these dependencies manually is to use the [depends system](../../depends) with the `MULTIPROCESS=1` [dependency option](../../depends#dependency-options) passed to make: +## Proposed Architecture +The new architecture divides the existing code into three specialized executables: + +- `bitcoin-node`: Manages the P2P node, indexes, and JSON-RPC server. +- `bitcoin-wallet`: Handles all wallet functionality. +- `bitcoin-gui`: Provides a standalone Qt-based GUI. + +This modular approach is designed to enhance security through component isolation and improve usability by allowing independent operation of each module. This allows for new use-cases, such as running the node on a dedicated machine and operating wallets and GUIs on separate machines with the flexibility to start and stop them as needed. + +This subdivision could be extended in the future. For example, indexes could be removed from the `bitcoin-node` executable and run in separate executables. And JSON-RPC servers could be added to wallet and index executables, so they can listen and respond to RPC requests on their own ports, without needing to forward RPC requests through `bitcoin-node`. + +<table><tr><td> + +```mermaid +flowchart LR + node[bitcoin-node] -- listens on --> socket["<datadir>/node.sock"] + wallet[bitcoin-wallet] -- connects to --> socket + gui[bitcoin-gui] -- connects to --> socket +``` + +</td></tr><tr><td> +Processes and socket connection. +</td></tr></table> + +## Component Overview: Navigating the IPC Framework + +This section describes the major components of the Inter-Process Communication (IPC) framework covering the relevant source files, generated files, tools, and libraries. + +### Abstract C++ Classes in [`src/interfaces/`](../../src/interfaces/) +- The foundation of the IPC implementation lies in the abstract C++ classes within the [`src/interfaces/`](../../src/interfaces/) directory. These classes define pure virtual methods that code in [`src/node/`](../../src/node/), [`src/wallet/`](../../src/wallet/), and [`src/qt/`](../../src/qt/) directories call to interact with each other. +- Each abstract class in this directory represents a distinct interface that the different modules (node, wallet, GUI) implement and use for cross-process communication. +- The classes are written following conventions described in [Internal Interface + Guidelines](../developer-notes.md#internal-interface-guidelines) to ensure + compatibility with Cap'n Proto. + +### Cap’n Proto Files in [`src/ipc/capnp/`](../../src/ipc/capnp/) +- Corresponding to each abstract class, there are `.capnp` files within the [`src/ipc/capnp/`](../../src/ipc/capnp/) directory. These files are used as input to the `mpgen` tool (described below) to generate C++ code. +- These Cap’n Proto files ([learn more about Cap'n Proto RPC](https://capnproto.org/rpc.html)) define the structure and format of messages that are exchanged over IPC. They serve as blueprints for generating C++ code that bridges the gap between high-level C++ interfaces and low-level socket communication. + +### The `mpgen` Code Generation Tool +- A central component of the IPC framework is the `mpgen` tool which is part the [`libmultiprocess` project](https://github.com/chaincodelabs/libmultiprocess). This tool takes the `.capnp` files as input and generates C++ code. +- The generated code handles IPC communication, translating interface calls into socket reads and writes. + +### C++ Client Subclasses in Generated Code +- In the generated code, we have C++ client subclasses that inherit from the abstract classes in [`src/interfaces/`](../../src/interfaces/). These subclasses are the workhorses of the IPC mechanism. +- They implement all the methods of the interface, marshalling arguments into a structured format, sending them as requests to the IPC server via a UNIX socket, and handling the responses. +- These subclasses effectively mask the complexity of IPC, presenting a familiar C++ interface to developers. +- Internally, the client subclasses generated by the `mpgen` tool wrap [client classes generated by Cap'n Proto](https://capnproto.org/cxxrpc.html#clients), and use them to send IPC requests. + +### C++ Server Classes in Generated Code +- On the server side, corresponding generated C++ classes receive IPC requests. These server classes are responsible for unmarshalling method arguments, invoking the corresponding methods in the local [`src/interfaces/`](../../src/interfaces/) objects, and creating the IPC response. +- The server classes ensure that return values (including output argument values and thrown exceptions) are marshalled and sent back to the client, completing the communication cycle. +- Internally, the server subclasses generated by the `mpgen` tool inherit from [server classes generated by Cap'n Proto](https://capnproto.org/cxxrpc.html#servers), and use them to process IPC requests. + +### The `libmultiprocess` Runtime Library +- **Core Functionality**: The `libmultiprocess` runtime library's primary function is to instantiate the generated client and server classes as needed. +- **Bootstrapping IPC Connections**: It provides functions for starting new IPC connections, specifically binding generated client and server classes for an initial `interfaces::Init` interface (defined in [`src/interfaces/init.h`](../../src/interfaces/init.h)) to a UNIX socket. This initial interface has methods returning other interfaces that different Bitcoin Core modules use to communicate after the bootstrapping phase. +- **Asynchronous I/O and Thread Management**: The library is also responsible for managing I/O and threading. Particularly, it ensures that IPC requests never block each other and that new threads on either side of a connection can always make client calls. It also manages worker threads on the server side of calls, ensuring that calls from the same client thread always execute on the same server thread (to avoid locking issues and support nested callbacks). + +### Type Hooks in [`src/ipc/capnp/*-types.h`](../../src/ipc/capnp/) +- **Custom Type Conversions**: In [`src/ipc/capnp/*-types.h`](../../src/ipc/capnp/), function overloads of two `libmultiprocess` C++ functions, `mp::CustomReadField` and `mp::CustomBuildFields`, are defined. These overloads are used for customizing the conversion of specific C++ types to and from Cap’n Proto types. +- **Handling Special Cases**: The `mpgen` tool and `libmultiprocess` library can convert most C++ types to and from Cap’n Proto types automatically, including interface types, primitive C++ types, standard C++ types like `std::vector`, `std::set`, `std::map`, `std::tuple`, and `std::function`, as well as simple C++ structs that consist of aforementioned types and whose fields correspond 1:1 with Cap’n Proto struct fields. For other types, `*-types.h` files provide custom code to convert between C++ and Cap’n Proto data representations. + +### Protocol-Agnostic IPC Code in [`src/ipc/`](../../src/ipc/) +- **Broad Applicability**: Unlike the Cap’n Proto-specific code in [`src/ipc/capnp/`](../../src/ipc/capnp/), the code in the [`src/ipc/`](../../src/ipc/) directory is protocol-agnostic. This enables potential support for other protocols, such as gRPC or a custom protocol in the future. +- **Process Management and Socket Operations**: The main purpose of this component is to provide functions for spawning new processes and creating and connecting to UNIX sockets. +- **ipc::Exception Class**: This code also defines an `ipc::Exception` class which is thrown from the generated C++ client class methods when there is an unexpected IPC error, such as a disconnection. + +<table><tr><td> + +```mermaid +flowchart TD + capnpFile[ipc/capnp/chain.capnp] -->|Input to| mpgenTool([mpgen Tool]) + mpgenTool -->|Generates| proxyTypesH[ipc/capnp/chain.capnp.proxy-types.h] + mpgenTool --> proxyClientCpp[ipc/capnp/chain.capnp.proxy-client.c++] + mpgenTool --> proxyServerCpp[ipc/capnp/chain.capnp.proxy-server.c++] + proxyTypesH -.->|Includes| interfaces/chain.h + proxyClientCpp -.-> interfaces/chain.h + proxyServerCpp -.-> interfaces/chain.h ``` -cd <BITCOIN_SOURCE_DIRECTORY> -make -C depends NO_QT=1 MULTIPROCESS=1 -CONFIG_SITE=$PWD/depends/x86_64-pc-linux-gnu/share/config.site ./configure -make -src/bitcoin-node -regtest -printtoconsole -debug=ipc -BITCOIND=bitcoin-node test/functional/test_runner.py + +</td></tr><tr><td> +Diagram showing generated source files and includes. +</td></tr></table> + +## Design Considerations + +### Selection of Cap’n Proto +The choice to use [Cap’n Proto](https://capnproto.org/) for IPC was primarily influenced by its support for passing object references and managing object lifetimes, which would have to be implemented manually with a framework that only supported plain requests and responses like [gRPC](https://grpc.io/). The support is especially helpful for passing callback objects like `std::function` and enabling bidirectional calls between processes. + +The choice to use an RPC framework at all instead of a custom protocol was necessitated by the size of Bitcoin Core internal interfaces which consist of around 150 methods that pass complex data structures and are called in complicated ways (in parallel, and from callbacks that can be nested and stored). Writing a custom protocol to wrap these complicated interfaces would be a lot more work, akin to writing a new RPC framework. + +### Hiding IPC + +The IPC mechanism is deliberately isolated from the rest of the codebase so less code has to be concerned with IPC. + +Building Bitcoin Core with IPC support is optional, and node, wallet, and GUI code can be compiled to either run in the same process or separate processes. The build system also ensures Cap’n Proto library headers can only be used within the [`src/ipc/capnp/`](../../src/ipc/capnp/) directory, not in other parts of the codebase. + +The libmultiprocess runtime is designed to place as few constraints as possible on IPC interfaces and to make IPC calls act like normal function calls. Method arguments, return values, and exceptions are automatically serialized and sent between processes. Object references and `std::function` arguments are tracked to allow invoked code to call back into invoking code at any time. And there is a 1:1 threading model where every client thread has a corresponding server thread responsible for executing incoming calls from that thread (there can be multiple calls from the same thread due to callbacks) without blocking, and holding the same thread-local variables and locks so behavior is the same whether IPC is used or not. + +### Interface Definition Maintenance + +The choice to maintain interface definitions and C++ type mappings as `.capnp` files in the [`src/ipc/capnp/`](../../src/ipc/capnp/) was mostly done for convenience, and probably something that could be improved in the future. + +In the current design, class names, method names, and parameter names are duplicated between C++ interfaces in [`src/interfaces/`](../../src/interfaces/) and Cap’n Proto files in [`src/ipc/capnp/`](../../src/ipc/capnp/). While this keeps C++ interface headers simple and free of references to IPC, it is a maintenance burden because it means inconsistencies between C++ declarations and Cap’n Proto declarations will result in compile errors. (Static type checking ensures these are not runtime errors.) + +An alternate approach could use custom [C++ Attributes](https://en.cppreference.com/w/cpp/language/attributes) embedded in interface declarations to automatically generate `.capnp` files from C++ headers. This has not been pursued because parsing C++ headers is more complicated than parsing Cap’n Proto interface definitions, especially portably on multiple platforms. + +In the meantime, the developer guide [Internal interface guidelines](../developer-notes.md#internal-interface-guidelines) can provide guidance on keeping interfaces consistent and functional and avoiding compile errors. + +### Interface Stability + +The currently defined IPC interfaces are unstable, and can change freely with no backwards compatibility. The decision to allow this stems from the recognition that our current interfaces are still evolving and not yet ideal for external use. As these interfaces mature and become more refined, there may be an opportunity to declare them stable and use Cap’n Proto's support for protocol evolution ([Cap'n Proto - Evolving Your Protocol](https://capnproto.org/language.html#evolving-your-protocol)) to allow them to be extended while remaining backwards compatible. This could allow different versions of node, GUI, and wallet binaries to interoperate, and potentially open doors for external tools to utilize these interfaces, such as creating custom indexes through a stable indexing interface. However, for now, the priority is to improve the interfaces internally. Given their current state and the advantages of using JSON-RPC for most common tasks, it's more practical to focus on internal development rather than external applicability. + +## Security Considerations + +The integration of [Cap’n Proto](https://capnproto.org/) and [libmultiprocess](https://github.com/chaincodelabs/libmultiprocess) into the Bitcoin Core architecture increases its potential attack surface. Cap’n Proto, being a complex and substantial new dependency, introduces potential sources of vulnerability, particularly through the creation of new UNIX sockets. The inclusion of libmultiprocess, while a smaller external dependency, also contributes to this risk. However, plans are underway to incorporate libmultiprocess as a git subtree, aligning it more closely with the project's well-reviewed internal libraries. While adopting these multiprocess features does introduce some risk, it's worth noting that they can be disabled, allowing builds without these new dependencies. This flexibility ensures that users can balance functionality with security considerations as needed. + +## Example Use Cases and Flows + +### Retrieving a Block Hash + +Let’s walk through an example where the `bitcoin-wallet` process requests the hash of a block at a specific height from the `bitcoin-node` process. This example demonstrates the practical application of the IPC mechanism, specifically the interplay between C++ method calls and Cap’n Proto-generated RPC calls. + +<table><tr><td> + +```mermaid +sequenceDiagram + box "bitcoin-wallet process" + participant WalletCode as Wallet code + participant ChainClient as Generated Chain client class<br/>ProxyClient<messages::Chain> + end + box "bitcoin-node process" + participant ChainServer as Generated Chain server class<br/>ProxyServer<messages::Chain> + participant LocalChain as Chain object<br/>node::ChainImpl + end + + WalletCode->>ChainClient: getBlockHash(height) + ChainClient->>ChainServer: Send RPC getBlockHash request + ChainServer->>LocalChain: getBlockHash(height) + LocalChain->>ChainServer: Return block hash + ChainServer->>ChainClient: Send response with block hash + ChainClient->>WalletCode: Return block hash ``` -The configure script will pick up settings and library locations from the depends directory, so there is no need to pass `--enable-multiprocess` as a separate flag when using the depends system (it's controlled by the `MULTIPROCESS=1` option). - -Alternately, you can install [Cap'n Proto](https://capnproto.org/) and [libmultiprocess](https://github.com/chaincodelabs/libmultiprocess) packages on your system, and just run `./configure --enable-multiprocess` without using the depends system. The configure script will be able to locate the installed packages via [pkg-config](https://www.freedesktop.org/wiki/Software/pkg-config/). See [Installation](https://github.com/chaincodelabs/libmultiprocess/blob/master/doc/install.md) section of the libmultiprocess readme for install steps. See [build-unix.md](../build-unix.md) and [build-osx.md](../build-osx.md) for information about installing dependencies in general. - -## IPC implementation details - -Cross process Node, Wallet, and Chain interfaces are defined in -[`src/interfaces/`](../../src/interfaces/). These are C++ classes which follow -[conventions](../developer-notes.md#internal-interface-guidelines), like passing -serializable arguments so they can be called from different processes, and -making methods pure virtual so they can have proxy implementations that forward -calls between processes. - -When Wallet, Node, and Chain code is running in the same process, calling any -interface method invokes the implementation directly. When code is running in -different processes, calling an interface method invokes a proxy interface -implementation that communicates with a remote process and invokes the real -implementation in the remote process. The -[libmultiprocess](https://github.com/chaincodelabs/libmultiprocess) code -generation tool internally generates proxy client classes and proxy server -classes for this purpose that are thin wrappers around Cap'n Proto -[client](https://capnproto.org/cxxrpc.html#clients) and -[server](https://capnproto.org/cxxrpc.html#servers) classes, which handle the -actual serialization and socket communication. - -As much as possible, calls between processes are meant to work the same as -calls within a single process without adding limitations or requiring extra -implementation effort. Processes communicate with each other by calling regular -[C++ interface methods](../../src/interfaces/README.md). Method arguments and -return values are automatically serialized and sent between processes. Object -references and `std::function` arguments are automatically tracked and mapped -to allow invoked code to call back into invoking code at any time, and there is -a 1:1 threading model where any thread invoking a method in another process has -a corresponding thread in the invoked process responsible for executing all -method calls from the source thread, without blocking I/O or holding up another -call, and using the same thread local variables, locks, and callbacks between -calls. The forwarding, tracking, and threading is implemented inside the -[libmultiprocess](https://github.com/chaincodelabs/libmultiprocess) library -which has the design goal of making calls between processes look like calls in -the same process to the extent possible. +</td></tr><tr><td> +<code>Chain::getBlockHash</code> call diagram +</td></tr></table> + +1. **Initiation in bitcoin-wallet** + - The wallet process calls the `getBlockHash` method on a `Chain` object. This method is defined as a virtual method in [`src/interfaces/chain.h`](../../src/interfaces/chain.h). + +2. **Translation to Cap’n Proto RPC** + - The `Chain::getBlockHash` virtual method is overridden by the `Chain` [client subclass](#c-client-subclasses-in-generated-code) to translate the method call into a Cap’n Proto RPC call. + - The client subclass is automatically generated by the `mpgen` tool from the [`chain.capnp`](https://github.com/ryanofsky/bitcoin/blob/pr/ipc/src/ipc/capnp/chain.capnp) file in [`src/ipc/capnp/`](../../src/ipc/capnp/). + +3. **Request Preparation and Dispatch** + - The `getBlockHash` method of the generated `Chain` client subclass in `bitcoin-wallet` populates a Cap’n Proto request with the `height` parameter, sends it to `bitcoin-node` process, and waits for a response. + +4. **Handling in bitcoin-node** + - Upon receiving the request, the Cap'n Proto dispatching code in the `bitcoin-node` process calls the `getBlockHash` method of the `Chain` [server class](#c-server-classes-in-generated-code). + - The server class is automatically generated by the `mpgen` tool from the [`chain.capnp`](https://github.com/ryanofsky/bitcoin/blob/pr/ipc/src/ipc/capnp/chain.capnp) file in [`src/ipc/capnp/`](../../src/ipc/capnp/). + - The `getBlockHash` method of the generated `Chain` server subclass in `bitcoin-wallet` receives a Cap’n Proto request object with the `height` parameter, and calls the `getBlockHash` method on its local `Chain` object with the provided `height`. + - When the call returns, it encapsulates the return value in a Cap’n Proto response, which it sends back to the `bitcoin-wallet` process, + +5. **Response and Return** + - The `getBlockHash` method of the generated `Chain` client subclass in `bitcoin-wallet` which sent the request now receives the response. + - It extracts the block hash value from the response, and returns it to the original caller. + +## Future Enhancements + +Further improvements are possible such as: + +- Separating indexes from `bitcoin-node`, and running indexing code in separate processes (see [indexes: Stop using node internal types #24230](https://github.com/bitcoin/bitcoin/pull/24230)). +- Enabling wallet processes to listen for JSON-RPC requests on their own ports instead of needing the node process to listen and forward requests to them. +- Automatically generating `.capnp` files from C++ interface definitions (see [Interface Definition Maintenance](#interface-definition-maintenance)). +- Simplifying and stabilizing interfaces (see [Interface Stability](#interface-stability)). +- Adding sandbox features, restricting subprocess access to resources and data (see [https://eklitzke.org/multiprocess-bitcoin](https://eklitzke.org/multiprocess-bitcoin)). +- Using Cap'n Proto's support for [other languages](https://capnproto.org/otherlang.html), such as [Rust](https://github.com/capnproto/capnproto-rust), to allow code written in other languages to call Bitcoin Core C++ code, and vice versa (see [How to rustify libmultiprocess? #56](https://github.com/chaincodelabs/libmultiprocess/issues/56)). + +## Conclusion + +This modularization represents an advancement in Bitcoin Core's architecture, offering enhanced security, flexibility, and maintainability. The project invites collaboration and feedback from the community. + +## Appendices + +### Glossary of Terms + +- **abstract class**: A class in C++ that consists of virtual functions. In the Bitcoin Core project, they define interfaces for inter-component communication. + +- **asynchronous I/O**: A form of input/output processing that allows a program to continue other operations while a transmission is in progress. + +- **Cap’n Proto**: A high-performance data serialization and RPC library, chosen for its support for object references and bidirectional communication. + +- **Cap’n Proto interface**: A set of methods defined in Cap’n Proto to facilitate structured communication between different software components. + +- **Cap’n Proto struct**: A structured data format used in Cap’n Proto, similar to structs in C++, for organizing and transporting data across different processes. + +- **client class (in generated code)**: A C++ class generated from a Cap’n Proto interface which inherits from a Bitcoin core abstract class, and implements each virtual method to send IPC requests to another process. (see also [components section](#c-client-subclasses-in-generated-code)) + +- **IPC (inter-process communication)**: Mechanisms that enable processes to exchange requests and data. + +- **ipc::Exception class**: A class within Bitcoin Core's protocol-agnostic IPC code that is thrown by client class methods when there is an IPC error. + +- **libmultiprocess**: A custom library and code generation tool used for creating IPC interfaces and managing IPC connections. + +- **marshalling**: Transforming an object’s memory representation for transmission. + +- **mpgen tool**: A tool within the `libmultiprocess` suite that generates C++ code from Cap’n Proto files, facilitating IPC. + +- **protocol-agnostic code**: Generic IPC code in [`src/ipc/`](../../src/ipc/) that does not rely on Cap’n Proto and could be used with other protocols. Distinct from code in [`src/ipc/capnp/`](../../src/ipc/capnp/) which relies on Cap’n Proto. + +- **RPC (remote procedure call)**: A protocol that enables a program to request a service from another program in a different address space or network. Bitcoin Core uses [JSON-RPC](https://en.wikipedia.org/wiki/JSON-RPC) for RPC. + +- **server class (in generated code)**: A C++ class generated from a Cap’n Proto interface which handles requests sent by a _client class_ in another process. The request handled by calling a local Bitcoin Core interface method, and the return values (if any) are sent back in a response. (see also: [components section](#c-server-classes-in-generated-code)) + +- **unix socket**: Communication endpoint which is a filesystem path, used for exchanging data between processes running on the same host. + +- **virtual method**: A function or method whose behavior can be overridden within an inheriting class by a function with the same signature. + +## References + +- **Cap’n Proto RPC protocol description**: https://capnproto.org/rpc.html +- **libmultiprocess project page**: https://github.com/chaincodelabs/libmultiprocess + +## Acknowledgements + +This design doc was written by @ryanofsky, who is grateful to all the reviewers who gave feedback and tested [multiprocess PRs](https://github.com/bitcoin/bitcoin/pull/28722), and everyone else who's helped with this project. Particular thanks to @ariard who deeply reviewed IPC code and improved the design of the IPC library and initialization process. @jnewbery who championed the early refactoring PRs and helped guide them through development and review. @sjors who has reviewed and repeatedly tested multiprocess code, reporting many issues and helping debug them. @hebasto, @fanquake, and @maflcko who made significant improvements to the build system and fixed countless build issues. @vasild and @jamesob who were brave contributors to the libmultiprocess library. And Chaincode Labs for making this work possible. Also thanks to ChatGPT, who actually wrote most of this document (not @ryanofsky). diff --git a/doc/developer-notes.md b/doc/developer-notes.md index 00a3b2ea6d..3458a71467 100644 --- a/doc/developer-notes.md +++ b/doc/developer-notes.md @@ -113,6 +113,10 @@ code. between integer types, use functional casts such as `int(x)` or `int{x}` instead of `(int) x`. When casting between more complex types, use `static_cast`. Use `reinterpret_cast` and `const_cast` as appropriate. + - Prefer [`list initialization ({})`](https://isocpp.github.io/CppCoreGuidelines/CppCoreGuidelines#Res-list) where possible. + For example `int x{0};` instead of `int x = 0;` or `int x(0);` + - Recursion is checked by clang-tidy and thus must be made explicit. Use + `NOLINTNEXTLINE(misc-no-recursion)` to suppress the check. For function calls a namespace should be specified explicitly, unless such functions have been declared within it. Otherwise, [argument-dependent lookup](https://en.cppreference.com/w/cpp/language/adl), also known as ADL, could be @@ -138,7 +142,7 @@ int main() Block style example: ```c++ -int g_count = 0; +int g_count{0}; namespace foo { class Class @@ -150,7 +154,7 @@ public: { // Comment summarising what this section of code does for (int i = 0; i < n; ++i) { - int total_sum = 0; + int total_sum{0}; // When something fails, return early if (!Something()) return false; ... @@ -330,7 +334,7 @@ Recommendations: - Avoid linking to external documentation; links can break. - Javadoc and all valid Doxygen comments are stripped from Doxygen source code - previews (`STRIP_CODE_COMMENTS = YES` in [Doxyfile.in](doc/Doxyfile.in)). If + previews (`STRIP_CODE_COMMENTS = YES` in [Doxyfile.in](/doc/Doxyfile.in)). If you want a comment to be preserved, it must instead use `//` or `/* */`. ### Generating Documentation @@ -356,7 +360,7 @@ produce better debugging builds. ### Show sources in debugging If you have ccache enabled, absolute paths are stripped from debug information -with the -fdebug-prefix-map and -fmacro-prefix-map options (if supported by the +with the `-fdebug-prefix-map` and `-fmacro-prefix-map` options (if supported by the compiler). This might break source file detection in case you move binaries after compilation, debug from the directory other than the project root or use an IDE that only supports absolute paths for debugging. @@ -575,13 +579,6 @@ export UBSAN_OPTIONS="suppressions=$(pwd)/test/sanitizer_suppressions/ubsan:prin See the CI config for more examples, and upstream documentation for more information about any additional options. -There are a number of known problems when using the `address` sanitizer. The -address sanitizer is known to fail in -[sha256_sse4::Transform](/src/crypto/sha256_sse4.cpp) which makes it unusable -unless you also use `--disable-asm` when running configure. We would like to fix -sanitizer issues, so please send pull requests if you can fix any errors found -by the address sanitizer (or any other sanitizer). - Not all sanitizer options can be enabled at the same time, e.g. trying to build with `--with-sanitizers=address,thread` will fail in the configure script as these sanitizers are mutually incompatible. Refer to your compiler manual to @@ -725,6 +722,47 @@ General Bitcoin Core - *Explanation*: If the test suite is to be updated for a change, this has to be done first. +Logging +------- + +The macros `LogInfo`, `LogDebug`, `LogTrace`, `LogWarning` and `LogError` are available for +logging messages. They should be used as follows: + +- `LogDebug(BCLog::CATEGORY, fmt, params...)` is what you want + most of the time, and it should be used for log messages that are + useful for debugging and can reasonably be enabled on a production + system (that has sufficient free storage space). They will be logged + if the program is started with `-debug=category` or `-debug=1`. + Note that `LogPrint(BCLog::CATEGORY, fmt, params...)` is a deprecated + alias for `LogDebug`. + +- `LogInfo(fmt, params...)` should only be used rarely, e.g. for startup + messages or for infrequent and important events such as a new block tip + being found or a new outbound connection being made. These log messages + are unconditional, so care must be taken that they can't be used by an + attacker to fill up storage. Note that `LogPrintf(fmt, params...)` is + a deprecated alias for `LogInfo`. + +- `LogError(fmt, params...)` should be used in place of `LogInfo` for + severe problems that require the node (or a subsystem) to shut down + entirely (e.g., insufficient storage space). + +- `LogWarning(fmt, params...)` should be used in place of `LogInfo` for + severe problems that the node admin should address, but are not + severe enough to warrant shutting down the node (e.g., system time + appears to be wrong, unknown soft fork appears to have activated). + +- `LogTrace(BCLog::CATEGORY, fmt, params...)` should be used in place of + `LogDebug` for log messages that would be unusable on a production + system, e.g. due to being too noisy in normal use, or too resource + intensive to process. These will be logged if the startup + options `-debug=category -loglevel=category:trace` or `-debug=1 + -loglevel=trace` are selected. + +Note that the format strings and parameters of `LogDebug` and `LogTrace` +are only evaluated if the logging category is enabled, so you must be +careful to avoid side-effects in those expressions. + Wallet ------- @@ -887,7 +925,7 @@ Strings and formatting `wcstoll`, `wcstombs`, `wcstoul`, `wcstoull`, `wcstoumax`, `wcswidth`, `wcsxfrm`, `wctob`, `wctomb`, `wctrans`, `wctype`, `wcwidth`, `wprintf` -- For `strprintf`, `LogPrint`, `LogPrintf` formatting characters don't need size specifiers. +- For `strprintf`, `LogInfo`, `LogDebug`, etc formatting characters don't need size specifiers. - *Rationale*: Bitcoin Core uses tinyformat, which is type safe. Leave them out to avoid confusion. @@ -899,7 +937,7 @@ Strings and formatting - *Rationale*: Although this is guaranteed to be safe starting with C++11, `.data()` communicates the intent better. - - Do not use it when passing strings to `tfm::format`, `strprintf`, `LogPrint[f]`. + - Do not use it when passing strings to `tfm::format`, `strprintf`, `LogInfo`, `LogDebug`, etc. - *Rationale*: This is redundant. Tinyformat handles strings. @@ -1297,8 +1335,7 @@ Release notes should be written for any PR that: Release notes should be added to a PR-specific release note file at `/doc/release-notes-<PR number>.md` to avoid conflicts between multiple PRs. -All `release-notes*` files are merged into a single -[/doc/release-notes.md](/doc/release-notes.md) file prior to the release. +All `release-notes*` files are merged into a single `release-notes-<version>.md` file prior to the release. RPC interface guidelines -------------------------- diff --git a/doc/dnsseed-policy.md b/doc/dnsseed-policy.md index 55a5c28258..99f48f2670 100644 --- a/doc/dnsseed-policy.md +++ b/doc/dnsseed-policy.md @@ -44,7 +44,7 @@ related to the DNS seed operation. If these expectations cannot be satisfied the operator should discontinue providing services and contact the active Bitcoin Core development team as well as posting on -[bitcoin-dev](https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev). +[bitcoin-dev](https://groups.google.com/g/bitcoindev). Behavior outside of these expectations may be reasonable in some situations but should be discussed in public in advance. diff --git a/doc/external-signer.md b/doc/external-signer.md index de44cdd880..1468e852ef 100644 --- a/doc/external-signer.md +++ b/doc/external-signer.md @@ -150,6 +150,9 @@ Example, display the first native SegWit receive address on Testnet: The command MUST be able to figure out the address type from the descriptor. +The command MUST return an object containing `{"address": "[the address]"}`. +As a sanity check, for devices that support this, it SHOULD ask the device to derive the address. + If <descriptor> contains a master key fingerprint, the command MUST fail if it does not match the fingerprint known by the device. If <descriptor> contains an xpub, the command MUST fail if it does not match the xpub known by the device. diff --git a/doc/fuzzing.md b/doc/fuzzing.md index a4b0198dd9..c9fb918c8f 100644 --- a/doc/fuzzing.md +++ b/doc/fuzzing.md @@ -127,11 +127,6 @@ The default Clang/LLVM version supplied by Apple on macOS does not include fuzzing libraries, so macOS users will need to install a full version, for example using `brew install llvm`. -Should you run into problems with the address sanitizer, it is possible you -may need to run `./configure` with `--disable-asm` to avoid errors -with certain assembly code from Bitcoin Core's code. See [developer notes on sanitizers](https://github.com/bitcoin/bitcoin/blob/master/doc/developer-notes.md#sanitizers) -for more information. - You may also need to take care of giving the correct path for `clang` and `clang++`, like `CC=/path/to/clang CXX=/path/to/clang++` if the non-systems `clang` does not come first in your path. @@ -139,7 +134,7 @@ You may also need to take care of giving the correct path for `clang` and Full configure that was tested on macOS with `brew` installed `llvm`: ```sh -./configure --enable-fuzz --with-sanitizers=fuzzer,address,undefined --disable-asm CC=$(brew --prefix llvm)/bin/clang CXX=$(brew --prefix llvm)/bin/clang++ +./configure --enable-fuzz --with-sanitizers=fuzzer,address,undefined CC=$(brew --prefix llvm)/bin/clang CXX=$(brew --prefix llvm)/bin/clang++ ``` Read the [libFuzzer documentation](https://llvm.org/docs/LibFuzzer.html) for more information. This [libFuzzer tutorial](https://github.com/google/fuzzing/blob/master/tutorial/libFuzzerTutorial.md) might also be of interest. diff --git a/doc/i2p.md b/doc/i2p.md index b6c07388b7..b769a74df4 100644 --- a/doc/i2p.md +++ b/doc/i2p.md @@ -79,8 +79,8 @@ one of the networks has issues. ## Persistent vs transient I2P addresses The first time Bitcoin Core connects to the I2P router, it automatically -generates a persistent I2P address and its corresponding private key by default -or if `-i2pacceptincoming=1` is set. The private key is saved in a file named +generates a persistent I2P address and its corresponding private key by default, +unless `-i2pacceptincoming=0` is set. The private key is saved in a file named `i2p_private_key` in the Bitcoin Core data directory. The persistent I2P address is used for making outbound connections and accepting inbound connections. diff --git a/doc/managing-wallets.md b/doc/managing-wallets.md index 22e006c963..b99d88877b 100644 --- a/doc/managing-wallets.md +++ b/doc/managing-wallets.md @@ -122,6 +122,22 @@ $ bitcoin-cli -rpcwallet="restored-wallet" getwalletinfo The restored wallet can also be loaded in the GUI via `File` ->`Open wallet`. +## Wallet Passphrase + +Understanding wallet security is crucial for safely storing your Bitcoin. A key aspect is the wallet passphrase, used for encryption. Let's explore its nuances, role, encryption process, and limitations. + +- **Not the Seed:** +The wallet passphrase and the seed are two separate components in wallet security. The seed, or HD seed, functions as a master key for deriving private and public keys in a hierarchical deterministic (HD) wallet. In contrast, the passphrase serves as an additional layer of security specifically designed to secure the private keys within the wallet. The passphrase serves as a safeguard, demanding an additional layer of authentication to access funds in the wallet. + +- **Protection Against Unauthorized Access:** +The passphrase serves as a protective measure, securing your funds in situations where an unauthorized user gains access to your unlocked computer or device while your wallet application is active. Without the passphrase, they would be unable to access your wallet's funds or execute transactions. However, it's essential to be aware that someone with access can potentially compromise the security of your passphrase by installing a keylogger. + +- **Doesn't Encrypt Metadata or Public Keys:** +It's important to note that the passphrase primarily secures the private keys and access to funds within the wallet. It does not encrypt metadata associated with transactions or public keys. Information about your transaction history and the public keys involved may still be visible. + +- **Risk of Fund Loss if Forgotten or Lost:** +If the wallet passphrase is too complex and is subsequently forgotten or lost, there is a risk of losing access to the funds permanently. A forgotten passphrase will result in the inability to unlock the wallet and access the funds. + ## Migrating Legacy Wallets to Descriptor Wallets Legacy wallets (traditional non-descriptor wallets) can be migrated to become Descriptor wallets diff --git a/doc/multiprocess.md b/doc/multiprocess.md new file mode 100644 index 0000000000..7ba89b3ff5 --- /dev/null +++ b/doc/multiprocess.md @@ -0,0 +1,34 @@ +# Multiprocess Bitcoin + +_This document describes usage of the multiprocess feature. For design information, see the [design/multiprocess.md](design/multiprocess.md) file._ + +## Build Option + +On unix systems, the `--enable-multiprocess` build option can be passed to `./configure` to build new `bitcoin-node`, `bitcoin-wallet`, and `bitcoin-gui` executables alongside existing `bitcoind` and `bitcoin-qt` executables. + +## Debugging + +The `-debug=ipc` command line option can be used to see requests and responses between processes. + +## Installation + +The multiprocess feature requires [Cap'n Proto](https://capnproto.org/) and [libmultiprocess](https://github.com/chaincodelabs/libmultiprocess) as dependencies. A simple way to get started using it without installing these dependencies manually is to use the [depends system](../depends) with the `MULTIPROCESS=1` [dependency option](../depends#dependency-options) passed to make: + +``` +cd <BITCOIN_SOURCE_DIRECTORY> +make -C depends NO_QT=1 MULTIPROCESS=1 +CONFIG_SITE=$PWD/depends/x86_64-pc-linux-gnu/share/config.site ./configure +make +src/bitcoin-node -regtest -printtoconsole -debug=ipc +BITCOIND=bitcoin-node test/functional/test_runner.py +``` + +The configure script will pick up settings and library locations from the depends directory, so there is no need to pass `--enable-multiprocess` as a separate flag when using the depends system (it's controlled by the `MULTIPROCESS=1` option). + +Alternately, you can install [Cap'n Proto](https://capnproto.org/) and [libmultiprocess](https://github.com/chaincodelabs/libmultiprocess) packages on your system, and just run `./configure --enable-multiprocess` without using the depends system. The configure script will be able to locate the installed packages via [pkg-config](https://www.freedesktop.org/wiki/Software/pkg-config/). See [Installation](https://github.com/chaincodelabs/libmultiprocess/blob/master/doc/install.md) section of the libmultiprocess readme for install steps. See [build-unix.md](build-unix.md) and [build-osx.md](build-osx.md) for information about installing dependencies in general. + +## Usage + +`bitcoin-node` is a drop-in replacement for `bitcoind`, and `bitcoin-gui` is a drop-in replacement for `bitcoin-qt`, and there are no differences in use or external behavior between the new and old executables. But internally after [#10102](https://github.com/bitcoin/bitcoin/pull/10102), `bitcoin-gui` will spawn a `bitcoin-node` process to run P2P and RPC code, communicating with it across a socket pair, and `bitcoin-node` will spawn `bitcoin-wallet` to run wallet code, also communicating over a socket pair. This will let node, wallet, and GUI code run in separate address spaces for better isolation, and allow future improvements like being able to start and stop components independently on different machines and environments. +[#19460](https://github.com/bitcoin/bitcoin/pull/19460) also adds a new `bitcoin-node` `-ipcbind` option and an `bitcoind-wallet` `-ipcconnect` option to allow new wallet processes to connect to an existing node process. +And [#19461](https://github.com/bitcoin/bitcoin/pull/19461) adds a new `bitcoin-gui` `-ipcconnect` option to allow new GUI processes to connect to an existing node process. diff --git a/doc/offline-signing-tutorial.md b/doc/offline-signing-tutorial.md index 4761cf256b..401ee52c1c 100644 --- a/doc/offline-signing-tutorial.md +++ b/doc/offline-signing-tutorial.md @@ -114,7 +114,7 @@ At this point, it's important to understand that both the `offline_wallet` and o tb1qtu5qgc6ddhmqm5yqjvhg83qgk2t4ewajg0h6yh ``` -2. Visit a faucet like https://signet.bc-2.jp and enter your address from the previous command to receive a small amount of signet coins to this address. +2. Visit a faucet like https://signetfaucet.com and enter your address from the previous command to receive a small amount of signet coins to this address. 3. Confirm that coins were received using the online `watch_only_wallet`. Note that the transaction may take a few moments before being received on your local node, depending on its connectivity. Just re-run the command periodically until the transaction is received. @@ -252,4 +252,4 @@ You can also show transactions related to the wallet using `listtransactions` { ... } -```
\ No newline at end of file +``` diff --git a/doc/policy/README.md b/doc/policy/README.md index 27536407e7..4392ffb610 100644 --- a/doc/policy/README.md +++ b/doc/policy/README.md @@ -2,8 +2,8 @@ **Policy** (Mempool or Transaction Relay Policy) is the node's set of validation rules, in addition to consensus, enforced for unconfirmed transactions before submitting them to the mempool. These -rules are local to the node and configurable (e.g. `-minrelaytxfee`, `-limitancestorsize`, -`-incrementalrelayfee`). Policy may include restrictions on the transaction itself, the transaction +rules are local to the node and configurable, see "Node relay options" when running `-help`. +Policy may include restrictions on the transaction itself, the transaction in relation to the current chain tip, and the transaction in relation to the node's mempool contents. Policy is *not* applied to transactions in blocks. diff --git a/doc/policy/mempool-replacements.md b/doc/policy/mempool-replacements.md index b3c4239b73..3fd7ff2ad2 100644 --- a/doc/policy/mempool-replacements.md +++ b/doc/policy/mempool-replacements.md @@ -11,7 +11,8 @@ their in-mempool descendants (together, "original transactions") if, in addition other consensus and policy rules, each of the following conditions are met: 1. The directly conflicting transactions all signal replaceability explicitly. A transaction is - signaling replaceability if any of its inputs have an nSequence number less than (0xffffffff - 1). + signaling BIP125 replaceability if any of its inputs have an nSequence number less than (0xffffffff - 1). + A transaction also signals replaceability if its nVersion field is set to 3. *Rationale*: See [BIP125 explanation](https://github.com/bitcoin/bips/blob/master/bip-0125.mediawiki#motivation). diff --git a/doc/release-notes-27114.md b/doc/release-notes-27114.md new file mode 100644 index 0000000000..980ffd78a4 --- /dev/null +++ b/doc/release-notes-27114.md @@ -0,0 +1,2 @@ +- Additional flags "in" and "out" have been added to `-whitelist` to control whether + permissions apply to incoming connections and/or manual (default: incoming only).
\ No newline at end of file diff --git a/doc/release-notes-27375.md b/doc/release-notes-27375.md new file mode 100644 index 0000000000..e3f4ebdf77 --- /dev/null +++ b/doc/release-notes-27375.md @@ -0,0 +1,6 @@ +P2P +--- + +UNIX domain sockets can now be used for proxy connections. Set `-onion` or `-proxy` +to the local socket path with the prefix `unix:` (e.g. `-onion=unix:/home/me/torsocket`). +(#27375)
\ No newline at end of file diff --git a/doc/release-notes-27679.md b/doc/release-notes-27679.md new file mode 100644 index 0000000000..dbbb30428c --- /dev/null +++ b/doc/release-notes-27679.md @@ -0,0 +1,2 @@ +- unix socket paths are now accepted for `-zmqpubrawblock` and `-zmqpubrawtx` with +the format `-zmqpubrawtx=unix:/path/to/file`
\ No newline at end of file diff --git a/doc/release-notes-28207.md b/doc/release-notes-28207.md deleted file mode 100644 index 56b88da16a..0000000000 --- a/doc/release-notes-28207.md +++ /dev/null @@ -1,7 +0,0 @@ -mempool.dat compatibility -======================== - -The `mempool.dat` file created by -persistmempool or the savemempool RPC will -be written in a new format, which can not be read by previous software -releases. To allow for a downgrade, a temporary setting `-persistmempoolv1` has -been added to fall back to the legacy format. diff --git a/doc/release-notes-29845.md b/doc/release-notes-29845.md new file mode 100644 index 0000000000..4994d0a34d --- /dev/null +++ b/doc/release-notes-29845.md @@ -0,0 +1,8 @@ +RPC +--- + +- the `warnings` field in `getblockchaininfo`, `getmininginfo` and + `getnetworkinfo` now returns all the active node warnings as an array + of strings, instead of just a single warning. The current behaviour + can temporarily be restored by running bitcoind with configuration + option `-deprecatedrpc=warnings`.
\ No newline at end of file diff --git a/doc/release-notes/release-notes-25.2.md b/doc/release-notes/release-notes-25.2.md new file mode 100644 index 0000000000..3f050ebaef --- /dev/null +++ b/doc/release-notes/release-notes-25.2.md @@ -0,0 +1,74 @@ +25.2 Release Notes +================== + +Bitcoin Core version 25.2 is now available from: + + <https://bitcoincore.org/bin/bitcoin-core-25.2> + +This release includes various bug fixes and performance +improvements, as well as updated translations. + +Please report bugs using the issue tracker at GitHub: + + <https://github.com/bitcoin/bitcoin/issues> + +To receive security and update notifications, please subscribe to: + + <https://bitcoincore.org/en/list/announcements/join/> + +How to Upgrade +============== + +If you are running an older version, shut it down. Wait until it has completely +shut down (which might take a few minutes in some cases), then run the +installer (on Windows) or just copy over `/Applications/Bitcoin-Qt` (on macOS) +or `bitcoind`/`bitcoin-qt` (on Linux). + +Upgrading directly from a version of Bitcoin Core that has reached its EOL is +possible, but it might take some time if the data directory needs to be migrated. Old +wallet versions of Bitcoin Core are generally supported. + +Compatibility +============== + +Bitcoin Core is supported and extensively tested on operating systems +using the Linux kernel, macOS 10.15+, and Windows 7 and newer. Bitcoin +Core should also work on most other Unix-like systems but is not as +frequently tested on them. It is not recommended to use Bitcoin Core on +unsupported systems. + +Notable changes +=============== + +### Gui + +- gui#774 Fix crash on selecting "Mask values" in transaction view + +### RPC + +- #29003 rpc: fix getrawtransaction segfault + +### Wallet + +- #29176 wallet: Fix use-after-free in WalletBatch::EraseRecords +- #29510 wallet: `getrawchangeaddress` and `getnewaddress` failures should not affect keypools for descriptor wallets + +### P2P and network changes + +- #29412 p2p: Don't process mutated blocks +- #29524 p2p: Don't consider blocks mutated if they don't connect to known prev block + +Credits +======= + +Thanks to everyone who directly contributed to this release: + +- Martin Zumsande +- Sebastian Falbesoner +- MarcoFalke +- UdjinM6 +- dergoegge +- Greg Sanders + +As well as to everyone that helped with translations on +[Transifex](https://www.transifex.com/bitcoin/bitcoin/). diff --git a/doc/release-notes/release-notes-26.1.md b/doc/release-notes/release-notes-26.1.md new file mode 100644 index 0000000000..b5c6e63007 --- /dev/null +++ b/doc/release-notes/release-notes-26.1.md @@ -0,0 +1,106 @@ +26.1 Release Notes +================== + +Bitcoin Core version 26.1 is now available from: + + <https://bitcoincore.org/bin/bitcoin-core-26.1/> + +This release includes various bug fixes and performance +improvements, as well as updated translations. + +Please report bugs using the issue tracker at GitHub: + + <https://github.com/bitcoin/bitcoin/issues> + +To receive security and update notifications, please subscribe to: + + <https://bitcoincore.org/en/list/announcements/join/> + +How to Upgrade +============== + +If you are running an older version, shut it down. Wait until it has completely +shut down (which might take a few minutes in some cases), then run the +installer (on Windows) or just copy over `/Applications/Bitcoin-Qt` (on macOS) +or `bitcoind`/`bitcoin-qt` (on Linux). + +Upgrading directly from a version of Bitcoin Core that has reached its EOL is +possible, but it might take some time if the data directory needs to be migrated. Old +wallet versions of Bitcoin Core are generally supported. + +Compatibility +============== + +Bitcoin Core is supported and extensively tested on operating systems +using the Linux kernel, macOS 11.0+, and Windows 7 and newer. Bitcoin +Core should also work on most other Unix-like systems but is not as +frequently tested on them. It is not recommended to use Bitcoin Core on +unsupported systems. + +Notable changes +=============== + +### Wallet + +- #28994 wallet: skip BnB when SFFO is enabled +- #28920 wallet: birth time update during tx scanning +- #29176 wallet: Fix use-after-free in WalletBatch::EraseRecords +- #29510 wallet: getrawchangeaddress and getnewaddress failures should not affect keypools for descriptor wallets + +### RPC + +- #29003 rpc: fix getrawtransaction segfault +- #28784 rpc: keep .cookie file if it was not generated + +### Logs + +- #29227 log mempool loading progress + +### P2P and network changes + +- #29200 net: create I2P sessions using both ECIES-X25519 and ElGamal encryption +- #29412 p2p: Don't process mutated blocks +- #29524 p2p: Don't consider blocks mutated if they don't connect to known prev block + +### Build + +- #29127 Use hardened runtime on macOS release builds. +- #29195 build: Fix -Xclang -internal-isystem option + +### CI + +- #28992 ci: Use Ubuntu 24.04 Noble for asan,tsan,tidy,fuzz +- #29080 ci: Set HOMEBREW_NO_INSTALLED_DEPENDENTS_CHECK to avoid unrelated failures +- #29610 ci: Fix "macOS native" job + +### Miscellaneous + +- #28391 refactor: Simplify CTxMempool/BlockAssembler fields, remove some external mapTx access +- #29179 test: wallet rescan with reorged parent + IsFromMe child in mempool +- #28791 snapshots: don't core dump when running -checkblockindex after loadtxoutset +- #29357 test: Drop x modifier in fsbridge::fopen call for MinGW builds +- #29529 fuzz: restrict fopencookie usage to Linux & FreeBSD + +Credits +======= + +Thanks to everyone who directly contributed to this release: + +- dergoegge +- fanquake +- furszy +- glozow +- Greg Sanders +- Hennadii Stepanov +- Jon Atack +- MarcoFalke +- Mark Friedenbach +- Martin Zumsande +- Murch +- Roman Zeyde +- stickies-v +- UdjinM6 + +As well as to everyone that helped with translations on +[Transifex](https://www.transifex.com/bitcoin/bitcoin/). + diff --git a/doc/release-notes/release-notes-27.0.md b/doc/release-notes/release-notes-27.0.md new file mode 100644 index 0000000000..5060068328 --- /dev/null +++ b/doc/release-notes/release-notes-27.0.md @@ -0,0 +1,217 @@ +Bitcoin Core version 27.0 is now available from: + + <https://bitcoincore.org/bin/bitcoin-core-27.0/> + +This release includes new features, various bug fixes and performance +improvements, as well as updated translations. + +Please report bugs using the issue tracker at GitHub: + + <https://github.com/bitcoin/bitcoin/issues> + +To receive security and update notifications, please subscribe to: + + <https://bitcoincore.org/en/list/announcements/join/> + +How to Upgrade +============== + +If you are running an older version, shut it down. Wait until it has completely +shut down (which might take a few minutes in some cases), then run the +installer (on Windows) or just copy over `/Applications/Bitcoin-Qt` (on macOS) +or `bitcoind`/`bitcoin-qt` (on Linux). + +Upgrading directly from a version of Bitcoin Core that has reached its EOL is +possible, but it might take some time if the data directory needs to be migrated. Old +wallet versions of Bitcoin Core are generally supported. + +Compatibility +============== + +Bitcoin Core is supported and extensively tested on operating systems +using the Linux Kernel 3.17+, macOS 11.0+, and Windows 7 and newer. Bitcoin +Core should also work on most other Unix-like systems but is not as +frequently tested on them. It is not recommended to use Bitcoin Core on +unsupported systems. + +Notable changes +=============== + +libbitcoinconsensus +------------------- + +- libbitcoinconsensus is deprecated and will be removed for v28. This library has + existed for nearly 10 years with very little known uptake or impact. It has + become a maintenance burden. + + The underlying functionality does not change between versions, so any users of + the library can continue to use the final release indefinitely, with the + understanding that Taproot is its final consensus update. + + In the future, libbitcoinkernel will provide a much more useful API that is + aware of the UTXO set, and therefore be able to fully validate transactions and + blocks. (#29189) + +mempool.dat compatibility +------------------------- + +- The `mempool.dat` file created by -persistmempool or the savemempool RPC will + be written in a new format. This new format includes the XOR'ing of transaction + contents to mitigate issues where external programs (such as anti-virus) attempt + to interpret and potentially modify the file. + + This new format can not be read by previous software releases. To allow for a + downgrade, a temporary setting `-persistmempoolv1` has been added to fall back + to the legacy format. (#28207) + +P2P and network changes +----------------------- + +- BIP324 v2 transport is now enabled by default. It remains possible to disable v2 + by running with `-v2transport=0`. (#29347) +- Manual connection options (`-connect`, `-addnode` and `-seednode`) will + now follow `-v2transport` to connect with v2 by default. They will retry with + v1 on failure. (#29058) + +- Network-adjusted time has been removed from consensus code. It is replaced + with (unadjusted) system time. The warning for a large median time offset + (70 minutes or more) is kept. This removes the implicit security assumption of + requiring an honest majority of outbound peers, and increases the importance + of the node operator ensuring their system time is (and stays) correct to not + fall out of consensus with the network. (#28956) + +Mempool Policy Changes +---------------------- + +- Opt-in Topologically Restricted Until Confirmation (TRUC) Transactions policy + (aka v3 transaction policy) is available for use on test networks when + `-acceptnonstdtxn=1` is set. By setting the transaction version number to 3, TRUC transactions + request the application of limits on spending of their unconfirmed outputs. These + restrictions simplify the assessment of incentive compatibility of accepting or + replacing TRUC transactions, thus ensuring any replacements are more profitable for + the node and making fee-bumping more reliable. TRUC transactions are currently + nonstandard and can only be used on test networks where the standardness rules are + relaxed or disabled (e.g. with `-acceptnonstdtxn=1`). (#28948) + +External Signing +---------------- + +- Support for external signing on Windows has been disabled. It will be re-enabled + once the underlying dependency (Boost Process), has been replaced with a different + library. (#28967) + +Updated RPCs +------------ + +- The addnode RPC now follows the `-v2transport` option (now on by default, see above) for making connections. + It remains possible to specify the transport type manually with the v2transport argument of addnode. (#29239) + +Build System +------------ + +- A C++20 capable compiler is now required to build Bitcoin Core. (#28349) +- MacOS releases are configured to use the hardened runtime libraries (#29127) + +Wallet +------ + +- The CoinGrinder coin selection algorithm has been introduced to mitigate unnecessary + large input sets and lower transaction costs at high feerates. CoinGrinder + searches for the input set with minimal weight. Solutions found by + CoinGrinder will produce a change output. CoinGrinder is only active at + elevated feerates (default: 30+ sat/vB, based on `-consolidatefeerate`×3). (#27877) +- The Branch And Bound coin selection algorithm will be disabled when the subtract fee + from outputs feature is used. (#28994) +- If the birth time of a descriptor is detected to be later than the first transaction + involving that descriptor, the birth time will be reset to the earlier time. (#28920) + +Low-level changes +================= + +Pruning +------- + +- When pruning during initial block download, more blocks will be pruned at each + flush in order to speed up the syncing of such nodes. (#20827) + +Init +---- + +- Various fixes to prevent issues where subsequent instances of Bitcoin Core would + result in deletion of files in use by an existing instance. (#28784, #28946) +- Improved handling of empty `settings.json` files. (#29144) + +Credits +======= + +Thanks to everyone who directly contributed to this release: + +- 22388o⚡️ +- Aaron Clauson +- Amiti Uttarwar +- Andrew Toth +- Anthony Towns +- Antoine Poinsot +- Ava Chow +- Brandon Odiwuor +- brunoerg +- Chris Stewart +- Cory Fields +- dergoegge +- djschnei21 +- Fabian Jahr +- fanquake +- furszy +- Gloria Zhao +- Greg Sanders +- Hennadii Stepanov +- Hernan Marino +- iamcarlos94 +- ismaelsadeeq +- Jameson Lopp +- Jesse Barton +- John Moffett +- Jon Atack +- josibake +- jrakibi +- Justin Dhillon +- Kashif Smith +- kevkevin +- Kristaps Kaupe +- L0la L33tz +- Luke Dashjr +- Lőrinc +- marco +- MarcoFalke +- Mark Friedenbach +- Marnix +- Martin Leitner-Ankerl +- Martin Zumsande +- Max Edwards +- Murch +- muxator +- naiyoma +- Nikodemas Tuckus +- ns-xvrn +- pablomartin4btc +- Peter Todd +- Pieter Wuille +- Richard Myers +- Roman Zeyde +- Russell Yanofsky +- Ryan Ofsky +- Sebastian Falbesoner +- Sergi Delgado Segura +- Sjors Provoost +- stickies-v +- stratospher +- Supachai Kheawjuy +- TheCharlatan +- UdjinM6 +- Vasil Dimov +- w0xlt +- willcl-ark + + +As well as to everyone that helped with translations on +[Transifex](https://www.transifex.com/bitcoin/bitcoin/). diff --git a/doc/release-process.md b/doc/release-process.md index 95ef08a26f..1e6d49100e 100644 --- a/doc/release-process.md +++ b/doc/release-process.md @@ -5,17 +5,17 @@ Release Process ### Before every release candidate -* Update translations see [translation_process.md](https://github.com/bitcoin/bitcoin/blob/master/doc/translation_process.md#synchronising-translations). * Update release candidate version in `configure.ac` (`CLIENT_VERSION_RC`). * Update manpages (after rebuilding the binaries), see [gen-manpages.py](https://github.com/bitcoin/bitcoin/blob/master/contrib/devtools/README.md#gen-manpagespy). -* Update bitcoin.conf and commit, see [gen-bitcoin-conf.sh](https://github.com/bitcoin/bitcoin/blob/master/contrib/devtools/README.md#gen-bitcoin-confsh). +* Update bitcoin.conf and commit changes if they exist, see [gen-bitcoin-conf.sh](https://github.com/bitcoin/bitcoin/blob/master/contrib/devtools/README.md#gen-bitcoin-confsh). ### Before every major and minor release * Update [bips.md](bips.md) to account for changes since the last release. * Update version in `configure.ac` (don't forget to set `CLIENT_VERSION_RC` to `0`). * Update manpages (see previous section) -* Write release notes (see "Write the release notes" below). +* Write release notes (see "Write the release notes" below) in doc/release-notes.md. If necessary, + archive the previous release notes as doc/release-notes/release-notes-${VERSION}.md. ### Before every major release @@ -28,6 +28,7 @@ Release Process #### Before branch-off +* Update translations see [translation_process.md](https://github.com/bitcoin/bitcoin/blob/master/doc/translation_process.md#synchronising-translations). * Update hardcoded [seeds](/contrib/seeds/README.md), see [this pull request](https://github.com/bitcoin/bitcoin/pull/27488) for an example. * Update the following variables in [`src/kernel/chainparams.cpp`](/src/kernel/chainparams.cpp) for mainnet, testnet, and signet: - `m_assumed_blockchain_size` and `m_assumed_chain_state_size` with the current size plus some overhead (see @@ -161,6 +162,8 @@ Then open a Pull Request to the [guix.sigs repository](https://github.com/bitcoi ### macOS codesigner only: Create detached macOS signatures (assuming [signapple](https://github.com/achow101/signapple/) is installed and up to date with master branch) +In the `guix-build-${VERSION}/output/x86_64-apple-darwin` and `guix-build-${VERSION}/output/arm64-apple-darwin` directories: + tar xf bitcoin-osx-unsigned.tar.gz ./detached-sig-create.sh /path/to/codesign.p12 Enter the keychain password and authorize the signature @@ -168,6 +171,8 @@ Then open a Pull Request to the [guix.sigs repository](https://github.com/bitcoi ### Windows codesigner only: Create detached Windows signatures +In the `guix-build-${VERSION}/output/x86_64-w64-mingw32` directory: + tar xf bitcoin-win-unsigned.tar.gz ./detached-sig-create.sh -key /path/to/codesign.key Enter the passphrase for the key when prompted @@ -175,18 +180,22 @@ Then open a Pull Request to the [guix.sigs repository](https://github.com/bitcoi ### Windows and macOS codesigners only: test code signatures It is advised to test that the code signature attaches properly prior to tagging by performing the `guix-codesign` step. -However if this is done, once the release has been tagged in the bitcoin-detached-sigs repo, the `guix-codesign` step must be performed again in order for the guix attestation to be valid when compared against the attestations of non-codesigner builds. +However if this is done, once the release has been tagged in the bitcoin-detached-sigs repo, the `guix-codesign` step must be performed again in order for the guix attestation to be valid when compared against the attestations of non-codesigner builds. The directories created by `guix-codesign` will need to be cleared prior to running `guix-codesign` again. ### Windows and macOS codesigners only: Commit the detached codesign payloads ```sh pushd ./bitcoin-detached-sigs -# checkout the appropriate branch for this release series -rm -rf ./* +# checkout or create the appropriate branch for this release series +git checkout --orphan <branch> +# if you are the macOS codesigner +rm -rf osx tar xf signature-osx.tar.gz +# if you are the windows codesigner +rm -rf win tar xf signature-win.tar.gz git add -A -git commit -m "point to ${VERSION}" +git commit -m "<version>: {osx,win} signature for {rc,final}" git tag -s "v${VERSION}" HEAD git push the current branch and new tag popd @@ -216,45 +225,36 @@ popd Then open a Pull Request to the [guix.sigs repository](https://github.com/bitcoin-core/guix.sigs). -## After 3 or more people have guix-built and their results match +## After 6 or more people have guix-built and their results match -Combine the `all.SHA256SUMS.asc` file from all signers into `SHA256SUMS.asc`: +After verifying signatures, combine the `all.SHA256SUMS.asc` file from all signers into `SHA256SUMS.asc`: ```bash cat "$VERSION"/*/all.SHA256SUMS.asc > SHA256SUMS.asc ``` -- Upload to the bitcoincore.org server (`/var/www/bin/bitcoin-core-${VERSION}/`): - 1. The contents of each `./bitcoin/guix-build-${VERSION}/output/${HOST}/` directory, except for - `*-debug*` files. +- Upload to the bitcoincore.org server: + 1. The contents of each `./bitcoin/guix-build-${VERSION}/output/${HOST}/` directory. Guix will output all of the results into host subdirectories, but the SHA256SUMS file does not include these subdirectories. In order for downloads via torrent to verify without directory structure modification, all of the uploaded files need to be in the same directory as the SHA256SUMS file. - The `*-debug*` files generated by the guix build contain debug symbols - for troubleshooting by developers. It is assumed that anyone that is - interested in debugging can run guix to generate the files for - themselves. To avoid end-user confusion about which file to pick, as well - as save storage space *do not upload these to the bitcoincore.org server, - nor put them in the torrent*. - - ```sh - find guix-build-${VERSION}/output/ -maxdepth 2 -type f -not -name "SHA256SUMS.part" -and -not -name "*debug*" -exec scp {} user@bitcoincore.org:/var/www/bin/bitcoin-core-${VERSION} \; - ``` + Wait until all of these files have finished uploading before uploading the SHA256SUMS(.asc) files. 2. The `SHA256SUMS` file - 3. The `SHA256SUMS.asc` combined signature file you just created + 3. The `SHA256SUMS.asc` combined signature file you just created. -- Create a torrent of the `/var/www/bin/bitcoin-core-${VERSION}` directory such - that at the top level there is only one file: the `bitcoin-core-${VERSION}` - directory containing everything else. Name the torrent - `bitcoin-${VERSION}.torrent` (note that there is no `-core-` in this name). +- After uploading release candidate binaries, notify the bitcoin-core-dev mailing list and + bitcoin-dev group that a release candidate is available for testing. Include a link to the release + notes draft. - Optionally help seed this torrent. To get the `magnet:` URI use: +- The server will automatically create an OpenTimestamps file and torrent of the directory. + +- Optionally help seed this torrent. To get the `magnet:` URI use: ```sh transmission-show -m <torrent file> @@ -265,32 +265,33 @@ cat "$VERSION"/*/all.SHA256SUMS.asc > SHA256SUMS.asc Also put it into the `optional_magnetlink:` slot in the YAML file for bitcoincore.org. -- Update other repositories and websites for new version - - - bitcoincore.org blog post +- Archive the release notes for the new version to `doc/release-notes/release-notes-${VERSION}.md` + (branch `master` and branch of the release). - - bitcoincore.org maintained versions update: - [table](https://github.com/bitcoin-core/bitcoincore.org/commits/master/_includes/posts/maintenance-table.md) +- Update the bitcoincore.org website - - Delete post-EOL [release branches](https://github.com/bitcoin/bitcoin/branches/all) and create a tag `v${branch_name}-final`. + - blog post - - Delete ["Needs backport" labels](https://github.com/bitcoin/bitcoin/labels?q=backport) for non-existing branches. + - maintained versions [table](https://github.com/bitcoin-core/bitcoincore.org/commits/master/_includes/posts/maintenance-table.md) - - bitcoincore.org RPC documentation update + - RPC documentation update - See https://github.com/bitcoin-core/bitcoincore.org/blob/master/contrib/doc-gen/ + +- Update repositories + + - Delete post-EOL [release branches](https://github.com/bitcoin/bitcoin/branches/all) and create a tag `v${branch_name}-final`. + + - Delete ["Needs backport" labels](https://github.com/bitcoin/bitcoin/labels?q=backport) for non-existing branches. + - Update packaging repo - Push the flatpak to flathub, e.g. https://github.com/flathub/org.bitcoincore.bitcoin-qt/pull/2 - Push the snap, see https://github.com/bitcoin-core/packaging/blob/main/snap/local/build.md - - This repo - - - Archive the release notes for the new version to `doc/release-notes/` (branch `master` and branch of the release) - - - Create a [new GitHub release](https://github.com/bitcoin/bitcoin/releases/new) with a link to the archived release notes + - Create a [new GitHub release](https://github.com/bitcoin/bitcoin/releases/new) with a link to the archived release notes - Announce the release: diff --git a/doc/shared-libraries.md b/doc/shared-libraries.md deleted file mode 100644 index 07aee46f0b..0000000000 --- a/doc/shared-libraries.md +++ /dev/null @@ -1,77 +0,0 @@ -Shared Libraries -================ - -## bitcoinconsensus - -The purpose of this library is to make the verification functionality that is critical to Bitcoin's consensus available to other applications, e.g. to language bindings. - -### API - -The interface is defined in the C header `bitcoinconsensus.h` located in `src/script/bitcoinconsensus.h`. - -#### Version - -`bitcoinconsensus_version` returns an `unsigned int` with the API version *(currently `2`)*. - -#### Script Validation - -`bitcoinconsensus_verify_script`, `bitcoinconsensus_verify_script_with_amount` and `bitcoinconsensus_verify_script_with_spent_outputs` return an `int` with the status of the verification. It will be `1` if the input script correctly spends the previous output `scriptPubKey`. - -##### Parameters -###### bitcoinconsensus_verify_script -- `const unsigned char *scriptPubKey` - The previous output script that encumbers spending. -- `unsigned int scriptPubKeyLen` - The number of bytes for the `scriptPubKey`. -- `const unsigned char *txTo` - The transaction with the input that is spending the previous output. -- `unsigned int txToLen` - The number of bytes for the `txTo`. -- `unsigned int nIn` - The index of the input in `txTo` that spends the `scriptPubKey`. -- `unsigned int flags` - The script validation flags *(see below)*. -- `bitcoinconsensus_error* err` - Will have the error/success code for the operation *(see below)*. - -###### bitcoinconsensus_verify_script_with_amount -- `const unsigned char *scriptPubKey` - The previous output script that encumbers spending. -- `unsigned int scriptPubKeyLen` - The number of bytes for the `scriptPubKey`. -- `int64_t amount` - The amount spent in the input -- `const unsigned char *txTo` - The transaction with the input that is spending the previous output. -- `unsigned int txToLen` - The number of bytes for the `txTo`. -- `unsigned int nIn` - The index of the input in `txTo` that spends the `scriptPubKey`. -- `unsigned int flags` - The script validation flags *(see below)*. -- `bitcoinconsensus_error* err` - Will have the error/success code for the operation *(see below)*. - -###### bitcoinconsensus_verify_script_with_spent_outputs -- `const unsigned char *scriptPubKey` - The previous output script that encumbers spending. -- `unsigned int scriptPubKeyLen` - The number of bytes for the `scriptPubKey`. -- `int64_t amount` - The amount spent in the input -- `const unsigned char *txTo` - The transaction with the input that is spending the previous output. -- `unsigned int txToLen` - The number of bytes for the `txTo`. -- `UTXO *spentOutputs` - Previous outputs spent in the transaction. `UTXO` is a struct composed by `const unsigned char *scriptPubKey`, `unsigned int scriptPubKeySize` (the number of bytes for the `scriptPubKey`) and `unsigned int value`. -- `unsigned int spentOutputsLen` - The number of bytes for the `spentOutputs`. -- `unsigned int nIn` - The index of the input in `txTo` that spends the `scriptPubKey`. -- `unsigned int flags` - The script validation flags *(see below)*. -- `bitcoinconsensus_error* err` - Will have the error/success code for the operation *(see below)*. - -##### Script Flags -- `bitcoinconsensus_SCRIPT_FLAGS_VERIFY_NONE` -- `bitcoinconsensus_SCRIPT_FLAGS_VERIFY_P2SH` - Evaluate P2SH ([BIP16](https://github.com/bitcoin/bips/blob/master/bip-0016.mediawiki)) subscripts -- `bitcoinconsensus_SCRIPT_FLAGS_VERIFY_DERSIG` - Enforce strict DER ([BIP66](https://github.com/bitcoin/bips/blob/master/bip-0066.mediawiki)) compliance -- `bitcoinconsensus_SCRIPT_FLAGS_VERIFY_NULLDUMMY` - Enforce NULLDUMMY ([BIP147](https://github.com/bitcoin/bips/blob/master/bip-0147.mediawiki)) -- `bitcoinconsensus_SCRIPT_FLAGS_VERIFY_CHECKLOCKTIMEVERIFY` - Enable CHECKLOCKTIMEVERIFY ([BIP65](https://github.com/bitcoin/bips/blob/master/bip-0065.mediawiki)) -- `bitcoinconsensus_SCRIPT_FLAGS_VERIFY_CHECKSEQUENCEVERIFY` - Enable CHECKSEQUENCEVERIFY ([BIP112](https://github.com/bitcoin/bips/blob/master/bip-0112.mediawiki)) -- `bitcoinconsensus_SCRIPT_FLAGS_VERIFY_WITNESS` - Enable WITNESS ([BIP141](https://github.com/bitcoin/bips/blob/master/bip-0141.mediawiki)) -- `bitcoinconsensus_SCRIPT_FLAGS_VERIFY_TAPROOT` - Enable TAPROOT ([BIP340](https://github.com/bitcoin/bips/blob/master/bip-0340.mediawiki), [BIP341](https://github.com/bitcoin/bips/blob/master/bip-0341.mediawiki), [BIP342](https://github.com/bitcoin/bips/blob/master/bip-0342.mediawiki)) - -##### Errors -- `bitcoinconsensus_ERR_OK` - No errors with input parameters *(see the return value of `bitcoinconsensus_verify_script` for the verification status)* -- `bitcoinconsensus_ERR_TX_INDEX` - An invalid index for `txTo` -- `bitcoinconsensus_ERR_TX_SIZE_MISMATCH` - `txToLen` did not match with the size of `txTo` -- `bitcoinconsensus_ERR_DESERIALIZE` - An error deserializing `txTo` -- `bitcoinconsensus_ERR_AMOUNT_REQUIRED` - Input amount is required if WITNESS is used -- `bitcoinconsensus_ERR_INVALID_FLAGS` - Script verification `flags` are invalid (i.e. not part of the libconsensus interface) -- `bitcoinconsensus_ERR_SPENT_OUTPUTS_REQUIRED` - Spent outputs are required if TAPROOT is used -- `bitcoinconsensus_ERR_SPENT_OUTPUTS_MISMATCH` - Spent outputs size doesn't match tx inputs size - -### Example Implementations -- [NBitcoin](https://github.com/MetacoSA/NBitcoin/blob/5e1055cd7c4186dee4227c344af8892aea54faec/NBitcoin/Script.cs#L979-#L1031) (.NET Bindings) -- [node-libbitcoinconsensus](https://github.com/bitpay/node-libbitcoinconsensus) (Node.js Bindings) -- [java-libbitcoinconsensus](https://github.com/dexX7/java-libbitcoinconsensus) (Java Bindings) -- [bitcoinconsensus-php](https://github.com/Bit-Wasp/bitcoinconsensus-php) (PHP Bindings) -- [rust-bitcoinconsensus](https://github.com/rust-bitcoin/rust-bitcoinconsensus) (Rust Bindings)
\ No newline at end of file diff --git a/doc/translation_process.md b/doc/translation_process.md index 97a8fbfff2..e5ed7f4e0a 100644 --- a/doc/translation_process.md +++ b/doc/translation_process.md @@ -44,21 +44,7 @@ Visit the [Transifex Signup](https://www.transifex.com/signup/) page to create a You can find the Bitcoin translation project at [https://www.transifex.com/bitcoin/bitcoin/](https://www.transifex.com/bitcoin/bitcoin/). ### Installing the Transifex client command-line tool -The client is used to fetch updated translations. If you are having problems, or need more details, see [https://docs.transifex.com/client/installing-the-client](https://docs.transifex.com/client/installing-the-client) - -`pip install transifex-client` - -Setup your Transifex client config as follows. Please *ignore the token field*. - -```ini -nano ~/.transifexrc - -[https://www.transifex.com] -hostname = https://www.transifex.com -password = PASSWORD -token = -username = USERNAME -``` +The client is used to fetch updated translations. Please check installation instructions and any other details at https://developers.transifex.com/docs/cli. The Transifex Bitcoin project config file is included as part of the repo. It can be found at `.tx/config`, however you shouldn’t need to change anything. |