diff options
Diffstat (limited to 'doc')
| -rw-r--r-- | doc/JSON-RPC-interface.md | 79 | ||||
| -rw-r--r-- | doc/README.md | 5 | ||||
| -rw-r--r-- | doc/README_osx.md | 97 | ||||
| -rw-r--r-- | doc/REST-interface.md | 7 | ||||
| -rw-r--r-- | doc/build-osx.md | 100 | ||||
| -rw-r--r-- | doc/dependencies.md | 2 | ||||
| -rw-r--r-- | doc/developer-notes.md | 47 | ||||
| -rw-r--r-- | doc/fuzzing.md | 46 | ||||
| -rw-r--r-- | doc/init.md | 18 | ||||
| -rw-r--r-- | doc/release-notes-14060.md | 21 | ||||
| -rw-r--r-- | doc/release-notes-14477.md | 5 | ||||
| -rw-r--r-- | doc/release-notes-14565.md | 5 | ||||
| -rw-r--r-- | doc/release-notes-14941.md | 5 | ||||
| -rw-r--r-- | doc/release-notes-14982.md | 5 | ||||
| -rw-r--r-- | doc/release-notes-pr13381.md | 29 | ||||
| -rw-r--r-- | doc/release-notes.md | 122 | ||||
| -rw-r--r-- | doc/release-notes/release-notes-pr12255.md | 17 | ||||
| -rw-r--r-- | doc/release-process.md | 6 | ||||
| -rw-r--r-- | doc/translation_process.md | 20 |
19 files changed, 430 insertions, 206 deletions
diff --git a/doc/JSON-RPC-interface.md b/doc/JSON-RPC-interface.md index 59df541567..982afd5d56 100644 --- a/doc/JSON-RPC-interface.md +++ b/doc/JSON-RPC-interface.md @@ -5,6 +5,85 @@ The headless daemon `bitcoind` has the JSON-RPC API enabled by default, the GUI option. In the GUI it is possible to execute RPC methods in the Debug Console Dialog. +## Security + +The RPC interface allows other programs to control Bitcoin Core, +including the ability to spend funds from your wallets, affect consensus +verification, read private data, and otherwise perform operations that +can cause loss of money, data, or privacy. This section suggests how +you should use and configure Bitcoin Core to reduce the risk that its +RPC interface will be abused. + +- **Securing the executable:** Anyone with physical or remote access to + the computer, container, or virtual machine running Bitcoin Core can + compromise either the whole program or just the RPC interface. This + includes being able to record any passphrases you enter for unlocking + your encrypted wallets or changing settings so that your Bitcoin Core + program tells you that certain transactions have multiple + confirmations even when they aren't part of the best block chain. For + this reason, you should not use Bitcoin Core for security sensitive + operations on systems you do not exclusively control, such as shared + computers or virtual private servers. + +- **Securing local network access:** By default, the RPC interface can + only be accessed by a client running on the same computer and only + after the client provides a valid authentication credential (username + and passphrase). Any program on your computer with access to the file + system and local network can obtain this level of access. + Additionally, other programs on your computer can attempt to provide + an RPC interface on the same port as used by Bitcoin Core in order to + trick you into revealing your authentication credentials. For this + reason, it is important to only use Bitcoin Core for + security-sensitive operations on a computer whose other programs you + trust. + +- **Securing remote network access:** You may optionally allow other + computers to remotely control Bitcoin Core by setting the `rpcallowip` + and `rpcbind` configuration parameters. These settings are only meant + for enabling connections over secure private networks or connections + that have been otherwise secured (e.g. using a VPN or port forwarding + with SSH or stunnel). **Do not enable RPC connections over the public + Internet.** Although Bitcoin Core's RPC interface does use + authentication, it does not use encryption, so your login credentials + are sent as clear text that can be read by anyone on your network + path. Additionally, the RPC interface has not been hardened to + withstand arbitrary Internet traffic, so changing the above settings + to expose it to the Internet (even using something like a Tor hidden + service) could expose you to unconsidered vulnerabilities. See + `bitcoind -help` for more information about these settings and other + settings described in this document. + + Related, if you use Bitcoin Core inside a Docker container, you may + need to expose the RPC port to the host system. The default way to + do this in Docker also exposes the port to the public Internet. + Instead, expose it only on the host system's localhost, for example: + `-p 127.0.0.1:8332:8332` + +- **Secure authentication:** By default, Bitcoin Core generates unique + login credentials each time it restarts and puts them into a file + readable only by the user that started Bitcoin Core, allowing any of + that user's RPC clients with read access to the file to login + automatically. The file is `.cookie` in the Bitcoin Core + configuration directory, and using these credentials is the preferred + RPC authentication method. If you need to generate static login + credentials for your programs, you can use the script in the + `share/rpcauth` directory in the Bitcoin Core source tree. As a final + fallback, you can directly use manually-chosen `rpcuser` and + `rpcpassword` configuration parameters---but you must ensure that you + choose a strong and unique passphrase (and still don't use insecure + networks, as mentioned above). + +- **Secure string handling:** The RPC interface does not guarantee any + escaping of data beyond what's necessary to encode it as JSON, + although it does usually provide serialized data using a hex + representation of the bytes. If you use RPC data in your programs or + provide its data to other programs, you must ensure any problem + strings are properly escaped. For example, multiple websites have + been manipulated because they displayed decoded hex strings that + included HTML `<script>` tags. For this reason, and other + non-security reasons, it is recommended to display all serialized data + in hex form only. + ## RPC consistency guarantees State that can be queried via RPCs is guaranteed to be at least up-to-date with diff --git a/doc/README.md b/doc/README.md index 344b1be5c4..51950d4a13 100644 --- a/doc/README.md +++ b/doc/README.md @@ -5,7 +5,7 @@ Setup --------------------- Bitcoin Core is the original Bitcoin client and it builds the backbone of the network. It downloads and, by default, stores the entire history of Bitcoin transactions, which requires a few hundred gigabytes of disk space. Depending on the speed of your computer and network connection, the synchronization process can take anywhere from a few hours to a day or more. -To download Bitcoin Core, visit [bitcoincore.org](https://bitcoincore.org/en/releases/). +To download Bitcoin Core, visit [bitcoincore.org](https://bitcoincore.org/en/download/). Running --------------------- @@ -41,9 +41,10 @@ The following are developer notes on how to build Bitcoin Core on your native pl - [macOS Build Notes](build-osx.md) - [Unix Build Notes](build-unix.md) - [Windows Build Notes](build-windows.md) +- [FreeBSD Build Notes](build-freebsd.md) - [OpenBSD Build Notes](build-openbsd.md) - [NetBSD Build Notes](build-netbsd.md) -- [Gitian Building Guide](gitian-building.md) +- [Gitian Building Guide (External Link)](https://github.com/bitcoin-core/docs/blob/master/gitian-building.md) Development --------------------- diff --git a/doc/README_osx.md b/doc/README_osx.md deleted file mode 100644 index 739e22d634..0000000000 --- a/doc/README_osx.md +++ /dev/null @@ -1,97 +0,0 @@ -Deterministic macOS DMG Notes. - -Working macOS DMGs are created in Linux by combining a recent clang, -the Apple binutils (ld, ar, etc) and DMG authoring tools. - -Apple uses clang extensively for development and has upstreamed the necessary -functionality so that a vanilla clang can take advantage. It supports the use -of -F, -target, -mmacosx-version-min, and --sysroot, which are all necessary -when building for macOS. - -Apple's version of binutils (called cctools) contains lots of functionality -missing in the FSF's binutils. In addition to extra linker options for -frameworks and sysroots, several other tools are needed as well such as -install_name_tool, lipo, and nmedit. These do not build under linux, so they -have been patched to do so. The work here was used as a starting point: -[mingwandroid/toolchain4](https://github.com/mingwandroid/toolchain4). - -In order to build a working toolchain, the following source packages are needed -from Apple: cctools, dyld, and ld64. - -These tools inject timestamps by default, which produce non-deterministic -binaries. The ZERO_AR_DATE environment variable is used to disable that. - -This version of cctools has been patched to use the current version of clang's -headers and its libLTO.so rather than those from llvmgcc, as it was -originally done in toolchain4. - -To complicate things further, all builds must target an Apple SDK. These SDKs -are free to download, but not redistributable. -To obtain it, register for a developer account, then download the [Xcode 7.3.1 dmg](https://developer.apple.com/devcenter/download.action?path=/Developer_Tools/Xcode_7.3.1/Xcode_7.3.1.dmg). - -This file is several gigabytes in size, but only a single directory inside is -needed: -``` -Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk -``` - -Unfortunately, the usual linux tools (7zip, hpmount, loopback mount) are incapable of opening this file. -To create a tarball suitable for Gitian input, there are two options: - -Using macOS, you can mount the dmg, and then create it with: -``` - $ hdiutil attach Xcode_7.3.1.dmg - $ tar -C /Volumes/Xcode/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/ -czf MacOSX10.11.sdk.tar.gz MacOSX10.11.sdk -``` - -Alternatively, you can use 7zip and SleuthKit to extract the files one by one. -The script contrib/macdeploy/extract-osx-sdk.sh automates this. First ensure -the dmg file is in the current directory, and then run the script. You may wish -to delete the intermediate 5.hfs file and MacOSX10.11.sdk (the directory) when -you've confirmed the extraction succeeded. - -```bash -apt-get install p7zip-full sleuthkit -contrib/macdeploy/extract-osx-sdk.sh -rm -rf 5.hfs MacOSX10.11.sdk -``` - -The Gitian descriptors build 2 sets of files: Linux tools, then Apple binaries -which are created using these tools. The build process has been designed to -avoid including the SDK's files in Gitian's outputs. All interim tarballs are -fully deterministic and may be freely redistributed. - -genisoimage is used to create the initial DMG. It is not deterministic as-is, -so it has been patched. A system genisoimage will work fine, but it will not -be deterministic because the file-order will change between invocations. -The patch can be seen here: [theuni/osx-cross-depends](https://raw.githubusercontent.com/theuni/osx-cross-depends/master/patches/cdrtools/genisoimage.diff). -No effort was made to fix this cleanly, so it likely leaks memory badly. But -it's only used for a single invocation, so that's no real concern. - -genisoimage cannot compress DMGs, so afterwards, the 'dmg' tool from the -libdmg-hfsplus project is used to compress it. There are several bugs in this -tool and its maintainer has seemingly abandoned the project. It has been forked -and is available (with fixes) here: [theuni/libdmg-hfsplus](https://github.com/theuni/libdmg-hfsplus). - -The 'dmg' tool has the ability to create DMGs from scratch as well, but this -functionality is broken. Only the compression feature is currently used. -Ideally, the creation could be fixed and genisoimage would no longer be necessary. - -Background images and other features can be added to DMG files by inserting a -.DS_Store before creation. This is generated by the script -contrib/macdeploy/custom_dsstore.py. - -As of OS X 10.9 Mavericks, using an Apple-blessed key to sign binaries is a -requirement in order to satisfy the new Gatekeeper requirements. Because this -private key cannot be shared, we'll have to be a bit creative in order for the -build process to remain somewhat deterministic. Here's how it works: - -- Builders use Gitian to create an unsigned release. This outputs an unsigned - dmg which users may choose to bless and run. It also outputs an unsigned app - structure in the form of a tarball, which also contains all of the tools - that have been previously (deterministically) built in order to create a - final dmg. -- The Apple keyholder uses this unsigned app to create a detached signature, - using the script that is also included there. Detached signatures are available from this [repository](https://github.com/bitcoin-core/bitcoin-detached-sigs). -- Builders feed the unsigned app + detached signature back into Gitian. It - uses the pre-built tools to recombine the pieces into a deterministic dmg. diff --git a/doc/REST-interface.md b/doc/REST-interface.md index 44df698382..d21df36130 100644 --- a/doc/REST-interface.md +++ b/doc/REST-interface.md @@ -27,6 +27,7 @@ For full TX query capability, one must enable the transaction index via "txindex `GET /rest/block/notxdetails/<BLOCK-HASH>.<bin|hex|json>` Given a block hash: returns a block, in binary, hex-encoded binary or JSON formats. +Responds with 404 if the block doesn't exist. The HTTP request and response are both handled entirely in-memory, thus making maximum memory usage at least 2.66MB (1 MB max block, plus hex encoding) per request. @@ -36,6 +37,12 @@ With the /notxdetails/ option JSON response will only contain the transaction ha `GET /rest/headers/<COUNT>/<BLOCK-HASH>.<bin|hex|json>` Given a block hash: returns <COUNT> amount of blockheaders in upward direction. +Returns empty if the block doesn't exist or it isn't in the active chain. + +#### Blockhash by height +`GET /rest/blockhashbyheight/<HEIGHT>.<bin|hex|json>` + +Given a height: returns hash of block in best-block-chain at height provided. #### Chaininfos `GET /rest/chaininfo.json` diff --git a/doc/build-osx.md b/doc/build-osx.md index c9a59bab83..119896dc67 100644 --- a/doc/build-osx.md +++ b/doc/build-osx.md @@ -105,3 +105,103 @@ Notes * Tested on OS X 10.10 Yosemite through macOS 10.13 High Sierra on 64-bit Intel processors only. * Building with downloaded Qt binaries is not officially supported. See the notes in [#7714](https://github.com/bitcoin/bitcoin/issues/7714) + +Deterministic macOS DMG Notes +----------------------------- + +Working macOS DMGs are created in Linux by combining a recent clang, +the Apple binutils (ld, ar, etc) and DMG authoring tools. + +Apple uses clang extensively for development and has upstreamed the necessary +functionality so that a vanilla clang can take advantage. It supports the use +of -F, -target, -mmacosx-version-min, and --sysroot, which are all necessary +when building for macOS. + +Apple's version of binutils (called cctools) contains lots of functionality +missing in the FSF's binutils. In addition to extra linker options for +frameworks and sysroots, several other tools are needed as well such as +install_name_tool, lipo, and nmedit. These do not build under linux, so they +have been patched to do so. The work here was used as a starting point: +[mingwandroid/toolchain4](https://github.com/mingwandroid/toolchain4). + +In order to build a working toolchain, the following source packages are needed +from Apple: cctools, dyld, and ld64. + +These tools inject timestamps by default, which produce non-deterministic +binaries. The ZERO_AR_DATE environment variable is used to disable that. + +This version of cctools has been patched to use the current version of clang's +headers and its libLTO.so rather than those from llvmgcc, as it was +originally done in toolchain4. + +To complicate things further, all builds must target an Apple SDK. These SDKs +are free to download, but not redistributable. +To obtain it, register for a developer account, then download the [Xcode 7.3.1 dmg](https://developer.apple.com/devcenter/download.action?path=/Developer_Tools/Xcode_7.3.1/Xcode_7.3.1.dmg). + +This file is several gigabytes in size, but only a single directory inside is +needed: +``` +Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk +``` + +Unfortunately, the usual linux tools (7zip, hpmount, loopback mount) are incapable of opening this file. +To create a tarball suitable for Gitian input, there are two options: + +Using macOS, you can mount the dmg, and then create it with: +``` + $ hdiutil attach Xcode_7.3.1.dmg + $ tar -C /Volumes/Xcode/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/ -czf MacOSX10.11.sdk.tar.gz MacOSX10.11.sdk +``` + +Alternatively, you can use 7zip and SleuthKit to extract the files one by one. +The script contrib/macdeploy/extract-osx-sdk.sh automates this. First ensure +the dmg file is in the current directory, and then run the script. You may wish +to delete the intermediate 5.hfs file and MacOSX10.11.sdk (the directory) when +you've confirmed the extraction succeeded. + +```bash +apt-get install p7zip-full sleuthkit +contrib/macdeploy/extract-osx-sdk.sh +rm -rf 5.hfs MacOSX10.11.sdk +``` + +The Gitian descriptors build 2 sets of files: Linux tools, then Apple binaries +which are created using these tools. The build process has been designed to +avoid including the SDK's files in Gitian's outputs. All interim tarballs are +fully deterministic and may be freely redistributed. + +genisoimage is used to create the initial DMG. It is not deterministic as-is, +so it has been patched. A system genisoimage will work fine, but it will not +be deterministic because the file-order will change between invocations. +The patch can be seen here: [theuni/osx-cross-depends](https://raw.githubusercontent.com/theuni/osx-cross-depends/master/patches/cdrtools/genisoimage.diff). +No effort was made to fix this cleanly, so it likely leaks memory badly. But +it's only used for a single invocation, so that's no real concern. + +genisoimage cannot compress DMGs, so afterwards, the 'dmg' tool from the +libdmg-hfsplus project is used to compress it. There are several bugs in this +tool and its maintainer has seemingly abandoned the project. It has been forked +and is available (with fixes) here: [theuni/libdmg-hfsplus](https://github.com/theuni/libdmg-hfsplus). + +The 'dmg' tool has the ability to create DMGs from scratch as well, but this +functionality is broken. Only the compression feature is currently used. +Ideally, the creation could be fixed and genisoimage would no longer be necessary. + +Background images and other features can be added to DMG files by inserting a +.DS_Store before creation. This is generated by the script +contrib/macdeploy/custom_dsstore.py. + +As of OS X 10.9 Mavericks, using an Apple-blessed key to sign binaries is a +requirement in order to satisfy the new Gatekeeper requirements. Because this +private key cannot be shared, we'll have to be a bit creative in order for the +build process to remain somewhat deterministic. Here's how it works: + +- Builders use Gitian to create an unsigned release. This outputs an unsigned + dmg which users may choose to bless and run. It also outputs an unsigned app + structure in the form of a tarball, which also contains all of the tools + that have been previously (deterministically) built in order to create a + final dmg. +- The Apple keyholder uses this unsigned app to create a detached signature, + using the script that is also included there. Detached signatures are available from this [repository](https://github.com/bitcoin-core/bitcoin-detached-sigs). +- Builders feed the unsigned app + detached signature back into Gitian. It + uses the pre-built tools to recombine the pieces into a deterministic dmg. + diff --git a/doc/dependencies.md b/doc/dependencies.md index 50dde02fad..b833e9151f 100644 --- a/doc/dependencies.md +++ b/doc/dependencies.md @@ -26,5 +26,5 @@ These are the dependencies currently used by Bitcoin Core. You can find instruct | Qt | [5.9.7](https://download.qt.io/official_releases/qt/) | [5.2](https://github.com/bitcoin/bitcoin/pull/14725) | No | | | | XCB | | | | | [Yes](https://github.com/bitcoin/bitcoin/blob/master/depends/packages/qt.mk#L87) (Linux only) | | xkbcommon | | | | | [Yes](https://github.com/bitcoin/bitcoin/blob/master/depends/packages/qt.mk#L86) (Linux only) | -| ZeroMQ | [4.2.5](https://github.com/zeromq/libzmq/releases) | 4.0.0 | No | | | +| ZeroMQ | [4.3.1](https://github.com/zeromq/libzmq/releases) | 4.0.0 | No | | | | zlib | [1.2.11](https://zlib.net/) | | | | No | diff --git a/doc/developer-notes.md b/doc/developer-notes.md index 40861608a6..1deb5d791a 100644 --- a/doc/developer-notes.md +++ b/doc/developer-notes.md @@ -7,8 +7,8 @@ Developer Notes - [Developer Notes](#developer-notes) - [Coding Style (General)](#coding-style-general) - [Coding Style (C++)](#coding-style-c) - - [Doxygen comments](#doxygen-comments) - [Coding Style (Python)](#coding-style-python) + - [Coding Style (Doxygen-compatible comments)](#coding-style-doxygen-compatible-comments) - [Development tips and tricks](#development-tips-and-tricks) - [Compiling for debugging](#compiling-for-debugging) - [Compiling for gprof profiling](#compiling-for-gprof-profiling) @@ -36,6 +36,7 @@ Developer Notes - [Subtrees](#subtrees) - [Git and GitHub tips](#git-and-github-tips) - [Scripted diffs](#scripted-diffs) + - [Release notes](#release-notes) - [RPC interface guidelines](#rpc-interface-guidelines) <!-- markdown-toc end --> @@ -121,10 +122,17 @@ public: } // namespace foo ``` -Doxygen comments ------------------ +Coding Style (Python) +--------------------- + +Refer to [/test/functional/README.md#style-guidelines](/test/functional/README.md#style-guidelines). + +Coding Style (Doxygen-compatible comments) +------------------------------------------ -To facilitate the generation of documentation, use doxygen-compatible comment blocks for functions, methods and fields. +Bitcoin Core uses [Doxygen](http://www.doxygen.nl/) to generate its official documentation. + +Use Doxygen-compatible comment blocks for functions, methods, and fields. For example, to describe a function use: ```c++ @@ -157,7 +165,7 @@ int var; //!< Detailed description after the member ``` or -```cpp +```c++ //! Description before the member int var; ``` @@ -177,15 +185,15 @@ Not OK (used plenty in the current source, but not picked up): // ``` -A full list of comment syntaxes picked up by doxygen can be found at http://www.stack.nl/~dimitri/doxygen/manual/docblocks.html, -but if possible use one of the above styles. +A full list of comment syntaxes picked up by Doxygen can be found at https://www.stack.nl/~dimitri/doxygen/manual/docblocks.html, +but the above styles are favored. -Documentation can be generated with `make docs` and cleaned up with `make clean-docs`. +Documentation can be generated with `make docs` and cleaned up with `make clean-docs`. The resulting files are located in `doc/doxygen/html`; open `index.html` to view the homepage. -Coding Style (Python) ---------------------- - -Refer to [/test/functional/README.md#style-guidelines](/test/functional/README.md#style-guidelines). +Before running `make docs`, you will need to install dependencies `doxygen` and `dot`. For example, on MacOS via Homebrew: +``` +brew install doxygen --with-graphviz +``` Development tips and tricks --------------------------- @@ -913,6 +921,21 @@ test/lint/commit-script-check.sh origin/master..HEAD Commit [`bb81e173`](https://github.com/bitcoin/bitcoin/commit/bb81e173) is an example of a scripted-diff. +Release notes +------------- + +Release notes should be written for any PR that: + +- introduces a notable new feature +- fixes a significant bug +- changes an API or configuration model +- makes any other visible change to the end-user experience. + +Release notes should be added to a PR-specific release note file at +`/doc/release-notes-<PR number>.md` to avoid conflicts between multiple PRs. +All `release-notes*` files are merged into a single +[/doc/release-notes.md](/doc/release-notes.md) file prior to the release. + RPC interface guidelines -------------------------- diff --git a/doc/fuzzing.md b/doc/fuzzing.md index 5dedcb51c8..08b73d3b3c 100644 --- a/doc/fuzzing.md +++ b/doc/fuzzing.md @@ -1,12 +1,13 @@ Fuzz-testing Bitcoin Core ========================== -A special test harness `test_bitcoin_fuzzy` is provided to provide an easy -entry point for fuzzers and the like. In this document we'll describe how to -use it with AFL. +A special test harness in `src/test/fuzz/` is provided for each fuzz target to +provide an easy entry point for fuzzers and the like. In this document we'll +describe how to use it with AFL and libFuzzer. -Building AFL -------------- +## AFL + +### Building AFL It is recommended to always use the latest version of afl: ``` @@ -17,16 +18,15 @@ make export AFLPATH=$PWD ``` -Instrumentation ----------------- +### Instrumentation To build Bitcoin Core using AFL instrumentation (this assumes that the `AFLPATH` was set as above): ``` -./configure --disable-ccache --disable-shared --enable-tests CC=${AFLPATH}/afl-gcc CXX=${AFLPATH}/afl-g++ +./configure --disable-ccache --disable-shared --enable-tests --enable-fuzz CC=${AFLPATH}/afl-gcc CXX=${AFLPATH}/afl-g++ export AFL_HARDEN=1 cd src/ -make test/test_bitcoin_fuzzy +make ``` We disable ccache because we don't want to pollute the ccache with instrumented objects, and similarly don't want to use non-instrumented cached objects linked @@ -35,12 +35,11 @@ in. The fuzzing can be sped up significantly (~200x) by using `afl-clang-fast` and `afl-clang-fast++` in place of `afl-gcc` and `afl-g++` when compiling. When compiling using `afl-clang-fast`/`afl-clang-fast++` the resulting -`test_bitcoin_fuzzy` binary will be instrumented in such a way that the AFL +binary will be instrumented in such a way that the AFL features "persistent mode" and "deferred forkserver" can be used. See https://github.com/mcarpenter/afl/tree/master/llvm_mode for details. -Preparing fuzzing ------------------- +### Preparing fuzzing AFL needs an input directory with examples, and an output directory where it will place examples that it found. These can be anywhere in the file system, @@ -60,13 +59,30 @@ Example inputs are available from: Extract these (or other starting inputs) into the `inputs` directory before starting fuzzing. -Fuzzing --------- +### Fuzzing To start the actual fuzzing use: ``` -$AFLPATH/afl-fuzz -i ${AFLIN} -o ${AFLOUT} -m52 -- test/test_bitcoin_fuzzy +$AFLPATH/afl-fuzz -i ${AFLIN} -o ${AFLOUT} -m52 -- test/fuzz/fuzz_target_foo ``` You may have to change a few kernel parameters to test optimally - `afl-fuzz` will print an error and suggestion if so. + +## libFuzzer + +A recent version of `clang`, the address sanitizer and libFuzzer is needed (all +found in the `compiler-rt` runtime libraries package). + +To build the `test/test_bitcoin_fuzzy` executable run + +``` +./configure --disable-ccache --enable-fuzz --with-sanitizers=fuzzer,address CC=clang CXX=clang++ +make +``` + +The fuzzer needs some inputs to work on, but the inputs or seeds can be used +interchangeably between libFuzzer and AFL. + +See https://llvm.org/docs/LibFuzzer.html#running on how to run the libFuzzer +instrumented executable. diff --git a/doc/init.md b/doc/init.md index 5778b09d05..a6c9bb94d8 100644 --- a/doc/init.md +++ b/doc/init.md @@ -56,7 +56,7 @@ All three configurations assume several paths that might need to be adjusted. Binary: `/usr/bin/bitcoind` Configuration file: `/etc/bitcoin/bitcoin.conf` Data directory: `/var/lib/bitcoind` -PID file: `/var/run/bitcoind/bitcoind.pid` (OpenRC and Upstart) or `/var/lib/bitcoind/bitcoind.pid` (systemd) +PID file: `/var/run/bitcoind/bitcoind.pid` (OpenRC and Upstart) or `/run/bitcoind/bitcoind.pid` (systemd) Lock file: `/var/lock/subsys/bitcoind` (CentOS) The configuration file, PID directory (if applicable) and data directory @@ -65,6 +65,22 @@ reasons to make the configuration file and data directory only readable by the bitcoin user and group. Access to bitcoin-cli and other bitcoind rpc clients can then be controlled by group membership. +NOTE: When using the systemd .service file, the creation of the aforementioned +directories and the setting of their permissions is automatically handled by +systemd. Directories are given a permission of 710, giving the bitcoin group +access to files under it _if_ the files themselves give permission to the +bitcoin group to do so (e.g. when `-sysperms` is specified). This does not allow +for the listing of files under the directory. + +NOTE: It is not currently possible to override `datadir` in +`/etc/bitcoin/bitcoin.conf` with the current systemd, OpenRC, and Upstart init +files out-of-the-box. This is because the command line options specified in the +init files take precedence over the configurations in +`/etc/bitcoin/bitcoin.conf`. However, some init systems have their own +configuration mechanisms that would allow for overriding the command line +options specified in the init files (e.g. setting `BITCOIND_DATADIR` for +OpenRC). + ### macOS Binary: `/usr/local/bin/bitcoind` diff --git a/doc/release-notes-14060.md b/doc/release-notes-14060.md deleted file mode 100644 index 2cc5ab49fb..0000000000 --- a/doc/release-notes-14060.md +++ /dev/null @@ -1,21 +0,0 @@ -Configuration -------------- - -The outbound message high water mark of the ZMQ PUB sockets are now -configurable via the options: - -`-zmqpubhashtxhwm=n` - -`-zmqpubhashblockhwm=n` - -`-zmqpubrawblockhwm=n` - -`-zmqpubrawtxhwm=n` - -Each high water mark value must be an integer greater than or equal to 0. -The high water mark limits the maximum number of messages that ZMQ will -queue in memory for any single subscriber. A value of 0 means no limit. -When not specified, the default value continues to be 1000. -When a ZMQ PUB socket reaches its high water mark for a subscriber, then -additional messages to the subscriber are dropped until the number of -queued messages again falls below the high water mark value. diff --git a/doc/release-notes-14477.md b/doc/release-notes-14477.md deleted file mode 100644 index bb8c0a623e..0000000000 --- a/doc/release-notes-14477.md +++ /dev/null @@ -1,5 +0,0 @@ -Miscellaneous RPC changes ------------- - -- `getaddressinfo` now reports `solvable`, a boolean indicating whether all information necessary for signing is present in the wallet (ignoring private keys). -- `getaddressinfo`, `listunspent`, and `scantxoutset` have a new output field `desc`, an output descriptor that encapsulates all signing information and key paths for the address (only available when `solvable` is true for `getaddressinfo` and `listunspent`). diff --git a/doc/release-notes-14565.md b/doc/release-notes-14565.md deleted file mode 100644 index 38d76fee46..0000000000 --- a/doc/release-notes-14565.md +++ /dev/null @@ -1,5 +0,0 @@ -Low-level RPC changes ---------------------- - -The `importmulti` RPC will now contain a new per-request `warnings` field with strings -that explain when fields are being ignored or inconsistant, if any. diff --git a/doc/release-notes-14941.md b/doc/release-notes-14941.md new file mode 100644 index 0000000000..c3820d0368 --- /dev/null +++ b/doc/release-notes-14941.md @@ -0,0 +1,5 @@ +Miscellaneous RPC changes +------------ + +- The `unloadwallet` RPC is now synchronous, meaning that it blocks until the + wallet is fully unloaded. diff --git a/doc/release-notes-14982.md b/doc/release-notes-14982.md new file mode 100644 index 0000000000..3f0bf8aacd --- /dev/null +++ b/doc/release-notes-14982.md @@ -0,0 +1,5 @@ +New RPCs +-------- + +- The RPC `getrpcinfo` returns runtime details of the RPC server. At the moment + it returns the active commands and the corresponding execution time. diff --git a/doc/release-notes-pr13381.md b/doc/release-notes-pr13381.md deleted file mode 100644 index 75faad9906..0000000000 --- a/doc/release-notes-pr13381.md +++ /dev/null @@ -1,29 +0,0 @@ -RPC importprivkey: new label behavior -------------------------------------- - -Previously, `importprivkey` automatically added the default empty label -("") to all addresses associated with the imported private key. Now it -defaults to using any existing label for those addresses. For example: - -- Old behavior: you import a watch-only address with the label "cold - wallet". Later, you import the corresponding private key using the - default settings. The address's label is changed from "cold wallet" - to "". - -- New behavior: you import a watch-only address with the label "cold - wallet". Later, you import the corresponding private key using the - default settings. The address's label remains "cold wallet". - -In both the previous and current case, if you directly specify a label -during the import, that label will override whatever previous label the -addresses may have had. Also in both cases, if none of the addresses -previously had a label, they will still receive the default empty label -(""). Examples: - -- You import a watch-only address with the label "temporary". Later you - import the corresponding private key with the label "final". The - address's label will be changed to "final". - -- You use the default settings to import a private key for an address that - was not previously in the wallet. Its addresses will receive the default - empty label (""). diff --git a/doc/release-notes.md b/doc/release-notes.md index 6b4bb478e9..c23a7f6e0a 100644 --- a/doc/release-notes.md +++ b/doc/release-notes.md @@ -72,16 +72,52 @@ Mining - Calls to `getblocktemplate` will fail if the segwit rule is not specified. Calling `getblocktemplate` without segwit specified is almost certainly a misconfiguration since doing so results in lower rewards for the miner. - -Command line option changes ---------------------------- - -The `-enablebip61` command line option (introduced in Bitcoin Core 0.17.0) is -used to toggle sending of BIP 61 reject messages. Reject messages have no use -case on the P2P network and are only logged for debugging by most network -nodes. The option will now by default be off for improved privacy and security -as well as reduced upload usage. The option can explicitly be turned on for -local-network debugging purposes. + Failed calls will produce an error message describing how to enable the + segwit rule. + +Configuration option changes +---------------------------- + +- A warning is printed if an unrecognized section name is used in the + configuration file. Recognized sections are `[test]`, `[main]`, and + `[regtest]`. + +- Four new options are available for configuring the maximum number of + messages that ZMQ will queue in memory (the "high water mark") before + dropping additional messages. The default value is 1,000, the same as + was used for previous releases. See the [ZMQ + documentation](https://github.com/bitcoin/bitcoin/blob/master/doc/zmq.md#usage) + for details. + +- The `enablebip61` option (introduced in Bitcoin Core 0.17.0) is + used to toggle sending of BIP 61 reject messages. Reject messages have no use + case on the P2P network and are only logged for debugging by most network + nodes. The option will now by default be off for improved privacy and security + as well as reduced upload usage. The option can explicitly be turned on for + local-network debugging purposes. + +- The `rpcallowip` option can no longer be used to automatically listen + on all network interfaces. Instead, the `rpcbind` parameter must also + be used to specify the IP addresses to listen on. Listening for RPC + commands over a public network connection is insecure and should be + disabled, so a warning is now printed if a user selects such a + configuration. If you need to expose RPC in order to use a tool + like Docker, ensure you only bind RPC to your localhost, e.g. `docker + run [...] -p 127.0.0.1:8332:8332` (this is an extra `:8332` over the + normal Docker port specification). + +- The `rpcpassword` option now causes a startup error if the password + set in the configuration file contains a hash character (#), as it's + ambiguous whether the hash character is meant for the password or as a + comment. + +- The `whitelistforcerelay` option is used to relay transactions from + whitelisted peers even when not accepted to the mempool. This option now + defaults to being off, so that changes in policy and disconnect/ban behavior + will not cause a node that is whitelisting another to be dropped by peers. + Users can still explicitly enable this behavior with the command line option + (and may want to consider letting the Bitcoin Core project know about their + use-case, as this feature could be deprecated in the future). Documentation ------------- @@ -168,7 +204,7 @@ Updated RPCs Note: some low-level RPC changes mainly useful for testing are described in the Low-level Changes section below. -- The `getpeerinfo` RPC now returns an additional "minfeefilter" field +- The `getpeerinfo` RPC now returns an additional `minfeefilter` field set to the peer's BIP133 fee filter. You can use this to detect that you have peers that are willing to accept transactions below the default minimum relay fee. @@ -190,8 +226,62 @@ in the Low-level Changes section below. P2SH-P2WPKH, and P2SH-P2WSH. Requests for P2WSH and P2SH-P2WSH accept an additional `witnessscript` parameter. +- The `importmulti` RPC now returns an additional `warnings` field for + each request with an array of strings explaining when fields are being + ignored or are inconsistent, if there are any. + +- The `getaddressinfo` RPC now returns an additional `solvable` boolean + field when Bitcoin Core knows enough about the address's scriptPubKey, + optional redeemScript, and optional witnessScript in order for the + wallet to be able to generate an unsigned input spending funds sent to + that address. + +- The `getaddressinfo`, `listunspent`, and `scantxoutset` RPCs now + return an additional `desc` field that contains an output descriptor + containing all key paths and signing information for the address + (except for the private key). The `desc` field is only returned for + `getaddressinfo` and `listunspent` when the address is solvable. + +- The `importprivkey` RPC will preserve previously-set labels for + addresses or public keys corresponding to the private key being + imported. For example, if you imported a watch-only address with the + label "cold wallet" in earlier releases of Bitcoin Core, subsequently + importing the private key would default to resetting the address's + label to the default empty-string label (""). In this release, the + previous label of "cold wallet" will be retained. If you optionally + specify any label besides the default when calling `importprivkey`, + the new label will be applied to the address. + - See the [Mining](#mining) section for changes to `getblocktemplate`. +- The `getrawtransaction` RPC & REST endpoints no longer check the + unspent UTXO set for a transaction. The remaining behaviors are as + follows: 1. If a blockhash is provided, check the corresponding block. + 2. If no blockhash is provided, check the mempool. 3. If no blockhash + is provided but txindex is enabled, also check txindex. + +Graphical User Interface (GUI) +------------------------------ + +- A new Window menu is added alongside the existing File, Settings, and + Help menus. Several items from the other menus that opened new + windows have been moved to this new Window menu. + +- In the Send tab, the checkbox for "pay only the required fee" + has been removed. Instead, the user can simply decrease the value in + the Custom Feerate field all the way down to the node's configured + minimum relay fee. + +- In the Overview tab, the watch-only balance will be the only + balance shown if the wallet was created using the `createwallet` RPC + and the `disable_private_keys` parameter was set to true. + +- The launch-on-startup option is no longer available on macOS if + compiled with macosx min version greater than 10.11 (use + CXXFLAGS="-mmacosx-version-min=10.11" + CFLAGS="-mmacosx-version-min=10.11" for setting the deployment + sdk version) + Low-level changes ================= @@ -216,6 +306,16 @@ Configuration deterministic wallets. This release makes specifying `-usehd` an invalid configuration option. +Changes for particular platforms +-------------------------------- + +- On macOS, Bitcoin Core now opts out of application CPU throttling + ("app nap") during initial blockchain download, when catching up from + over 100 blocks behind the current chain tip, or when reindexing chain + data. This helps prevent these operations from taking an excessively + long time because the operating system is attempting to conserve + power. + Credits ======= diff --git a/doc/release-notes/release-notes-pr12255.md b/doc/release-notes/release-notes-pr12255.md new file mode 100644 index 0000000000..5ac8b44283 --- /dev/null +++ b/doc/release-notes/release-notes-pr12255.md @@ -0,0 +1,17 @@ +systemd init file +========= + +The systemd init file (`contrib/init/bitcoind.service`) has been changed to use +`/var/lib/bitcoind` as the data directory instead of `~bitcoin/.bitcoin`. This +change makes Bitcoin Core more consistent with other services, and makes the +systemd init config more consistent with existing Upstart and OpenRC configs. + +The configuration, PID, and data directories are now completely managed by +systemd, which will take care of their creation, permissions, etc. See +[`systemd.exec (5)`](https://www.freedesktop.org/software/systemd/man/systemd.exec.html#RuntimeDirectory=) +for more details. + +When using the provided init files under `contrib/init`, overriding the +`datadir` option in `/etc/bitcoin/bitcoin.conf` will have no effect. This is +because the command line arguments specified in the init files take precedence +over the options specified in `/etc/bitcoin/bitcoin.conf`. diff --git a/doc/release-process.md b/doc/release-process.md index 97fedb6e24..d20a3dc6b3 100644 --- a/doc/release-process.md +++ b/doc/release-process.md @@ -23,7 +23,7 @@ Before every minor and major release: Before every major release: * Update hardcoded [seeds](/contrib/seeds/README.md), see [this pull request](https://github.com/bitcoin/bitcoin/pull/7415) for an example. -* Update [`BLOCK_CHAIN_SIZE`](/src/qt/intro.cpp) to the current size plus some overhead. +* Update [`src/chainparams.cpp`](/src/chainparams.cpp) m_assumed_blockchain_size and m_assumed_chain_state_size with the current size plus some overhead. * Update `src/chainparams.cpp` chainTxData with statistics about the transaction count and rate. Use the output of the RPC `getchaintxstats`, see [this pull request](https://github.com/bitcoin/bitcoin/pull/12270) for an example. Reviewers can verify the results by running `getchaintxstats <window_block_count> <window_last_block_hash>` with the `window_block_count` and `window_last_block_hash` from your output. * Update version of `contrib/gitian-descriptors/*.yml`: usually one'd want to do this on master after branching off the release - but be sure to at least do it before a new major release @@ -87,10 +87,12 @@ Ensure gitian-builder is up-to-date: pushd ./gitian-builder mkdir -p inputs wget -P inputs https://bitcoincore.org/cfields/osslsigncode-Backports-to-1.7.1.patch + echo 'a8c4e9cafba922f89de0df1f2152e7be286aba73f78505169bc351a7938dd911 inputs/osslsigncode-Backports-to-1.7.1.patch' | sha256sum -c wget -P inputs https://downloads.sourceforge.net/project/osslsigncode/osslsigncode/osslsigncode-1.7.1.tar.gz + echo 'f9a8cdb38b9c309326764ebc937cba1523a3a751a7ab05df3ecc99d18ae466c9 inputs/osslsigncode-1.7.1.tar.gz' | sha256sum -c popd -Create the macOS SDK tarball, see the [macOS readme](README_osx.md) for details, and copy it into the inputs directory. +Create the macOS SDK tarball, see the [macOS build instructions](build-osx.md#deterministic-macos-dmg-notes) for details, and copy it into the inputs directory. ### Optional: Seed the Gitian sources cache and offline git repositories diff --git a/doc/translation_process.md b/doc/translation_process.md index 19f145e9bf..9692832842 100644 --- a/doc/translation_process.md +++ b/doc/translation_process.md @@ -68,11 +68,21 @@ The Transifex Bitcoin project config file is included as part of the repo. It ca To assist in updating translations, we have created a script to help. 1. `python contrib/devtools/update-translations.py` -2. Update `src/qt/bitcoin_locale.qrc` manually or via - `ls src/qt/locale/*ts|xargs -n1 basename|sed 's/\(bitcoin_\(.*\)\).ts/<file alias="\2">locale\/\1.qm<\/file>/'` -3. Update `src/Makefile.qt.include` manually or via - `ls src/qt/locale/*ts|xargs -n1 basename|sed 's/\(bitcoin_\(.*\)\).ts/ qt\/locale\/\1.ts \\/'` -4. `git add` new translations from `src/qt/locale/` +2. `git add` new translations from `src/qt/locale/` +3. Update `src/qt/bitcoin_locale.qrc` manually or via +```bash +git ls-files src/qt/locale/*ts|xargs -n1 basename|sed 's/\(bitcoin_\(.*\)\).ts/<file alias="\2">locale\/\1.qm<\/file>/' +``` +4. Update `src/Makefile.qt.include` manually or via +```bash +git ls-files src/qt/locale/*ts|xargs -n1 basename|sed 's/\(bitcoin_\(.*\)\).ts/ qt\/locale\/\1.ts \\/' +``` +5. Update `build_msvc/libbitcoin_qt/libbitcoin_qt.vcxproj` or via +```bash +git ls-files src/qt/locale/*ts|xargs -n1 basename | + sed 's/@/%40/' | + sed 's/\(bitcoin_\(.*\)\).ts/ <None Include="..\\..\\src\\qt\\locale\\\1.ts">\n <DeploymentContent>true<\/DeploymentContent>\n <\/None>/' +``` **Do not directly download translations** one by one from the Transifex website, as we do a few post-processing steps before committing the translations. |