aboutsummaryrefslogtreecommitdiff
path: root/test/lint
AgeCommit message (Collapse)Author
13 dayslint: remove unneeded trailing line fixwillcl-ark
13 dayslint: ignore files ignored by git in mlcwillcl-ark
Updating to MLC v0.18.0 includes a new feature which will ignore all files ignored by git: `--gitignore`. This helps avoid false-positives flagged by this linter in non-project files, such as a developer might expect to have in their directory (e.g. guix-builds, python venvs, etc.)
2024-06-04Support running individual lint checksDavid Gumberg
Add support for passing `--lint=LINT_TO_RUN` to the lint runner and add corresponding help message.
2024-05-24lint: add markdown hyperlink checkerwillcl-ark
This adds a markdown hyperlink check task to the lint test_runner. It relies on having the [`mlc`](https://crates.io/crates/mlc) binary found on $PATH, but will fail with `success` if the binary is not found. `mlc` is also added to the ci/04_install.sh script run by the containerfile. Note that broken markdown hyperlinks will be detected in untracked markdown files found in a dirty working directory (including e.g. .venv).
2024-05-08Merge bitcoin/bitcoin#30025: doc: fix broken relative md linksmerge-script
4b9f49da2b120e81516ddc3dc577d7a2e58e02d3 doc: fix broken relative md links (willcl-ark) Pull request description: These relative links in our documentation are broken, fix them. ACKs for top commit: maflcko: ACK 4b9f49da2b120e81516ddc3dc577d7a2e58e02d3 ryanofsky: Code review ACK 4b9f49da2b120e81516ddc3dc577d7a2e58e02d3. Thanks for the updates! ismaelsadeeq: Re ACK 4b9f49da2b120e81516ddc3dc577d7a2e58e02d3 Tree-SHA512: df4ef5ddece6c21125ce719ed6a4f69aba4f884c353ff7a8445ecb6438ed6bf0ff8268a1ae19cdd910adaadc189c6861c445b4d469f92ee81874d810dcbd0846
2024-05-07Merge bitcoin/bitcoin#29494: build: Assume HAVE_CONFIG_H, Add IWYU pragma ↵Ava Chow
keep to bitcoin-config.h includes fa09451f8e6799682d7e7c863f25334fd1c7dce3 Add lint check for bitcoin-config.h include IWYU pragma (MarcoFalke) dddd40ba8267dea11a3eb03d5cf8b51dbb99be5d scripted-diff: Add IWYU pragma keep to bitcoin-config.h includes (MarcoFalke) Pull request description: The `bitcoin-config.h` includes have issues: * The header is incompatible with iwyu, because symbols may be defined or not defined. So the `IWYU pragma: keep` is needed to keep the include when a symbol is not defined on a platform. Compare the previous discussion in https://github.com/bitcoin/bitcoin/pull/29408#issuecomment-1948959711 * Guarding the includes by `HAVE_CONFIG_H` is verbose and brittle. Now that all build config dependencies have been removed from low level headers, the benefits are questionable, and the guard can be removed. The linter could also be tricked by guarding the include by `#if defined(HAVE_C0NFIG_H)` (`O` replaced by `0`). Compare the previous discussion in https://github.com/bitcoin/bitcoin/pull/29404#discussion_r1483189853 . ACKs for top commit: achow101: ACK fa09451f8e6799682d7e7c863f25334fd1c7dce3 TheCharlatan: ACK fa09451f8e6799682d7e7c863f25334fd1c7dce3 hebasto: re-ACK fa09451f8e6799682d7e7c863f25334fd1c7dce3, only rebased since my recent [review](https://github.com/bitcoin/bitcoin/pull/29494#pullrequestreview-2028864535) (`timedata.cpp` removed in https://github.com/bitcoin/bitcoin/pull/29623). Tree-SHA512: 47cb973f7f24bc625acc4e78683371863675d186780236d55d886cf4130e05a78bb04f1d731aae7088313b8e963a9677cc77cf518187dbd99d776f6421ca9b52
2024-05-03doc: fix broken relative md linkswillcl-ark
These relative links in our documentation are broken, fix them.
2024-05-01Add lint check for bitcoin-config.h include IWYU pragmaMarcoFalke
Also, remove the no longer needed, remaining definitions and checks of HAVE_CONFIG_H.
2024-05-01lint: [doc] Clarify Windows line endings (CR LF) not to be usedMarcoFalke
2024-04-25Merge bitcoin/bitcoin#29689: lint: scripted-diff verification also requires ↵Ava Chow
GNU grep 3bf4f8db669e1e274ce2633cf84add2938b9914b lint: scripted-diff verification also requires GNU grep (Sjors Provoost) Pull request description: I noticed while trying to verify all historical `scripted-diff:` commits on macOS that some scripts require GNU sed. For example 0d6d2b650d1017691f48c9109a6cd020ab46aa73 uses `git grep --perl-regexp`. ACKs for top commit: hernanmarino: cr ACK 3bf4f8db669e1e274ce2633cf84add2938b9914b maflcko: utACK 3bf4f8db669e1e274ce2633cf84add2938b9914b achow101: ACK 3bf4f8db669e1e274ce2633cf84add2938b9914b alfonsoromanz: Tested ACK 3bf4f8db669e1e274ce2633cf84add2938b9914b kristapsk: cr utACK 3bf4f8db669e1e274ce2633cf84add2938b9914b Tree-SHA512: 09a060ab1bafad03df60d0f20c3dd1451850868dbd66ea38b18178b6230c1f06cf48622db82d9c51422d5689962ee0cd7aae0a31f84bd6d878215e6d73c1d47e
2024-04-11Fix typos in `subprocess.hpp`Hennadii Stepanov
2024-03-27external_signer: replace boost::process with cpp-subprocessSebastian Falbesoner
This primarily affects the `RunCommandParseJSON` utility function.
2024-03-27Add `cpp-subprocess` header-only libraryHennadii Stepanov
Upstream repo: https://github.com/arun11299/cpp-subprocess Commit: 4025693decacaceb9420efedbf4967a04cb028e7 The "Convenience Functions" section is unused in our codebase, so it has been removed.
2024-03-27Merge bitcoin/bitcoin#29479: test: Refactor subtree exclusion in lint testsfanquake
80fa7da21c470302165c47cc4a6a62fb44f997ef test: Refactor subtree exclusion in lint tests (Brandon Odiwuor) Pull request description: Fixes https://github.com/bitcoin/bitcoin/issues/17413 Refactor subtree exclusion in lint tests to one place Second attempt after PR: https://github.com/bitcoin/bitcoin/pull/24435 ACKs for top commit: fjahr: re-ACK 80fa7da21c470302165c47cc4a6a62fb44f997ef maflcko: lgtm ACK 80fa7da21c470302165c47cc4a6a62fb44f997ef davidgumberg: ACK https://github.com/bitcoin/bitcoin/commit/80fa7da21c470302165c47cc4a6a62fb44f997ef Tree-SHA512: deff7457dd19ca5ea440d3d53feae047e8863b9ddeb6494a3c94605a5d16edc91db8f99a435b4fab2ef89aedee42439562be006da647fb85bbf3def903a3ce50
2024-03-26test: Refactor subtree exclusion in lint testsBrandon Odiwuor
2024-03-21lint: Fix COMMIT_RANGE issuesMarcoFalke
2024-03-21lint: scripted-diff verification also requires GNU grepSjors Provoost
2024-03-17lint: Clarify lint runner rust dependencyFabian Jahr
2024-03-13lint: Use git --no-pager to print any output in one goMarcoFalke
2024-03-12lint: Fix lint-whitespace issuesMarcoFalke
2024-03-05docs: Fix broken reference to CI setup in test/lint/README.mdnaiyoma
2024-02-20lint: Check for missing or redundant bitcoin-config.h includesMarcoFalke
2024-02-20lint: Make lint error easier to spot in outputMarcoFalke
2024-02-20doc: Add missing RUST_BACKTRACE=1MarcoFalke
This will print a backtrace when an internal code error happened.
2024-02-20lint: Add get_subtrees() helperMarcoFalke
This is needed for a future change.
2024-01-16test: Remove all-lint.py scriptMarcoFalke
2024-01-16doc: move-only lint docs to one placeMarcoFalke
Can be reviewed with --color-moved=dimmed-zebra
2024-01-10Merge bitcoin/bitcoin#28318: logging: Simplify API for level based loggingAva Chow
e60fc7d5d34f23cccbff6e4f5f3d716fa8dad50c logging: Replace uses of LogPrintfCategory (Anthony Towns) f7ce5ac08c669ac763e275bb7c82dcfb2b1b6c33 logging: add LogError, LogWarning, LogInfo, LogDebug, LogTrace (Anthony Towns) fbd7642c8e5b70327e019382320f5ef0a651ecc5 logging: add -loglevelalways=1 option (Anthony Towns) 782bb6a05663ad7a53908e910d0f42b49b881e09 logging: treat BCLog::ALL like BCLog::NONE (Anthony Towns) 667ce3e3297645527b07314e1d5a82275fb25845 logging: Drop BCLog::Level::None (Anthony Towns) ab34dc6012351e7b8aab871dd9d2b38ade1cd9bc logging: Log Info messages unconditionally (Anthony Towns) dfe98b6874da04e45f68d17575c1e8a5431ca9bc logging: make [cat:debug] and [info] implicit (Anthony Towns) c5c76dc615677d226c9f6b3f2b66d833315d40da logging: refactor: pull prefix code out (Anthony Towns) Pull request description: Replace `LogPrint*` functions with severity based logging functions: * `LogInfo(...)`, `LogWarning(...)`, `LogError(...)` for unconditional (uncategorised) logging (replaces `LogPrintf`) * `LogDebug(CATEGORY, ...)` and `LogTrace(CATEGORY, ...)` for conditional logging (replaces `LogPrint`) * `LogPrintLevel(CATEGORY, LEVEL, ...)` for when the level isn't known in advance, or a category needs to be added for an info/warning/error log message (mostly unchanged, but rarely needed) Logs look roughly as they do now with `LogInfo` not having an `[info]` prefix, and `LogDebug` having a `[cat]` prefix, rather than a `[cat:debug]` prefix. This removes `BCLog::Level::None` entirely -- for `LogFlags::NONE` just use `Level::Info`, for any actual category, use `Level::Debug`. Adds docs to developer-notes about when to use which level. Adds `-loglevelalways=1` option so that you get `[net:debug]`, `[all:info]`, `[all:warning]` etc, which might be helpful for automated parsing, or just if you like everything to be consistent. Defaults to off to reduce noise in the default config, and to avoid unnecessary changes on upgrades. Changes the behaviour of `LogPrintLevel(CATEGORY, BCLog::Level::Info, ...)` to be logged unconditionally, rather than only being an additional optional logging level in addition to trace and debug. Does not change the behaviour of `LogPrintLevel(NONE, Debug, ...)` and `LogPrintLevel(NONE, Trace, ...)` being no-ops. ACKs for top commit: maflcko: re-ACK e60fc7d5d34f23cccbff6e4f5f3d716fa8dad50c 🌚 achow101: ACK e60fc7d5d34f23cccbff6e4f5f3d716fa8dad50c stickies-v: ACK e60fc7d5d34f23cccbff6e4f5f3d716fa8dad50c jamesob: ACK e60fc7d5d34f23cccbff6e4f5f3d716fa8dad50c ([`jamesob/ackr/28318.1.ajtowns.logging_simplify_api_for`](https://github.com/jamesob/bitcoin/tree/ackr/28318.1.ajtowns.logging_simplify_api_for)) Tree-SHA512: e7a4588779b148242495b7b6f64198a00c314cd57100affab11c43e9d39c9bbf85118ee2002792087fdcffdea08c84576e20844b3079f27083e26ddd7ca15d7f
2023-12-20logging: add LogError, LogWarning, LogInfo, LogDebug, LogTraceAnthony Towns
These provide simple and clear ways to write the most common logging operations: LogInfo("msg"); LogDebug(BCLog::LogFlags::NET, "msg"); LogError("msg"); LogWarning("msg"); LogTrace(BCLog::LogFlags::NET, "msg"); For cases where the level cannot be hardcoded, LogPrintLevel(category, level, ...) remains available.
2023-11-22Merge bitcoin/bitcoin#28862: lint: Report all lint errors instead of early exitfanquake
fa01f884d3ac128f09bfa57ac2648a19a19d854e ci: Add missing COPY for ./test/lint/test_runner (MarcoFalke) faff3e3b4604519375e122c103b156ec13eef80f lint: Report all lint errors instead of early exit (MarcoFalke) Pull request description: `all-lint.py` currently collects all failures. However, the `06_script.sh` does not, since July this year (https://github.com/bitcoin/bitcoin/pull/28103#discussion_r1268115806). Fix this by printing all failures before exiting. Can be tested by modifying (for example) two subtrees in the same commit and then running the linters. ACKs for top commit: kevkevinpal: ACK [fa01f88](https://github.com/bitcoin/bitcoin/pull/28862/commits/fa01f884d3ac128f09bfa57ac2648a19a19d854e) TheCharlatan: lgtm ACK fa01f884d3ac128f09bfa57ac2648a19a19d854e Tree-SHA512: c0f3110f2907d87e29c755e3b77a67dfae1f8a25833fe6ef8f2f2c58cfecf1aa46f1a20881576b62252b04930140a9e416c78b4edba0780d3c4fa7aaebabba81
2023-11-17Merge bitcoin/bitcoin#28725: test: refactor: use built-in collection types ↵fanquake
for type hints (Python 3.9 / PEP 585) a478c817b2f62b7334b36e331a2e37fe8380c754 test: replace `Callable`/`Iterable` with their `collections.abc` alternative (PEP 585) (stickies-v) 4b9afb18e6b9e16d7b299820f3a1382986a451d4 scripted-diff: use PEP 585 built-in collection types for verify-binary script (Sebastian Falbesoner) d516cf83ed2da86dfefb395cd46f8a894907b88e test: use built-in collection types for type hints (Python 3.9 / PEP 585) (Sebastian Falbesoner) Pull request description: With Python 3.9 / [PEP 585](https://peps.python.org/pep-0585/), [type hinting has become a little less awkward](https://docs.python.org/3.9/whatsnew/3.9.html#type-hinting-generics-in-standard-collections), as for collection types one doesn't need to import the corresponding capitalized types (`Dict`, `List`, `Set`, `Tuple`, ...) anymore, but can use the built-in types directly (see https://peps.python.org/pep-0585/#implementation for the full list). This PR applies the replacement for all Python scripts (i.e. in the contrib and test folders) for the basic types, i.e.: - typing.Dict -> dict - typing.List -> list - typing.Set -> set - typing.Tuple -> tuple For an additional check, I ran mypy 1.6.1 on both master and the PR branch via ``` $ mypy --ignore-missing-imports --explicit-package-bases $(git ls-files "*.py") ``` and verified that the output is identical -- (from the 22 identified problems, most look like false-positives, it's probably worth it to go deeper here and address them in a follow-up though). ACKs for top commit: stickies-v: ACK a478c817b2f62b7334b36e331a2e37fe8380c754 fanquake: ACK a478c817b2f62b7334b36e331a2e37fe8380c754 Tree-SHA512: 6948c905f6abd644d84f09fcb3661d7edb2742e8f2b28560008697d251d77a61a1146ab4b070e65b0d27acede7a5256703da7bf6eb1c7c3a897755478c76c6e8
2023-11-16Merge bitcoin/bitcoin#28605: Fix typosfanquake
43de4d3630274e1287179c86896ed4c2d8b9eff4 doc: fix typos (Sjors Provoost) Pull request description: This PR fixes typos found by lint-spelling.py using codespell 2.2.6. Our CI linter job uses codespell 2.2.5 and found fewer typos that I did locally. In any case it's happy now. ACKs for top commit: pablomartin4btc: re ACK 43de4d3630274e1287179c86896ed4c2d8b9eff4 Tree-SHA512: c032fe86cb49c924a468385653b31f309a9db68c478d70335bba3e65a1ff3826abe80284fe00a090ab5a509e1edbf17e476f6922fb15d055e50f1103dad2ccb0
2023-11-13lint: Report all lint errors instead of early exitMarcoFalke
2023-11-13Merge bitcoin/bitcoin#28076: util: Replace std::filesystem with util/fs.hfanquake
bbbbdb0cd57d75a06357d2811363d30a498f4499 ci: Add filesystem lint check (MarcoFalke) fada2f91108a56cc5c447bd6b6fac411e4d5cdca refactor: Replace <filesystem> with <util/fs.h> (MarcoFalke) Pull request description: Using `std::filesystem` is problematic: * There is a `fs` namespace wrapper for it. So having two ways to achieve the same is confusing. * Not using the `fs` wrapper is dangerous and buggy, because it disables known bugs by deleting problematic functions. Fix all issues by removing use of it and adding a linter to avoid using it again in the future. ACKs for top commit: TheCharlatan: ACK bbbbdb0cd57d75a06357d2811363d30a498f4499 fanquake: ACK bbbbdb0cd57d75a06357d2811363d30a498f4499 🦀 Tree-SHA512: 0e2d49742b08eb2635e6fce41485277cb9c40fe20b81017c391d3472a43787db1278a236825714ca1e41c9d2f59913865cfb0c649e3c8ab1fb598c849f80c660
2023-11-07doc: fix typosSjors Provoost
As found by lint-spelling.py using codespell 2.2.6.
2023-11-03init: completely remove `-zapwallettxes` (remaining hidden option)Sebastian Falbesoner
The `-zapwallettxes` functionality has been removed in v0.21.0 (see commit 3340dbadd38f5624642cf0e14dddbe6f83a3863b / PR #19671), with the parameter being kept as hidden option, to inform users via an exit error that `abandontransaction` should be used instead. As any guides that still suggest to use `-zapwallettxes` would refer to a Bitcoin Core version that is EOL since many years (i.e. <= v0.20.x), it is highly unlikely that the error caused by the option is still relevant for any user, hence it seems fine to remove it now.
2023-11-03Merge bitcoin/bitcoin#28762: MiniMiner changes for package linearizationAndrew Chow
d9cc99d04e813caed51c5f7b6ebdc4c39c8823c9 [test] MiniMiner::Linearize and manual construction (glozow) dfd6a3788c35be121eba1ad84f20effadcb7e7dc [refactor] unify fee amounts in miniminer_tests (glozow) f4b1b24a3bcd0199a6d2e828ad46033e1368336e [MiniMiner] track inclusion order and add Linearize() function (glozow) 004075963f12f17c3c399d81e3b48d6a8151aebd [test] add case for MiniMiner working with negative fee txns (glozow) fe6332c0badf07e99044b00084fc6b07f735a051 [MiniMiner] make target_feerate optional (glozow) 5a83f55c96661a886dd6f5231920b2f730cf6773 [MiniMiner] allow manual construction with non-mempool txns (glozow) e3b2e630b219ca15fe0b2640ca422712c86ac33d [refactor] change MiniMinerMempoolEntry ctor to take values, update includes (glozow) 4aa98b79b266cd526efa577762b0bcfccbdeda11 [lint] update expected boost includes (glozow) Pull request description: This is part of #27463. It splits off the `MiniMiner`-specific changes from #26711 for ease of review, as suggested in https://github.com/bitcoin/bitcoin/pull/26711#issuecomment-1786392253. - Allow using `MiniMiner` on transactions that aren't in the mempool. - Make `target_feerate` param of `BuildMockTemplate` optional, meaning "don't stop building the template until all the transactions have been selected." - Add clarification for how this is different from `target_feerate=0` (https://github.com/bitcoin/bitcoin/pull/26711#discussion_r1377019133) - Track the order in which transactions are included in the template to get the "linearization order" of the transactions. - Tests Reviewers can take a look at #26711 to see how these functions are used to linearize the `AncestorPackage` there. ACKs for top commit: TheCharlatan: ACK d9cc99d04e813caed51c5f7b6ebdc4c39c8823c9 kevkevinpal: reACK [d9cc99d](https://github.com/bitcoin/bitcoin/pull/28762/commits/d9cc99d04e813caed51c5f7b6ebdc4c39c8823c9) achow101: re-ACK d9cc99d04e813caed51c5f7b6ebdc4c39c8823c9 Tree-SHA512: 32b80064b6679536ac573d674825c5ca0cd6245e49c2fd5eaf260dc535335a57683c74ddd7ce1f249b5b12b2683de4362a7b0f1fc0814c3b3b9f14c682665583
2023-11-03[lint] update expected boost includesglozow
2023-11-01refactor: Remove unused circular include dependency from validation.cppMarcoFalke
2023-10-25Merge bitcoin-core/gui#771: Avoid error-prone leading whitespace in ↵Hennadii Stepanov
translatable strings 856325fac17465d102da621f1282b6d8ed02f679 lint: Add `lint-qt-translation.py` (Hennadii Stepanov) 294a018bf5106b03af39a2a8cfa4d5f2ebf6912b qt: Avoid error prone leading spaces in translatable strings (Hennadii Stepanov) d8298e7f069f961fc077ceacff2c332d58734688 qt, refactor: Drop superfluous type conversions (Hennadii Stepanov) Pull request description: While working on the GUI translation via Transifex web interface, I found it error-prone to have leading whitespace in translatable strings. This is because it is very easy to unintentionally drop them in translations unnoticed. Fixed all current cases. Added a linter to prevent similar cases in the future. ACKs for top commit: furszy: utACK 856325f Tree-SHA512: b1ca5effb2db6649e1e99382de79acf3a9f81cc9dad434db5623338489e597897e8addd60c1ab3dcc7506ae62753a7a4ad5a41d7a865f8fcdf94348b54baa7e7
2023-10-25test: use built-in collection types for type hints (Python 3.9 / PEP 585)Sebastian Falbesoner
Since Python 3.9, type hinting has become a little less awkward, as for collection types one doesn't need to import the corresponding capitalized types (`Dict`, `List`, `Set`, `Tuple`, ...) anymore, but can use the built-in types directly. [1] [2] This commit applies the replacement for all Python scripts (i.e. in the contrib and test folders) for the basic types: - typing.Dict -> dict - typing.List -> list - typing.Set -> set - typing.Tuple -> tuple [1] https://docs.python.org/3.9/whatsnew/3.9.html#type-hinting-generics-in-standard-collections [2] https://peps.python.org/pep-0585/#implementation for a list of type
2023-10-24Merge bitcoin/bitcoin#28211: Bump python minimum supported version to 3.9fanquake
fa25e8b0a1610553014c786428f146ef9c694678 doc: Recommend lint image build on every call (MarcoFalke) faf70c1f330a92612cf381d32c791e9ba445d3f2 Bump python minimum version to 3.9 (MarcoFalke) fa8996b930886da712c09ffe4b58016b36c2ae5b ci: Bump i686_multiprocess.sh to latest Ubuntu LTS (MarcoFalke) Pull request description: All supported operating systems ship with python 3.9 (or later), so bumping the minimum should not cause any issues. A bump will allow new code to use new python 3.9 features. For reference: * https://packages.debian.org/bullseye/python3 * https://packages.ubuntu.com/focal/python3.9 * FreeBSD 12/13 also ships with 3.9 * CentOS-like 8/9 also ships with 3.9 (and 3.11) * OpenSuse Leap also ships with 3.9 (and 3.11) https://software.opensuse.org/package/python311-base This is for Bitcoin Core 27.0 in 2024 (next year), not the soon upcoming 26.0 next month. ACKs for top commit: Sjors: ACK fa25e8b0a1610553014c786428f146ef9c694678 jamesob: ACK fa25e8b0a1610553014c786428f146ef9c694678 ([`jamesob/ackr/28211.1.MarcoFalke.bump_python_minimum_supp`](https://github.com/jamesob/bitcoin/tree/ackr/28211.1.MarcoFalke.bump_python_minimum_supp)) Tree-SHA512: 86c9f6ac4b5ba94a62ee6a6062dd48a8295d8611a39cdb5829f4f0dbc77aaa1a51edccc7a99275bf699143ad3a6fe826de426d413e5a465e3b0e82b86d10c32e
2023-10-23lint: Add `lint-qt-translation.py`Hennadii Stepanov
2023-10-13lint: Include test_utxo_snapshots in lint_shellFabian Jahr
2023-10-03Merge bitcoin/bitcoin#28304: doc: Remove confusing assert linterfanquake
fa6e6a3f03a38f8b431bf694268ed344d1815b3b doc: Remove confusing assert linter (MarcoFalke) Pull request description: The `assert()` documentation and linter are redundant and confusing: * The source code already refuses to compile with `assert()` disabled. * They violate the assumptions about `Assert()`, which *requires* side effects. * The existing linter doesn't enforce the guideline, only checking for `++` and `--` side effects. Fix all issues by removing the docs and the linter. See also https://github.com/bitcoin/bitcoin/pull/26684#discussion_r1287370102 Going forward everyone is free to use whatever code in this regard they think is the easiest to read. Also, everyone is still free to share style-nits, if they think it is a good use of their time and of the pull request author. Finally, the author is still free to dismiss or ignore this style-nit, or any other style-nit. ACKs for top commit: hebasto: ACK fa6e6a3f03a38f8b431bf694268ed344d1815b3b, I have reviewed the code and it looks OK. theStack: ACK fa6e6a3f03a38f8b431bf694268ed344d1815b3b Tree-SHA512: 686738d71e1316cc95e5d3f71869b55a02bfb137c795cc0875057f4410e564bc8eff03c985a2087b007fb08fc84551c7da1e8b30c7a9c3f2b14e5e44a5970236
2023-10-02Merge bitcoin/bitcoin#27596: assumeutxo (2)Andrew Chow
edbed31066e3674ba52b8c093ab235625527f383 chainparams: add signet assumeutxo param at height 160_000 (Sjors Provoost) b8cafe38713cbf10d15459042f7f911bcc1b1e4e chainparams: add testnet assumeutxo param at height 2_500_000 (Sjors Provoost) 99839bbfa7110c7abf22e587ae2f72c9c57d3c85 doc: add note about confusing HaveTxsDownloaded name (James O'Beirne) 7ee46a755f1d57ce9d51975d3b54dc9ac3d08d52 contrib: add script to demo/test assumeutxo (James O'Beirne) 42cae39356fd20d521aaf99aff1ed85856f3c9f3 test: add feature_assumeutxo functional test (James O'Beirne) 0f64bac6030334d798ae205cd7af4bf248feddd9 rpc: add getchainstates (James O'Beirne) bb0585779472962f40d9cdd9c6532132850d371c refuse to activate a UTXO snapshot if mempool not empty (James O'Beirne) ce585a9a158476b0ad3296477b922e79f308e795 rpc: add loadtxoutset (James O'Beirne) 62ac519e718eb7a31dca1102a96ba219fbc7f95d validation: do not activate snapshot if behind active chain (James O'Beirne) 9511fb3616b7bbe1d0d2f54a45ea0a650ba0367b validation: assumeutxo: swap m_mempool on snapshot activation (James O'Beirne) 7fcd21544a333ffdf1910b65c573579860be6a36 blockstorage: segment normal/assumedvalid blockfiles (James O'Beirne) 4c3b8ca35c2e4a441264749bb312df2bd054b5b8 validation: populate nChainTx value for assumedvalid chainstates (James O'Beirne) 49ef778158c43859946a592e11ec34fe1b93a5b6 test: adjust chainstate tests to use recognized snapshot base (James O'Beirne) 1019c399825b0d512c1fd751c376d46fed4992b9 validation: pruning for multiple chainstates (James O'Beirne) 373cf91531b84bfdd06fdf8abf4dca228029ce6b validation: indexing changes for assumeutxo (James O'Beirne) 1fffdd76a1bca908f55d73b64983655b14cf7432 net_processing: validationinterface: ignore some events for bg chain (James O'Beirne) fbe0a7d7ca680358237b6c2369b3fd2b43221113 wallet: validationinterface: only handle active chain notifications (James O'Beirne) f073917a9e7ba423643dcae0339776470b628f65 validationinterface: only send zmq notifications for active (James O'Beirne) 4d8f4dcb450d31e4847804e62bf91545b949fa14 validation: pass ChainstateRole for validationinterface calls (James O'Beirne) 1e59acdf17309f567c370885f0cf02605e2baa58 validation: only call UpdatedBlockTip for active chainstate (James O'Beirne) c6af23c5179cc383f8e6c275373af8d11e6a989f validation: add ChainstateRole (James O'Beirne) 9f2318c76cc6986d48e13831cf5bd8dab194fdf4 validation: MaybeRebalanceCaches when chain leaves IBD (James O'Beirne) 434495a8c1496ca23fe35b84499f3daf668d76b8 chainparams: add blockhash to AssumeutxoData (James O'Beirne) c711ca186f8d8a28810be0beedcb615ddcf93163 assumeutxo: remove snapshot during -reindex{-chainstate} (James O'Beirne) c93ef43e4fd4fbc1263cdc9e98ae5856830fe89e bugfix: correct is_snapshot_cs in VerifyDB (James O'Beirne) b73d3bbd23220857bf17cbb6401275bf58013b72 net_processing: Request assumeutxo background chain blocks (Suhas Daftuar) Pull request description: - Background and FAQ: https://github.com/jamesob/assumeutxo-docs/tree/2019-04-proposal/proposal - Prior progress/project: https://github.com/bitcoin/bitcoin/projects/11 - Replaces https://github.com/bitcoin/bitcoin/pull/15606, which was closed due to Github slowness. Original description and commentary can be found there. --- This changeset finishes the first phase of the assumeutxo project. It makes UTXO snapshots loadable via RPC (`loadtxoutset`) and adds `assumeutxo` parameters to chainparams. It contains all the remaining changes necessary to both use an assumedvalid snapshot chainstate and do a full validation sync in the background. This may look like a lot to review, but note that - ~200 lines are a (non-essential) demo shell script - Many lines are functional test, documentation, and relatively dilute RPC code. So it shouldn't be as burdensome to review as the linecount might suggest. - **P2P**: minor changes are made to `init.cpp` and `net_processing.cpp` to make simultaneous IBD across multiple chainstates work. - **Pruning**: implement correct pruning behavior when using a background chainstate - **Blockfile separation**: to prevent "fragmentation" in blockfile storage, have background chainstates use separate blockfiles from active snapshot chainstates to avoid interleaving heights and impairing pruning. - **Indexing**: some `CValidationInterface` events are given with an additional parameter, ChainstateRole, and all indexers ignore events from ChainstateRole::ASSUMEDVALID so that indexation only happens sequentially. - Have `-reindex` properly wipe snapshot chainstates. - **RPC**: introduce RPC commands `loadtxoutset` and (hidden) `getchainstates`. - **Release docs & first assumeutxo commitment**: add notes and a particular assumeutxo hash value for first AU-enabled release. - This will complete the project and allow use of UTXO snapshots for faster node bootstrap. The next phase, if it were to be pursued, would be coming up with a way to distribute the UTXO snapshots over the P2P network. --- ### UTXO snapshots Create your own with `./contrib/devtools/utxo_snapshot.sh`, e.g. ```shell ./contrib/devtools/utxo_snapshot.sh 788000 utxo.dat ./src/bitcoin-cli -datadir=$(pwd)/testdata`) ``` or use the pre-generated ones listed below. - Testnet: **2'500'000** (Sjors): - torrent: `magnet:?xt=urn:btih:511e09f4bf853aefab00de5c070b1e031f0ecbe9&dn=utxo-testnet-2500000.dat&tr=udp%3A%2F%2Ftracker.bitcoin.sprovoost.nl%3A6969` - sha256: `79db4b025448cc0ac388d8589a28eab02de53055d181e34eb47391717aa16388` - Signet: **160'000** (Sjors): - torrent: `magnet:?xt=urn:btih:9da986cb27b3980ea7fd06b21e199b148d486880&dn=utxo-signet-160000.dat&tr=udp%3A%2F%2Ftracker.bitcoin.sprovoost.nl%3A6969` - sha256: `eeeca845385ba91e84ef58c09d38f98f246a24feadaad57fe1e5874f3f92ef8c` - Mainnet: **800'000** (Sjors): - Note: this needs the following commit cherry-picked in: https://github.com/Sjors/bitcoin/commit/24deb2022b822f22fba9fcbee201e37a83225eb2 - torrent: `magnet:?xt=urn:btih:50ee955bef37f5ec3e5b0df4cf0288af3d715a2e&dn=utxo-800000.dat&tr=udp%3A%2F%2Ftracker.bitcoin.sprovoost.nl%3A6969` ### Testing #### For fun (~5min) If you want to do a quick test, you can run `./contrib/devtools/test_utxo_snapshots.sh` and follow the instructions. This is mostly obviated by the functional tests, though. #### For real (longer) If you'd like to experience a real usage of assumeutxo, you can do that too. I've cut a new snapshot at height 788'000 (http://img.jameso.be/utxo-788000.dat - but you can do it yourself with `./contrib/devtools/utxo_snapshot.sh` if you want). Download that, and then create a datadir for testing: ```sh $ cd ~/src/bitcoin # or whatever # get the snapshot $ curl http://img.jameso.be/utxo-788000.dat > utxo-788000.dat # you'll want to do this if you like copy/pasting $ export AU_DATADIR=/home/${USER}/au-test # or wherever $ mkdir ${AU_DATADIR} $ vim ${AU_DATADIR}/bitcoin.conf dbcache=8000 # or, you know, something high blockfilterindex=1 coinstatsindex=1 prune=3000 logthreadnames=1 ``` Obtain this branch, build it, and then start bitcoind: ```sh $ git remote add jamesob https://github.com/jamesob/bitcoin $ git fetch jamesob assumeutxo $ git checkout jamesob/assumeutxo $ ./configure $conf_args && make # (whatever you like to do here) # start 'er up and watch the logs $ ./src/bitcoind -datadir=${AU_DATADIR} ``` Then, in some other window, load the snapshot ```sh $ ./src/bitcoin-cli -datadir=${AU_DATADIR} loadtxoutset $(pwd)/utxo-788000.dat ``` You'll see some log messages about headers retrieval and waiting to see the snapshot in the headers chain. Once you get the full headers chain, you'll spend a decent amount of time (~10min) loading the snapshot, checking it, and flushing it to disk. After all that happens, you should be syncing to tip in pretty short order, and you'll see the occasional `[background validation]` log message go by. In yet another window, you can check out chainstate status with ```sh $ ./src/bitcoin-cli -datadir=${AU_DATADIR} getchainstates ``` as well as usual favorites like `getblockchaininfo`. ACKs for top commit: achow101: ACK edbed31066e3674ba52b8c093ab235625527f383 Tree-SHA512: 6086fb9a38dc7df85fedc76b30084dd8154617a2a91e89a84fb41326d34ef8e7d7ea593107afba01369093bf8cc91770621d98f0ea42a5b3b99db868d2f14dc2
2023-09-30contrib: add script to demo/test assumeutxoJames O'Beirne
Add the script to the shellcheck exception list since the quoted variables rule needs to be violated in order to get bitcoind to pick up on $CHAIN_HACK_FLAGS.
2023-09-28lint: fix custom mypy cache dir settingFabian Jahr
2023-09-14doc: Remove confusing assert linterMarcoFalke
2023-09-14ci: Add filesystem lint checkMarcoFalke
generated by cgit v1.2.3 (git 2.26.2) at 2024-07-16 14:39:13 +0000