aboutsummaryrefslogtreecommitdiff
path: root/src/validation.cpp
AgeCommit message (Collapse)Author
2023-08-07Merge bitcoin/bitcoin#28186: kernel: Prune leveldb headersfanquake
d8f1222ac50f089a0af29eaf8ce0555bad8366ef refactor: Correct dbwrapper key naming (TheCharlatan) be8f159ac59b9e700cbd3314ed71ebf39bd5b67a build: Remove leveldb from BITCOIN_INCLUDES (TheCharlatan) c95b37d641b1eed4a62d55ca5342a6ed8c7a1ce7 refactor: Move CDBWrapper leveldb members to their own context struct (TheCharlatan) c534a615e93452a5f509aaf5f68c600391a98d6a refactor: Split dbwrapper CDBWrapper::EstimateSize implementation (TheCharlatan) 586448888b72f7c87db4dcd30fc4e4044afae13b refactor: Move HandleError to dbwrapper implementation (TheCharlatan) dede0eef7adb7413f62f5abd68cac8e01635ba4a refactor: Split dbwrapper CDBWrapper::Exists implementation (TheCharlatan) a5c2eb57484314b04ec94523d14e0ef0c6c46d4f refactor: Fix logging.h includes (TheCharlatan) 84058e0eed9c05bc30984b39131e88ad1425628f refactor: Split dbwrapper CDBWrapper::Read implementation (TheCharlatan) e4af2408f2ac59788567b6fc8cb3a68fc43da9fe refactor: Pimpl leveldb::Iterator for CDBIterator (TheCharlatan) ef941ff1281e76308c3e746e592375bec023e9e4 refactor: Split dbwrapper CDBIterator::GetValue implementation (TheCharlatan) b7a1ab5cb4e60230f62c94efb3a10d07c9af4883 refactor: Split dbwrapper CDBIterator::GetKey implementation (TheCharlatan) d7437908cdf242626263ba9d5541addcddadc594 refactor: Split dbwrapper CDBIterator::Seek implementation (TheCharlatan) ea8135de7e617259cda3fc7b1c8e7569d454fd57 refactor: Pimpl leveldb::batch for CDBBatch (TheCharlatan) b9870c920dc475ec759eaf7339ea42aecba92138 refactor: Split dbwrapper CDBatch::Erase implementation (TheCharlatan) 532ee812a499e13b123af6b8415d8de1f3804f0f refactor: Split dbwrapper CDBBatch::Write implementation (TheCharlatan) afc534df9adbf5599b286b5dc3531a4b9ac2d056 refactor: Wrap DestroyDB in dbwrapper helper (TheCharlatan) Pull request description: Leveldb headers are currently included in the `dbwrapper.h` file and thus available to many of Bitcoin Core's source files. However, leveldb-specific functionality should be abstracted by the `dbwrapper` and does not need to be available to the rest of the code. Having leveldb included in a widely-used header such as `dbwrapper.h` bloats the entire project's header tree. The `dbwrapper` is a key component of the libbitcoinkernel library. Future users of this library would not want to contend with having the leveldb headers exposed and potentially polluting their project's namespace. For these reasons, the leveldb headers are removed from the `dbwrapper` by moving leveldb-specific code to the implementation file and creating a [pimpl](https://en.cppreference.com/w/cpp/language/pimpl) where leveldb member variables are indispensable. As a final step, the leveldb include flags are removed from the `BITCOIN_INCLUDES` and moved to places where the dbwrapper is compiled. --- This pull request is part of the [libbitcoinkernel project](https://github.com/bitcoin/bitcoin/issues/27587), and more specifically its stage 1 step 3 "Decouple most non-consensus headers from libbitcoinkernel". ACKs for top commit: stickies-v: re-ACK https://github.com/bitcoin/bitcoin/commit/d8f1222ac50f089a0af29eaf8ce0555bad8366ef MarcoFalke: ACK d8f1222ac50f089a0af29eaf8ce0555bad8366ef 🔠 Tree-SHA512: 0f58309be165af0162e648233451cd80fda88726fc10c0da7bfe4ec2ffa9afe63fbf7ffae9493698d3f39653b4ad870c372eee652ecc90ab1c29d86c387070f3
2023-08-03refactor: fix unterminated LogPrintf()sfanquake
2023-08-03lint: remove /* Continued */ markers from codebasefanquake
2023-08-01refactor: Wrap DestroyDB in dbwrapper helperTheCharlatan
Wrap leveldb::DestroyDB in a helper function without exposing leveldb-specifics. Also, add missing optional include. The context of this commit is an effort to decouple the dbwrapper header file from leveldb includes. To this end, the includes are moved to the dbwrapper implementation file. This is done as part of the kernel project to reduce the number of required includes for users of the kernel.
2023-07-31Merge bitcoin/bitcoin#27746: Rework validation logic for assumeutxoRyan Ofsky
a733dd79e29068ad1e0532ac42a45188a040a7b9 Remove unused function `reliesOnAssumedValid` (Suhas Daftuar) d4a11abb1972b54f0babdccfbb2fde97ab885933 Cache block index entry corresponding to assumeutxo snapshot base blockhash (Suhas Daftuar) 3556b850221bc0e597d7dd749d4d47ab58dc8083 Move CheckBlockIndex() from Chainstate to ChainstateManager (Suhas Daftuar) 0ce805b632dcb98944a931f758f76f530f5ce5f2 Documentation improvements for assumeutxo (Ryan Ofsky) 768690b7ce551cd403f8e2a099372915f6022ad4 Fix initialization of setBlockIndexCandidates when working with multiple chainstates (Suhas Daftuar) d43a1f1a2fa35d377c7a9ad7ab92d1ae325bde3d Tighten requirements for adding elements to setBlockIndexCandidates (Suhas Daftuar) d0d40ea9a6478d81d7531b7cfc52a8bdaa0883d6 Move block-storage-related logic to ChainstateManager (Suhas Daftuar) 3cfc75366e6596942cbc84f354f42dfd7fc5c073 test: Clear block index flags when testing snapshots (Suhas Daftuar) 272fbc370c4e133d31d9f1d34e327cc265c5fad2 Update CheckBlockIndex invariants for chains based on an assumeutxo snapshot (Suhas Daftuar) 10c05710ce1602d932037f72dc6c4bbc3f6f34ba Add wrapper for adding entries to a chainstate's block index candidates (Suhas Daftuar) 471da5f6e74bac71aeffe2ebc5faff145a6cbcea Move block-arrival information / preciousblock counters to ChainstateManager (Suhas Daftuar) 1cfc887d00c5d1d4281107e3b3ff4641c6c34631 Remove CChain dependency in node/blockstorage (Suhas Daftuar) fe86a7cd480b32463da900db764d2d11a2bea095 Explicitly track maximum block height stored in undo files (Suhas Daftuar) Pull request description: This PR proposes a clean up of the relationship between block storage and the chainstate objects, by moving the decision of whether to store a block on disk to something that is not chainstate-specific. Philosophically, the decision of whether to store a block on disk is related to validation rules that do not require any UTXO state; for anti-DoS reasons we were using some chainstate-specific heuristics, and those have been reworked here to achieve the proposed separation. This PR also fixes a bug in how a chainstate's `setBlockIndexCandidates` was being initialized; it should always have all the HAVE_DATA block index entries that have more work than the chain tip. During startup, we were not fully populating `setBlockIndexCandidates` in some scenarios involving multiple chainstates. Further, this PR establishes a concept that whenever we have 2 chainstates, that we always know the snapshotted chain's base block and the base block's hash must be an element of our block index. Given that, we can establish a new invariant that the background validation chainstate only needs to consider blocks leading to that snapshotted block entry as potential candidates for its tip. As a followup I would imagine that when writing net_processing logic to download blocks for the background chainstate, that we would use this concept to only download blocks towards the snapshotted entry as well. ACKs for top commit: achow101: ACK a733dd79e29068ad1e0532ac42a45188a040a7b9 jamesob: reACK a733dd79e29068ad1e0532ac42a45188a040a7b9 ([`jamesob/ackr/27746.5.sdaftuar.rework_validation_logic`](https://github.com/jamesob/bitcoin/tree/ackr/27746.5.sdaftuar.rework_validation_logic)) Sjors: Code review ACK a733dd79e29068ad1e0532ac42a45188a040a7b9. ryanofsky: Code review ACK a733dd79e29068ad1e0532ac42a45188a040a7b9. Just suggested changes since the last review. There are various small things that could be followed up on, but I think this is ready for merge. Tree-SHA512: 9ec17746f22b9c27082743ee581b8adceb2bd322fceafa507b428bdcc3ffb8b4c6601fc61cc7bb1161f890c3d38503e8b49474da7b5ab1b1f38bda7aa8668675
2023-07-24Cache block index entry corresponding to assumeutxo snapshot base blockhashSuhas Daftuar
This is to (a) avoid repeated lookups into the block index for an entry that should never change and (b) emphasize that the snapshot base should always exist when set and not change during the runtime of the program. Thanks to Russ Yanofsky for suggesting this approach.
2023-07-24Move CheckBlockIndex() from Chainstate to ChainstateManagerSuhas Daftuar
Also rewrite CheckBlockIndex() to perform tests on all chainstates. This increases sanity-check coverage, as any place in our code where we were invoke CheckBlockIndex() on a single chainstate will now invoke the sanity checks on all chainstates. This change also tightens up the checks on setBlockIndexCandidates and mapBlocksUnlinked, to more precisely match what we aim for even in the presence of assumed-valid blocks.
2023-07-24Fix initialization of setBlockIndexCandidates when working with multiple ↵Suhas Daftuar
chainstates When using assumeutxo and multiple chainstates are active, the background chainstate should consider all HAVE_DATA blocks that are ancestors of the snapshotted block and that have more work than the tip as potential candidates.
2023-07-24Tighten requirements for adding elements to setBlockIndexCandidatesSuhas Daftuar
When using assumeutxo, we only need the background chainstate to consider blocks that are on the chain leading to the snapshotted block. Note that this introduces the new invariant that we can only have an assumeutxo snapshot where the snapshotted blockhash is in our block index. Unknown block hashes that are somehow passed in will cause assertion failures when processing new blocks. Includes test fixes and improvements by Andrew Chow and Fabian Jahr.
2023-07-21Move block-storage-related logic to ChainstateManagerSuhas Daftuar
Separate the notion of which blocks are stored on disk, and what data is in our block index, from what tip a chainstate might be able to get to. We can use chainstate-agnostic data to determine when to store a block on disk (primarily, an anti-DoS set of criteria) and let the chainstates figure out for themselves when a block is of interest for being a candidate tip. Note: some of the invariants in CheckBlockIndex are modified, but more work is needed (ie to move CheckBlockIndex to ChainstateManager, as most of what CheckBlockIndex is doing is checking the consistency of the block index, which is outside of Chainstate).
2023-07-17validation: use noexcept instead of deprecated throw()fanquake
```bash CXX libbitcoin_node_a-validation.o validation.cpp:5164:30: warning: dynamic exception specifications are deprecated [-Wdeprecated-dynamic-exception-spec] const char* what() const throw() override ^~~~~~~ validation.cpp:5164:30: note: use 'noexcept' instead const char* what() const throw() override ^~~~~~~ noexcept ```
2023-07-14Update CheckBlockIndex invariants for chains based on an assumeutxo snapshotSuhas Daftuar
2023-07-14Add wrapper for adding entries to a chainstate's block index candidatesSuhas Daftuar
2023-07-14Move block-arrival information / preciousblock counters to ChainstateManagerSuhas Daftuar
Block arrival information (and the preciousblock RPC, a related concept) are both chainstate-agnostic, so these are moved to ChainstateManager. This should just be a refactor, without any observable behavior changes.
2023-07-14Remove CChain dependency in node/blockstorageSuhas Daftuar
2023-07-11kernel: Remove StartShutdown calls from validation codeRyan Ofsky
This change drops the last kernel dependency on shutdown.cpp. It also adds new hooks for libbitcoinkernel applications to be able to interrupt kernel operations when the chain tip changes. This is a refactoring that does not affect behavior. (Looking at the code it can appear like the new break statement in the ActivateBestChain function is a change in behavior, but actually the previous StartShutdown call was indirectly triggering a break before, because it was causing m_chainman.m_interrupt to be true. The new code just makes the break more obvious.)
2023-06-28kernel: Add fatalError method to notificationsTheCharlatan
FatalError replaces what previously was the AbortNode function in shutdown.cpp. This commit is part of the libbitcoinkernel project and further removes the shutdown's and, more generally, the kernel library's dependency on interface_ui with a kernel notification method. By removing interface_ui from the kernel library, its dependency on boost is reduced to just boost::multi_index. At the same time it also takes a step towards de-globalising the interrupt infrastructure. Co-authored-by: Russell Yanofsky <russ@yanofsky.org> Co-authored-by: TheCharlatan <seb.kung@gmail.com>
2023-06-28kernel: Pass interrupt reference to chainmanTheCharlatan
This and the following commit seek to decouple the libbitcoinkernel library from the shutdown code. As a library, it should it should have its own flexible interrupt infrastructure without relying on node-wide globals. The commit takes the first step towards this goal by de-globalising `ShutdownRequested` calls in kernel code. Co-authored-by: Russell Yanofsky <russ@yanofsky.org> Co-authored-by: TheCharlatan <seb.kung@gmail.com>
2023-06-23Merge bitcoin/bitcoin#26828: assumeutxo: catch and log fs::remove error ↵Andrew Chow
instead of two exist checks 0e21b56a44d53cec9080edb04410a692717f1ddc assumeutxo: catch and log fs::remove error instead of two exist checks (Andrew Toth) Pull request description: Fixes a block of code which seems to be incorrectly performing two existence checks instead of catching and logging errors. `fs::remove` returns `false` only if the file being removed does not exist, so it is redundant with the `fs::exists` check. If an error does occur when trying to remove an existing file, `fs::remove` will throw. See https://en.cppreference.com/w/cpp/filesystem/remove. Also see https://github.com/bitcoin/bitcoin/blob/master/src/init.cpp#L326-L332 for a similar pattern. ACKs for top commit: MarcoFalke: lgtm ACK 0e21b56a44d53cec9080edb04410a692717f1ddc jamesob: ACK https://github.com/bitcoin/bitcoin/pull/26828/commits/0e21b56a44d53cec9080edb04410a692717f1ddc achow101: ACK 0e21b56a44d53cec9080edb04410a692717f1ddc Tree-SHA512: 137d0be5266cfd947e5e50ec93b895ac659adadf9413bef3468744bfdacee8dbe7d9bdfaf91784c45708610325d2241a114f4be4e622a108a639b3672b618fd2
2023-06-22Merge bitcoin/bitcoin#27862: validation: Stricter assumeutxo error handling ↵Andrew Chow
when renaming chainstates 1c7d08b9acd33aff343228ada7e058e606cb1062 validation: Stricter assumeutxo error handling in InvalidateCoinsDBOnDisk (Ryan Ofsky) 9047337d369d800e6eca4d3b686139073a8e8905 validation: Stricter assumeutxo error handling in LoadChainstate (Ryan Ofsky) Pull request description: There are two places in assumeutxo code where it is calling `AbortNode` to trigger asynchronous shutdowns without returning errors to calling functions. One case, in `LoadChainstate`, happens when snapshot validation succeeds, and there is an error trying to replace the background chainstate with the snapshot chainstate. The other case, in `InvalidateCoinsDBOnDisk`, happens when snapshot validatiion fails, and there is an error trying to remove the snapshot chainstate. In both cases the node is being forced to shut down, so it makes sense for these functions to raise errors so callers can know that an error happened without having to infer it from the shutdown state. Noticed these cases while reviewing #27861, which replaces the `AbortNode` function with a `FatalError` function. ACKs for top commit: achow101: ACK 1c7d08b9acd33aff343228ada7e058e606cb1062 TheCharlatan: ACK 1c7d08b9acd33aff343228ada7e058e606cb1062 jamesob: ACK 1c7d08b9acd33aff343228ada7e058e606cb1062 ([`jamesob/ackr/27862.1.ryanofsky.validation_stricter_assu`](https://github.com/jamesob/bitcoin/tree/ackr/27862.1.ryanofsky.validation_stricter_assu)) Tree-SHA512: fb1dcde3fa0e77b4ba0c48507d289552b939c2866781579c8e994edc209abc3cd29cf81c89380057199323a8eec484956abb1fd3a43c957ecd0e7f7bbfd63fd8
2023-06-16validation: add missing insert to m_dirty_blockindexMartin Zumsande
...in FindMostWorkChain(). Before this, it was possible that the change to the block index wouldn't be persisted to disk.
2023-06-15validation: Stricter assumeutxo error handling in InvalidateCoinsDBOnDiskRyan Ofsky
Currently InvalidateCoinsDBOnDisk is calling AbortNode without an error to the caller if it fails. Change it to return just return util::Result, and update the caller to handle the error itself. This causes the secondary error to be shown below the main error instead of the other way around.
2023-06-12Merge bitcoin/bitcoin#27357: validation: Move warningcache to ↵fanquake
ChainstateManager and rename to m_warningcache 552684976b6df34ce563458f73812e6e494e3b0e validation: Move warningcache to ChainstateManager (dimitaracev) Pull request description: Removes `warningcache` and moves it to `ChainstateManager`. Also removes the respective `TODO` completely. ACKs for top commit: ajtowns: ACK 552684976b6df34ce563458f73812e6e494e3b0e dimitaracev: > ACK [5526849](https://github.com/bitcoin/bitcoin/commit/552684976b6df34ce563458f73812e6e494e3b0e) TheCharlatan: ACK 552684976b6df34ce563458f73812e6e494e3b0e ryanofsky: Code review ACK 552684976b6df34ce563458f73812e6e494e3b0e Tree-SHA512: 6869bd7aa4f0b59324e12eb8e3df47f2c9a3f3b0d9b7d45857426ec9e8b71c5573bdcf71db822f8c10aff7d8679a00a4bedc7a256c28f325e744e5d7267b41e9
2023-05-30refactor: Remove gArgs access from validation.cppTheCharlatan
This is done in the context of the libbitcoinkernel project, wherein reliance of libbitcoinkernel code on the global gArgs is incrementally removed.
2023-05-30refactor: Add path argument to FindSnapshotChainstateDirTheCharlatan
Remove access to the global gArgs for getting the directory in utxo_snapshot. This is done in the context of the libbitcoinkernel project, wherein reliance of libbitcoinkernel code on the global gArgs is incrementally removed.
2023-05-30refactor: Add stop_at_height option in ChainstateManagerTheCharlatan
Remove access to the global gArgs for the stopatheight argument and replace it by adding a field to the existing ChainstateManager Options struct. This should eventually allow users of the ChainstateManager to not rely on the global gArgs and instead pass in their own options.
2023-05-30Merge bitcoin/bitcoin#27636: kernel: Remove util/system from kernel library, ↵fanquake
interface_ui from validation. 7d3b35004b039f2bd606bb46a540de7babdbc41e refactor: Move system from util to common library (TheCharlatan) 7eee356c0a7fefd70c8de21689efa335f52a69ba refactor: Split util::AnyPtr into its own file (TheCharlatan) 44de325d95447498036479c3112ba741caf45bf6 refactor: Split util::insert into its own file (TheCharlatan) 9ec5da36b62276ae22e348f26f88aaf646357d6d refactor: Move ScheduleBatchPriority to its own file (TheCharlatan) f871c69191dfe1331861ebcdbadb6bd47e45c8b1 kernel: Add warning method to notifications (TheCharlatan) 4452707edec91c7d7991f486dd41ef3edb4f7fbf kernel: Add progress method to notifications (TheCharlatan) 84d71457e7250ab25c0a11d1ad1c7657197ffd90 kernel: Add headerTip method to notifications (TheCharlatan) 447761c8228d58f948aae7e73ed079c028cacb97 kernel: Add notification interface (TheCharlatan) Pull request description: This pull request is part of the `libbitcoinkernel` project https://github.com/bitcoin/bitcoin/issues/27587 https://github.com/bitcoin/bitcoin/projects/18 and more specifically its "Step 2: Decouple most non-consensus code from libbitcoinkernel". --- It removes the kernel library's dependency on `util/system` and `interface_ui`. `util/system` contains networking and shell-related code that should not be part of the kernel library. The following pull requests prepared `util/system` for this final step: https://github.com/bitcoin/bitcoin/pull/27419 https://github.com/bitcoin/bitcoin/pull/27254 https://github.com/bitcoin/bitcoin/pull/27238. `interface_ui` defines functions for a more general node interface and has a dependency on `boost/signals2`. After applying the patches from this pull request, the kernel's reliance on boost is down to `boost::multiindex`. The approach implemented here introduces some indirection, which makes the code a bit harder to read. Any suggestions for improving or reworking this pull request to make it more concise, or even reworking it into a more proper interface, are appreciated. ACKs for top commit: MarcoFalke: re-ACK 7d3b35004b039f2bd606bb46a540de7babdbc41e (no change) 🎋 stickies-v: Code Review ACK 7d3b35004b039f2bd606bb46a540de7babdbc41e hebasto: re-ACK 7d3b35004b039f2bd606bb46a540de7babdbc41e, only last two commits dropped since my [recent](https://github.com/bitcoin/bitcoin/pull/27636#pullrequestreview-1435394620) review. Tree-SHA512: c8cfc698dc9d78e20191c444708f2d957501229abe95e5806106d1126fb9c5fbcee686fb55645658c0107ce71f10646f37a2fdf7fde16bbf22cbf1ac885dd08d
2023-05-20kernel: Add warning method to notificationsTheCharlatan
This commit is part of the libbitcoinkernel project and seeks to remove the ChainstateManager's and, more generally, the kernel library's dependency on interface_ui with options methods in this and the following few commits. By removing interface_ui from the kernel library, its dependency on boost is reduced to just boost::multi_index. The DoWarning and AlertNotify functions are moved out of the validation.cpp file, which removes its dependency on interface_ui as well as util/system.
2023-05-20kernel: Add progress method to notificationsTheCharlatan
This commit is part of the libbitcoinkernel project and seeks to remove the ChainstateManager's and, more generally, the kernel library's dependency on interface_ui with options methods in this and the following few commits. By removing interface_ui from the kernel library, its dependency on boost is reduced to just boost::multi_index.
2023-05-20kernel: Add headerTip method to notificationsTheCharlatan
This commit is part of the libbitcoinkernel project and seeks to remove the ChainstateManager's and, more generally, the kernel library's dependency on interface_ui with options methods in this and the following few commits. By removing interface_ui from the kernel library, its dependency on boost is reduced to just boost::multi_index.
2023-05-20kernel: Add notification interfaceTheCharlatan
This commit is part of the libbitcoinkernel project and seeks to remove the ChainstateManager's and, more generally, the kernel library's dependency on interface_ui with options methods in this and the following few commits. By removing interface_ui from the kernel library, its dependency on boost is reduced to just boost::multi_index. Define a new kernel notification class with virtual methods for notifying about internal kernel events. Create a new file in the node library for defining a function creating the default set of notification methods such that these do not need to be re-defined all over the codebase. As a first step, add a `blockTip` method, wrapping `uiInterface.NotifyBlockTip`.
2023-05-16ConnectTip: don't log total disk read time in benchSjors Provoost
This incorrectly assumed num_blocks_total would be greater than 0. This is not guaranteed until the ConnectBlock call right below it. The total and average metric is not very useful because it does not distinguish between blocks read from disk and those loaded from memory. So rather than fixing the divide by zero issue, we just drop the metric.
2023-05-11Merge bitcoin/bitcoin#27125: refactor, kernel: Decouple ArgsManager from ↵fanquake
blockstorage 5ff63a09a9edd1204b2cc56cf6f48a44adab7bb3 refactor, blockstorage: Replace stopafterblockimport arg (TheCharlatan) 18e5ba7c8002bcd473ee29ce4b5bfc56df6142a4 refactor, blockstorage: Replace blocksdir arg (TheCharlatan) 02a0899527ba3d31329e56c791c9dbf36075bb84 refactor, BlockManager: Replace fastprune from arg with options (TheCharlatan) a498d699e3fdac5bfdb33020a1fd6c4a79989752 refactor/iwyu: Complete includes for blockmanager_args (TheCharlatan) f0bb1021f0d60f5f19176e67a66fcf7c325f88d1 refactor: Move functions to BlockManager methods (TheCharlatan) cfbb2124939822e95265a39242ffca3d86bac6e8 zmq: Pass lambda to zmq's ZMQPublishRawBlockNotifier (TheCharlatan) 8ed4ff8e05d61a8e954d72cebdc2e1d1ab24fb84 refactor: Declare g_zmq_notification_interface as unique_ptr (TheCharlatan) Pull request description: The libbitcoin_kernel library should not rely on the `ArgsManager`, but rather use option structs that can be passed to the various classes it uses. This PR removes reliance on the `ArgsManager` from the `blockstorage.*` files. Like similar prior work, it uses the options struct in the `BlockManager` that can be populated with `ArgsManager` values. Some related prior work: https://github.com/bitcoin/bitcoin/pull/26889 https://github.com/bitcoin/bitcoin/pull/25862 https://github.com/bitcoin/bitcoin/pull/25527 https://github.com/bitcoin/bitcoin/pull/25487 Related PR removing blockstorage globals: https://github.com/bitcoin/bitcoin/pull/25781 ACKs for top commit: ryanofsky: Code review ACK 5ff63a09a9edd1204b2cc56cf6f48a44adab7bb3. Since last ACK just added std::move and fixed commit title. Sorry for the noise! mzumsande: Code Review ACK 5ff63a09a9edd1204b2cc56cf6f48a44adab7bb3 Tree-SHA512: 4bde8fd140a40b97eca923e9016d85dcea6fad6fd199731f158376294af59c3e8b163a0725aa47b4be3519b61828044e0a042deea005e0c28de21d8b6c3e1ea7
2023-05-10Merge bitcoin/bitcoin#19690: util: improve FindByte() performanceAndrew Chow
72efc26439da9a1344a19569fb0cab01f82ae7d1 util: improve streams.h:FindByte() performance (Larry Ruane) 604df63f6c70b9692b067777ddb38d946ac0b2fc [bench] add streams findbyte (gzhao408) Pull request description: This PR is strictly a performance improvement; there is no functional change. The `CBufferedFile::FindByte()` method searches for the next occurrence of the given byte in the file. Currently, this is done by explicitly inspecting each byte in turn. This PR takes advantage of `std::find()` to do the same more efficiently, improving its CPU runtime by a factor of about 25 in typical use. ACKs for top commit: achow101: re-ACK 72efc26439da9a1344a19569fb0cab01f82ae7d1 stickies-v: re-ACK 72efc26439da9a1344a19569fb0cab01f82ae7d1 Tree-SHA512: ddf0bff335cc8aa34f911aa4e0558fa77ce35d963d602e4ab1c63090b4a386faf074548daf06ee829c7f2c760d06eed0125cf4c34e981c6129cea1804eb3b719
2023-05-10refactor: Move functions to BlockManager methodsTheCharlatan
This is a commit in preparation for the next few commits. The functions are moved to methods to avoid their re-declaration for the purpose of passing in BlockManager options. The functions that were now moved into the BlockManager should no longer use the params as an argument, but instead use the member variable. In the moved ReadBlockFromDisk and UndoReadFromDisk, change the function signature to accept a reference to a CBlockIndex instead of a raw pointer. The pointer is expected to be non-null, so reflect that in the type. To allow for the move of functions to BlockManager methods all call sites require an instantiated BlockManager, or a callback to one.
2023-05-05util: improve streams.h:FindByte() performanceLarry Ruane
Avoid use of the expensive mod operator (%) when calculating the buffer offset. No functional difference. Co-authored-by: Hennadii Stepanov <32963518+hebasto@users.noreply.github.com>
2023-05-04Remove unused chainparams from BlockManager methodsMarcoFalke
Also, replace pointer with reference while touching the signature.
2023-05-03Merge bitcoin/bitcoin#24957: prune, import: allow pruning to work during ↵Andrew Chow
loadblock import c4981e7f63a3e0aeec1ef3dec040261e993dd724 prune, import: fixes #23852 (mruddy) Pull request description: Fixes #23852 This allows pruning to work during the `-loadblock` import process. An example use case is where you have a clean set of block files and you want to create a pruned node from them, but you don't want to alter the input set of block files. #23852 noted that pruning was not working reliably during the loadblock import process. The reason why the loadblock process was not pruning regularly as it progressed is that the pruning process (`BlockManager::FindFilesToPrune`) checks the tip height of the active chainstate, and `CChainState::ActivateBestChain` was not called (which updates that tip height) in `ThreadImport` until after all the import files were processed. An example bash command line that makes it easy to import a bunch of block files: ``` ./src/qt/bitcoin-qt -debug -logthreadnames -datadir=/tmp/btc -prune=550 -loadblock=/readonly/btc/main/blk{00000..00043}.dat ``` One interesting side note is that `CChainState::ActivateBestChain` can be called while the import process is running (in the `loadblk` thread) by concurrent network message processing activity in the `msghand` thread. For example, one way to reproduce this easily is with the `getblockfrompeer` RPC (requesting a block with height greater than 100000) run from a node connected to an importing node. There are other ways too, but this is an easy way. I only mention this to explain how the `max_prune_height=225719` log message in the original issue came to occur. ACKs for top commit: achow101: re-ACK c4981e7f63a3e0aeec1ef3dec040261e993dd724 Tree-SHA512: d287c7753952c22f598ba782914c47f45ad44ce60b0fbce9561354e701f1a2a98bafaaaa106c8428690b814e281305ca3622b177ed3cb2eb7559f07c958ab537
2023-04-26Merge bitcoin/bitcoin#26933: mempool: disallow txns under min relay fee, ↵glozow
even in packages bf77fc9cb45209b9c560208c65abc94209cd7919 [test] mempool full in package accept (glozow) b51ebccc28e66c1822ab22d2d178be55c6618196 [validation] set PackageValidationState when mempool full (glozow) 563a2ee4f564c8ea5f8313d711b196e260568c04 [policy] disallow transactions under min relay fee, even in packages (glozow) c4554fe894d7af8e666f5d424deccddf516713ef [test] package cpfp bumps parents <mempoolminfee but >=minrelaytxfee (glozow) ac463e87df728689701810c3961155c49fdc5b31 [test util] mock mempool minimum feerate (glozow) Pull request description: Part of package relay, see #27463. Note that this still allows packages to bump transactions that are below the dynamic mempool minimum feerate, which means this still solves the "mempool is congested and my presigned 1sat/vB tx is screwed" problem for all transactions. On master, the package policy (only accessible through regtest-only RPC submitpackage) allows 0-fee (or otherwise below min relay feerate) transactions if they are bumped by a child. However, with default package limits, we don't yet have a DoS-resistant way of ensuring these transactions remain bumped throughout their time in the mempool. Primarily, the fee-bumping child may later be replaced by another transaction that doesn't bump the parent(s). The parent(s) could potentially stay bumped by other transactions, but not enough to ever be selected by the `BlockAssembler` (due to `blockmintxfee`). For example, (tested [here](https://github.com/glozow/bitcoin/commits/26933-motivation)): - The mempool accepts 24 below-minrelayfeerate transactions ("0-fee parents"), all bumped by a single high-fee transaction ("the fee-bumping child"). The fee-bumping child also spends a confirmed UTXO. - Two additional children are added to each 0-fee parent. These children each pay a feerate slightly above the minimum relay feerate (e.g. 1.9sat/vB) such that, for each 0-fee parent, the total fees of its two children divided by the total size of the children and parent is above the minimum relay feerate. - If a block template is built now, all transactions would be selected. - A transaction replaces the the fee-bumping child, spending only the confirmed UTXO and not any of the outputs from the 0-fee parents. - The 0-fee parents now each have 2 children. Their descendant feerates are above minrelayfeerate, which means that they remain in the mempool, even if the mempool evicts all below-minrelayfeerate packages. - If a block template is built now, none of the 0-fee parents or their children would be selected. - Even more low-feerate descendants can be added to these below-minrelayfeerate packages and they will not be evicted until they expire or the mempool reaches capacity. Unless we have a DoS-resistant way of ensuring package CPFP-bumped transactions are always bumped, allowing package CPFP to bump below-minrelayfeerate transactions can result in these problematic situations. See #27018 which proposes a partial solution with some limitations, and contains discussion about potential improvements to eviction strategy. While no adequate solution exists, for now, avoid these situations by requiring all transactions to meet min relay feerate. ACKs for top commit: ajtowns: reACK bf77fc9cb45209b9c560208c65abc94209cd7919 instagibbs: re-ACK https://github.com/bitcoin/bitcoin/pull/26933/commits/bf77fc9cb45209b9c560208c65abc94209cd7919 Tree-SHA512: 28940f41493a9e280b010284316fb8caf1ed7b2090ba9a4ef8a3b2eafc5933601074b142f4f7d4e3c6c4cce99d3146f5c8e1393d9406c6f2070dd41c817985c9
2023-04-21Merge bitcoin/bitcoin#27419: move-only: Extract common/args from util/systemfanquake
be55f545d53d44fdcf2d8ae802e9eae551d120c6 move-only: Extract common/args and common/config.cpp from util/system (TheCharlatan) Pull request description: This pull request is part of the `libbitcoinkernel` project https://github.com/bitcoin/bitcoin/issues/24303 https://github.com/bitcoin/bitcoin/projects/18 and more specifically its "Step 2: Decouple most non-consensus code from libbitcoinkernel". It is part of a series of patches splitting up the `util/system` files. Its preceding pull request is https://github.com/bitcoin/bitcoin/pull/27254. The pull request contains an extraction of ArgsManager related functions from util/system into their own common/ file. The background of this commit is an ongoing effort to decouple the libbitcoinkernel library from the ArgsManager. The ArgsManager belongs into the common library, since the kernel library should not depend on it. See [doc/design/libraries.md](https://github.com/bitcoin/bitcoin/blob/master/doc/design/libraries.md) for more information on this rationale. ACKs for top commit: MarcoFalke: re-ACK be55f545d53d44fdcf2d8ae802e9eae551d120c6 🚲 ryanofsky: Code review ACK be55f545d53d44fdcf2d8ae802e9eae551d120c6. Just small cleanups since the last review. hebasto: ACK be55f545d53d44fdcf2d8ae802e9eae551d120c6, I have reviewed the code and it looks OK, I agree it can be merged. Tree-SHA512: 90eb03334af0155b823030b4f2ecf286d35058d700ee2ddbbaa445be19e31eb0fe982656f35bd14ecee3ad2c3d0db3746855cb8f3777eff7253713e42873e111
2023-04-20Merge bitcoin/bitcoin#25325: Add pool based memory resourceAndrew Chow
9f947fc3d4b779f017332135323b34e8f216f613 Use PoolAllocator for CCoinsMap (Martin Leitner-Ankerl) 5e4ac5abf54f8e6d6330df0c73119aa0cca4c103 Call ReallocateCache() on each Flush() (Martin Leitner-Ankerl) 1afca6b663bb54022afff193fd9d83856606b189 Add PoolResource fuzzer (Martin Leitner-Ankerl) e19943f049ed8aa4f32a1d8440a9fbf160367f0f Calculate memory usage correctly for unordered_maps that use PoolAllocator (Martin Leitner-Ankerl) b8401c3281978beed6198b2f9782b6a8dd35cbd7 Add pool based memory resource & allocator (Martin Leitner-Ankerl) Pull request description: A memory resource similar to `std::pmr::unsynchronized_pool_resource`, but optimized for node-based containers. The goal is to be able to cache more coins with the same memory usage, and allocate/deallocate faster. This is a reimplementation of #22702. The goal was to implement it in a way that is simpler to review & test * There is now a generic `PoolResource` for allocating/deallocating memory. This has practically the same API as `std::pmr::memory_resource`. (Unfortunately I cannot use std::pmr because libc++ simply doesn't implement that API). * Thanks to sipa there is now a fuzzer for PoolResource! On a fast machine I ran it for ~770 million executions without finding any issue. * The estimation of the correct node size is now gone, PoolResource now has multiple pools and just needs to be created large enough to have space for the unordered_map nodes. I ran benchmarks with #22702, mergebase, and this PR. Frequency locked Intel i7-8700, clang++ 13.0.1 to reindex up to block 690000. ```sh bitcoind -dbcache=5000 -assumevalid=00000000000000000002a23d6df20eecec15b21d32c75833cce28f113de888b7 -reindex-chainstate -printtoconsole=0 -stopatheight=690000 ``` The performance is practically identical with #22702, just 0.4% slower. It's ~21% faster than master: ![Progress in Million Transactions over Time(2)](https://user-images.githubusercontent.com/14386/173288685-91952ade-f304-4825-8bfb-0725a71ca17b.png) ![Size of Cache in MiB over Time](https://user-images.githubusercontent.com/14386/173291421-e6b410be-ac77-479b-ad24-5fafcebf81eb.png) Note that on cache drops mergebase's memory doesnt go so far down because it does not free the `CCoinsMap` bucket array. ![Size of Cache in Million tx over Time(1)](https://user-images.githubusercontent.com/14386/173288703-a80c9c9e-93c8-4a16-9df8-610c89c61cc4.png) ACKs for top commit: LarryRuane: ACK 9f947fc3d4b779f017332135323b34e8f216f613 achow101: re-ACK 9f947fc3d4b779f017332135323b34e8f216f613 john-moffett: ACK 9f947fc3d4b779f017332135323b34e8f216f613 jonatack: re-ACK 9f947fc3d4b779f017332135323b34e8f216f613 Tree-SHA512: 48caf57d1775875a612b54388ef64c53952cd48741cacfe20d89049f2fb35301b5c28e69264b7d659a3ca33d4c714d47bafad6fd547c4075f08b45acc87c0f45
2023-04-19move-only: Extract common/args and common/config.cpp from util/systemTheCharlatan
This is an extraction of ArgsManager related functions from util/system into their own common file. Config file related functions are moved to common/config.cpp. The background of this commit is an ongoing effort to decouple the libbitcoinkernel library from the ArgsManager. The ArgsManager belongs into the common library, since the kernel library should not depend on it. See doc/design/libraries.md for more information on this rationale.
2023-04-17[validation] set PackageValidationState when mempool fullglozow
2023-04-17[policy] disallow transactions under min relay fee, even in packagesglozow
Avoid adding transactions below min relay feerate because, even if they were bumped through CPFP when entering the mempool, we do not have a DoS-resistant way of ensuring they always remain bumped. In the future, this rule can be relaxed (e.g. to allow packages to bump 0-fee transactions) if we find a way to do so.
2023-04-03Merge bitcoin/bitcoin#27254: refactor: Extract util/fs from util/systemfanquake
00e9b97f37e0bdf4c647236838c10b68b7ad5be3 refactor: Move fs.* to util/fs.* (TheCharlatan) 106b46d9d25b5228ef009fbbe6f9a7ae35090d15 Add missing fs.h includes (TheCharlatan) b202b3dd6393b415fa68e18dc49c9431dc6b58b2 Add missing cstddef include in assumptions.h (TheCharlatan) 18fb36367a28819bd5ab402344802796a1248979 refactor: Extract util/fs_helpers from util/system (Ben Woosley) Pull request description: This pull request is part of the `libbitcoinkernel` project https://github.com/bitcoin/bitcoin/issues/24303 https://github.com/bitcoin/bitcoin/projects/18 and more specifically its "Step 2: Decouple most non-consensus code from libbitcoinkernel". This commit was originally authored by empact and is taken from its parent PR #25152. #### Context There is an ongoing effort to decouple the `ArgsManager` used for command line parsing user-provided arguments from the libbitcoinkernel library (https://github.com/bitcoin/bitcoin/pull/25290, https://github.com/bitcoin/bitcoin/pull/25487, https://github.com/bitcoin/bitcoin/pull/25527, https://github.com/bitcoin/bitcoin/pull/25862, https://github.com/bitcoin/bitcoin/pull/26177, and https://github.com/bitcoin/bitcoin/pull/27125). The `ArgsManager` is defined in `system.h`. A similar pull request extracting functionality from `system.h` has been merged in https://github.com/bitcoin/bitcoin/pull/27238. #### Changes Next to providing better code organization, this PR removes some reliance of the tree of libbitcoinkernel header includes on `system.h` (and thus the `ArgsManager` definition) by moving filesystem related functions out of the `system.*` files. There is already a pair of `fs.h` / `fs.cpp` in the top-level `src/` directory. They were not combined with the files introduced here, to keep the patch cleaner and more importantly because they are often included without the utility functions. The new files are therefore named `fs_helpers` and the existing `fs` files are moved into the util directory. Further commits splitting more functionality out of `system.h` are still in #25152 and will be submitted in separate PRs once this PR has been processed. ACKs for top commit: hebasto: ACK 00e9b97f37e0bdf4c647236838c10b68b7ad5be3 Tree-SHA512: 31422f148d14ba3c843b99b1550a6fd77c77f350905ca324f93d4f97b652246bc58fa9696c64d1201979cf88733e40be02d262739bb7d417cf22bf506fdb7666
2023-03-29validation: Move warningcache to ChainstateManagerdimitaracev
2023-03-23Call ReallocateCache() on each Flush()Martin Leitner-Ankerl
This frees up all associated memory with the map, not only the nodes. This is necessary in preparation for using the PoolAllocator for CCoinsMap, which does not actually free any memory on clear().
2023-03-23refactor: Move fs.* to util/fs.*TheCharlatan
The fs.* files are already part of the libbitcoin_util library. With the introduction of the fs_helpers.* it makes sense to move fs.* into the util/ directory as well.
2023-03-23refactor: Extract util/fs_helpers from util/systemBen Woosley
This is an extraction of filesystem related functions from util/system into their own utility file. The background of this commit is an ongoing effort to decouple the libbitcoinkernel library from the ArgsManager defined in system.h. Moving these functions out of system.h allows including them from a separate source file without including the ArgsManager definitions from system.h.
2023-03-22Merge bitcoin/bitcoin#26749: refactor: Use move semantics instead of custom ↵fanquake
swap functions 95ad70ab652ddde7de65f633c36c1378b26a313a test: Default initialize `should_freeze` to `true` (Hennadii Stepanov) cea50521fe810111a8a3c84ad14f944eafb5b658 refactor: Drop no longer used `swap` member functions (Hennadii Stepanov) a87fb6bee5a7fb0879b3adea9a29997f1331acb0 clang-tidy: Fix modernize-use-default-member-init in `CScriptCheck` (Hennadii Stepanov) b4bed5c1f98c0eed18f52fdcea11a420c10ed98d refactor: Drop no longer used `CScriptCheck()` default constructor (Hennadii Stepanov) d8427cc28e3a9ac3319fb452b16661957c812b8f refactor: Use move semantics in `CCheckQueue::Loop` (Hennadii Stepanov) 9a0b5241396efe3b3ceb3931717c30bb94f99bfb clang-tidy, test: Fix bugprone-use-after-move in `Correct_Queue_range()` (Hennadii Stepanov) 04831fee6dca3eb86cd1d6b9ef879b296263fe35 refactor: Make move semantics explicit for callers (Hennadii Stepanov) 6c2d5972f3544c4f3e987828a99e88f27b62cf87 refactor: Use move semantics in `CCheckQueue::Add` (Hennadii Stepanov) 06820032142a75cc3c5b832045058bc6f6f74786 test, refactor: Avoid `CScriptCheck::swap` in `transaction_tests` (Hennadii Stepanov) 15209d97c6aad7d5c199fe007ad39b91c8ee6562 consensus, refactor: Avoid `CScriptCheck::swap` in `CheckInputScripts` (Hennadii Stepanov) Pull request description: This PR makes code more succinct and readable by using move semantics. ACKs for top commit: martinus: re-ACK 95ad70ab652ddde7de65f633c36c1378b26a313a achow101: ACK 95ad70ab652ddde7de65f633c36c1378b26a313a TheCharlatan: re-ACK https://github.com/bitcoin/bitcoin/commit/95ad70ab652ddde7de65f633c36c1378b26a313a MarcoFalke: re-ACK 95ad70ab652ddde7de65f633c36c1378b26a313a 🚥 Tree-SHA512: adda760891b12d252dc9b823fe7c41eed660364b6fb1a69f17607d7a31eb0bbb82a80d154a7acfaa241b5de37d42a293c2b6e059f26a8e92d88d3a87c99768fb
generated by cgit v1.2.3 (git 2.26.2) at 2024-09-18 06:55:52 +0000