aboutsummaryrefslogtreecommitdiff
path: root/src/txmempool.cpp
AgeCommit message (Collapse)Author
2024-05-06refactor: Simply include CTxMemPool::Options in CTxMemPool directly rather ↵Luke Dashjr
than duplicating definition
2024-04-22Avoid explicitly computing diagram; compare based on chunksPieter Wuille
2024-03-26CalculateFeerateDiagramsForRBF: update misleading description of old diagram ↵Greg Sanders
contents
2024-03-26CalculateFeerateDiagramsForRBF: remove size tie-breaking from chunking conflictsGreg Sanders
2024-03-18Implement ImprovesFeerateDiagramGreg Sanders
This new function takes the populated sets of direct and all conflicts computed in the current mempool, assuming the replacements are a single chunk, and computes a diagram check. The diagram check only works against cluster sizes of 2 or less, and fails if it encounters a different topology. Co-authored-by: Suhas Daftuar <sdaftuar@chaincode.com>
2024-02-15[refactor] Make signals optional in mempool and chainmanTheCharlatan
This is done in preparation for the next two commits, where the CMainSignals are de-globalized. This avoids adding new constructor arguments to the ChainstateManager and CTxMemPool classes over the next two commits. This could also allow future tests that are only interested in the internal behaviour of the classes to forgo instantiating the signals.
2024-02-09Merge bitcoin/bitcoin#28948: v3 transaction policy for anti-pinningAva Chow
29029df5c700e6940c712028303761d91ae15847 [doc] v3 signaling in mempool-replacements.md (glozow) e643ea795e4b6fea4a6bbb3d72870ee6a4c836b1 [fuzz] v3 transactions and sigop-adjusted vsize (glozow) 1fd16b5c62f54c7f4c60122acd65d852f63d1e8b [functional test] v3 transaction submission (glozow) 27c8786ba918a42c860e6a50eaee9fdf56d7c646 test framework: Add and use option for tx-version in MiniWallet methods (MarcoFalke) 9a1fea55b29fe025355b06b45e3d77d192acc635 [policy/validation] allow v3 transactions with certain restrictions (glozow) eb8d5a2e7d939dd3ee683486e98702079e0dfcc0 [policy] add v3 policy rules (glozow) 9a29d470fbb62bbb27d517efeafe46ff03c25f54 [rpc] return full string for package_msg and package-error (glozow) 158623b8e0726dff7eae4288138f1710e727db9c [refactor] change Workspace::m_conflicts and adjacent funcs/structs to use Txid (glozow) Pull request description: See #27463 for overall package relay tracking. Delving Bitcoin discussion thread: https://delvingbitcoin.org/t/v3-transaction-policy-for-anti-pinning/340 Delving Bitcoin discussion for LN usage: https://delvingbitcoin.org/t/lightning-transactions-with-v3-and-ephemeral-anchors/418 Rationale: - There are various pinning problems with RBF and our general ancestor/descendant limits. These policies help mitigate many pinning attacks and make package RBF feasible (see #28984 which implements package RBF on top of this). I would focus the most here on Rule 3 pinning. [1][2] - Switching to a cluster-based mempool (see #27677 and #28676) requires the removal of CPFP carve out, which applications depend on. V3 + package RBF + ephemeral anchors + 1-parent-1-child package relay provides an intermediate solution. V3 policy is for "Priority Transactions." [3][4] It allows users to opt in to more restrictive topological limits for shared transactions, in exchange for the more robust fee-bumping abilities that offers. Even though we don't have cluster limits, we are able to treat these transactions as having as having a maximum cluster size of 2. Immediate benefits: - You can presign a transaction with 0 fees (not just 1sat/vB!) and add a fee-bump later. - Rule 3 pinning is reduced by a significant amount, since the attacker can only attach a maximum of 1000vB to your shared transaction. This also enables some other cool things (again see #27463 for overall roadmap): - Ephemeral Anchors - Package RBF for these 1-parent-1-child packages. That means e.g. a commitment tx + child can replace another commitment tx using the child's fees. - We can transition to a "single anchor" universe without worrying about package limit pinning. So current users of CPFP carve out would have something else to use. - We can switch to a cluster-based mempool [5] (#27677 #28676), which removes CPFP carve out [6]. [1]: Original mailing list post and discussion about RBF pinning problems https://gist.github.com/glozow/25d9662c52453bd08b4b4b1d3783b9ff, https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2022-January/019817.html [2]: A FAQ is "we need this for cluster mempool, but is this still necessary afterwards?" There are some pinning issues that are fixed here and not fully fixed in cluster mempool, so we will still want this or something similar afterward. [3]: Mailing list post for v3 https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2022-September/020937.html [4]: Original PR #25038 also contains a lot of the discussion [5]: https://delvingbitcoin.org/t/an-overview-of-the-cluster-mempool-proposal/393/7 [6]: https://delvingbitcoin.org/t/an-overview-of-the-cluster-mempool-proposal/393#the-cpfp-carveout-rule-can-no-longer-be-supported-12 ACKs for top commit: sdaftuar: ACK 29029df5c700e6940c712028303761d91ae15847 achow101: ACK 29029df5c700e6940c712028303761d91ae15847 instagibbs: ACK 29029df5c700e6940c712028303761d91ae15847 modulo that Tree-SHA512: 9664b078890cfdca2a146439f8835c9d9ab483f43b30af8c7cd6962f09aa557fb1ce7689d5e130a2ec142235dbc8f21213881baa75241c5881660f9008d68450
2024-01-18refactor: remove CTxMemPool::queryHashes()stickies-v
Its only usage can easily be replaced with CTxMemPool::entryAll()
2024-01-16[refactor] change Workspace::m_conflicts and adjacent funcs/structs to use Txidglozow
It's preferable to use type-safe transaction identifiers to avoid confusing txid and wtxid. The next commit will add a reference to this set; we use this opportunity to change it to Txid ahead of time instead of adding new uses of uint256.
2023-12-17wallet, mempool: propagete `checkChainLimits` error message to walletismaelsadeeq
Update CheckPackageLimits to use util::Result to pass the error message instead of out parameter. Also update test to reflect the error message from `CTxMempool` `CheckPackageLimits` output.
2023-12-01Merge bitcoin/bitcoin#28368: Fee Estimator updates from Validation ↵Andrew Chow
Interface/CScheduler thread 91504cbe0de2b74ef1aa2709761aaf0597ec66a2 rpc: `SyncWithValidationInterfaceQueue` on fee estimation RPC's (ismaelsadeeq) 714523918ba2b853fc69bee6b04a33ba0c828bf5 tx fees, policy: CBlockPolicyEstimator update from `CValidationInterface` notifications (ismaelsadeeq) dff5ad3b9944cbb56126ba37a8da180d1327ba39 CValidationInterface: modify the parameter of `TransactionAddedToMempool` (ismaelsadeeq) 91532bd38223d7d04166e05de11d0d0b55e60f13 tx fees, policy: update `CBlockPolicyEstimator::processBlock` parameter (ismaelsadeeq) bfcd401368fc0dc43827a8969a37b7e038d5ca79 CValidationInterface, mempool: add new callback to `CValidationInterface` (ismaelsadeeq) 0889e07987294d4ef2814abfca16d8e2a0c5f541 tx fees, policy: cast with static_cast instead of C-Style cast (ismaelsadeeq) a0e3eb7549d2ba4dd3af12b9ce65e29158f59078 tx fees, policy: bugfix: move `removeTx` into reason != `BLOCK` condition (ismaelsadeeq) Pull request description: This is an attempt to #11775 This Pr will enable fee estimator to listen to ValidationInterface notifications to process new transactions added and removed from the mempool. This PR includes the following changes: - Added a new callback to the Validation Interface `MempoolTransactionsRemovedForConnectedBlock`, which notifies listeners about the transactions that have been removed due to a new block being connected, along with the height at which the transactions were removed. - Modified the `TransactionAddedToMempool` callback parameter to include additional information about the transaction needed for fee estimation. - Updated `CBlockPolicyEstimator` to process transactions using` CTransactionRef` instead of `CTxMempoolEntry.` - Implemented the `CValidationInterface` interface in `CBlockPolicyEstimater` and overridden the `TransactionAddedToMempool`, `TransactionRemovedFromMempool`, and `MempoolTransactionsRemovedForConnectedBlock` methods to receive updates from their notifications. Prior to this PR, the fee estimator updates from the mempool, i.e whenever a new block is connected all transactions in the block that are in our mempool are going to be removed using the `removeForBlock` function in `txmempool.cpp`. This removal triggered updates to the fee estimator. As a result, the fee estimator would block mempool's `cs` until it finished updating every time a new block was connected. Instead of being blocked only on mempool tx removal, we were blocking on both tx removal and fee estimator updating. If we want to further improve fee estimation, or add heavy-calulation steps to it, it is currently not viable as we would be slowing down block relay in the process This PR is smaller in terms of the changes made compared to #11775, as it focuses solely on enabling fee estimator updates from the validationInterface/cscheduler thread notifications. I have not split the validation interface because, as I understand it, the rationale behind the split in #11775 was to have `MempoolInterface` signals come from the mempool and `CValidationInterface` events come from validation. I believe this separation can be achieved in a separate refactoring PR when the need arises. Also left out some commits from #11775 - Some refactoring which are no longer needed. - Handle reorgs much better in fee estimator. - Track witness hash malleation in fee estimator I believe they are a separate change that can come in a follow-up after this. ACKs for top commit: achow101: ACK 91504cbe0de2b74ef1aa2709761aaf0597ec66a2 TheCharlatan: Re-ACK 91504cbe0de2b74ef1aa2709761aaf0597ec66a2 willcl-ark: ACK 91504cbe0de2b74ef1aa2709761aaf0597ec66a2 Tree-SHA512: 846dfb9da57a8a42458827b8975722d153907fe6302ad65748d74f311e1925557ad951c3d95fe71fb90ddcc8a3710c45abb343ab86b88780871cb9c38c72c7b1
2023-11-28Merge bitcoin/bitcoin#28903: refactor: Make CTxMemPoolEntry only explicitly ↵Andrew Chow
copyable 705e3f1de00bf30d728addd52a790a139d948e32 refactor: Make CTxMemPoolEntry only explicitly copyable (TheCharlatan) Pull request description: This has the goal of prohibiting users from accidentally creating runtime failures, e.g. by interacting with iterator_to with a copied entry. This was brought up here: https://github.com/bitcoin/bitcoin/pull/28886#issuecomment-1814794954. CTxMemPoolEntry is already implicitly not move-constructable. So be explicit about this and use a std::list to collect the values in the policy_estimator fuzz test instead of a std::vector. ACKs for top commit: maflcko: ACK 705e3f1de00bf30d728addd52a790a139d948e32 🌯 achow101: ACK 705e3f1de00bf30d728addd52a790a139d948e32 ajtowns: ACK 705e3f1de00bf30d728addd52a790a139d948e32 ismaelsadeeq: ACK 705e3f1de00bf30d728addd52a790a139d948e32 Tree-SHA512: 62056905c679c919d00f9ae065ed66ac986e7e7062015aea542843d8deecda57104d7a68d002f7b20afa3164f8e9215d2d2d002c167224129540e3b1bd0712cc
2023-11-22tx fees, policy: CBlockPolicyEstimator update from `CValidationInterface` ↵ismaelsadeeq
notifications `CBlockPolicyEstimator` will implement `CValidationInterface` and subscribe to its notification to process transactions added and removed from the mempool. Re-delegate calculation of `validForFeeEstimation` from validation to fee estimator. Also clean up the validForFeeEstimation arg thats no longer needed in `CTxMempool`. Co-authored-by: Matt Corallo <git@bluematt.me>
2023-11-22tx fees, policy: update `CBlockPolicyEstimator::processBlock` parameterismaelsadeeq
Update `processBlock` parameter to reference to a vector of `RemovedMempoolTransactionInfo`.
2023-11-22CValidationInterface, mempool: add new callback to `CValidationInterface`ismaelsadeeq
This commit adds a new callback `MempoolTransactionsRemovedForBlock` which notify its listeners of the transactions that are removed from the mempool because a new block is connected, along with the block height the transactions were removed. The transactions are in `RemovedMempoolTransactionInfo` format. `CTransactionRef`, base fee, virtual size, and height which the transaction was added to the mempool are all members of the struct called `RemovedMempoolTransactionInfo`. A struct `NewMempoolTransactionInfo`, which has fields similar to `RemovedMempoolTransactionInfo`, will be added in a later commit, create a struct `TransactionInfo` with all similar fields. They can both have a member with type `TransactionInfo`.
2023-11-22tx fees, policy: bugfix: move `removeTx` into reason != `BLOCK` conditionismaelsadeeq
If the removal reason of a transaction is BLOCK, then the `removeTx` boolean argument should be true. Before this PR, `CBlockPolicyEstimator` have to complete updating the fee stats before the mempool clears that's why having removeTx call outside reason!= `BLOCK` in `addUnchecked` was not a bug. But in a case where the `CBlockPolicyEstimator` update is asynchronous, the mempool might clear before we update the `CBlockPolicyEstimator` fee stats. Transactions that are removed for `BLOCK` reasons will also be incorrectly removed from `CBlockPolicyEstimator` stats as failures.
2023-11-21Use Txid in COutpointdergoegge
2023-11-17refactor: Make CTxMemPoolEntry only explicitly copyableTheCharlatan
This has the goal of prohibiting users from accidentally creating runtime failures, e.g. by interacting with iterator_to with a copied entry. CTxMemPoolEntry is already implicitly not move-constructable. So be explicit about this and use a std::list to collect the values in the policy_estimator fuzz test instead of a std::vector. Co-authored-by: Anthony Towns <aj@erisian.com.au>
2023-11-13Merge bitcoin/bitcoin#28207: mempool: Persist with XORAndrew Chow
fa6b053b5c964fb35935fa994cb782c0731a56f8 mempool: persist with XOR (MarcoFalke) Pull request description: Currently the `mempool.dat` file stores data received from remote peers as-is. This may be problematic when a program other than Bitcoin Core tries to interpret them by accident. For example, an anti-virus program or other program may scan the file and move it into quarantine, or delete it, or corrupt it. While the local wallet is expected to re-submit any pending transactions, unrelated transactions may be missing from the mempool after a restart. This may cause fee estimates to be off, or may cause block relay to be slower. Fix this, similar to https://github.com/bitcoin/bitcoin/pull/6650, by rolling a random XOR pattern over the dat file when writing or reading it. Obviously this can only protect against programs that accidentally and unintentionally are trying to mess with the dat file. Any program that intentionally wants to mess with the dat file can still trivially do so. ACKs for top commit: achow101: re-ACK fa6b053b5c964fb35935fa994cb782c0731a56f8 glozow: reACK fa6b053b5c964fb35935fa994cb782c0731a56f8 ismaelsadeeq: ACK fa6b053b5c964fb35935fa994cb782c0731a56f8 Tree-SHA512: ded2ce3d81bc944b828263534e3178a1e45a914fe8e024f4a14c6561a73e301820944ecc75dd704b3d4221a7a3a5c0597ccab79546250c1197609ee981fe324e
2023-11-10scripted-diff: rename vTxHashes to txns_randomizedTheCharlatan
-BEGIN VERIFY SCRIPT- git grep -l "vTxHashesIdx" src | xargs sed -i "s/vTxHashesIdx/idx_randomized/g" git grep -l "vTxHashes" src | xargs sed -i "s/vTxHashes/txns_randomized/g" -END VERIFY SCRIPT-
2023-11-10[refactor] rewrite vTxHashes as a vector of CTransactionRefglozow
vTxHashes exposes a complex mapTx iterator type that its external users don't need. Directly populate it with CTransactionRef instead.
2023-11-10[refactor] Add helper for retrieving mempool entryTheCharlatan
In places where the iterator is only needed for accessing the actual entry, it should not be required to first retrieve the iterator.
2023-11-10[refactor] Add helper for iterating through mempool entriesstickies-v
Instead of reaching into the mapTx data structure, use a helper method that provides the required vector of CTxMemPoolEntry pointers.
2023-11-09mempool: persist with XORMarcoFalke
2023-10-12tidy: modernize-use-emplaceMarcoFalke
2023-09-20Handle over-sized (in virtual bytes) packages with no in-mempool ancestorsGreg Sanders
2023-09-20Bugfix: Pass correct virtual size to CheckPackageLimitsLuke Dashjr
2023-09-14Remove MemPoolAccept::m_limits, only have local copies for carveoutsGreg Sanders
2023-09-13Merge bitcoin/bitcoin#28251: validation: fix coins disappearing mid-package ↵fanquake
evaluation 32c1dd1ad65af0ad4d36a56d2ca32a8481237e68 [test] mempool coins disappearing mid-package evaluation (glozow) a67f460c3fd1c7eb8070623666d887eefccff0d6 [refactor] split setup in mempool_limit test (glozow) d08696120e3647b4c2cd0ae8d6e57dea12418b7c [test framework] add ability to spend only confirmed utxos (glozow) 3ea71feb11c261f002ed918f91f3434fd8a23589 [validation] don't LimitMempoolSize in any subpackage submissions (glozow) d227b7234cd4cfd7c593ffcf8e2f24573d1ebea5 [validation] return correct result when already-in-mempool tx gets evicted (glozow) 9698b81828ff98820fa49c83ca364063233374c6 [refactor] back-fill results in AcceptPackage (glozow) 8ad7ad33929ee846a55a43c55732be0cb8973060 [validation] make PackageMempoolAcceptResult members mutable (glozow) 03b87c11ca0705e1d6147b90da33ce555f9f41c8 [validation] add AcceptSubPackage to delegate Accept* calls and clean up m_view (glozow) 3f01a3dab1c4ee37fd4093b6a0a3b622f53e231d [CCoinsViewMemPool] track non-base coins and allow Reset (glozow) 7d7f7a1189432b1b6245ba25df572229870567cb [policy] check for duplicate txids in package (glozow) Pull request description: While we are evaluating a package, we split it into "subpackages" for evaluation (currently subpackages all have size 1 except the last one). If a subpackage has size 1, we may add a tx to mempool and call `LimitMempoolSize()`, which evicts transactions if the mempool gets full. We handle the case where the just-submitted transaction is evicted immediately, but we don't handle the case in which a transaction from a previous subpackage (either just submitted or already in mempool) is evicted. Mainly, since the coins created by the evicted transaction are cached in `m_view`, we don't realize the UTXO has disappeared until `CheckInputsFromMempoolAndCache` asserts that they exist. Also, the returned `PackageMempoolAcceptResult` reports that the transaction is in mempool even though it isn't anymore. Fix this by not calling `LimitMempoolSize()` until the very end, and editing the results map with "mempool full" if things fall out. Pointed out by instagibbs in https://github.com/bitcoin/bitcoin/commit/faeed687e5cde5e32750d93818dd1d4add837f24 on top of the v3 PR. ACKs for top commit: instagibbs: reACK https://github.com/bitcoin/bitcoin/pull/28251/commits/32c1dd1ad65af0ad4d36a56d2ca32a8481237e68 Tree-SHA512: 61e7f69db4712e5e5bfa27d037ab66bdd97f1bf60a8d9ffb96adb1f0609af012c810d681102ee5c7baec7b5fe8cb7c304a60c63ccc445d00d86a2b7f0e7ddb90
2023-09-13[CCoinsViewMemPool] track non-base coins and allow Resetglozow
Temporary coins should not be available in separate subpackage submissions. Any mempool coins that are cached in m_view should be removed whenever mempool contents change, as they may be spent or no longer exist.
2023-08-17move-only: Create src/kernel/mempool_removal_reason.hMarcoFalke
This is needed for a future commit. Can be reviewed with: --color-moved=dimmed-zebra
2023-08-17Merge bitcoin/bitcoin#27675: p2p: Drop m_recently_announced_invs bloom filterfanquake
fb02ba3c5f5bcd96b5e3622ef001b8e57ce63fc0 mempool_entry: improve struct packing (Anthony Towns) 1a118062fbc4ec8f645f4ec4298d869a869c3344 net_processing: Clean up INVENTORY_BROADCAST_MAX constants (Anthony Towns) 6fa49937e488d0924044786c76b42324b659f351 test: Check tx from disconnected block is immediately requestable (glozow) e4ffabbffacc4b890d393aafcc8286916ef887d8 net_processing: don't add txids to m_tx_inventory_known_filter (Anthony Towns) 6ec1809d33bfc42b80cb6f35625dccd56be8d507 net_processing: drop m_recently_announced_invs bloom filter (Anthony Towns) a70beafdb22564043dc24fc98133fdadbaf77d8a validation: when adding txs due to a block reorg, allow immediate relay (Anthony Towns) 1e9684f39fba909b3501e9402d5b61f4bf744ff2 mempool_entry: add mempool entry sequence number (Anthony Towns) Pull request description: This PR replaces the `m_recently_announced_invs` bloom filter with a simple sequence number tracking the mempool state when we last considered sending an INV message to a node. This saves 33kB per peer (or more if we raise the rate at which we relay transactions over the network, in which case we would need to increase the size of the bloom filter proportionally). The philosophy here (compare with #18861 and #19109) is that we consider the rate limiting on INV messages to only be about saving bandwidth and not protecting privacy, and therefore after you receive an INV message, it's immediately fair game to request any transaction that was in the mempool at the time the INV message was sent. We likewise consider the BIP 133 feefilter and BIP 37 bloom filters to be bandwidth optimisations here, and treat transactions as requestable if they would have been announced without those filters. Given that philosophy, tracking the timestamp of the last INV message and comparing that against the mempool entry time allows removal of each of `m_recently_announced_invs`, `m_last_mempool_req` and `UNCONDITIONAL_RELAY_DELAY` and associated logic. ACKs for top commit: naumenkogs: ACK fb02ba3c5f5bcd96b5e3622ef001b8e57ce63fc0 amitiuttarwar: review ACK fb02ba3c5f5 glozow: reACK fb02ba3c5f5bcd96b5e3622ef001b8e57ce63fc0 Tree-SHA512: cbba5ee04c86df26b6057f3654c00a2b45ec94d354f4f157a769cecdaa0b509edaac02b3128afba39b023e82473fc5e28c915a787f84457ffe66638c6ac9c2d4
2023-08-03net_processing: drop m_recently_announced_invs bloom filterAnthony Towns
Rather than using a bloom filter to track announced invs, simply allow a peer to request any tx that entered the mempool prior to the last INV message we sent them. This also obsoletes the UNCONDITIONAL_RELAY_DELAY.
2023-07-22refactor: Make more transaction size variables signedHennadii Stepanov
This change gets rid of `static_cast`s and compiler warnings.
2023-06-14refactor: Make m_count_with_* in CTxMemPoolEntry int64_t, drop UBSAN suppMarcoFalke
This is a refactor as long as no signed integer overflow appears. In normal operation and absent bugs, signed integer overflow should never happen in the touched code paths. The main benefit of this refactor is to drop the file-wide ubsan suppression unsigned-integer-overflow:txmempool.cpp. For now, this only changes the internal private representation and the publicly returned type remains uint64_t.
2023-06-12Remove txmempool implicit-integer-sign-change sanitizer suppressionsHennadii Stepanov
2023-06-12Use `int32_t` type for most transaction size/weight valuesHennadii Stepanov
This change gets rid of a few casts and makes the following commit diff smaller.
2023-06-07Merge bitcoin/bitcoin#27501: mempool / rpc: add getprioritisedtransactions, ↵Andrew Chow
delete a mapDeltas entry when delta==0 67b7fecacd0489809690982c89ba2d0acdca938c [mempool] clear mapDeltas entry if prioritisetransaction sets delta to 0 (glozow) c1061acb9d502cdf8c6996c818d9a8a281cbe40c [functional test] prioritisation is not removed during replacement and expiry (glozow) 0e5874f0b06114d9b077e0ff582915e4f83059e6 [functional test] getprioritisedtransactions RPC (glozow) 99f8046829f699ff2eace266aa8cea1d9f7cb65a [rpc] add getprioritisedtransactions (glozow) 9e9ca36c80013749faaf2aa777d52bd07d9d24ec [mempool] add GetPrioritisedTransactions (glozow) Pull request description: Add an RPC to get prioritised transactions (also tells you whether the tx is in mempool or not), helping users clean up `mapDeltas` manually. When `CTxMemPool::PrioritiseTransaction` sets a delta to 0, remove the entry from `mapDeltas`. Motivation / Background - `mapDeltas` entries are never removed from mapDeltas except when the tx is mined in a block or conflicted. - Mostly it is a feature to allow `prioritisetransaction` for a tx that isn't in the mempool {yet, anymore}. A user can may resbumit a tx and it retains its priority, or mark a tx as "definitely accept" before it is seen. - Since #8448, `mapDeltas` is persisted to mempool.dat and loaded on restart. This is also good, otherwise we lose prioritisation on restart. - Note the removal due to block/conflict is only done when `removeForBlock` is called, i.e. when the block is received. If you load a mempool.dat containing `mapDeltas` with transactions that were mined already (e.g. the file was saved prior to the last few blocks), you don't delete them. - Related: #4818 and #6464. - There is no way to query the node for not-in-mempool `mapDeltas`. If you add a priority and forget what the value was, the only way to get that information is to inspect mempool.dat. - Calling `prioritisetransaction` with an inverse value does not remove it from `mapDeltas`, it just sets the value to 0. It disappears on a restart (`LoadMempool` checks if delta is 0), but that might not happen for a while. Added together, if a user calls `prioritisetransaction` very regularly and not all those transactions get mined/conflicted, `mapDeltas` might keep lots of entries of delta=0 around. A user should clean up the not-in-mempool prioritisations, but that's currently difficult without keeping track of what those txids/amounts are. ACKs for top commit: achow101: ACK 67b7fecacd0489809690982c89ba2d0acdca938c theStack: Code-review ACK 67b7fecacd0489809690982c89ba2d0acdca938c instagibbs: code review ACK 67b7fecacd0489809690982c89ba2d0acdca938c ajtowns: ACK 67b7fecacd0489809690982c89ba2d0acdca938c code review only, some nits Tree-SHA512: 9df48b622ef27f33db1a2748f682bb3f16abe8172fcb7ac3c1a3e1654121ffb9b31aeaad5570c4162261f7e2ff5b5912ddc61a1b8beac0e9f346a86f5952260a
2023-05-20refactor: Move system from util to common libraryTheCharlatan
Since the kernel library no longer depends on the system file, move it to the common library instead in accordance to the diagram in doc/design/libraries.md.
2023-05-19Merge bitcoin/bitcoin#27021: Implement Mini version of BlockAssembler to ↵glozow
calculate mining scores 6b605b91c1faf2c7f7cc0c9d39b4fcfd66dc2965 [fuzz] Add MiniMiner target + diff fuzz against BlockAssembler (glozow) 3f3f2d59ea2946a7b7cc8cb0222fb602d62645d0 [unit test] GatherClusters and MiniMiner unit tests (glozow) 59afcc83548ea67a863dac7b75d000bc8f6a7023 Implement Mini version of BlockAssembler to calculate mining scores (glozow) 56484f0fdc44261e723563f59df886d5acdd851f [mempool] find connected mempool entries with GatherClusters(…) (glozow) Pull request description: Implement Mini version of BlockAssembler to calculate mining scores Run the mining algorithm on a subset of the mempool, only disturbing the mempool to copy out fee information for relevant entries. Intended to be used by wallet to calculate amounts needed for fee-bumping unconfirmed transactions. From comments of sipa and glozow below: > > In what way does the code added here differ from the real block assembly code? > > * Only operates on the relevant transactions rather than full mempool > * Has the ability to remove transactions that will be replaced so they don't impact their ancestors > * Does not hold mempool lock outside of the constructor, makes copies of the entries it needs instead (though I'm not sure if this has an effect in practice) > * Doesn't do the sanity checks like keeping weight within max block weight and `IsFinalTx()` > * After the block template is built, additionally calculates fees to bump remaining ancestor packages to target feerate ACKs for top commit: achow101: ACK 6b605b91c1faf2c7f7cc0c9d39b4fcfd66dc2965 Xekyo: > ACK [6b605b9](https://github.com/bitcoin/bitcoin/commit/6b605b91c1faf2c7f7cc0c9d39b4fcfd66dc2965) modulo `miniminer_overlap` test. furszy: ACK 6b605b91 modulo `miniminer_overlap` test. theStack: Code-review ACK 6b605b91c1faf2c7f7cc0c9d39b4fcfd66dc2965 Tree-SHA512: f86a8b4ae0506858a7b15d90f417ebceea5038b395c05c825e3796123ad3b6cb8a98ebb948521316802a4c6d60ebd7041093356b1e2c2922a06b3b96b3b8acb6
2023-05-10[mempool] clear mapDeltas entry if prioritisetransaction sets delta to 0glozow
It's unnecessary to keep the data around, as it doesn't do anything. If prioritisetransaction is called again, we'll make a new entry in mapDeltas. These entries are only deleted when the transaction is mined or conflicted from a block (i.e. not in replacement or eviction), are persisted in mempool.dat, and never expire. If node operators use the RPC to regularly prioritise/de-prioritise transactions, these (meaningless) map entries may hang around forever and take up valuable mempool memory.
2023-05-10[mempool] add GetPrioritisedTransactionsglozow
2023-05-10txmempool: have CompareDepthAndScore sort missing txs firstAnthony Towns
We use CompareDepthAndScore to choose an order of txs to inv. Rather than sorting txs that have been evicted from the mempool at the end of the list, sort them at the beginning so they are removed from the queue immediately.
2023-04-19move-only: Extract common/args and common/config.cpp from util/systemTheCharlatan
This is an extraction of ArgsManager related functions from util/system into their own common file. Config file related functions are moved to common/config.cpp. The background of this commit is an ongoing effort to decouple the libbitcoinkernel library from the ArgsManager. The ArgsManager belongs into the common library, since the kernel library should not depend on it. See doc/design/libraries.md for more information on this rationale.
2023-03-27[mempool] find connected mempool entries with GatherClusters(…)glozow
We limit GatherClusters’s result to a maximum of 500 transactions as clusters can be made arbitrarily large by third parties. Co-authored-by: Murch <murch@murch.one>
2023-03-20Merge bitcoin/bitcoin#26531: mempool: Add mempool tracepointsAndrew Chow
4b7aec2951fe4595946cdc804b0dec1921d79d05 Add mempool tracepoints (virtu) Pull request description: This PR adds multiple mempool tracepoints. | tracepoint | description | | ------------- | ------------- | | `mempool:added` | Is called when a transaction enters the mempool | | `mempool:removed` | ... when a transaction is removed from the mempool | | `mempool:replaced` | ... when a transaction is replaced in the mempool | | `mempool:rejected` | ... when a transaction is rejected from entering the mempool | The tracepoints are further documented in `docs/tracing.md`. Usage is demonstrated in the example script `contrib/tracing/mempool_monitor.py`. Interface tests are provided in `test/functional/interface_usdt_mempool.py`. The rationale for passing the removal reason as a string instead of numerically is that the benefits of not having to maintain a redundant enum-string mapping seem to outweigh the small cost of string generation. The reject reason is passed as string as well, although in this instance the string does not have to be generated but is readily available. ACKs for top commit: 0xB10C: ACK 4b7aec2951fe4595946cdc804b0dec1921d79d05 achow101: ACK 4b7aec2951fe4595946cdc804b0dec1921d79d05 Tree-SHA512: 6deb3ba2d1a061292fb9b0f885f7a5c4d11b109b838102d8a8f4828cd68f5cd03fa3fc64adc6fdf54a08a1eaccce261b0aa90c2b8c33cd5fd3828c8f74978958
2023-03-20Add mempool tracepointsvirtu
Tracepoints for added, removed, replaced, and rejected transactions. The removal reason is passed as string instead of a numeric value, since the benefits of not having to maintain a redundant enum-string mapping seem to outweigh the small cost of string generation. The reject reason is passed as string as well, although here the string does not have to be generated but is readily available. So far, tracepoint PRs typically included two demo scripts: a naive bpftrace script to show raw tracepoint data and a bcc script for a more refined view. However, as some of the ongoing changes to bpftrace introduce a certain degree of unreliability (running some of the existing bpftrace scripts was not possible with standard kernels and bpftrace packages on latest stable Ubuntu, Debian, and NixOS), this PR includes only a single bcc script that fuses the functionality of former bpftrace and bcc scripts.
2023-02-01Fix clang-tidy readability-const-return-type violationsMarcoFalke
2023-01-05refactor: Work around Werror=free-nonheap-object in ↵MarcoFalke
AssumeCalculateMemPoolAncestors
2023-01-04Merge bitcoin/bitcoin#19909: refactor: Remove unused CTxMemPool::clear() helperglozow
fa818e103c0ddb515f29ae9ce8de44931e12e69e txmempool: Remove unused clear() member function (MarcoFalke) Pull request description: Seems odd to have code in Bitcoin Core that is unused. Moreover the function was broken (see https://github.com/bitcoin/bitcoin/pull/24145) and is brittle, as there is nothing that prevents similar bugs from re-appearing. Fix both issues by replacing it with C++11 member initializers. ACKs for top commit: glozow: ACK fa818e103c0ddb515f29ae9ce8de44931e12e69e Tree-SHA512: e79e44cac7d5a84d9ecc8e3f3b0b9a50e1e3ebec358b20ba5dac175ef07d1fbe338a20f83ee80f746f7c726c79e77f8be49e14bca57a41063da8a5302123c3a9