aboutsummaryrefslogtreecommitdiff
path: root/src/net_processing.cpp
AgeCommit message (Collapse)Author
2020-10-08Avoid 'timing mishap' warnings when mockingPieter Wuille
2020-10-08Use mockable time everywhere in net_processingPieter Wuille
2020-10-07Merge #19339: validation: re-delegate absurd fee checking from mempool to ↵fanquake
clients b048b275d9711f70847afaea5450f17a0f7e673a [validation] Remove absurdfee from accepttomempool (John Newbery) 932564b9cfda8446a957649c2316a52e868ad5d4 scripted-diff: update max-fee-exceeded error message to include RPC (gzhao408) 8f1290c60159a3171c27250bc95687548c5c1b84 [rpc/node] check for high fee before ATMP in clients (gzhao408) Pull request description: Picked up from #15810. Add separate fee-checking logic for clients that need to enforce max fee rates, then remove the `absurdFee` logic from ATMP. ATMP's `nAbsurdFee` argument is used to enforce user-specific behavior (it is not policy since it isn't applied consistently: it is only ever used in RPC and wallet, and set to 0 everywhere else internally). It should be removed from `AcceptToMemoryPool` because (1) validation results/mempool behavior should not be user-specific and (2) enforcing a max fee rate should be the responsibility of the client instead of the mempool. Note: this PR does not intend to _remove_ protection from high fees, just re-delegate the responsibility to clients. ACKs for top commit: jnewbery: utACK b048b275d9711f70847afaea5450f17a0f7e673a LarryRuane: re-ACK b048b275d9711f70847afaea5450f17a0f7e673a MarcoFalke: re-ACK b048b275d9 , only change is squashing one commit 🏦 instagibbs: utACK https://github.com/bitcoin/bitcoin/pull/19339/commits/b048b275d9711f70847afaea5450f17a0f7e673a Tree-SHA512: 57c17ba16d230a4cae2896dd6a64c924f307757824e35784bf96da7b10aff2d8ea910710edf35e981035623a155f8766209a92a0fdb856549fde78bc3eaae4d2
2020-10-05[validation] Remove absurdfee from accepttomempoolJohn Newbery
Mempool behavior should not be user-specific. Checking that txfee is acceptable should be the responsibility of the wallet or client, not the mempool.
2020-10-04Merge #19723: Ignore unknown messages before VERACKMarcoFalke
675e55e01392971aa56bda56cb09498b466d0902 Ignore unknown messages before VERACK (Suhas Daftuar) Pull request description: This allows for feature negotiation to take place with messages between VERSION and VERACK in the future, without requiring additional software changes to specifically ignore messages for features that are unimplemented by our software. ACKs for top commit: sipa: utACK 675e55e01392971aa56bda56cb09498b466d0902 practicalswift: ACK 675e55e01392971aa56bda56cb09498b466d0902: patch looks correct MarcoFalke: ACK 675e55e01392971aa56bda56cb09498b466d0902 hebasto: ACK 675e55e01392971aa56bda56cb09498b466d0902, the offender peer will be eventually disconnected due to the timeout. Tree-SHA512: 8d2b1d8b9843f2ee26b2c30f7c5ff0bfcfbe3f46b32cd0369c48ece26624151091237e83ce3f18c6da004099026602cfab1642ac916db777f047d170b365c007
2020-10-02Merge #19871: doc: Clarify scope of eviction protection of outbound ↵Wladimir J. van der Laan
block-relay peers d76925478efd35e6fd835370639f2139b28381e4 [doc] Clarify semantic of peer's m_protect w.r.t to outbound eviction logics (Antoine Riard) ac71fe936da290adf5a3155fe8db5f78b485f1f1 [doc] Clarify scope of eviction protection of outbound block-relay peers (Antoine Riard) Pull request description: Block-relay-only peers were introduced by #15759. According to its author, it was intented to make them only immune to outbound peer rotation-based eviction and not from all eviction as modified comment leans to think of. Clearly indicate that outbound block-relay peers aren't protected from eviction by the bad/lagging chain logic. Fix #19863 ACKs for top commit: naumenkogs: ACK d76925478efd35e6fd835370639f2139b28381e4 jonatack: ACK d76925478efd35e6fd835370639f2139b28381e4 Tree-SHA512: 597fbd62838a6e39276024165b11514cad20a2e9d33cf9202d261cbadcb62b2df427c858e0cb57e585840d4c1d4600104aa53916bb868541f2580e4eed9b4b52
2020-09-30Merge #19498: Tidy up ProcessOrphanTxMarcoFalke
001343f4bc8b22fa9e313bd2867756eb9d614fa3 ProcessOrphanTx: Move AddToCompactExtraTransactions call into ProcessOrphanTx (John Newbery) 4fce726bd1e35a686cd9d48add5da22b1b5e25e1 ProcessOrphanTx: Remove aliases (John Newbery) e07c5d94231cefb748f9534ab8ff0b3e2b04c4d8 ProcessOrphanTx: Remove outdated commented (John Newbery) 4763b51bca86fb9e49175619a47cdbef34feaf99 ProcessOrphanTx: remove useless setMisbehaving set (John Newbery) 55c79a9cefb6c83cdebbf6c538c471607695b457 ProcessOrphanTx: remove useless done variable (John Newbery) 6e8dd99ef1c147898bd06fee7014afdff6618f18 [net processing] Add doxygen comments for orphan data and function (John Newbery) Pull request description: Originally a follow-up to #19364, this simplifies the logic in ProcessOrphanTx() and removes unused variables. ACKs for top commit: troygiorshev: ACK 001343f4bc8b22fa9e313bd2867756eb9d614fa3 sipa: utACK 001343f4bc8b22fa9e313bd2867756eb9d614fa3 MarcoFalke: ACK 001343f4bc8b22fa9e313bd2867756eb9d614fa3 🌮 Tree-SHA512: be558457f2e08ebb6bddcd49bdd75bd410c3650da44a76c688fc9f07822f94d5a1af93fa1342678052b2c8163cdb9745c352c7884325ab0a41fa593c3eb89116
2020-09-29Merge #19107: p2p: Move all header verification into the network layer, ↵fanquake
extend logging deb52711a17236d0fca302701b5af585341ab42a Remove header checks out of net_processing (Troy Giorshev) 52d4ae46ab822d0f54e246a6f2364415cda149bd Give V1TransportDeserializer CChainParams& member (Troy Giorshev) 5bceef6b12fa16d20287693be377dace3dfec3e5 Change CMessageHeader Constructor (Troy Giorshev) 1ca20c1af8f08f07c407c3183c37b467ddf0f413 Add doxygen comment for ReceiveMsgBytes (Troy Giorshev) 890b1d7c2b8312d41d048d2db124586c5dbc8a49 Move checksum check from net_processing to net (Troy Giorshev) 2716647ebf60cea05fc9edce6a18dcce4e7727ad Give V1TransportDeserializer an m_node_id member (Troy Giorshev) Pull request description: Inspired by #15206 and #15197, this PR moves all message header verification from the message processing layer and into the network/transport layer. In the previous PRs there is a change in behavior, where we would disconnect from peers upon a single failed checksum check. In various discussions there was concern over whether this was the right choice, and some expressed a desire to see how this would look if it was made to be a pure refactor. For more context, see https://bitcoincore.reviews/15206.html#l-81. This PR improves the separation between the p2p layers, helping improvements like [BIP324](https://github.com/bitcoin/bitcoin/pull/18242) and #18989. ACKs for top commit: ryanofsky: Code review ACK deb52711a17236d0fca302701b5af585341ab42a just rebase due to conflict on adjacent line jnewbery: Code review ACK deb52711a17236d0fca302701b5af585341ab42a. Tree-SHA512: 1a3b7ae883b020cfee1bef968813e04df651ffdad9dd961a826bd80654f2c98676ce7f4721038a1b78d8790e4cebe8060419e3d8affc97ce2b9b4e4b72e6fa9f
2020-09-26Merge #19725: [RPC] Add connection type to getpeerinfo, improve logsMarcoFalke
a512925e19a70d7f6b80ac530a169f45ffaafa1c [doc] Release notes (Amiti Uttarwar) 50f94b34a33c954f6e207f509c93d33267a5c3e2 [rpc] Deprecate getpeerinfo addnode field (Amiti Uttarwar) df091b9b509f0b10e4315c0bfa2da0cc0c31c22f [refactor] Rename test file to allow any getpeerinfo deprecations. (Amiti Uttarwar) 395acfa83a5436790c1a722a5609ac9d48df235f [rpc] Add connection type to getpeerinfo RPC, update tests (Amiti Uttarwar) 49c10a9ca40967d28ae16dfea9cccc6f3a6624a1 [log] Add connection type to log statement (Amiti Uttarwar) Pull request description: After #19316, we can more directly expose information about the connection type on the `getpeerinfo` RPC. Doing so also makes the existing addnode field redundant, so this PR begins the process of deprecating this field. This PR also includes one commit that improves a log message, as both use a shared function to return the connection type as a string. Suggested by sdaftuar- https://github.com/bitcoin/bitcoin/pull/19316#discussion_r468001604 & https://github.com/bitcoin/bitcoin/pull/19316#discussion_r468018093 ACKs for top commit: jnewbery: Code review ACK a512925e19a70d7f6b80ac530a169f45ffaafa1c. sipa: utACK a512925e19a70d7f6b80ac530a169f45ffaafa1c guggero: Tested and code review ACK a512925e. MarcoFalke: cr ACK a512925e19a70d7f6b80ac530a169f45ffaafa1c 🌇 promag: Code review ACK a512925e19a70d7f6b80ac530a169f45ffaafa1c. Tree-SHA512: 601a7a38aee235ee59aca690784f886dc2ae4e418b2e6422c4b58cd597376c00f74910f66920b08a08a0bec28bf8022e71a1435785ff6ba8a188954261aba78e
2020-09-23Merge #19979: Replace LockAssertion with AssertLockHeld, remove LockAssertionMarcoFalke
0bd1184adf6610c0bd14f4e9a25c0a200e65ae25 Remove unused LockAssertion struct (Hennadii Stepanov) ab2a44297fd0796bf5797ae2a477e8e56d9c3c12 Replace LockAssertion with a proper thread safety annotations (Hennadii Stepanov) 73f71e19965e07534eb47701f2b23c9ed59ef475 refactor: Use explicit function type instead of template (Hennadii Stepanov) Pull request description: This PR replaces `LockAssertion` with `AssertLockHeld`, and removes `LockAssertion`. This PR is compared with alternatives in https://github.com/bitcoin-core/bitcoin-devwiki/wiki/AssertLockHeld-PRs ACKs for top commit: MarcoFalke: ACK 0bd1184adf6610c0bd14f4e9a25c0a200e65ae25 ajtowns: ACK 0bd1184adf6610c0bd14f4e9a25c0a200e65ae25 vasild: ACK 0bd1184ad Tree-SHA512: ef7780dd689faf0bb479fdb97c49bc652e2dd10c148234bb95502dfbb676442d8565ee37864d923ca21a25f9dc2a335bf46ee82c095e387b59a664ab05c0ae41
2020-09-22Remove header checks out of net_processingTroy Giorshev
This moves header size and netmagic checking out of net_processing and into net. This check now runs in ReadHeader, so that net can exit early out of receiving bytes from the peer. IsValid is now slimmed down, so it no longer needs a MessageStartChars& parameter. Additionally this removes the rest of the m_valid_* members from CNetMessage.
2020-09-22Move checksum check from net_processing to netTroy Giorshev
This removes the m_valid_checksum member from CNetMessage. Instead, GetMessage() returns an Optional. Additionally, GetMessage() has been given an out parameter to be used to hold error information. For now it is specifically a uint32_t used to hold the raw size of the corrupt message. The checksum check is now done in GetMessage.
2020-09-21[log] Add connection type to log statementAmiti Uttarwar
In addition to adding more specificity to the log statement about the type of connection, this change also consolidates two statements into one. Previously, the second one should have never been hit, since block-relay connections would match the "!IsInboundConn()" condition and return early.
2020-09-22Merge #17785: p2p: Unify Send and Receive protocol versionsWladimir J. van der Laan
ddefb5c0b759950942ac03f28c43b548af7b4033 p2p: Use the greatest common version in peer logic (Hennadii Stepanov) e084d45562b94827b3a7873895882fcaae9f4d48 p2p: Remove SetCommonVersion() from VERACK handler (Hennadii Stepanov) 8d2026796a6f7add0c2cda9806e759817d1eae6f refactor: Rename local variable nSendVersion (Hennadii Stepanov) e9a6d8b13b0558b17cdafbd32fd2663b4138ff11 p2p: Unify Send and Receive protocol versions (Hennadii Stepanov) Pull request description: On master (6fef85bfa3cd7f76e83b8b57f9e4acd63eb664ec) `CNode` has two members to keep protocol version: - `nRecvVersion` for received messages - `nSendVersion` for messages to send After exchanging with `VERSION` and `VERACK` messages via protocol version `INIT_PROTO_VERSION`, both nodes set `nRecvVersion` _and_ `nSendVersion` to _the same_ value which is the greatest common protocol version. This PR: - replaces two `CNode` members, `nRecvVersion` `nSendVersion`, with `m_greatest_common_version` - removes duplicated getter and setter There is no change in behavior on the P2P network. ACKs for top commit: jnewbery: ACK ddefb5c0b759950942ac03f28c43b548af7b4033 naumenkogs: ACK ddefb5c0b759950942ac03f28c43b548af7b4033 fjahr: Code review ACK ddefb5c0b759950942ac03f28c43b548af7b4033 amitiuttarwar: code review but untested ACK ddefb5c0b7 benthecarman: utACK `ddefb5c` Tree-SHA512: 5305538dbaa5426b923b0afd20bdef4f248d310855d1d78427210c00716c67b7cb691515c421716b6157913e453076e293b10ff5fd2cd26a8e5375d42da7809d
2020-09-21Merge #19697: Improvements on ADDR cachingWladimir J. van der Laan
0d04784af151de249bbbcbad51e6e8ad9af8f5a3 Refactor the functional test (Gleb Naumenko) 83ad65f31b5c9441ae1618614082e584854a14e1 Address nits in ADDR caching (Gleb Naumenko) 81b00f87800f40cb14f2131ff27668bd2bb9e551 Add indexing ADDR cache by local socket addr (Gleb Naumenko) 42ec5585424ceb91bed07826dde15697c020661a Justify the choice of ADDR cache lifetime (Gleb Naumenko) Pull request description: This is a follow-up on #18991 which does 3 things: - improves privacy of a node listening to multiple addresses via adding cache index by local socket addr (suggested [here](https://github.com/bitcoin/bitcoin/pull/18991#issuecomment-668219345)) - documents on the choice of 24h cache lifetime - addresses nits from #18991 ACKs for top commit: jnewbery: utACK 0d04784af151de249bbbcbad51e6e8ad9af8f5a3 vasild: ACK 0d04784 jonatack: Code review ACK 0d04784 Tree-SHA512: bb65a34dd1ce2811186d3e4469bc33e8399cebaaa494ce13041c7cff23275870e4176a719f7a72f8d779c49f8b2344bf4fa1aeb3ea4e2626d5ae76514f00a750
2020-09-19Replace LockAssertion with a proper thread safety annotationsHennadii Stepanov
2020-09-16Merge #19879: [p2p] miscellaneous wtxid followupsfanquake
a8a64acaf32ac21feeb885671772282b531ef9a2 [BroadcastTransaction] Remove unsafe move operator (Amiti Uttarwar) 125c0381266e0e05a408f8e1818501ab73d29110 [p2p] Remove dead code (Amiti Uttarwar) fc66d0a65cdc52a3b259effe0c29b5eafb1b5ff5 [p2p] Check for nullptr before dereferencing pointer (Adam Jonas) cb79b9dbf4cd06e17c8c65b36bf15c3ea2641de4 [mempool] Revert unbroadcast set to tracking just txid (Amiti Uttarwar) Pull request description: Addresses some outstanding review comments from #18044 - reverts unbroadcast txids to a set instead of a map (simpler, communicates intent better, takes less space, no efficiency advantages of map) - adds safety around two touchpoints (check for nullptr before dereferencing pointer, remove an inaccurate std::move operator) - removes some dead code Links to comments on wtxid PR: [1](https://github.com/bitcoin/bitcoin/pull/18044#discussion_r460495254) [2](https://github.com/bitcoin/bitcoin/pull/18044#discussion_r460496023) [3](https://github.com/bitcoin/bitcoin/pull/18044#discussion_r463532611) thanks to jnewbery & adamjonas for flagging these ! ! ACKs for top commit: sdaftuar: utACK a8a64acaf32ac21feeb885671772282b531ef9a2 naumenkogs: utACK a8a64acaf32ac21feeb885671772282b531ef9a2 jnewbery: utACK a8a64acaf32ac21feeb885671772282b531ef9a2 Tree-SHA512: 7be669cb30cc17fb9e06b50e636ef7887c6a27354697987e4e4d38dba4b8f50e175647587430cd9bc3295bec01ce8b1e6639a50a4249d8fff9b1ca1b9ead3277
2020-09-10[doc] Clarify semantic of peer's m_protect w.r.t to outbound eviction logicsAntoine Riard
The field m_protect is used to protect from eviction both by bad/lagging chain and extra outbound peers logics. Outbound block-relay peers are always excluded from this protection.
2020-09-08Do not pass chain params to CheckForStaleTipAndEvictPeers twiceMarcoFalke
2020-09-07ProcessOrphanTx: Move AddToCompactExtraTransactions call into ProcessOrphanTxJohn Newbery
2020-09-07ProcessOrphanTx: Remove aliasesJohn Newbery
2020-09-07ProcessOrphanTx: Remove outdated commentedJohn Newbery
Also rename orphan_state to state. Both the comment and the variable name are leftover from when this logic was part of ProcessMessage().
2020-09-07ProcessOrphanTx: remove useless setMisbehaving setJohn Newbery
This starts empty, and is only added to if we're about to exit the function (so we never read from it).
2020-09-07ProcessOrphanTx: remove useless done variableJohn Newbery
There is a keyword that allows us to break out of loops. Use it. There's a small change in behaviour here: if we process multiple orphans that are still orphans, then we'll only call mempool.check() once at the end, instead of after processing each tx.
2020-09-07[net processing] Add doxygen comments for orphan data and functionJohn Newbery
2020-09-07p2p: Use the greatest common version in peer logicHennadii Stepanov
2020-09-07p2p: Remove SetCommonVersion() from VERACK handlerHennadii Stepanov
SetCommonVersion() is already called from the VERSION message handler. There is no change in behavior on the P2P network.
2020-09-07refactor: Rename local variable nSendVersionHennadii Stepanov
2020-09-07p2p: Unify Send and Receive protocol versionsHennadii Stepanov
There is no change in behavior on the P2P network.
2020-09-07Merge #19791: [net processing] Move Misbehaving() to PeerManagerMarcoFalke
bb6a32ce9983c72afa90f41a43a47ffd703ca006 [net processing] Move Misbehaving() to PeerManager (John Newbery) aa114b1c9b06c2bd3ed936bbb9fb32b31f75bdb2 [net_processing] Move SendBlockTransactions into PeerManager (John Newbery) 3115e00f75b41d9765dcbb376e367b25f61a1d58 [net processing] Move MaybePunishPeerForTx to PeerManager (John Newbery) e662e2d42afaf9c67c898634a0f3bc200255b6ea [net processing] Move ProcessOrphanTx to PeerManager (John Newbery) b70cd890e375e904b7f36b3d959e5656f5a5cbcd [net processing] Move MaybePunishNodeForBlock into PeerManager (John Newbery) d7778351bf60a21925a97b7fc4e9df5541b6d995 [net processing] Move ProcessHeadersMessage to PeerManager (John Newbery) 64f6162651420be2f4aa1498f0378a86780bc089 [whitespace] tidy up indentation after scripted diff (John Newbery) 58bd369b0ddd3383f7bdf7840912d18b96545f91 scripted-diff: [net processing] Rename PeerLogicValidation to PeerManager (John Newbery) 2297b26b3ce95e935c0ebb8c38dabf19965054a5 [net_processing] Pass chainparams to PeerLogicValidation constructor (John Newbery) 824bbd1ffba3df7ffa6f5bfaa31298cd484473b1 [move only] Collect all private members of PeerLogicValidation together (John Newbery) Pull request description: Continues the work of moving net_processing logic into PeerLogicValidation. See https://github.com/bitcoin/bitcoin/pull/19704 and https://github.com/bitcoin/bitcoin/pull/19607#discussion_r462032894 for motivation. This PR also renames `PeerLogicValidation` to `PeerManager` as suggested in https://github.com/bitcoin/bitcoin/pull/10756#pullrequestreview-53892618. ACKs for top commit: MarcoFalke: re-ACK bb6a32ce99 only change is rebase due to conflict in struct NodeContext and variable rename 🤸 hebasto: re-ACK bb6a32ce9983c72afa90f41a43a47ffd703ca006, only rebased, and added renaming `s/peer_logic/peerman/` into scripted-diff since my [previous](https://github.com/bitcoin/bitcoin/pull/19791#pullrequestreview-483118079) review (verified with `git range-diff`). Tree-SHA512: a2de4a521688fd25125b401e5575402c52b328a0fa27b3010567008d4f596b960aabbd02b2d81f42658f88f4365443fadb1008150a62fbcea123fb42d85a2c21
2020-09-07[doc] Clarify scope of eviction protection of outbound block-relay peersAntoine Riard
Block-relay-only peers were introduced by #15759. According to its author, it was intented to make them only immune to outbound peer rotation-based eviction and not from all eviction as modified comment leans to think of. Clearly indicate that outbound block-relay peers aren't protected from eviction by the bad/lagging chain logic.
2020-09-07[net processing] Move Misbehaving() to PeerManagerJohn Newbery
2020-09-07[net_processing] Move SendBlockTransactions into PeerManagerJohn Newbery
2020-09-07[net processing] Move MaybePunishPeerForTx to PeerManagerJohn Newbery
2020-09-07[net processing] Move ProcessOrphanTx to PeerManagerJohn Newbery
2020-09-07[net processing] Move MaybePunishNodeForBlock into PeerManagerJohn Newbery
2020-09-07[net processing] Move ProcessHeadersMessage to PeerManagerJohn Newbery
2020-09-07[whitespace] tidy up indentation after scripted diffJohn Newbery
2020-09-07scripted-diff: [net processing] Rename PeerLogicValidation to PeerManagerJohn Newbery
-BEGIN VERIFY SCRIPT- sed -i 's/PeerLogicValidation/PeerManager/g' $(git grep -l PeerLogicValidation ./src ./test) sed -i 's/peer_logic/peerman/g' $(git grep -l peer_logic ./src ./test) -END VERIFY SCRIPT- PeerLogicValidation was originally net_processing's implementation to the validation interface. It has since grown to contain much of net_processing's logic. Therefore rename it to reflect its responsibilities. Suggested in https://github.com/bitcoin/bitcoin/pull/10756#pullrequestreview-53892618.
2020-09-07[net_processing] Pass chainparams to PeerLogicValidation constructorJohn Newbery
Keep a references to chainparams, rather than calling the global Params() function every time it's needed. This is fine, since globalChainParams does not get updated once it's been set, and it's available at the point of constructing the PeerLogicValidation object.
2020-09-07[move only] Collect all private members of PeerLogicValidation togetherJohn Newbery
We don't have a project style for ordering class members, but it always makes sense to have no more than one of each public/protected/private specifier. Also move documentation for MaybeDiscourageAndDisconnect to the header.
2020-09-07Merge #19478: Remove CTxMempool::mapLinks data structure memberMarcoFalke
296be8f58e02b39a58f017c52294aceed22c3ffd Get rid of unused functions CTxMemPool::GetMemPoolChildren, CTxMemPool::GetMemPoolParents (Jeremy Rubin) 46d955d196043cc297834baeebce31ff778dff80 Remove mapLinks in favor of entry inlined structs with iterator type erasure (Jeremy Rubin) Pull request description: Currently we have a peculiar data structure in the mempool called maplinks. Maplinks job is to track the in-pool children and parents of each transaction. This PR can be primarily understood and reviewed as a simple refactoring to remove this extra data structure, although it comes with a nice memory and performance improvement for free. Maplinks is particularly peculiar because removing it is not as simple as just moving it's inner structure to the owning CTxMempoolEntry. Because TxLinks (the class storing the setEntries for parents and children) store txiters to each entry in the mempool corresponding to the parent or child, it means that the TxLinks type is "aware" of the boost multiindex (mapTx) it's coming from, which is in turn, aware of the entry type stored in mapTx. Thus we used maplinks to store this entry associated data we in an entirely separate data structure just to avoid a circular type reference caused by storing a txiter inside a CTxMempoolEntry. It turns out, we can kill this circular reference by making use of iterator_to multiindex function and std::reference_wrapper. This allows us to get rid of the maplinks data structure and move the ownership of the parents/child sets to the entries themselves. The benefit of this good all around, for any of the reasons given below the change would be acceptable, and it doesn't make the code harder to reason about or worse in any respect (as far as I can tell, there's no tradeoff). ### Simpler ownership model No longer having to consistency check that mapLinks did have records for our CTxMempoolEntry, impossible to have a mapLinks entry outlive or incorrectly die before a CTxMempoolEntry. ### Memory Usage We get rid of a O(Transactions) sized map in the mempool, which is a long lived data structure. ### Performance If you have a CTxMemPoolEntry, you immediately know the address of it's children/parents, rather than having to do a O(log(Transactions)) lookup via maplinks (which we do very often). We do it in *so many* places that a true benchmark has to look at a full running node, but it is easy enough to show an improvement in this case. The ComplexMemPool shows a good coherence check that we see the expected result of it being 12.5% faster / 1.14x faster. ``` Before: # Benchmark, evals, iterations, total, min, max, median ComplexMemPool, 5, 1, 1.40462, 0.277222, 0.285339, 0.279793 After: # Benchmark, evals, iterations, total, min, max, median ComplexMemPool, 5, 1, 1.22586, 0.243831, 0.247076, 0.244596 ``` The ComplexMemPool benchmark only checks doing addUnchecked and TrimToSize for 800 transactions. While this bench does a good job of hammering the relevant types of function, it doesn't test everything. Subbing in 5000 transactions shows a that the advantage isn't completely wiped out by other asymptotic factors (this isn't the only bottleneck in growing the mempool), but it's only a bit proportionally slower (10.8%, 1.12x), which adds evidence that this will be a good change for performance minded users. ``` # Benchmark, evals, iterations, total, min, max, median ComplexMemPool, 5, 1, 59.1321, 11.5919, 12.235, 11.7068 # Benchmark, evals, iterations, total, min, max, median ComplexMemPool, 5, 1, 52.1307, 10.2641, 10.5206, 10.4306 ``` I don't think it's possible to come up with an example of where a maplinks based design would have better performance, but it's something for reviewers to consider. # Discussion ## Why maplinks in the first place? I spoke with the author of mapLinks (sdaftuar) a while back, and my recollection from our conversation was that it was implemented because he did not know how to resolve the circular dependency at the time, and there was no other reason for making it a separate map. ## Is iterator_to weird? iterator_to is expressly for this purpose, see https://www.boost.org/doc/libs/1_51_0/libs/multi_index/doc/tutorial/indices.html#iterator_to > iterator_to provides a way to retrieve an iterator to an element from a pointer to the element, thus making iterators and pointers interchangeable for the purposes of element pointing (not so for traversal) in many situations. This notwithstanding, it is not the aim of iterator_to to promote the usage of pointers as substitutes for real iterators: the latter are specifically designed for handling the elements of a container, and not only benefit from the iterator orientation of container interfaces, but are also capable of exposing many more programming bugs than raw pointers, both at compile and run time. iterator_to is thus meant to be used in scenarios where access via iterators is not suitable or desireable: > > - Interoperability with preexisting APIs based on pointers or references. > - Publication of pointer-based interfaces (for instance, when designing a C-compatible library). > - The exposure of pointers in place of iterators can act as a type erasure barrier effectively decoupling the user of the code from the implementation detail of which particular container is being used. Similar techniques, like the famous Pimpl idiom, are used in large projects to reduce dependencies and build times. > - Self-referencing contexts where an element acts upon its owner container and no iterator to itself is available. In other words, iterator_to is the perfect tool for the job by the last reason given. Under the hood it should just be a simple pointer cast and have no major runtime overhead (depending on if the function call is inlined). Edit by laanwj: removed at sign from the description ACKs for top commit: jonatack: re-ACK 296be8f per `git range-diff ab338a19 3ba1665 296be8f`, sanity check gcc 10.2 debug build is clean. hebasto: re-ACK 296be8f58e02b39a58f017c52294aceed22c3ffd, only rebased since my [previous](https://github.com/bitcoin/bitcoin/pull/19478#pullrequestreview-482400727) review (verified with `git range-diff`). Tree-SHA512: f5c30a4936fcde6ae32a02823c303b3568a747c2681d11f87df88a149f984a6d3b4c81f391859afbeb68864ef7f6a3d8779f74a58e3de701b3d51f78e498682e
2020-09-05Remove mempool global from p2pMarcoFalke
2020-09-05Merge #19728: Increase the ip address relay branching factor for unreachable ↵Wladimir J. van der Laan
networks 86d4cf42d97abf4c436d1eabf29e2ed150f69c1e Increase the ip address relay branching factor for unreachable networks (Pieter Wuille) Pull request description: Onion addresses propagate very badly among the IPv4/IPv6 network, resulting in difficulty for those to find each other. The branching factor 1 is probably so low that propagations die out before they reach another onion peer. Increase it to 1.5 on average. ACKs for top commit: practicalswift: ACK 86d4cf42d97abf4c436d1eabf29e2ed150f69c1e -- patch looks correct naumenkogs: ACK 86d4cf4 jonatack: ACK 86d4cf42d97abf4c436d1eabf29e2ed150f69c1e. Code review, built and running with some sanity check logging. `RelayAddress()` is called by `ProcessMessage() ADDR` msg handling, from within the loop while processing each new address to relay it to a limited number of other nodes. According to git blame, the line setting `nRelayNodes` hasn't been touched since 2016 in e736772c56a *Move network-msg-processing code out of main to its own file*, which moved the line but otherwise did not change it. Running a mixed clearnet/onion node with this patch and the logging below, I'm only seeing values of `fReachable 1, nRelayNodes 2`. IIUC, I need to use the settings in `init.cpp` that call `SetReachable(*, false)`. *Edit:* with `onlynet=onion` am now seeing entries of `fReachable 0` with `nRelayNodes` values of 1 and 2. vasild: ACK 86d4cf42d Tree-SHA512: 22391e16d60bcfdec9a9336728da39d68a24a183b3d1b0e8fbc038d265ca6ddf71d16db018f3678745fd9f3e9281049e42197fa0a29124833c50a9170ed6f793
2020-09-04[p2p] Remove dead codeAmiti Uttarwar
The else clause is dead code because the only way to not enter the if branch is if TX_WITNESS_STRIPPED is true. In that case, it would not have a witness to match the `tx.HasWitness()` else condition. Co-authored-by: Adam Jonas <jonas@chaincode.com> Co-authored-by: John Newbery <john@johnnewbery.com>
2020-09-04[p2p] Check for nullptr before dereferencing pointerAdam Jonas
2020-09-04[mempool] Revert unbroadcast set to tracking just txidAmiti Uttarwar
When I originally implemented the unbroadcast set in 18038, it just tracked txids. After 18038 was merged, I offered a patch to 18044 to make the unbroadcast changes compatible with wtxid relay. In this patch, I updated `unbroadcast_txids` to a map of txid -> wtxid. Post merge review comments shed light on the fact that this update was unnecessary, and distracting. So, this commit updates the unbroadcast ids back to a set.
2020-09-04Remove mapLinks in favor of entry inlined structs with iterator type erasureJeremy Rubin
2020-09-02[refactor] Simplify check for block-relay-only connection.Amiti Uttarwar
Previously we deduced it was a block-relay-only based on presence of the m_tx_relay structure. Now we have the ability to identify it directly via a connection type accessor function.
2020-09-02[trivial] Small style updatesAmiti Uttarwar