aboutsummaryrefslogtreecommitdiff
AgeCommit message (Collapse)Author
2020-09-14build: AX_PTHREAD serial 27fanquake
2020-09-14build: split PTHREAD_* flags out of AM_LDFLAGSfanquake
Note that with this change we are no-longer including PTHREAD_* flags when building libbitcoinconsensus. Also note that we are including PTHREAD_LIBS in AM_PTHREAD_FLAGS
2020-09-14scripted-diff: add FUZZ_SUITE_LDFLAGS_COMMONfanquake
-BEGIN VERIFY SCRIPT- sed -i -e 's/\$(RELDFLAGS) \$(AM_LDFLAGS) \$(LIBTOOL_APP_LDFLAGS)$/\$(FUZZ_SUITE_LDFLAGS_COMMON)/' src/Makefile.test.include patch -p1 << "EOF" --- a/src/Makefile.test.include +++ b/src/Makefile.test.include @@ -323,6 +323,8 @@ endif if ENABLE_FUZZ +FUZZ_SUITE_LDFLAGS_COMMON = $(RELDFLAGS) $(AM_LDFLAGS) $(LIBTOOL_APP_LDFLAGS) + test_fuzz_addition_overflow_CPPFLAGS = $(AM_CPPFLAGS) $(BITCOIN_INCLUDES) test_fuzz_addition_overflow_CXXFLAGS = $(AM_CXXFLAGS) $(PIE_FLAGS) test_fuzz_addition_overflow_LDADD = $(FUZZ_SUITE_LD_COMMON) EOF -END VERIFY SCRIPT-
2020-09-14build: add PTHREAD_LIBS to LDFLAGS configure outputfanquake
Also moves $PTHREAD_CFLAGS to the CFLAGS.
2020-09-14Merge #19944: Update secp256k1 subtree (including BIP340 support)fanquake
b9c1a7648131c5deec9704ee9acd00ec1820b9ce Squashed 'src/secp256k1/' changes from 2ed54da18a..8ab24e8dad (Pieter Wuille) Pull request description: This updates our src/secp256k1 subtree to the latest libsecp256k1 upstream version. As it adds BIP340 support (see https://github.com/bitcoin-core/secp256k1/pull/558), this is a prerequisite for #17977. In particular, it contains: * A few generic library improvements * Support for x-only public keys as used by BIP340. * Support for "key pair" objects, making signing more efficient by using a precomputed public key. * Signing support for BIP340 Schnorr (single-party) signatures. * Verification support for BIP340 Schnorr signatures. * Support for verifying tweaked x-only keys, as used by BIP341's Taproot construction. Things that are not included: * MuSig, nor any kind of multisignatures, threshold signatures, ... on top. * Batch verification. * Support for variable-length messages in BIP340 (which are still being discussed, but won't affect BIP341, or Bitcoin Core). * A few more generic improvements that are still in the pipeline, including faster modular inversions. ACKs for top commit: instagibbs: ACK 894fb33f4c1b24667891f7d2aff9f486177b1173 fanquake: ACK 894fb33f4c1b24667891f7d2aff9f486177b1173. Any Valgrind concerns will be addressed upstream, see discussion in https://github.com/bitcoin-core/secp256k1/pull/813, and if necessary, can be pulled into our tree prior to the 0.21.0 branch off. They are not a blocker for merging this PR in it's current state. benthecarman: ACK `894fb33` Tree-SHA512: 6dc992f4477069b7fbd223316f1be955750923be1479c38adad2312649fdca1f316edb375c42ef9d97cea2407caaef49fb8c93abd6c037fe1a522910cbbc2479
2020-09-13Merge #19919: bugfix: make LoadWallet assigns status alwaysSamuel Dobson
8b39a875581bed1c2f40a7d9616bdb7cc642bf59 bugfix: make LoadWallet assigns status always (Akio Nakamura) Pull request description: In my enviroment, ```test/functional/wallet_multiwallet.py``` failed in line 237 for master( 147d50d63 ). It got an expected rpc-error-message, but error code was not (-4) but (-18). This is because that although loadwallet() in rpcwallet.cpp assumes LoadWallet() always assign some value to the 'status', but LoadWallet() does not do so in some situation. This PR intends to fix above and prevends loadwallet() returns ambiguous error code. ACKs for top commit: hebasto: re-ACK 8b39a875581bed1c2f40a7d9616bdb7cc642bf59, that is the same as 1728059730abef04f3fa84de0b6e20044be7a9d6. ryanofsky: Code review ACK 8b39a875581bed1c2f40a7d9616bdb7cc642bf59 (same as previous) meshcollider: utACK 8b39a875581bed1c2f40a7d9616bdb7cc642bf59 Tree-SHA512: a75d8240f60325bfdb69a07d392269fec97de743f38fe108371eb63a0aba5d8ce3cc484ecc69e81febf8040f5ab64f3a9450b98f8e07a0c17803784bb6f342bf
2020-09-11Update src/secp256k1 subtree to upstream libsecp256k1Pieter Wuille
2020-09-11Squashed 'src/secp256k1/' changes from 2ed54da18a..8ab24e8dadPieter Wuille
8ab24e8dad Merge #558: Add schnorrsig module which implements BIP-340 compliant signatures f3733c5433 Merge #797: Fix Jacobi benchmarks and other benchmark improvements cb5524adc5 Add benchmark for secp256k1_ge_set_gej_var 5c6af60ec5 Make jacobi benchmarks vary inputs d0fdd5f009 Randomize the Z coordinates in bench_internal c7a3424c5f Rename bench_internal variables 875d68b95f Merge #699: Initialize field elements when resulting in infinity 54caf2e74f Merge #799: Add fallback LE/BE for architectures with known endianness + SHA256 selftest f431b3f28a valgrind_ctime_test: Add schnorrsig_sign 16ffa9d97c schnorrsig: Add taproot test case 8dfd53ee3f schnorrsig: Add benchmark for sign and verify 4e43520026 schnorrsig: Add BIP-340 compatible signing and verification 7332d2db6b schnorrsig: Add BIP-340 nonce function 7a703fd97d schnorrsig: Init empty experimental module eabd9bc46a Allow initializing tagged sha256 6fcb5b845d extrakeys: Add keypair_xonly_tweak_add 58254463f9 extrakeys: Add keypair struct with create, pub and pub_xonly f0010349b8 Separate helper functions for pubkey_create and seckey_tweak_add 910d9c284c extrakeys: Add xonly_pubkey_tweak_add & xonly_pubkey_tweak_add_test 176bfb1110 Separate helper function for ec_pubkey_tweak_add 4cd2ee474d extrakeys: Add xonly_pubkey with serialize, parse and from_pubkey f49c9896b0 Merge #806: Trivial: Add test logs to gitignore aabf00c155 Merge #648: Prevent ints from wrapping around in scratch space functions f5adab16a9 Merge #805: Remove the extremely outdated TODO file. bceefd6547 Add test logs to gitignore 1c325199d5 Remove the extremely outdated TODO file. 47e6618e11 extrakeys: Init empty experimental module 3e08b02e2a Make the secp256k1_declassify argument constant 8bc6aeffa9 Add SHA256 selftest 670cdd3f8b Merge #798: Check assumptions on integer implementation at compile time 5e5fb28b4a Use additional system macros to figure out endianness 7c068998ba Compile-time check assumptions on integer types 02b6c87b52 Add support for (signed) __int128 979961c506 Merge #787: Use preprocessor macros instead of autoconf to detect endianness 887bd1f8b6 Merge #793: Make scalar/field choice depend on C-detected __int128 availability 0dccf98a21 Use preprocessor macros instead of autoconf to detect endianness b2c8c42cf1 Merge #795: Avoid linking libcrypto in the valgrind ct test. 57d3a3c64c Avoid linking libcrypto in the valgrind ct test. 79f1f7a4f1 Autodetect __int128 availability on the C side 0d7727f95e Add SECP256K1_FE_STORAGE_CONST_GET to 5x52 field 805082de11 Merge #696: Run a Travis test on s390x (big endian) 39295362cf Test travis s390x (big endian) 6034a04fb1 Merge #778: secp256k1_gej_double_nonzero supports infinity f60915906d Merge #779: travis: Fix argument quoting for ./configure 9e49a9b255 travis: Fix argument quoting for ./configure 18d36327fd secp256k1_gej_double_nonzero supports infinity 214cb3c321 Merge #772: Improve constant-timeness on PowerPC 40412b1930 Merge #774: tests: Abort if malloc() fails during context cloning tests 2e1b9e0458 tests: Abort if malloc() fails during context cloning tests 67a429f31f Suppress a harmless variable-time optimization by clang in _int_cmov 5b196338f0 Remove redundant "? 1 : 0" after comparisons in scalar code 3e5cfc5c73 Merge #741: Remove unnecessary sign variable from wnaf_const 66bb9320c0 Merge #773: Fix some compile problems on weird/old compilers. 1309c03c45 Fix some compile problems on weird/old compilers. 2309c7dd4a Merge #769: Undef HAVE___INT128 in basic-config.h to fix gen_context compilation 22e578bb11 Undef HAVE___INT128 in basic-config.h to fix gen_context compilation 3f4a5a10e4 Merge #765: remove dead store in ecdsa_signature_parse_der_lax f00d6575ca remove dead store in ecdsa_signature_parse_der_lax dbd41db16a Merge #759: Fix uninitialized variables in ecmult_multi test 2e7fc5b537 Fix uninitialized variables in ecmult_multi test 37dba329c6 Remove unnecessary sign variable from wnaf_const 6bb0b77e15 Fix test_constant_wnaf for -1 and add a test for it. 47a7b8382f Clear field elements when writing infinity 61d1ecb028 Added test with additions resulting in infinity 60f7f2de5d Don't assume that ALIGNMENT > 1 in tests ada6361dec Use ROUND_TO_ALIGN in scratch_create 8ecc6ce50e Add check preventing rounding to alignment from wrapping around in scratch_alloc 4edaf06fb0 Add check preventing integer multiplication wrapping around in scratch_max_allocation git-subtree-dir: src/secp256k1 git-subtree-split: 8ab24e8dad9d43fc6661842149899e3cc9213b24
2020-09-11Merge #19922: test: Run rpc_txoutproof.py even with wallet disabledWladimir J. van der Laan
faf251d854e3a670533ea3e9087e82c92f3ae533 test: gettxoutproof duplicate txid (João Barbosa) faf5eb45c4a08fbfd9a8c12534bca8adfe756ef2 test: Test empty array in gettxoutproof (MarcoFalke) fa56e866e8ac08b35e775a4e37a4e5849e093c7d test: Run rpc_txoutproof.py even with wallet disabled (MarcoFalke) faba790bd40b5a9e8849997785020790ff60571b test: MiniWallet: Default fee_rate in send_self_transfer, Pass in utxo_to_spend (MarcoFalke) fa65a11d0c9a34ff7f4cc4efd53367794e751749 test: bugfix: Actually pick largest utxo (MarcoFalke) Pull request description: Run the consensus test even when the wallet was not compiled. Also: * Minor bugfix in MiniWallet * Two new test cases (one cherry-picked from #19847) ACKs for top commit: jnewbery: utACK faf251d854e3a670533ea3e9087e82c92f3ae533. Thanks Marco! kristapsk: ACK faf251d854e3a670533ea3e9087e82c92f3ae533 Tree-SHA512: a5ab33695c88cfb3c369021d4506069c08ce298e24e891db55159130693ed3817444c72f6aad3f472235aa4597b2c601010af714411c2ec8ad9c2d2e0b00ecbc
2020-09-11Merge #19916: build: allow user to specify DIR_FUZZ_SEED_CORPUS for cov_fuzzfanquake
fb3bacce69a39bf86288ece239c1be63518e11fb .gitignore: ignore qa-assets/ folder (eugene) a9f201439a84096a209bebc11705656b08d1b488 build: use DIR_FUZZ_SEED_CORPUS if specified for cov_fuzz target (eugene) Pull request description: This PR contains two commits: - The cov_fuzz target now uses `DIR_FUZZ_SEED_CORPUS` as the seed directory instead of the hard-coded `qa-assets/fuzz_seed_corpus`. Otherwise, running it requires me to copy the corpus to the bitcoin directory first. In case `DIR_FUZZ_SEED_CORPUS` is not specified, the original default is used. - add qa-assets folder to gitignore Example usage: `make cov_fuzz DIR_FUZZ_SEED_CORPUS=~/workspace/qa-assets/fuzz_seed_corpus` It can also just be an environment variable. ACKs for top commit: fanquake: ACK fb3bacce69a39bf86288ece239c1be63518e11fb - looks fine. practicalswift: ACK fb3bacce69a39bf86288ece239c1be63518e11fb - patch looks correct Tree-SHA512: 19ad7c6a2b0c088df14fb71a217d956e66a69eea78f016cd1e914d39c5d7cc196766a637e6c941c6706322663010e7162e85f57e888b8f3b05d0c37d44740847
2020-09-11Merge #19870: doc: update PyZMQ install instructions, fix zmq_sub.py file ↵fanquake
permissions 062e6699c4ac48c3d46516190ec411dec3680a0d script: fix zmq_sub.py file permissions (Jon Atack) 36f8e0cce700576865e61035626e08c5e845a38a doc: update PyZMQ installation instructions, ZeroMQ link (Jon Atack) Pull request description: Seen while reviewing #19572. ACKs for top commit: theStack: ACK 062e6699c4ac48c3d46516190ec411dec3680a0d 🧷 fanquake: ACK 062e6699c4ac48c3d46516190ec411dec3680a0d Tree-SHA512: 2210d92385377d066984d0a83882c3ece9f0f41c901b7eb375af9cdb57296f50f227c68193ccf35926073c2b788d58976442791a9fce2fc0f76452804d5cee6a
2020-09-10Merge #19841: Implement Keccak and SHA3_256Wladimir J. van der Laan
ab654c7d587b33d62230394663020439f80cee28 Unroll Keccak-f implementation (Pieter Wuille) 3f01ddb01bfffd49dfa131898d1c674ac5d0ac99 Add SHA3 benchmark (Pieter Wuille) 2ac8bf95834c8a43ebf365f09fb610829733134b Implement keccak-f[1600] and SHA3-256 (Pieter Wuille) Pull request description: Add a simple (and initially unoptimized) Keccak/SHA3 implementation based on https://github.com/mjosaarinen/tiny_sha3/blob/master/sha3.c, as one will be needed for TORv3 support (the conversion from BIP155 encoding to .onion notation uses a SHA3-based checksum). In follow-up commits, a benchmark is added, and the Keccakf function is unrolled for a (for me) 4.9x speedup. Test vectors are taken from https://csrc.nist.gov/projects/cryptographic-algorithm-validation-program/secure-hashing#sha3vsha3vss. ACKs for top commit: practicalswift: ACK ab654c7d587b33d62230394663020439f80cee28 -- patch looks correct and no sanitizer complaints when doing some basic fuzz testing of the added code (remember: **don't trust: fuzz!**) :) laanwj: re-ACK ab654c7d587b33d62230394663020439f80cee28 vasild: ACK ab654c7 Tree-SHA512: 8a91b18c46e8fb178b7ff82046cff626180362337e515b92fbbd771876e795da2ed4e3995eb4849773040287f6e687237f469a90474ac53f521fc12e0f5031d9
2020-09-10bugfix: make LoadWallet assigns status alwaysAkio Nakamura
Although loadwallet() in rpcwallet.cpp assumes LoadWallet() always assign some value to the 'status', but LoadWallet() does not do so in some situation. This fixes above and prevends loadwallet() returns ambiguous error code.
2020-09-09test: gettxoutproof duplicate txidJoão Barbosa
2020-09-09test: Test empty array in gettxoutproofMarcoFalke
2020-09-09test: Run rpc_txoutproof.py even with wallet disabledMarcoFalke
2020-09-09test: MiniWallet: Default fee_rate in send_self_transfer, Pass in utxo_to_spendMarcoFalke
Adds two new features to MiniWallet: * The fee rate is irrelevant sometimes, so just set an arbitrary default * The utxo to spend needs to be selected manually sometimes
2020-09-09test: bugfix: Actually pick largest utxoMarcoFalke
2020-09-09Merge #19800: test: MockwalletMarcoFalke
fa188c9c59b8c3e43c31be01797f073e27a7bc10 test: Use MiniWalet in p2p_feefilter (MarcoFalke) fa39c62eb7f39e7d249b8d46c075c4e7a9aec684 test: inline hashToHex (MarcoFalke) Pull request description: This introduces a minimalistic test wallet, which can be used as a drop in replacement for the Bitcoin Core wallet to create dummy transactions with a given fee rate. ACKs for top commit: jnewbery: utACK fa188c9c59b8c3e43c31be01797f073e27a7bc10 Tree-SHA512: 0aad9cb14eea4f0055bd6a47cc8c8f82a16941b152598c3bf1e083aae84cca4ffa23f0b854a362a68be1b917deba1b5ec7c0207b63b0805d747ba9a7d1d82efe
2020-09-08Merge #19914: refactor: Do not pass chain params to ↵MarcoFalke
CheckForStaleTipAndEvictPeers twice fa7e407b504bc60c77341f02636ed9d6a4b53d79 Do not pass chain params to CheckForStaleTipAndEvictPeers twice (MarcoFalke) Pull request description: `PeerManager` already keeps a reference to the chain params as a member variable. No need to pass it in once again as a function parameter. ACKs for top commit: naumenkogs: utACK fa7e407b504bc60c77341f02636ed9d6a4b53d79 jnewbery: code review ACK fa7e407b504bc60c77341f02636ed9d6a4b53d79 epson121: Code review ACK fa7e407b504bc60c77341f02636ed9d6a4b53d79 Tree-SHA512: 640c2d8adf9f1d54d0bfbdf81989064be2f5ba4b534d07d42258b372dc130f7b9c3fd087c7d28f0439678d124127f5d6f82f3139b1766f59f5ed661e7ac2a923
2020-09-08.gitignore: ignore qa-assets/ foldereugene
This commit ignores the qa-assets/ folder in case a user is running the cov_fuzz target without DIR_FUZZ_SEED_CORPUS set. In this case, the qa-assets folder is assumed to live in the bitcoin directory and should be ignored by git.
2020-09-08build: use DIR_FUZZ_SEED_CORPUS if specified for cov_fuzz targeteugene
This commit allows the user to specify the location of the fuzz_seed_corpus directory on their machine when running the cov_fuzz target. If DIR_FUZZ_SEED_CORPUS is specified, then this will be used. Otherwise, qa-assets/fuzz_seed_corpus is assumed to be in the bitcoin directory.
2020-09-08Do not pass chain params to CheckForStaleTipAndEvictPeers twiceMarcoFalke
2020-09-07Merge #19791: [net processing] Move Misbehaving() to PeerManagerMarcoFalke
bb6a32ce9983c72afa90f41a43a47ffd703ca006 [net processing] Move Misbehaving() to PeerManager (John Newbery) aa114b1c9b06c2bd3ed936bbb9fb32b31f75bdb2 [net_processing] Move SendBlockTransactions into PeerManager (John Newbery) 3115e00f75b41d9765dcbb376e367b25f61a1d58 [net processing] Move MaybePunishPeerForTx to PeerManager (John Newbery) e662e2d42afaf9c67c898634a0f3bc200255b6ea [net processing] Move ProcessOrphanTx to PeerManager (John Newbery) b70cd890e375e904b7f36b3d959e5656f5a5cbcd [net processing] Move MaybePunishNodeForBlock into PeerManager (John Newbery) d7778351bf60a21925a97b7fc4e9df5541b6d995 [net processing] Move ProcessHeadersMessage to PeerManager (John Newbery) 64f6162651420be2f4aa1498f0378a86780bc089 [whitespace] tidy up indentation after scripted diff (John Newbery) 58bd369b0ddd3383f7bdf7840912d18b96545f91 scripted-diff: [net processing] Rename PeerLogicValidation to PeerManager (John Newbery) 2297b26b3ce95e935c0ebb8c38dabf19965054a5 [net_processing] Pass chainparams to PeerLogicValidation constructor (John Newbery) 824bbd1ffba3df7ffa6f5bfaa31298cd484473b1 [move only] Collect all private members of PeerLogicValidation together (John Newbery) Pull request description: Continues the work of moving net_processing logic into PeerLogicValidation. See https://github.com/bitcoin/bitcoin/pull/19704 and https://github.com/bitcoin/bitcoin/pull/19607#discussion_r462032894 for motivation. This PR also renames `PeerLogicValidation` to `PeerManager` as suggested in https://github.com/bitcoin/bitcoin/pull/10756#pullrequestreview-53892618. ACKs for top commit: MarcoFalke: re-ACK bb6a32ce99 only change is rebase due to conflict in struct NodeContext and variable rename 🤸 hebasto: re-ACK bb6a32ce9983c72afa90f41a43a47ffd703ca006, only rebased, and added renaming `s/peer_logic/peerman/` into scripted-diff since my [previous](https://github.com/bitcoin/bitcoin/pull/19791#pullrequestreview-483118079) review (verified with `git range-diff`). Tree-SHA512: a2de4a521688fd25125b401e5575402c52b328a0fa27b3010567008d4f596b960aabbd02b2d81f42658f88f4365443fadb1008150a62fbcea123fb42d85a2c21
2020-09-07test: Use MiniWalet in p2p_feefilterMarcoFalke
2020-09-07[net processing] Move Misbehaving() to PeerManagerJohn Newbery
2020-09-07[net_processing] Move SendBlockTransactions into PeerManagerJohn Newbery
2020-09-07[net processing] Move MaybePunishPeerForTx to PeerManagerJohn Newbery
2020-09-07[net processing] Move ProcessOrphanTx to PeerManagerJohn Newbery
2020-09-07[net processing] Move MaybePunishNodeForBlock into PeerManagerJohn Newbery
2020-09-07[net processing] Move ProcessHeadersMessage to PeerManagerJohn Newbery
2020-09-07[whitespace] tidy up indentation after scripted diffJohn Newbery
2020-09-07scripted-diff: [net processing] Rename PeerLogicValidation to PeerManagerJohn Newbery
-BEGIN VERIFY SCRIPT- sed -i 's/PeerLogicValidation/PeerManager/g' $(git grep -l PeerLogicValidation ./src ./test) sed -i 's/peer_logic/peerman/g' $(git grep -l peer_logic ./src ./test) -END VERIFY SCRIPT- PeerLogicValidation was originally net_processing's implementation to the validation interface. It has since grown to contain much of net_processing's logic. Therefore rename it to reflect its responsibilities. Suggested in https://github.com/bitcoin/bitcoin/pull/10756#pullrequestreview-53892618.
2020-09-07[net_processing] Pass chainparams to PeerLogicValidation constructorJohn Newbery
Keep a references to chainparams, rather than calling the global Params() function every time it's needed. This is fine, since globalChainParams does not get updated once it's been set, and it's available at the point of constructing the PeerLogicValidation object.
2020-09-07[move only] Collect all private members of PeerLogicValidation togetherJohn Newbery
We don't have a project style for ordering class members, but it always makes sense to have no more than one of each public/protected/private specifier. Also move documentation for MaybeDiscourageAndDisconnect to the header.
2020-09-07Merge #19478: Remove CTxMempool::mapLinks data structure memberMarcoFalke
296be8f58e02b39a58f017c52294aceed22c3ffd Get rid of unused functions CTxMemPool::GetMemPoolChildren, CTxMemPool::GetMemPoolParents (Jeremy Rubin) 46d955d196043cc297834baeebce31ff778dff80 Remove mapLinks in favor of entry inlined structs with iterator type erasure (Jeremy Rubin) Pull request description: Currently we have a peculiar data structure in the mempool called maplinks. Maplinks job is to track the in-pool children and parents of each transaction. This PR can be primarily understood and reviewed as a simple refactoring to remove this extra data structure, although it comes with a nice memory and performance improvement for free. Maplinks is particularly peculiar because removing it is not as simple as just moving it's inner structure to the owning CTxMempoolEntry. Because TxLinks (the class storing the setEntries for parents and children) store txiters to each entry in the mempool corresponding to the parent or child, it means that the TxLinks type is "aware" of the boost multiindex (mapTx) it's coming from, which is in turn, aware of the entry type stored in mapTx. Thus we used maplinks to store this entry associated data we in an entirely separate data structure just to avoid a circular type reference caused by storing a txiter inside a CTxMempoolEntry. It turns out, we can kill this circular reference by making use of iterator_to multiindex function and std::reference_wrapper. This allows us to get rid of the maplinks data structure and move the ownership of the parents/child sets to the entries themselves. The benefit of this good all around, for any of the reasons given below the change would be acceptable, and it doesn't make the code harder to reason about or worse in any respect (as far as I can tell, there's no tradeoff). ### Simpler ownership model No longer having to consistency check that mapLinks did have records for our CTxMempoolEntry, impossible to have a mapLinks entry outlive or incorrectly die before a CTxMempoolEntry. ### Memory Usage We get rid of a O(Transactions) sized map in the mempool, which is a long lived data structure. ### Performance If you have a CTxMemPoolEntry, you immediately know the address of it's children/parents, rather than having to do a O(log(Transactions)) lookup via maplinks (which we do very often). We do it in *so many* places that a true benchmark has to look at a full running node, but it is easy enough to show an improvement in this case. The ComplexMemPool shows a good coherence check that we see the expected result of it being 12.5% faster / 1.14x faster. ``` Before: # Benchmark, evals, iterations, total, min, max, median ComplexMemPool, 5, 1, 1.40462, 0.277222, 0.285339, 0.279793 After: # Benchmark, evals, iterations, total, min, max, median ComplexMemPool, 5, 1, 1.22586, 0.243831, 0.247076, 0.244596 ``` The ComplexMemPool benchmark only checks doing addUnchecked and TrimToSize for 800 transactions. While this bench does a good job of hammering the relevant types of function, it doesn't test everything. Subbing in 5000 transactions shows a that the advantage isn't completely wiped out by other asymptotic factors (this isn't the only bottleneck in growing the mempool), but it's only a bit proportionally slower (10.8%, 1.12x), which adds evidence that this will be a good change for performance minded users. ``` # Benchmark, evals, iterations, total, min, max, median ComplexMemPool, 5, 1, 59.1321, 11.5919, 12.235, 11.7068 # Benchmark, evals, iterations, total, min, max, median ComplexMemPool, 5, 1, 52.1307, 10.2641, 10.5206, 10.4306 ``` I don't think it's possible to come up with an example of where a maplinks based design would have better performance, but it's something for reviewers to consider. # Discussion ## Why maplinks in the first place? I spoke with the author of mapLinks (sdaftuar) a while back, and my recollection from our conversation was that it was implemented because he did not know how to resolve the circular dependency at the time, and there was no other reason for making it a separate map. ## Is iterator_to weird? iterator_to is expressly for this purpose, see https://www.boost.org/doc/libs/1_51_0/libs/multi_index/doc/tutorial/indices.html#iterator_to > iterator_to provides a way to retrieve an iterator to an element from a pointer to the element, thus making iterators and pointers interchangeable for the purposes of element pointing (not so for traversal) in many situations. This notwithstanding, it is not the aim of iterator_to to promote the usage of pointers as substitutes for real iterators: the latter are specifically designed for handling the elements of a container, and not only benefit from the iterator orientation of container interfaces, but are also capable of exposing many more programming bugs than raw pointers, both at compile and run time. iterator_to is thus meant to be used in scenarios where access via iterators is not suitable or desireable: > > - Interoperability with preexisting APIs based on pointers or references. > - Publication of pointer-based interfaces (for instance, when designing a C-compatible library). > - The exposure of pointers in place of iterators can act as a type erasure barrier effectively decoupling the user of the code from the implementation detail of which particular container is being used. Similar techniques, like the famous Pimpl idiom, are used in large projects to reduce dependencies and build times. > - Self-referencing contexts where an element acts upon its owner container and no iterator to itself is available. In other words, iterator_to is the perfect tool for the job by the last reason given. Under the hood it should just be a simple pointer cast and have no major runtime overhead (depending on if the function call is inlined). Edit by laanwj: removed at sign from the description ACKs for top commit: jonatack: re-ACK 296be8f per `git range-diff ab338a19 3ba1665 296be8f`, sanity check gcc 10.2 debug build is clean. hebasto: re-ACK 296be8f58e02b39a58f017c52294aceed22c3ffd, only rebased since my [previous](https://github.com/bitcoin/bitcoin/pull/19478#pullrequestreview-482400727) review (verified with `git range-diff`). Tree-SHA512: f5c30a4936fcde6ae32a02823c303b3568a747c2681d11f87df88a149f984a6d3b4c81f391859afbeb68864ef7f6a3d8779f74a58e3de701b3d51f78e498682e
2020-09-07Merge #19556: Remove mempool globalMarcoFalke
fafb381af8279b2d2ca768df0bf68d7eb036a2f9 Remove mempool global (MarcoFalke) fa0359c5b30730744aa8a7cd9ffab79ded91041f Remove mempool global from p2p (MarcoFalke) eeee1104d78eb59a582ee1709ff4ac2c33ee1190 Remove mempool global from init (MarcoFalke) Pull request description: This refactor unlocks some nice potential features, such as, but not limited to: * Removing the fee estimates global (would avoid slightly fragile workarounds such as #18766) * Making the mempool optional for a "blocksonly" operation mode Even absent those features, the new code without the global should be easier to maintain, read and write tests for. ACKs for top commit: jnewbery: utACK fafb381af8279b2d2ca768df0bf68d7eb036a2f9 hebasto: ACK fafb381af8279b2d2ca768df0bf68d7eb036a2f9, I have reviewed the code and it looks OK, I agree it can be merged. darosior: ACK fafb381af8279b2d2ca768df0bf68d7eb036a2f9 Tree-SHA512: a2e696dc377e2e81eaf9c389e6d13dde4a48d81f3538df88f4da502d3012dd61078495140ab5a5854f360a06249fe0e1f6a094c4e006d8b5cc2552a946becf26
2020-09-07Merge #19738: wallet: Avoid multiple BerkeleyBatch in DelAddressBookSamuel Dobson
abac4367607d8d2b628e4db6a9663c960bacdacc wallet: Avoid multiple BerkeleyBatch in DelAddressBook (João Barbosa) Pull request description: ACKs for top commit: achow101: ACK abac4367607d8d2b628e4db6a9663c960bacdacc jonatack: ACK abac4367607d8d2b628e4db6a9663c960bacdacc meshcollider: re-utACK abac4367607d8d2b628e4db6a9663c960bacdacc Tree-SHA512: 92309fb74c48694160807326c0fe9793044a75cd77ed19400cceab54a7eefeb54ffc9334535e6021b3af7b9a364dbbeda3a9173540fff8144dfd437e96d76b5c
2020-09-06Unroll Keccak-f implementationPieter Wuille
2020-09-06Add SHA3 benchmarkPieter Wuille
2020-09-06Implement keccak-f[1600] and SHA3-256Pieter Wuille
2020-09-07Merge #19619: Remove wallet.dat path handling from wallet.cpp, rpcwallet.cppSamuel Dobson
7bf6dfbb484adfda3b8df26ee3e2ebda239dd263 wallet: Remove path checking code from bitcoin-wallet tool (Russell Yanofsky) 77d5bb72b8722ec7a6c7c33479a532cbd5870ba4 wallet: Remove path checking code from createwallet RPC (Russell Yanofsky) a987438e9d9cad0b5530e218a447928485f3fd93 wallet: Remove path checking code from loadwallet RPC (Russell Yanofsky) 8b5e7297c02f3100a9cb27bfe206e3fc617ec173 refactor: Pass wallet database into CWallet::Create (Russell Yanofsky) 3c815cfe54087fd139169161d2fd175e99840e6a wallet: Remove Verify and IsLoaded methods (Russell Yanofsky) 0d94e6062547f288a75921d2433458a44a5f2297 refactor: Use DatabaseStatus and DatabaseOptions types (Russell Yanofsky) b5b414151af32e5a07b5757b64482d77519d77c0 wallet: Add MakeDatabase function (Russell Yanofsky) 288b4ffb6b291f0466d513ff3c40af6758ca7c88 Remove WalletLocation class (Russell Yanofsky) Pull request description: Get rid of file path handling in wallet application code and move it down to database layer. There is no change in behavior except for some changed error messages. Motivation for this change is to make code more understandable, but also to prepare for adding SQLite support in #19077 so SQLite implementation can be contained at the database layer and wallet loading code does not need to become more complicated. ACKs for top commit: achow101: ACK 7bf6dfbb484adfda3b8df26ee3e2ebda239dd263 meshcollider: Code re-review and functional test run ACK 7bf6dfbb484adfda3b8df26ee3e2ebda239dd263 Tree-SHA512: 23ad18324c9e8947f0cf88a3734c2e9fb25536b2cb4d552cf5d1a4ade320fbffb73bb2d1b3a99585c11630aa7092e0fcfc2dd4fe65b91e3a54161433a5cd13cb
2020-09-06Merge #19897: Change FILE_CHAR_BLOCKLIST to FILE_CHARS_DISALLOWEDWladimir J. van der Laan
637d8bce741213295bd9b9d1982cae663c701ba1 Change FILE_CHAR_BLOCKLIST to FILE_CHARS_DISALLOWED (Benoit Verret) Pull request description: Blocklist is ambiguous. It could mean a list of blocks. Example: "blocknotify" in the same file refers to Bitcoin blocks. ACKs for top commit: MarcoFalke: ACK 637d8bce741213295bd9b9d1982cae663c701ba1 laanwj: ACK 637d8bce741213295bd9b9d1982cae663c701ba1 — this is a clear variable name improvement theStack: ACK 637d8bce741213295bd9b9d1982cae663c701ba1 jonatack: ACK 637d8bce741213295bd9b9d1982cae663c701ba1 eriknylund: ACK 637d8bce741213295bd9b9d1982cae663c701ba1 promag: ACK 637d8bce741213295bd9b9d1982cae663c701ba1. Tree-SHA512: 028e7102eeaf61105736c55c119a7f5c05411f2b6715a7939c41cb9e8f13afb757bbb6e7a302b3aae21722e69dab91f6eff8099e5884d248299905b4c7687c02
2020-09-06Change FILE_CHAR_BLOCKLIST to FILE_CHARS_DISALLOWEDBenoit Verret
Blocklist is ambiguous. It could mean a list of blocks. Example: "blocknotify" in the same file refers to Bitcoin blocks.
2020-09-06Merge #19890: refactor: remove unused header <arpa/inet.h> in protocol.cppWladimir J. van der Laan
2f79e9d00206a5230377f49be7b2f6da70f80417 refactor: remove unused header <arpa/inet.h> in protocol.cpp (Sebastian Falbesoner) Pull request description: There is no code using types or functions related to "internet operations" anymore in protocol.cpp (since #735, more than 8 years ago!), hence the header include can be removed. ACKs for top commit: practicalswift: ACK 2f79e9d00206a5230377f49be7b2f6da70f80417 -- patch looks correct and CI is happy epson121: Code review ACK 2f79e9d00206a5230377f49be7b2f6da70f80417 laanwj: ACK 2f79e9d00206a5230377f49be7b2f6da70f80417 promag: Code review ACK 2f79e9d00206a5230377f49be7b2f6da70f80417. Tree-SHA512: b3f75fa080125a34ce224f11eb13ec7b914cd9930e3bbed24f550031ce92a7e0830e8ff20159d737ffe487dfd28c24c273ad5e89c6932c8c6960d7fadb6c5e54
2020-09-06Merge #19887: test: Fix flaky wallet_basic testMarcoFalke
56b018ca7f37d25041b74f1bec305bdf54a55b9b test: Fix flaky wallet_basic test (Fabian Jahr) Pull request description: Fixes #19853 I investigated the issue in #19876 and I still intend to fix the underlying issue of a race when using wallet RPCs right after starting a node in that PR. However, since that is a bit more complicated than I initially thought it makes sense to merge the fix of the test so the intermittent test failures stop. This fix in the test is going to be needed, either way, #19876 will only provide an error where before it was reporting a false balance. Top commit has no ACKs. Tree-SHA512: 52bb2388a3e77aa20d26ab0fd45796bc1781483b1cffe49cbb44e2488a72e76998edfb1198495373f9c6fd2ec26064d4176bd1a64dd59806622d5e50a4f4e870
2020-09-06wallet: Avoid multiple BerkeleyBatch in DelAddressBookJoão Barbosa
2020-09-06Merge #19881: ci: Double tsan CPU and Memory to avoid global timeoutMarcoFalke
fa8e1487144eab237ffd291397355ef4801f46f8 ci: Double tsan CPU and Memory to avoid global timeout (MarcoFalke) Pull request description: Fix #19864 ACKs for top commit: practicalswift: ACK fa8e1487144eab237ffd291397355ef4801f46f8 -- patch looks correct hebasto: ACK fa8e1487144eab237ffd291397355ef4801f46f8, according to https://cirrus-ci.org/guide/linux/ the limits are: Tree-SHA512: b6d522290bfe80ed7453387b811628bf42c7657aa6a84d2f5984c8bb16f9857a71eabc6b8a4d63b84227d59b41a8ed7dd85d86cae5628dc9cf6b85bd365248d7
2020-09-06refactor: remove unused header <arpa/inet.h> in protocol.cppSebastian Falbesoner
2020-09-05test: Fix flaky wallet_basic testFabian Jahr
Co-authored-by: João Barbosa <joao.paulo.barbosa@gmail.com>