aboutsummaryrefslogtreecommitdiff
path: root/src/test
AgeCommit message (Collapse)Author
2023-07-26Add BIP324Cipher, encapsulating key agreement, derivation, and stream/AEAD ↵Pieter Wuille
ciphers Co-authored-by: dhruv <856960+dhruv@users.noreply.github.com>
2023-07-26crypto: support split plaintext in ChaCha20Poly1305 Encrypt/DecryptPieter Wuille
2023-07-26crypto: add FSChaCha20Poly1305, rekeying wrapper around ChaCha20Poly1305Pieter Wuille
This adds the FSChaCha20Poly1305 AEAD as specified in BIP324, a wrapper around the ChaCha20Poly1305 AEAD (as specified in RFC8439 section 2.8) which automatically rekeys every N messages, and automatically increments the nonce every message.
2023-07-26crypto: add FSChaCha20, a rekeying wrapper around ChaCha20Pieter Wuille
This adds the FSChaCha20 stream cipher as specified in BIP324, a wrapper around the ChaCha20 stream cipher (specified in RFC8439 section 2.4) which automatically rekeys every N messages, and manages the nonces used for encryption. Co-authored-by: dhruv <856960+dhruv@users.noreply.github.com>
2023-07-26crypto: add the ChaCha20Poly1305 AEAD as specified in RFC8439Pieter Wuille
This adds an implementation of the ChaCha20Poly1305 AEAD exactly matching the version specified in RFC8439 section 2.8, including tests and official test vectors.
2023-07-26crypto: remove outdated variant of ChaCha20Poly1305 AEADPieter Wuille
Remove the variant of ChaCha20Poly1305 AEAD that was previously added in anticipation of BIP324 using it. BIP324 was updated to instead use rekeying wrappers around otherwise unmodified versions of the ChaCha20 stream cipher and the ChaCha20Poly1305 AEAD as specified in RFC8439.
2023-07-26Merge bitcoin/bitcoin#28127: refactor: Remove C-style const-violating cast, ↵fanquake
Use reinterpret_cast fa9108f85afdc926fd6a8b96cc2acff7ca25d7a8 refactor: Use reinterpret_cast where appropriate (MarcoFalke) 3333f950d49f13662842650ae76599a0dff052eb refactor: Avoid casting away constness (MarcoFalke) fa6394dd10ae71755e46fd523dd43c2a1f2b832d refactor: Remove unused C-style casts (MarcoFalke) Pull request description: Using a C-style cast to convert pointer types to a byte-like pointer type has many issues: * It may accidentally and silently throw away `const`. * It forces reviewers to check that it doesn't accidentally throw away `const`. For example, on current master a `const char*` is cast to `unsigned char*` (without `const`), see https://github.com/bitcoin/bitcoin/blob/d23fda05842ba4539b225bbab01b94df0060f697/src/span.h#L273 . This can lead to UB, and the only reason why it didn't lead to UB is because the return type added back the `const`. (Obviously this would break if the return type was deduced via `auto`) Fix all issues by adding back the `const` and using `reinterpret_cast` where appropriate. ACKs for top commit: darosior: re-utACK fa9108f85afdc926fd6a8b96cc2acff7ca25d7a8 hebasto: re-ACK fa9108f85afdc926fd6a8b96cc2acff7ca25d7a8. john-moffett: ACK fa9108f85afdc926fd6a8b96cc2acff7ca25d7a8 Tree-SHA512: 87f6e4b574f9bd96d4e0f2a0631fd0a9dc6096e5d4f1b95042fe9f197afc2fe9a24e333aeb34fed11feefcdb184a238fe1ea5aff10d580bb18d76bfe48b76a10
2023-07-26Merge bitcoin/bitcoin#28150: test: Avoid intermittent issues due to async ↵fanquake
events in validationinterface_tests faca9a3d5a6887517d02b994a43d0e1101b718bc test: Avoid intermittent issues due to async events in validationinterface_tests (MarcoFalke) Pull request description: Currently the tests have many issues: * They setup the genesis block, even though it is not needed * They queue an async `UpdatedBlockTip` even, which causes intermittent issues: https://github.com/bitcoin/bitcoin/issues/28146#issuecomment-1650064645 Fix all issues by trimming down the setup to just `ChainTestingSetup`. ACKs for top commit: Crypt-iQ: tACK faca9a3d5a6887517d02b994a43d0e1101b718bc Tree-SHA512: 4449040330f89bbaf5ce5b2052417c160b451c373987fdf1069596c07834ed81f0aea1506d53c7d2cd21062b27332d30679285dae194b272fd0cb9ce5ded32cf
2023-07-25Merge bitcoin/bitcoin#28113: kernel: Remove UniValue from kernel libraryAndrew Chow
6960c81cbfa6208d4098353e53b313e13a21cb49 kernel: Remove Univalue from kernel library (TheCharlatan) 10eb3a9faa977371facacee937b2e6dc26f008e0 kernel: Split ParseSighashString (TheCharlatan) Pull request description: Besides the build system changes, this is a mostly move-only change for moving the few UniValue-related functions out of kernel files. UniValue is not required by any of the kernel components and a JSON library should not need to be part of a consensus library. ACKs for top commit: achow101: ACK 6960c81cbfa6208d4098353e53b313e13a21cb49 theuni: Re-ACK 6960c81cbfa6208d4098353e53b313e13a21cb49 stickies-v: re-ACK https://github.com/bitcoin/bitcoin/commit/6960c81cbfa6208d4098353e53b313e13a21cb49 Tree-SHA512: d92e4cb4e12134c94b517751bd746d39f9b8da528ec3a1c94aaedcce93274a3bae9277832e8a7c0243c13df0397ca70ae7bbb24ede200018c569f8d81103c1da
2023-07-25kernel: Remove Univalue from kernel libraryTheCharlatan
It is not required by any of the kernel components. A JSON library should not need to be part of a consensus library.
2023-07-25kernel: Split ParseSighashStringTheCharlatan
This split is done in preparation for the next commit where the dependency on UniValue in the kernel library is removed.
2023-07-25test: Avoid intermittent issues due to async events in validationinterface_testsMarcoFalke
2023-07-24Fix initialization of setBlockIndexCandidates when working with multiple ↵Suhas Daftuar
chainstates When using assumeutxo and multiple chainstates are active, the background chainstate should consider all HAVE_DATA blocks that are ancestors of the snapshotted block and that have more work than the tip as potential candidates.
2023-07-24Tighten requirements for adding elements to setBlockIndexCandidatesSuhas Daftuar
When using assumeutxo, we only need the background chainstate to consider blocks that are on the chain leading to the snapshotted block. Note that this introduces the new invariant that we can only have an assumeutxo snapshot where the snapshotted blockhash is in our block index. Unknown block hashes that are somehow passed in will cause assertion failures when processing new blocks. Includes test fixes and improvements by Andrew Chow and Fabian Jahr.
2023-07-24[net processing] Introduce PeerManager optionsdergoegge
2023-07-24refactor: Remove unused C-style castsMarcoFalke
2023-07-22fuzz: use `ConnmanTestMsg` in `connman`brunoerg
Using `ConnmanTestMsg` we can add nodes and be more effective fuzzing functions like `DisconnectNode`, `FindNode`, `GetNodeStats` and other ones.
2023-07-21fuzz: increase coverage of the descriptor targetsAntoine Poinsot
Once a descriptor is successfully parsed, execute more of its methods. There is probably still room for improvements by checking for some invariants, but this is a low hanging fruit that significantly increases the code coverage of these targets.
2023-07-21fuzz: add a new, more efficient, descriptor parsing targetAntoine Poinsot
This new target focuses on fuzzing the actual descriptor parsing logic by not requiring the fuzzer to produce valid keys (nor a valid checksum for that matter). This should make it much more efficient to find bugs we could introduce moving forward. Using a character as a marker (here '%') to be able to search and replace in the string without having to mock the actual descriptor parsing logic was an insight from Pieter Wuille.
2023-07-21Move block-storage-related logic to ChainstateManagerSuhas Daftuar
Separate the notion of which blocks are stored on disk, and what data is in our block index, from what tip a chainstate might be able to get to. We can use chainstate-agnostic data to determine when to store a block on disk (primarily, an anti-DoS set of criteria) and let the chainstates figure out for themselves when a block is of interest for being a candidate tip. Note: some of the invariants in CheckBlockIndex are modified, but more work is needed (ie to move CheckBlockIndex to ChainstateManager, as most of what CheckBlockIndex is doing is checking the consistency of the block index, which is outside of Chainstate).
2023-07-21fuzz: make the parsed descriptor testing into a functionAntoine Poinsot
We'll be reusing it in the new target.
2023-07-19Merge bitcoin/bitcoin#28085: refactor: use Span for SipHash::WriteAndrew Chow
7d92b1430a6fd42c4438810640576830d0ff8d13 refactor: use Span for SipHash::Write (Sebastian Falbesoner) Pull request description: This simple refactoring PR changes the interface for the `SipHash` arbitrary-data `Write` method to take a `Span<unsigned char>` instead of having to pass data and length. (`Span<std::byte>` seems to be more modern, but vectors of `unsigned char` are still used prety much everywhere where SipHash is called, and I didn't find it very appealing having to clutter the code with `Make(Writable)ByteSpan` helpers). ACKs for top commit: sipa: utACK 7d92b1430a6fd42c4438810640576830d0ff8d13 MarcoFalke: lgtm ACK 7d92b1430a6fd42c4438810640576830d0ff8d13 achow101: ACK 7d92b1430a6fd42c4438810640576830d0ff8d13 Tree-SHA512: f17a27013c942aead4b09f5a64e0c3ff8dbc7e83fe63eb9a2e3ace8be9921c9cbba3ec67e3e83fbe3332ca941c42370efd059e702c060f9b508307e9657c66f2
2023-07-19streams: Teach AutoFile how to XORMarcoFalke
2023-07-19Merge bitcoin/bitcoin#27425: test: move remaining rand code from ↵fanquake
util/setup_common to util/random 1cd45d4e08c3dfd1d6423620c79169f1404ac12b test: move random.h include header from setup_common.h to cpp (Jon Atack) 1b246fdd145a95f5da479159f5e8eaf5a76bdc3a test: move remaining random test util code from setup_common to random (jonatack) Pull request description: and drop the `util/random` dependency on `util/setup_common`. This improves code separation and allows `util/setup_common` to call `util/random` functions without creating a circular dependency, thereby addressing https://github.com/bitcoin/bitcoin/pull/26940#issuecomment-1497266140 by glozow (thanks!) ACKs for top commit: MarcoFalke: lgtm ACK 1cd45d4e08c3dfd1d6423620c79169f1404ac12b 🌂 Tree-SHA512: 6ce63d9103ba9b04eebbd8ad02fe9aa79e356296533404034a1ae88e9b7ca0bc9a5c51fd754b71cf4e7b55b18bcd4d5474b2d588edee3851e3b3ce0e4d309a93
2023-07-17Merge bitcoin/bitcoin#27997: Descriptors: rule out unspendable miniscript ↵Andrew Chow
descriptors c7db88af71b3204171f33399aa4f33b40a4f7cd9 descriptor: assert we never parse a sane miniscript with no pubkey (Antoine Poinsot) a49402a9ec7431c286139b76f8759719a99a8551 qa: make sure we don't let unspendable Miniscript descriptors be imported (Antoine Poinsot) 639e3b6c9759a7a582c5c86fdbfa5ea99cb7bb16 descriptor: refuse to parse unspendable miniscript descriptors (Antoine Poinsot) e3280eae1b53006d74d11f3cf9d7a9dc7ff2c39e miniscript: make GetStackSize() and GetOps() return optionals (Antoine Poinsot) Pull request description: `IsSane()` in Miniscript does not ensure a Script is actually spendable. This is an issue as we would accept any sane Miniscript when parsing a descriptor. Fix this by explicitly checking a Miniscript descriptor is both sane and spendable when parsing it. This bug was exposed due to a check added in #22838 (https://github.com/bitcoin/bitcoin/pull/22838#discussion_r1226859880) that triggered a fuzz crash (https://github.com/bitcoin/bitcoin/pull/22838#issuecomment-1612510057). ACKs for top commit: sipa: utACK c7db88af71b3204171f33399aa4f33b40a4f7cd9 achow101: ACK c7db88af71b3204171f33399aa4f33b40a4f7cd9 Tree-SHA512: e79bc9f7842e98a4e8f358f05811fca51b15b4b80a171c0d2b17cf4bb1f578a18e4397bc2ece9817d392e0de0196ee6a054b7318441fd3566dd22e1f03eb64a5
2023-07-17Merge bitcoin/bitcoin#27993: Make poly1305 support incremental computation + ↵Andrew Chow
modernize 4e5c933f6a40c07d1c68115f7979b89a5b2ba580 Switch all callers from poly1305_auth to Poly1305 class (Pieter Wuille) 8871f7d1ae096839abcbf25a548319185acc01a2 tests: add more Poly1305 test vectors (Pieter Wuille) 40e6c5b9fce92ffe64e91c2aba38bb2ed57bfbfb crypto: add Poly1305 class with std::byte Span interface (Pieter Wuille) 50269b391fa18556bad72dc8c2fb4e2493a6a054 crypto: switch poly1305 to incremental implementation (Pieter Wuille) Pull request description: Our current Poly1305 code (src/crypto/poly1305.*) only supports computing the entire tag in one go (the `poly1305_auth` function takes a key and message, and outputs the tag). However, the RFC8439 authenticated encryption (as used in BIP324, see #27634) scheme makes use of Poly1305 in a way where the message consists of 3 different pieces: * The additionally authenticated data (AAD), padded to 16 bytes. * The ciphertext, padded to 16 bytes. * The length of the AAD and the length of the ciphertext, together another 16 bytes. Implementing RFC8439 using the existing `poly1305_auth` function requires creating a temporary copy with all these pieces of data concatenated just for the purpose of computing the tag (the approach used in #25361). This PR replaces the poly1305 code with new code from https://github.com/floodyberry/poly1305-donna (with minor adjustments to make it match our coding style and use our utility functions, documented in the commit) which supports incremental operation, and then adds a C++ wrapper interface using std::byte Spans around it, and adds tests that incremental and all-at-once computation match. ACKs for top commit: achow101: ACK 4e5c933f6a40c07d1c68115f7979b89a5b2ba580 theStack: ACK 4e5c933f6a40c07d1c68115f7979b89a5b2ba580 stratospher: tested ACK 4e5c933. Tree-SHA512: df6e9a2a4a38a480f9e4360d3e3def5311673a727a4a85b008a084cf6843b260dc82cec7c73e1cecaaccbf10f3521a0ae7dba388b65d0b086770f7fbc5223e2a
2023-07-17Merge bitcoin/bitcoin#28065: fuzz: Flatten all FUZZ_TARGET macros into onefanquake
fa6dfaaf45bde465969fa7d8fa6b6574a497a6ca scripted-diff: Use new FUZZ_TARGET macro everywhere (MarcoFalke) fa36ad8b091c70190491280dcf0794e94e34a9ed fuzz: Accept options in FUZZ_TARGET macro (MarcoFalke) Pull request description: The `FUZZ_TARGET` macros have many issues: * The developer will have to pick the right macro to pass the wanted option. * Adding a new option requires doubling the number of existing macros in the worst case. Fix all issues by using only a single macro. This refactor does not change behavior. ACKs for top commit: dergoegge: ACK fa6dfaaf45bde465969fa7d8fa6b6574a497a6ca Tree-SHA512: 49a34553867a1734ce89e616b2d7c29b784a67cd8990db6573f0c7b18957636ef0c81d3d0d444a04c12cdc98bc4c4aa7a2ec94e6232dc363620a746e28416444
2023-07-17refactor: use Span for SipHash::WriteSebastian Falbesoner
Co-authored-by: MarcoFalke <*~=`'#}+{/-|&$^_@721217.xyz>
2023-07-17fuzz: Bump FuzzedDataProvider.hMarcoFalke
From https://github.com/llvm/llvm-project/blob/fa8401f9bfe81f4853bf9b67bff42a2cebffc10f/compiler-rt/include/fuzzer/FuzzedDataProvider.h
2023-07-14test: Clear block index flags when testing snapshotsSuhas Daftuar
When simulating a snapshot, remove the HAVE_DATA status for blocks below the snapshot height, to simulate never having downloaded them at all. This makes tests more realistic (and more closely match what will happen when using assumeutxo).
2023-07-14Move block-arrival information / preciousblock counters to ChainstateManagerSuhas Daftuar
Block arrival information (and the preciousblock RPC, a related concept) are both chainstate-agnostic, so these are moved to ChainstateManager. This should just be a refactor, without any observable behavior changes.
2023-07-14Remove CChain dependency in node/blockstorageSuhas Daftuar
2023-07-13Merge bitcoin/bitcoin#27549: fuzz: addrman, add coverage for `network` field ↵Andrew Chow
in `Select()`, `Size()` and `GetAddr()` 35a2175ad8bec92b409ae2202c124e39b2f3f838 fuzz: addrman, add coverage for `network` field in `Select()`, `Size()` and `GetAddr()` (brunoerg) Pull request description: This PR adds fuzz coverage for `network` field in `Select()`, `Size()` and `GetAddr()`, there was only call to them without passing a network. https://marcofalke.github.io/b-c-cov/fuzz.coverage/src/addrman.cpp.gcov.html ACKs for top commit: amitiuttarwar: for the record, ACK 35a2175ad8bec92b409ae2202c124e39b2f3f838 - only small changes from the version (previously) proposed in 27213 achow101: ACK 35a2175ad8bec92b409ae2202c124e39b2f3f838 mzumsande: Code Review ACK 35a2175ad8bec92b409ae2202c124e39b2f3f838, haven't tested this yet, but I will let the fuzzer run for a while now. Tree-SHA512: dddb8322298d6c373c8e68d57538470b11825a9a310a355828c351d5c0b19ff6779d024a800e3ea90126d0c050e86f71fd22cd23d1a306c784cef0f82c45e3ca
2023-07-13scripted-diff: Use new FUZZ_TARGET macro everywhereMarcoFalke
-BEGIN VERIFY SCRIPT- ren() { sed --regexp-extended -i "s|$1|$2|g" $(git grep -l --extended-regexp "$1"); } # Replace FUZZ_TARGET_INIT ren 'FUZZ_TARGET_INIT\((.+), (.+)\)' 'FUZZ_TARGET(\1, .init = \2)' # Delete unused FUZZ_TARGET_INIT sed -i -e '37,39d' src/test/fuzz/fuzz.h -END VERIFY SCRIPT-
2023-07-13fuzz: Accept options in FUZZ_TARGET macroMarcoFalke
* This allows to reduce the number of total macros. * Also, adding a new option no longer requires doubling the number of macros in the worst case.
2023-07-13Merge bitcoin/bitcoin#27411: p2p: Restrict self-advertisements with privacy ↵Andrew Chow
networks to avoid fingerprinting e7cf8657e1165ea5da3911a9e543837cd8938f97 test: add unit test for local address advertising (Martin Zumsande) f4754b9dfb84859166843fb2a1888fb3cfebf73c net: restrict self-advertisements with privacy networks (Martin Zumsande) e4d541c7cfa65da77e80e6786fdcb197ab50d04b net, refactor: pass reference for peer address in GetReachabilityFrom (Martin Zumsande) 62d73f5370415f910c95a67b3d9f97bc85487bbe net, refactor: pass CNode instead of CNetAddr to GetLocalAddress (Martin Zumsande) Pull request description: The current logic for self-advertisements works such that we detect as many local addresses as we can, and then, using the scoring matrix from `CNetAddr::GetReachabilityFrom()`, self-advertise with the address that fits best to our peer. It is in general not hard for our peers to distinguish our self-advertisements from other addrs we send them, because we self-advertise every ~24h and because the first addr we send over a connection is likely our self-advertisement. `GetReachabilityFrom()` currently only takes into account actual reachability, but not whether we'd _want_ to announce our identity for one network to peers from other networks, which is not straightforward in connection with privacy networks. While the general approach is to prefer self-advertising with the address for the network our peer is on, there are several special situations in which we don't have one, and as a result could allow self-advertise other local addresses, for example: A) We run i2p and clearnet, use `-i2pacceptincoming=0` (so we have no local i2p address), and we have a local ipv4 address. In this case, we'd advertise the ipv4 address to our outbound i2p peers. B) Our `-discover` logic cannot detect any local clearnet addresses in our network environment, but we are actually reachable over clearnet. If we ran bitcoind clearnet-only, we'd always advertise the address our peer sees us with instead, and could get inbound peers this way. Now, if we also have an onion service running (but aren't using tor as a proxy for clearnet connections), we could advertise our onion address to clearnet peers, so that they would be able to connect our clearnet and onion identities. This PR tries to avoid these situations by 1.) never advertising our local Tor or I2P address to peers from other networks. 2.) never advertising local addresses from non-anonymity networks to peers from Tor or I2P Note that this affects only our own self-advertisements, the rules to forward other people's addrs are not changed. [Edit] after Initial [discussion](https://github.com/bitcoin/bitcoin/pull/27411#issuecomment-1497176155): CJDNS is not being treated like Tor and I2P at least for now, because it has different privacy properties and for the practical reason that it has still very few bitcoin nodes. ACKs for top commit: achow101: ACK e7cf8657e1165ea5da3911a9e543837cd8938f97 vasild: ACK e7cf8657e1165ea5da3911a9e543837cd8938f97 luke-jr: utACK e7cf8657e1165ea5da3911a9e543837cd8938f97 Tree-SHA512: 3db8415dea6f82223d11a23bd6cbb3b8cf68831321280e926034a1f110cbe22562570013925f6fa20d8f08e41d0202fd69c733d9f16217318a660d2a1a21b795
2023-07-12Switch all callers from poly1305_auth to Poly1305 classPieter Wuille
This also removes the old poly1305_auth interface, as it no longer serves any function. The new Poly1305 class based interface is more modern and safe.
2023-07-12tests: add more Poly1305 test vectorsPieter Wuille
2023-07-12crypto: add Poly1305 class with std::byte Span interfacePieter Wuille
2023-07-12Merge bitcoin/bitcoin#27985: Add support for RFC8439 variant of ChaCha20Andrew Chow
0bf87476f55dceb106563156c7c8d6bfb8162e29 test: add ChaCha20 test triggering 32-bit block counter overflow (Sebastian Falbesoner) 7f2a985147ef541123c65d5db1c3fc3e533fd4ce tests: improve ChaCha20 unit tests (Pieter Wuille) 511a8d406e3115b97c6d35e2c603af53b3f9da13 crypto: Implement RFC8439-compatible variant of ChaCha20 (Pieter Wuille) Pull request description: Based on and replaces part of #25361, part of the BIP324 project (#27634). See also #19225 for background. There are two variants of ChaCha20 in use. The currently implemented one uses a 64-bit nonce and a 64-bit block counter, while the one used in RFC8439 (and thus BIP324) uses a 96-bit nonce and 32-bit block counter. This PR changes the logic to use the 96-bit nonce variant, though in a way that's compatible with >256 GiB output (by automatically incrementing the first 32-bit part of the nonce when the block counter overflows). For those who reviewed the original PR, the biggest change is here that the 96-bit nonce is passed as a Nonce96 type (pair of 32-bit + 64-bit integer) rather than a 12-byte array. ACKs for top commit: achow101: ACK 0bf87476f55dceb106563156c7c8d6bfb8162e29 theStack: Code-review ACK 0bf87476f55dceb106563156c7c8d6bfb8162e29 Tree-SHA512: 62e4cbd5388b8d50ef1a0dc99b6f4ad36c7b4419032035f8e622dda63a62311dd923032217e20054bcd836865d4be5c074f9e5538ca158f94f08eab75c5519c1
2023-07-11Merge bitcoin/bitcoin#28044: test: indexes, fix on error infinite loopRyan Ofsky
89ba8905f5c68ae29412f9c4010314c5a113c234 test: indexes, fix on error infinite loop (furszy) Pull request description: Coming from https://github.com/bitcoin/bitcoin/pull/28036#issuecomment-1623813703, I thought that we were going to fix it there but seems that got merged without it for some reason. As index sync failures trigger a shutdown request without notifying `BaseIndex::BlockUntilSyncedToCurrentChain` in any way, we also need to check whether a shutdown was requested or not inside 'IndexWaitSynced'. Otherwise, any error inside the index sync process will hang the test forever. ACKs for top commit: MarcoFalke: lgtm ACK 89ba8905f5c68ae29412f9c4010314c5a113c234 jamesob: ACK 89ba890 ryanofsky: Code review ACK 89ba8905f5c68ae29412f9c4010314c5a113c234. Just comment update since last review Tree-SHA512: 1f6daf34e51d3fbc802799bfa4ac0ef0d8f774db5f9e2f5d35df18a77679778475c94efc3da1fb723ebaf3583e4075e4a5cbe4a5104ad0c50e2b32076e247b29
2023-07-10test: indexes, fix on error infinite loopfurszy
As index sync failures trigger a shutdown request without notifying BaseIndex::BlockUntilSyncedToCurrentChain in any way, we also need to check whether a shutdown was requested or not inside 'IndexWaitSynced'. Otherwise, any error inside the index sync process will hang the test forever.
2023-07-10Merge bitcoin/bitcoin#27607: index: make startup more efficientRyan Ofsky
ca91c244ef1ba7eac6643d66a5fc56d3a2a8b550 index: verify blocks data existence only once (furszy) fcbdaeef4d5a63e3e5b479c6fcad730eb86fb923 init: don't start indexes sync thread prematurely (furszy) 2ec89f1970935d27631bcd17b7923a79cdb1edbb refactor: simplify pruning violation check (furszy) c82ef91eae38f385d408b095ebbc8a180ce52ebb make GetFirstStoredBlock assert that 'start_block' always has data (furszy) 430e7027a18870a296abb0bbd9332cbe40d8fdc0 refactor: index, decouple 'Init' from 'Start' (furszy) 225e213110602b4fd1d345167f5f92d26557f6c1 refactor: init indexes, decouple 'Start()' from the creation step (furszy) 2ebc7e68cc9d347807b646f601b27940c9590c89 doc: describe 'init load' thread actions (Martin Zumsande) 04575106b2529f495ce8110ddf7ed2247d4bc339 scripted-diff: rename 'loadblk' thread name to 'initload' (furszy) ed4462cc78afd2065bbf5bd79728852b65b9ad7f init: start indexes sync earlier (furszy) Pull request description: Simplifies index startup code, eliminating the `g_indexes_ready_to_sync` variable, deduplicating code and moving the prune violation check out of the `BaseIndex` class. Also makes startup more efficient by running the prune violation check once for all indexes instead of once for each index, and by delaying the prune violation check and moving it off of the main thread so the node can start up faster and perform the block data availability verification even when the '-reindex" or the "-reindex-chainstate" flags are enabled (which hasn't being possible so far). ACKs for top commit: ryanofsky: Code review ACK ca91c244ef1ba7eac6643d66a5fc56d3a2a8b550. Just rebase and suggested changes since last review (Start return check, and code simplification) TheCharlatan: re-ACK ca91c244ef1ba7eac6643d66a5fc56d3a2a8b550 Tree-SHA512: e9c98ce89aeb29e8d0f505f17b34aa54fe44efefbf017f4746e3b446ab4de25ade4f707254a0bbe4b99b69731b04a4067ce529eb7aa834ced196784b694cf7ce
2023-07-10index: verify blocks data existence only oncefurszy
At present, during init, we traverse the chain (once per index) to confirm that all necessary blocks to sync each index up to the current tip are present. To make the process more efficient, we can fetch the oldest block from the indexers and perform the chain data existence check from that point only once. This also moves the pruning violation check to the end of the 'loadinit' thread, which is where the reindex, block loading and chain activation processes happen. Making the node's startup process faster, allowing us to remove the global g_indexes_ready_to_sync flag, and enabling the execution of the pruning violation verification even when the reindex or reindex-chainstate flags are enabled (which has being skipped so far).
2023-07-10refactor: simplify pruning violation checkfurszy
By generalizing 'GetFirstStoredBlock' and implementing 'CheckBlockDataAvailability' we can dedup code and avoid repeating work when multiple indexes are enabled. E.g. get the oldest block across all indexes and perform the pruning violation check from that point up to the tip only once (this feature is being introduced in a follow-up commit). This commit shouldn't change behavior in any way. Co-authored-by: Ryan Ofsky <ryan@ofsky.org>
2023-07-10make GetFirstStoredBlock assert that 'start_block' always has datafurszy
And transfer the responsibility of verifying whether 'start_block' has data or not to the caller. This is because the 'GetFirstStoredBlock' function responsibility is to return the first block containing data. And the current implementation can return 'start_block' when it has no data!. Which is misleading at least. Edge case behavior change: Previously, if the block tip lacked data but all preceding blocks contained data, there was no prune violation. And now, such scenario will result in a prune violation.
2023-07-10refactor: index, decouple 'Init' from 'Start'furszy
So indexes can be initialized without spawning the sync thread. This makes asynchronous indexes startup possible in the following commits.
2023-07-09test: add ChaCha20 test triggering 32-bit block counter overflowSebastian Falbesoner
Verify that our ChaCha20 implementation using the 96/32 split interface is compatible with >256 GiB outputs by triggering a 32-bit block counter overflow and checking that the keystream matches one created with an alternative implementation using a 64/64 split interface with the corresponding input data. The test case data was generated with the following Python script using the PyCryptodome library (version 3.15.0): ---------------------------------------------------------------------------------------------- from Crypto.Cipher import ChaCha20 key = bytes(list(range(32))); nonce = 0xdeadbeef12345678; pos = 2**32 - 1 c = ChaCha20.new(key=key, nonce=nonce.to_bytes(8, 'little')) c.seek(pos * 64); stream = c.encrypt(bytes([0])*128) print(f"Key: {key.hex()}\nNonce: {hex(nonce)}\nPos: {hex(pos)}\nStream: {stream.hex()}") ----------------------------------------------------------------------------------------------
2023-07-08test: make assumeUTXO test capture the expected fatal errorfurszy
The test is exercising the error, so it can capture it before the test framework displays it on the console as an unforeseen fatal error.
2023-07-07tests: improve ChaCha20 unit testsPieter Wuille