aboutsummaryrefslogtreecommitdiff
path: root/src/index
AgeCommit message (Collapse)Author
2024-04-17index: race fix, lock cs_main while 'm_synced' is subject to changeRyan Ofsky
This ensures that the index does not miss any 'new block' signals occurring in-between reading the 'next block' and setting 'm_synced'. Because, if this were to happen, the ignored blocks would never be indexed, thus stalling the index forever.
2024-04-01Fix #29767, set m_synced = true after Commit()nanlour
2024-03-22Merge bitcoin/bitcoin#29672: validation: Make translations of fatal errors ↵Ava Chow
consistent 824f47294a309ba8e58ba8d1da0af15d8d828f43 node: Use log levels in noui_ThreadSafeMessageBox (TheCharlatan) ddc7872c08b7ddf9b1e83abdb97c21303f4a9172 node: Make translations of fatal errors consistent (TheCharlatan) Pull request description: The extra `bilingual_str` argument of the fatal error notifications and `node::AbortNode()` is often unused and when used usually contains the same string as the message argument. It also seems to be confusing, since it is not consistently used for errors requiring user action. For example some assumeutxo fatal errors require the user to do something, but are not translated. So simplify the fatal error and abort node interfaces by only passing a translated string. This slightly changes the fatal errors displayed to the user. ACKs for top commit: stickies-v: re-ACK 824f47294a309ba8e58ba8d1da0af15d8d828f43 maflcko: ACK 824f47294a309ba8e58ba8d1da0af15d8d828f43 🔎 achow101: ACK 824f47294a309ba8e58ba8d1da0af15d8d828f43 hebasto: re-ACK 824f47294a309ba8e58ba8d1da0af15d8d828f43. Tree-SHA512: 2868ee7b045fe7f3ac582ce5039141b398480b7627734976201dafaaef7544b8461635a7292fee4a7f32ff1bfc26f9bd4d0c292dca424ba42fb7fc4483d7ce8d
2024-03-21node: Make translations of fatal errors consistentTheCharlatan
The extra `bilingual_str` argument of the fatal error notifications and `node::AbortNode()` is often unused and when used usually contains the same string as the message argument. It also seems to be confusing, since it is not consistently used for errors requiring user action. For example some assumeutxo fatal errors require the user to do something, but are not translated. So simplify the fatal error and abort node interfaces by only passing a translated string. This slightly changes the fatal errors displayed to the user. Also de-duplicate the abort error log since it is repeated in noui.cpp.
2024-03-20Merge bitcoin/bitcoin#29671: index: avoid "failed to commit" errors on ↵Ava Chow
initialization f65b0f6401091e4a4ca4c9f4db1cf388f0336bad index: Move last_locator_write_time and logging to end of threadsync loop (Fabian Jahr) Pull request description: In the index sync thread, when initializing an index for the first time, stop callng BaseIndex::Commit when m_best_block_index is null, to avoid a spurious "failed to commit" error from that function. This error started happening in commit https://github.com/bitcoin/bitcoin/commit/7878f97bf15b6e7c9b47d1c0d96419b97e1bdcbd from https://github.com/bitcoin/bitcoin/pull/25494 and was reported by pstratem in https://github.com/bitcoin/bitcoin/pull/26903 with an alternate fix. ACKs for top commit: achow101: ACK f65b0f6401091e4a4ca4c9f4db1cf388f0336bad ryanofsky: Code review ACK f65b0f6401091e4a4ca4c9f4db1cf388f0336bad. Just moved log "Syncing" log line since last commit to avoid having to call now() twice. furszy: ACK f65b0f6401091e4a4ca4c9f4db1cf388f0336bad TheCharlatan: ACK f65b0f6401091e4a4ca4c9f4db1cf388f0336bad Tree-SHA512: afa8f05786318d36346d167ff53ea0b3bc8abdb0ad04465d199dc3eb91e9f837369e24fcb7e24b5757b02d698ec504e61da6ac365eaf006c874fc07a424a7e20
2024-03-19index: Move last_locator_write_time and logging to end of threadsync loopFabian Jahr
This avoids having commit print a needless error message during init. Co-authored-by: furszy <mfurszy@protonmail.com>
2024-03-12index: decrease ThreadSync cs_main contentionfurszy
Only NextSyncBlock requires cs_main lock. The other function calls like Commit or Rewind will lock or not cs_main internally when they need it. Avoiding keeping cs_main locked when Commit() or Rewind() write data to disk.
2024-03-12index: cache last block filter headerfurszy
Avoid disk read operations on every new processed block.
2024-03-12index: blockfilter, decouple header lookup into its own functionfurszy
2024-03-12index: blockfilter, decouple Write into its own functionfurszy
2024-03-12bench: basic block filter index initial syncfurszy
Introduce benchmark for the block filter index sync. And makes synchronous 'Sync()' mechanism accessible.
2024-03-11scripted-diff: Replace error() with LogError()MarcoFalke
This fixes the log output when -logsourcelocations is used. Also, instead of 'ERROR:', the log will now say '[error]', like other errors logged with LogError. -BEGIN VERIFY SCRIPT- sed -i --regexp-extended 's! error\("([^"]+)"! LogError("\1\\n"!g' $( git grep -l ' error(' ./src/ ) -END VERIFY SCRIPT-
2024-03-11scripted-diff: return error(...); ==> error(...); return false;MarcoFalke
This is needed for the next commit. -BEGIN VERIFY SCRIPT- # Separate sed invocations to replace one-line, and two-line error(...) calls sed -i --regexp-extended 's!( +)return (error\(.*\);)!\1\2\n\1return false;!g' $( git grep -l 'return error(' ) sed -i --null-data --regexp-extended 's!( +)return (error\([^\n]*\n[^\n]*\);)!\1\2\n\1return false;!g' $( git grep -l 'return error(' ) -END VERIFY SCRIPT-
2024-03-11refactor: Add missing {} around error() callsMarcoFalke
This is required for the next commit to be correct.
2024-02-15refactor: De-globalize g_signalsTheCharlatan
2023-12-04refactor: Remove calls to StartShutdown from KernelNotificationsRyan Ofsky
Use SignalInterrupt object instead. There is a slight change in behavior here because the previous StartShutdown code used to abort on failure and the new code logs errors instead.
2023-11-18blockstorage: switch from CAutoFile to AutoFileAnthony Towns
Also bump includes per suggestions from iwyu.
2023-11-16serialize: Drop useless version param from GetSerializeSize()Anthony Towns
2023-11-14Use ParamsWrapper for witness serializationAnthony Towns
2023-10-20coinstats: Fix hash_serialized2 calculationFabian Jahr
The legacy serialization was vulnerable to maleation and is fixed by adopting the same serialization procedure as was already in use for MuHash. This also includes necessary test fixes where the hash_serialized2 was hardcoded as well as correction of the regtest chainparams. Co-authored-by: Sebastian Falbesoner <sebastian.falbesoner@gmail.com>
2023-09-30validation: indexing changes for assumeutxoJames O'Beirne
When using an assumedvalid chainstate, only process validationinterface callbacks from the background chainstate within indexes. This ensures that all indexes are built in-order. Later, we can possibly designate indexes which can be built out of order and continue their operation during snapshot use. Once the background sync has completed, restart the indexes so that they continue to index the now-validated snapshot chainstate.
2023-09-30validation: pass ChainstateRole for validationinterface callsJames O'Beirne
This allows consumers to decide how to handle events from background or assumedvalid chainstates.
2023-09-15Return CAutoFile from BlockManager::Open*File()MarcoFalke
This is a refactor.
2023-09-12Remove unused GetType() from CBufferedFile and CAutoFileMarcoFalke
GetType() is only called in tests, so it is unused and can be removed.
2023-09-12Merge bitcoin/bitcoin#28427: index: coinstats reorg, fail when block cannot ↵fanquake
be reversed c0bf667912064960df194ea94150976b34f7c267 index: add [nodiscard] attribute to functions writing to the db (furszy) eef595560e9ecf3a0d1db4d8ea7ecc33a49d839f index: coinstats reorg, fail when block cannot be reversed (furszy) Pull request description: Found it while reviewing https://github.com/bitcoin/bitcoin/pull/24230#discussion_r1310863359. During a reorg, continuing execution when a block cannot be reversed leaves the coinstats index in an inconsistent state. This was surely overlooked when 'CustomRewind' was implemented. ACKs for top commit: ryanofsky: Code review ACK c0bf667912064960df194ea94150976b34f7c267. Only change since last review is new commit adding [[nodiscard]] Tree-SHA512: f4fc8522508d23e4fff09a29c935971819b1bd3b2a260e08e2e2b72f9340980d74fbec742a58fe216baf61d27de057c7c8300e8fa075f8507cd1227f128af909
2023-09-08refactor: remove clientversion include from dbwrapper.hCory Fields
2023-09-08index: add [nodiscard] attribute to functions writing to the dbfurszy
2023-09-07index: coinstats reorg, fail when block cannot be reversedfurszy
During a reorg, continuing execution when a block cannot be reversed leaves the coinstats index in an inconsistent state, which was surely overlooked when 'CustomRewind' was implemented.
2023-08-28Replace READWRITEAS macro with AsBase wrapping functionMarcoFalke
Co-authored-by: Pieter Wuille <pieter@wuille.net>
2023-08-17Remove unused includes from blockfilter.hMarcoFalke
This removes unused includes, primitives/block found manually, and the others by iwyu: blockfilter.h should remove these lines: - #include <serialize.h> // lines 16-16 - #include <undo.h> // lines 18-18
2023-08-07Merge bitcoin/bitcoin#28186: kernel: Prune leveldb headersfanquake
d8f1222ac50f089a0af29eaf8ce0555bad8366ef refactor: Correct dbwrapper key naming (TheCharlatan) be8f159ac59b9e700cbd3314ed71ebf39bd5b67a build: Remove leveldb from BITCOIN_INCLUDES (TheCharlatan) c95b37d641b1eed4a62d55ca5342a6ed8c7a1ce7 refactor: Move CDBWrapper leveldb members to their own context struct (TheCharlatan) c534a615e93452a5f509aaf5f68c600391a98d6a refactor: Split dbwrapper CDBWrapper::EstimateSize implementation (TheCharlatan) 586448888b72f7c87db4dcd30fc4e4044afae13b refactor: Move HandleError to dbwrapper implementation (TheCharlatan) dede0eef7adb7413f62f5abd68cac8e01635ba4a refactor: Split dbwrapper CDBWrapper::Exists implementation (TheCharlatan) a5c2eb57484314b04ec94523d14e0ef0c6c46d4f refactor: Fix logging.h includes (TheCharlatan) 84058e0eed9c05bc30984b39131e88ad1425628f refactor: Split dbwrapper CDBWrapper::Read implementation (TheCharlatan) e4af2408f2ac59788567b6fc8cb3a68fc43da9fe refactor: Pimpl leveldb::Iterator for CDBIterator (TheCharlatan) ef941ff1281e76308c3e746e592375bec023e9e4 refactor: Split dbwrapper CDBIterator::GetValue implementation (TheCharlatan) b7a1ab5cb4e60230f62c94efb3a10d07c9af4883 refactor: Split dbwrapper CDBIterator::GetKey implementation (TheCharlatan) d7437908cdf242626263ba9d5541addcddadc594 refactor: Split dbwrapper CDBIterator::Seek implementation (TheCharlatan) ea8135de7e617259cda3fc7b1c8e7569d454fd57 refactor: Pimpl leveldb::batch for CDBBatch (TheCharlatan) b9870c920dc475ec759eaf7339ea42aecba92138 refactor: Split dbwrapper CDBatch::Erase implementation (TheCharlatan) 532ee812a499e13b123af6b8415d8de1f3804f0f refactor: Split dbwrapper CDBBatch::Write implementation (TheCharlatan) afc534df9adbf5599b286b5dc3531a4b9ac2d056 refactor: Wrap DestroyDB in dbwrapper helper (TheCharlatan) Pull request description: Leveldb headers are currently included in the `dbwrapper.h` file and thus available to many of Bitcoin Core's source files. However, leveldb-specific functionality should be abstracted by the `dbwrapper` and does not need to be available to the rest of the code. Having leveldb included in a widely-used header such as `dbwrapper.h` bloats the entire project's header tree. The `dbwrapper` is a key component of the libbitcoinkernel library. Future users of this library would not want to contend with having the leveldb headers exposed and potentially polluting their project's namespace. For these reasons, the leveldb headers are removed from the `dbwrapper` by moving leveldb-specific code to the implementation file and creating a [pimpl](https://en.cppreference.com/w/cpp/language/pimpl) where leveldb member variables are indispensable. As a final step, the leveldb include flags are removed from the `BITCOIN_INCLUDES` and moved to places where the dbwrapper is compiled. --- This pull request is part of the [libbitcoinkernel project](https://github.com/bitcoin/bitcoin/issues/27587), and more specifically its stage 1 step 3 "Decouple most non-consensus headers from libbitcoinkernel". ACKs for top commit: stickies-v: re-ACK https://github.com/bitcoin/bitcoin/commit/d8f1222ac50f089a0af29eaf8ce0555bad8366ef MarcoFalke: ACK d8f1222ac50f089a0af29eaf8ce0555bad8366ef 🔠 Tree-SHA512: 0f58309be165af0162e648233451cd80fda88726fc10c0da7bfe4ec2ffa9afe63fbf7ffae9493698d3f39653b4ad870c372eee652ecc90ab1c29d86c387070f3
2023-08-05refactor: Fix logging.h includesTheCharlatan
These were uncovered as missing by the next commit.
2023-08-03lint: remove /* Continued */ markers from codebasefanquake
2023-07-10index: verify blocks data existence only oncefurszy
At present, during init, we traverse the chain (once per index) to confirm that all necessary blocks to sync each index up to the current tip are present. To make the process more efficient, we can fetch the oldest block from the indexers and perform the chain data existence check from that point only once. This also moves the pruning violation check to the end of the 'loadinit' thread, which is where the reindex, block loading and chain activation processes happen. Making the node's startup process faster, allowing us to remove the global g_indexes_ready_to_sync flag, and enabling the execution of the pruning violation verification even when the reindex or reindex-chainstate flags are enabled (which has being skipped so far).
2023-07-10init: don't start indexes sync thread prematurelyfurszy
By moving the 'StartIndexes()' call into the 'initload' thread, we can remove the threads active wait. Optimizing the available resources. The only difference with the current state is that now the indexes threads will only be started when they can process work and not before it.
2023-07-10refactor: simplify pruning violation checkfurszy
By generalizing 'GetFirstStoredBlock' and implementing 'CheckBlockDataAvailability' we can dedup code and avoid repeating work when multiple indexes are enabled. E.g. get the oldest block across all indexes and perform the pruning violation check from that point up to the tip only once (this feature is being introduced in a follow-up commit). This commit shouldn't change behavior in any way. Co-authored-by: Ryan Ofsky <ryan@ofsky.org>
2023-07-10make GetFirstStoredBlock assert that 'start_block' always has datafurszy
And transfer the responsibility of verifying whether 'start_block' has data or not to the caller. This is because the 'GetFirstStoredBlock' function responsibility is to return the first block containing data. And the current implementation can return 'start_block' when it has no data!. Which is misleading at least. Edge case behavior change: Previously, if the block tip lacked data but all preceding blocks contained data, there was no prune violation. And now, such scenario will result in a prune violation.
2023-07-10refactor: index, decouple 'Init' from 'Start'furszy
So indexes can be initialized without spawning the sync thread. This makes asynchronous indexes startup possible in the following commits.
2023-06-28kernel: Add fatalError method to notificationsTheCharlatan
FatalError replaces what previously was the AbortNode function in shutdown.cpp. This commit is part of the libbitcoinkernel project and further removes the shutdown's and, more generally, the kernel library's dependency on interface_ui with a kernel notification method. By removing interface_ui from the kernel library, its dependency on boost is reduced to just boost::multi_index. At the same time it also takes a step towards de-globalising the interrupt infrastructure. Co-authored-by: Russell Yanofsky <russ@yanofsky.org> Co-authored-by: TheCharlatan <seb.kung@gmail.com>
2023-06-28scripted-diff: Rename FatalError to FatalErrorfTheCharlatan
This is done in preparation for the next commit where a new FatalError function is introduced. FatalErrorf follows common convention to append 'f' for functions accepting format arguments. -BEGIN VERIFY SCRIPT- sed -i 's/FatalError/FatalErrorf/g' $( git grep -l 'FatalError') -END VERIFY SCRIPT-
2023-06-16Remove the syscall sandboxfanquake
After initially being merged in #20487, it's no-longer clear that an internal syscall sandboxing mechanism is something that Bitcoin Core should have/maintain, especially when compared to better maintained/supported alterantives, i.e firejail. Note that given where it's used, the sandbox also gets dragged into the kernel. There is some related discussion in #24771. This should not require any sort of deprecation, as this was only ever an opt-in, experimental feature. Closes #24771.
2023-06-08refactor: index: use `AbortNode` in fatal error helperSebastian Falbesoner
Deduplicates code in the `FatalError` template function by using `AbortNode` which does the exact same thing if called without any user message (i.e. without second parameter specified). The template is still kept for ease-of-use w.r.t. not having to call `tfm::format(...)` at the call-side each time, and also to keep the diff minimal.
2023-05-22index: prevent race by calling 'CustomInit' prior setting 'synced' flagfurszy
The 'm_synced' flag enables 'BlockConnected' events to be processed by the index. If we set the flag before calling 'CustomInit', we could be dispatching a block connected event to an uninitialized index child class. e.g. BlockFilterIndex, initializes the next filter position inside 'CustomInit'. So, if `CustomInit` is not called prior receiving the block event, the index will use 'next_filter_position=0' which overwrites the first filter in disk.
2023-05-17index: Enable reindex-chainstate with active indexesMartin Zumsande
This is achieved by letting the index sync thread wait until reindex-chainstate is finished. This also disables the pruning check when reindexing the chainstate (which is incompatible with prune mode) because there would be no chain at this point in init.
2023-05-17index: Use first block from locator instead of looking for fork pointMartin Zumsande
The index sync code has logic to go back the chain to the forking point, while also updating index-specific state, which is necessary to prevent possible corruption of the coinstatsindex. Also add a test for this (a reorg happens while the index is deactivated) that would not pass before this change.
2023-05-10refactor: Move functions to BlockManager methodsTheCharlatan
This is a commit in preparation for the next few commits. The functions are moved to methods to avoid their re-declaration for the purpose of passing in BlockManager options. The functions that were now moved into the BlockManager should no longer use the params as an argument, but instead use the member variable. In the moved ReadBlockFromDisk and UndoReadFromDisk, change the function signature to accept a reference to a CBlockIndex instead of a raw pointer. The pointer is expected to be non-null, so reflect that in the type. To allow for the move of functions to BlockManager methods all call sites require an instantiated BlockManager, or a callback to one.
2023-04-19move-only: Extract common/args and common/config.cpp from util/systemTheCharlatan
This is an extraction of ArgsManager related functions from util/system into their own common file. Config file related functions are moved to common/config.cpp. The background of this commit is an ongoing effort to decouple the libbitcoinkernel library from the ArgsManager. The ArgsManager belongs into the common library, since the kernel library should not depend on it. See doc/design/libraries.md for more information on this rationale.
2023-04-03Merge bitcoin/bitcoin#27254: refactor: Extract util/fs from util/systemfanquake
00e9b97f37e0bdf4c647236838c10b68b7ad5be3 refactor: Move fs.* to util/fs.* (TheCharlatan) 106b46d9d25b5228ef009fbbe6f9a7ae35090d15 Add missing fs.h includes (TheCharlatan) b202b3dd6393b415fa68e18dc49c9431dc6b58b2 Add missing cstddef include in assumptions.h (TheCharlatan) 18fb36367a28819bd5ab402344802796a1248979 refactor: Extract util/fs_helpers from util/system (Ben Woosley) Pull request description: This pull request is part of the `libbitcoinkernel` project https://github.com/bitcoin/bitcoin/issues/24303 https://github.com/bitcoin/bitcoin/projects/18 and more specifically its "Step 2: Decouple most non-consensus code from libbitcoinkernel". This commit was originally authored by empact and is taken from its parent PR #25152. #### Context There is an ongoing effort to decouple the `ArgsManager` used for command line parsing user-provided arguments from the libbitcoinkernel library (https://github.com/bitcoin/bitcoin/pull/25290, https://github.com/bitcoin/bitcoin/pull/25487, https://github.com/bitcoin/bitcoin/pull/25527, https://github.com/bitcoin/bitcoin/pull/25862, https://github.com/bitcoin/bitcoin/pull/26177, and https://github.com/bitcoin/bitcoin/pull/27125). The `ArgsManager` is defined in `system.h`. A similar pull request extracting functionality from `system.h` has been merged in https://github.com/bitcoin/bitcoin/pull/27238. #### Changes Next to providing better code organization, this PR removes some reliance of the tree of libbitcoinkernel header includes on `system.h` (and thus the `ArgsManager` definition) by moving filesystem related functions out of the `system.*` files. There is already a pair of `fs.h` / `fs.cpp` in the top-level `src/` directory. They were not combined with the files introduced here, to keep the patch cleaner and more importantly because they are often included without the utility functions. The new files are therefore named `fs_helpers` and the existing `fs` files are moved into the util directory. Further commits splitting more functionality out of `system.h` are still in #25152 and will be submitted in separate PRs once this PR has been processed. ACKs for top commit: hebasto: ACK 00e9b97f37e0bdf4c647236838c10b68b7ad5be3 Tree-SHA512: 31422f148d14ba3c843b99b1550a6fd77c77f350905ca324f93d4f97b652246bc58fa9696c64d1201979cf88733e40be02d262739bb7d417cf22bf506fdb7666
2023-03-23refactor: Extract util/fs_helpers from util/systemBen Woosley
This is an extraction of filesystem related functions from util/system into their own utility file. The background of this commit is an ongoing effort to decouple the libbitcoinkernel library from the ArgsManager defined in system.h. Moving these functions out of system.h allows including them from a separate source file without including the ArgsManager definitions from system.h.
2023-03-21Refactor: Remove unused FlatFilePos::SetNullMarcoFalke
generated by cgit v1.2.3 (git 2.26.2) at 2024-08-27 20:26:57 +0000