aboutsummaryrefslogtreecommitdiff
path: root/src/txdb.cpp
AgeCommit message (Collapse)Author
2020-07-29Merge #18637: coins: allow cache resize after initMarcoFalke
f19fdd47a6371dcbe0760ef6f3c3c5adb31b1bb4 test: add test for CChainState::ResizeCoinsCaches() (James O'Beirne) 8ac3ef46999ed676ca3775f7b2f461d92f09a542 add ChainstateManager::MaybeRebalanceCaches() (James O'Beirne) f36aaa6392fdbdac6891d92202d3efeff98754f4 Add CChainState::ResizeCoinsCaches (James O'Beirne) b223111da2e0e9ceccef75df8a20252b0094b7bc txdb: add CCoinsViewDB::ChangeCacheSize (James O'Beirne) Pull request description: This is part of the [assumeutxo project](https://github.com/bitcoin/bitcoin/projects/11): Parent PR: #15606 Issue: #15605 Specification: https://github.com/jamesob/assumeutxo-docs/tree/master/proposal --- In the assumeutxo implementation draft (#15056), once a UTXO snapshot is loaded, a new chainstate object is created after initialization. This means that we have to reclaim some of the cache that we've allocated to the original chainstate (per `dbcache=`) to repurpose for the snapshot chainstate. Furthermore, it makes sense to have different cache allocations depending on which chainstate is more active. While the snapshot chainstate is working to get to the network tip (and the background validation chainstate is idle), it makes sense that the snapshot chainstate should have the majority of cache allocation. And contrariwise once the snapshot has reached network tip, most of the cache should be given to the background validation chainstate. This set of changes (detailed in the commit messages) allows us to dynamically resize the various coins caches. None of the functionality introduced here is used at the moment, but will be in the next AU PR (which introduces `ActivateSnapshot`). `ChainstateManager::MaybeRebalanceCaches()` defines the (somewhat normative) cache allocations between the snapshot and background validation chainstates. I'd be interested in feedback if anyone has thoughts on the proportions I've set there. ACKs for top commit: ajtowns: weak utACK f19fdd47a6371dcbe0760ef6f3c3c5adb31b1bb4 -- didn't find any major problems, but not super confident that I didn't miss anything fjahr: Code review ACK f19fdd4 ryanofsky: Code review ACK f19fdd47a6371dcbe0760ef6f3c3c5adb31b1bb4. Only change since last review is constructor cleanup (no change in behavior). I think the suggestions here from ajtowns and others are good, but shouldn't delay merging the PR (and hold up assumeutxo) Tree-SHA512: fffb7847fb6993dd4a1a41cf11179b211b0b20b7eb5f7cf6266442136bfe9d43b830bbefcafd475bfd4af273f5573500594aa41fff03e0ed5c2a1e8562ff9269
2020-07-01txdb: add CCoinsViewDB::ChangeCacheSizeJames O'Beirne
We'll need this to dynamically update the cache size of the existing CCoinsViewDB instance when we create a new one during snapshot activation. This requires us to keep the CDBWrapper instance as a pointer instead of a reference so that we're able to destruct it and create a new instance when the cache size changes. Also renames `db` to `m_db` since we're already modifying each usage. Includes feedback from Russ Yanofsky.
2020-06-27scripted-diff: Move ui_interface to the node libMarcoFalke
-BEGIN VERIFY SCRIPT- # Move files git mv src/ui_interface.h src/node/ui_interface.h git mv src/ui_interface.cpp src/node/ui_interface.cpp sed -i -e 's/BITCOIN_UI_INTERFACE_H/BITCOIN_NODE_UI_INTERFACE_H/g' src/node/ui_interface.h # Adjust includes and makefile sed -i -e 's|ui_interface|node/ui_interface|g' $(git grep -l ui_interface) # Sort includes git diff -U0 | clang-format-diff -p1 -i -v -END VERIFY SCRIPT-
2020-06-04txdb: Remove unused boost/threadMarcoFalke
2020-05-20Merge #18317: Serialization improvements step 6 (all except wallet/gui)MarcoFalke
f9ee0f37c28f604bc82dab502ce229c66ef5b3b9 Add comments to CustomUintFormatter (Pieter Wuille) 4eb5643e3538863c9d2ff261f49a9a1b248de243 Convert everything except wallet/qt to new serialization (Pieter Wuille) 2b1f85e8c52c8bc5a17eae4c809eaf61d724af98 Convert blockencodings_tests to new serialization (Pieter Wuille) 73747afbbeb013669faf4c4d2c0903cec4526fb0 Convert merkleblock to new serialization (Pieter Wuille) d06fedd1bc26bf5bf2b203d4445aeaebccca780e Add SER_READ and SER_WRITE for read/write-dependent statements (Russell Yanofsky) 6f9a1e5ad0a270d3b5a715f3e3ea0911193bf244 Extend CustomUintFormatter to support enums (Russell Yanofsky) 769ee5fa0011ae658770586442715452a656559d Merge BigEndian functionality into CustomUintFormatter (Pieter Wuille) Pull request description: The next step of changes from #10785. This: * Adds support for enum serialization to `CustomUintFormatter`, used in `CAddress` for service flags. * Merges `BigEndian` into `CustomUintFormatter`, used in `CNetAddr` for port numbers. * Converts everything (except wallet and gui) to use the new serialization framework. ACKs for top commit: MarcoFalke: re-ACK f9ee0f37c2, only change is new documentation commit for CustomUintFormatter 📂 ryanofsky: Code review ACK f9ee0f37c28f604bc82dab502ce229c66ef5b3b9. Just new commit adding comment since last review jonatack: Code review re-ACK f9ee0f37c28f604bc82dab502ce229c6 only change since last review is an additional commit adding Doxygen documentation for `CustomUintFormatter`. Tree-SHA512: e7a0a36afae592d5a4ff8c81ae04d858ac409388e361f2bc197d9a78abca45134218497ab2dfd6d031e0cce0ca586cf857077b7c6ce17fccf67e2d367c1b6cd4
2020-04-16scripted-diff: Bump copyright headersMarcoFalke
-BEGIN VERIFY SCRIPT- ./contrib/devtools/copyright_header.py update ./ -END VERIFY SCRIPT-
2020-03-30Convert everything except wallet/qt to new serializationPieter Wuille
2020-02-10Get rid of VARINT default argumentPieter Wuille
This removes the need for the GNU C++ extension of variadic macros.
2020-01-21Convert compression.h to new serialization frameworkPieter Wuille
2019-12-30scripted-diff: Bump copyright of files changed in 2019MarcoFalke
-BEGIN VERIFY SCRIPT- ./contrib/devtools/copyright_header.py update ./ -END VERIFY SCRIPT-
2019-10-16Add some general std::vector utility functionsPieter Wuille
Added are: * Vector(arg1,arg2,arg3,...) constructs a vector with the specified arguments as elements. The vector's type is derived from the arguments. If some of the arguments are rvalue references, they will be moved into place rather than copied (which can't be achieved using list initialization). * Cat(vector1,vector2) returns a concatenation of the two vectors, efficiently moving elements when relevant. Vector generalizes (and replaces) the Singleton function in src/descriptor.cpp, and Cat replaces the Cat function in bech32.cpp
2019-10-05util: Simplify path argument for CBlockTreeDB ctorHennadii Stepanov
This commit does not change behavior as GetBlocksDir() with unset "-blocksdir" returns the same path.
2019-08-15refactor: have CCoins* data managed under CChainStateJames O'Beirne
This change encapsulates UTXO set data within CChainState instances, removing global data `pcoinsTip` and `pcoinsviewdb`. This is necessary if we want to maintain multiple chainstates with their own rendering of the UTXO set. We introduce a class CoinsViews which consolidates the construction of a CCoins* hierarchy. Construction of its various pieces (db, coinscatcher, in-memory cache) is split up so that we avoid flushing bad state to disk if startup is interrupted. We also introduce `CChainState::CanFlushToDisk()` which tells us when it is safe to flush the chainstate based on this partial construction. This commit could be broken into smaller pieces, but it would require more ephemeral diffs to, e.g., temporarily change CCoinsViewDB's constructor invocations. Other changes: - A parameter has been added to the CCoinsViewDB constructor that allows the name of the corresponding leveldb directory to be specified. Thanks to Russell Yanofsky and Marco Falke for helpful feedback.
2019-07-24scripted-diff: Make translation bilingualHennadii Stepanov
-BEGIN VERIFY SCRIPT- sed -i 's/inline std::string _(const char\* psz)/inline bilingual_str _(const char\* psz)/' src/util/translation.h sed -i 's/return G_TRANSLATION_FUN ? (G_TRANSLATION_FUN)(psz) : psz;/return bilingual_str{psz, G_TRANSLATION_FUN ? (G_TRANSLATION_FUN)(psz) : psz};/' src/util/translation.h sed -i 's/\b_("\([^"]\|\\"\)*")/&.translated/g' $(git grep --files-with-matches '\b_("' src) echo Hard cases - multiline strings. sed -i 's/"Visit %s for further information about the software.")/&.translated/g' src/init.cpp sed -i "s/\"Only rebuild the block database if you are sure that your computer's date and time are correct\")/&.translated/g" src/init.cpp sed -i 's/" restore from a backup.")/&.translated/g' src/wallet/db.cpp sed -i 's/" or address book entries might be missing or incorrect.")/&.translated/g' src/wallet/wallet.cpp echo Special case. sed -i 's/_(COPYRIGHT_HOLDERS)/&.translated/' src/util/system.cpp test/lint/lint-format-strings.py -END VERIFY SCRIPT-
2019-07-24Refactor out translation.hHennadii Stepanov
This is a prerequisite for introducing bilingual error messages. Note: #includes are arranged by clang-format-diff.py script.
2019-07-08refactoring: remove mapBlockIndex globalJames O'Beirne
in lieu of ::BlockIndex().
2019-06-02Make reasoning about dependencies easier by not including unused dependenciespracticalswift
2019-05-19Merge #12980: Allow quicker shutdowns during LoadBlockIndex()Jonas Schnelli
af5fa82b6 Allow quicker shutdowns during LoadBlockIndex() (Jonas Schnelli) Pull request description: ACKs for commit af5fa8: promag: utACK af5fa82b676a36e60eda080ca0a946bdfffefd49. practicalswift: utACK af5fa82b676a36e60eda080ca0a946bdfffefd49 Tree-SHA512: 1c64dcc5d8a9d3411553257cd5a598dcd29be981660e5bca9283c1d957dc56798abcf41d9969cd573088137597a23e48e62a8c476c463d3f176b86a10048f47b
2018-11-04scripted-diff: Move util files to separate directory.Jim Posen
-BEGIN VERIFY SCRIPT- mkdir -p src/util git mv src/util.h src/util/system.h git mv src/util.cpp src/util/system.cpp git mv src/utilmemory.h src/util/memory.h git mv src/utilmoneystr.h src/util/moneystr.h git mv src/utilmoneystr.cpp src/util/moneystr.cpp git mv src/utilstrencodings.h src/util/strencodings.h git mv src/utilstrencodings.cpp src/util/strencodings.cpp git mv src/utiltime.h src/util/time.h git mv src/utiltime.cpp src/util/time.cpp sed -i 's/<util\.h>/<util\/system\.h>/g' $(git ls-files 'src/*.h' 'src/*.cpp') sed -i 's/<utilmemory\.h>/<util\/memory\.h>/g' $(git ls-files 'src/*.h' 'src/*.cpp') sed -i 's/<utilmoneystr\.h>/<util\/moneystr\.h>/g' $(git ls-files 'src/*.h' 'src/*.cpp') sed -i 's/<utilstrencodings\.h>/<util\/strencodings\.h>/g' $(git ls-files 'src/*.h' 'src/*.cpp') sed -i 's/<utiltime\.h>/<util\/time\.h>/g' $(git ls-files 'src/*.h' 'src/*.cpp') sed -i 's/BITCOIN_UTIL_H/BITCOIN_UTIL_SYSTEM_H/g' src/util/system.h sed -i 's/BITCOIN_UTILMEMORY_H/BITCOIN_UTIL_MEMORY_H/g' src/util/memory.h sed -i 's/BITCOIN_UTILMONEYSTR_H/BITCOIN_UTIL_MONEYSTR_H/g' src/util/moneystr.h sed -i 's/BITCOIN_UTILSTRENCODINGS_H/BITCOIN_UTIL_STRENCODINGS_H/g' src/util/strencodings.h sed -i 's/BITCOIN_UTILTIME_H/BITCOIN_UTIL_TIME_H/g' src/util/time.h sed -i 's/ util\.\(h\|cpp\)/ util\/system\.\1/g' src/Makefile.am sed -i 's/utilmemory\.\(h\|cpp\)/util\/memory\.\1/g' src/Makefile.am sed -i 's/utilmoneystr\.\(h\|cpp\)/util\/moneystr\.\1/g' src/Makefile.am sed -i 's/utilstrencodings\.\(h\|cpp\)/util\/strencodings\.\1/g' src/Makefile.am sed -i 's/utiltime\.\(h\|cpp\)/util\/time\.\1/g' src/Makefile.am sed -i 's/-> util ->/-> util\/system ->/' test/lint/lint-circular-dependencies.sh sed -i 's/src\/util\.cpp/src\/util\/system\.cpp/g' test/lint/lint-format-strings.py test/lint/lint-locale-dependence.sh sed -i 's/src\/utilmoneystr\.cpp/src\/util\/moneystr\.cpp/g' test/lint/lint-locale-dependence.sh sed -i 's/src\/utilstrencodings\.\(h\|cpp\)/src\/util\/strencodings\.\1/g' test/lint/lint-locale-dependence.sh sed -i 's/src\\utilstrencodings\.cpp/src\\util\\strencodings\.cpp/' build_msvc/libbitcoinconsensus/libbitcoinconsensus.vcxproj -END VERIFY SCRIPT-
2018-08-02Allow quicker shutdowns during LoadBlockIndex()Jonas Schnelli
2018-07-27Return void instead of bool for functions that cannot failpracticalswift
* CBlockTreeDB::ReadReindexing(...) * CChainState::ResetBlockFailureFlags(...) * CTxMemPool::addUnchecked(...) * CWallet::LoadDestData(...) * CWallet::LoadKeyMetadata(...) * CWallet::LoadScriptMetadata(...) * CWallet::LoadToWallet(...) * CWallet::SetHDChain(...) * CWallet::SetHDSeed(...) * RemoveLocal(...) * SetMinVersion(...) * StartHTTPServer(...) * StartRPC(...) * TorControlConnection::Disconnect(...)
2018-07-24scripted-diff: Remove trailing whitespacesJoão Barbosa
-BEGIN VERIFY SCRIPT- sed --in-place'' --regexp-extended 's/[[:space:]]+$//g' $(git grep -I --files-with-matches --extended-regexp '[[:space:]]+$' -- src test ':!*.svg' ':!src/crypto/sha256_sse4*' ':!src/leveldb' ':!src/qt/locale' ':!src/secp256k1' ':!src/univalue') -END VERIFY SCRIPT-
2018-06-25Break circular dependency: init -> * -> init by extracting shutdown.hBen Woosley
Most includers just wanted to react to pending shutdown. This isolates access to `fRequestShutdown` and limits access to the shutdown api functions, including the new `AbortShutdown` for setting it to `false`. Note I originally called `AbortShutdown` `CancelShutdown` but that name was already taken by winuser.h https://travis-ci.org/bitcoin/bitcoin/jobs/386913329 This change also triggered a build error in bench. Fixing it required moving LIBBITCOIN_SERVER after LIBBITCOIN_WALLET in bench_bench_bitcoin_LDADD To make server definitions in src/net.cpp available to wallet methods in src/wallet/wallet.cpp. Specifically, solving: libbitcoin_wallet.a(libbitcoin_wallet_a-wallet.o): In function `CWalletTx::RelayWalletTransaction(CConnman*)': wallet.cpp:(.text+0x3f0e): undefined reference to `CConnman::NodeFullyConnected(CNode const*)' collect2: error: ld returned 1 exit status https://travis-ci.org/bitcoin/bitcoin/jobs/392133581 Need for remaining init.h includes confirmed via a thorough search with a more specific regex: \bInterrupt\(\)|\bShutdown\(\)|\bInitLogging\(\)|\bInitParameterInteraction\(\)|\bAppInitBasicSetup\(\)|\bAppInitParameterInteraction\(\)|\bAppInitSanityChecks\(\)|\bAppInitLockDataDirectory\(\)|\bAppInitMain\(\)|\bSetupServerArgs\(\)|\bLicenseInfo\(\)|g_wallet_init_interface|init.h
2018-06-04index: Move index DBs into index/ directory.Jim Posen
2018-06-04db: Make reusable base class for index databases.Jim Posen
2018-06-04db: Remove obsolete methods from CBlockTreeDB.Jim Posen
2018-04-25[db] Migration for txindex data to new, separate database.Jim Posen
2018-04-25[db] Create separate database for txindex.Jim Posen
The new TxIndexDB class will be used by a future commit in this change set.
2018-04-07[logging] Comment all continuing logs.John Newbery
Most logs should terminated with a '\n'. Some logs are built up over multiple calls to logPrintf(), so do not need a newline terminater. Comment all of these 'continued' logs as a linter hing.
2018-03-27Merge #12653: Allow to optional specify the directory for the blocks storageWladimir J. van der Laan
a192636 -blocksdir: keep blockindex leveldb database in datadir (Jonas Schnelli) f38e4fd QA: Add -blocksdir test (Jonas Schnelli) 386a6b6 Allow to optional specify the directory for the blocks storage (Jonas Schnelli) Pull request description: Since the actual block files taking up more and more space, it may be desirable to have them stored in a different location then the data directory (use case: SSD for chainstate, etc., HD for blocks). This PR adds a `-blocksdir` option that allows one to keep the blockfiles and the blockindex external from the data directory (instead of creating symlinks). I fist had an option to keep the blockindex within the datadir, but seems to make no sense since accessing the index will (always) lead to access (r/w) the block files. Tree-SHA512: f8b9e1a681679eac25076dc30e45e6e12d4b2d9ac4be907cbea928a75af081dbcb0f1dd3e97169ab975f73d0bd15824c00c2a34638f3b284b39017171fce2409
2018-03-20-blocksdir: keep blockindex leveldb database in datadirJonas Schnelli
2018-03-15Add static_assert to prevent VARINT(<signed value>)Russell Yanofsky
Using VARINT with signed types is dangerous because negative values will appear to serialize correctly, but then deserialize as positive values mod 128. This commit changes the VARINT macro to trigger an error by default if called with an signed value, and updates broken uses of VARINT to pass a special flag that lets them keep working with no change in behavior.
2018-03-13Support deserializing into temporariesPieter Wuille
Currently, the READWRITE macro cannot be passed any non-const temporaries, as the SerReadWrite function only accepts lvalue references. Deserializing into a temporary is very common, however. See for example things like 's >> VARINT(n)'. The VARINT macro produces a temporary wrapper that holds a reference to n. Fix this by accepting non-const rvalue references instead of lvalue references. We don't propagate the rvalue-ness down, as there are no useful optimizations that only apply to temporaries. Then use this new functionality to get rid of many (but not all) uses of the 'REF' macro (which casts away constness).
2018-03-11Allow to optional specify the directory for the blocks storageJonas Schnelli
2018-01-03Increment MIT Licence copyright header year on files modified in 2017Akira Takizawa
2017-11-16scripted-diff: Replace #include "" with #include <> (ryanofsky)MeshCollider
-BEGIN VERIFY SCRIPT- for f in \ src/*.cpp \ src/*.h \ src/bench/*.cpp \ src/bench/*.h \ src/compat/*.cpp \ src/compat/*.h \ src/consensus/*.cpp \ src/consensus/*.h \ src/crypto/*.cpp \ src/crypto/*.h \ src/crypto/ctaes/*.h \ src/policy/*.cpp \ src/policy/*.h \ src/primitives/*.cpp \ src/primitives/*.h \ src/qt/*.cpp \ src/qt/*.h \ src/qt/test/*.cpp \ src/qt/test/*.h \ src/rpc/*.cpp \ src/rpc/*.h \ src/script/*.cpp \ src/script/*.h \ src/support/*.cpp \ src/support/*.h \ src/support/allocators/*.h \ src/test/*.cpp \ src/test/*.h \ src/wallet/*.cpp \ src/wallet/*.h \ src/wallet/test/*.cpp \ src/wallet/test/*.h \ src/zmq/*.cpp \ src/zmq/*.h do base=${f%/*}/ relbase=${base#src/} sed -i "s:#include \"\(.*\)\"\(.*\):if test -e \$base'\\1'; then echo \"#include <\"\$relbase\"\\1>\\2\"; else echo \"#include <\\1>\\2\"; fi:e" $f done -END VERIFY SCRIPT-
2017-09-07Merge #10770: Drop upgrade-cancel callback registration for a generic ↵Jonas Schnelli
"cancelable" ee4d1493e Drop upgrade-cancel callback registration for a generic "resumeable" (Matt Corallo) Pull request description: Instead of passing a StartShutdown reference all the way up from txdb, give ShowProgress a "cancelable" boolean, as StartShutdown is pretty much always what you'll want to use to cancel. Use the same boolean to allow cancel during initial block verification. Tree-SHA512: 515817aaa4b9e3e856200e00be9c2d44ecfa2d4f288fe3e02116105fe85de2650c13076ee7e45396ec1ce6ab45e53b0477cddda7cfdee5b3bd0589cb81a4c346
2017-08-20Drop upgrade-cancel callback registration for a generic "resumeable"Matt Corallo
Instead of passing a StartShutdown reference all the way up from txdb, give ShowProgress a "resumeable" boolean, which is used to inform the user if the action will be resumed, but cancel is always allowed by just calling StartShutdown().
2017-08-16Declare single-argument (non-converting) constructors "explicit"practicalswift
In order to avoid unintended implicit conversions.
2017-08-14scripted-diff: stop using the gArgs wrappersMarko Bencun
They were temporary additions to ease the transition. -BEGIN VERIFY SCRIPT- find src/ -name "*.cpp" ! -wholename "src/util.h" ! -wholename "src/util.cpp" | xargs perl -i -pe 's/(?<!\.)(ParseParameters|ReadConfigFile|IsArgSet|(Soft|Force)?(Get|Set)(|Bool|)Arg(s)?)\(/gArgs.\1(/g' -END VERIFY SCRIPT-
2017-08-07Skip remainder of init if upgrade is cancelledMatt Corallo
2017-07-27Force on-the-fly compaction during pertxout upgradePieter Wuille
2017-07-07Avoid dereference-of-casted-pointerPieter Wuille
2017-06-29Report [CANCELLED] instead of [DONE] when shut down during txdb upgradeJonas Schnelli
2017-06-29Report txdb upgrade not more often then every 10%Jonas Schnelli
2017-06-29Show txdb upgrade progress in debug logJonas Schnelli
2017-06-29Allow to cancel the txdb upgrade via splashscreen callbackJonas Schnelli
2017-06-29Allow to shut down during txdb upgradeJonas Schnelli
2017-06-26[qa] Test non-atomic chainstate writesSuhas Daftuar
Adds new functional test, dbcrash.py, which uses -dbcrashratio to exercise the logic for recovering from a crash during chainstate flush. dbcrash.py is added to the extended tests, as it may take ~10 minutes to run Use _Exit() instead of exit() for crash simulation This eliminates stderr output such as: terminate called without an active exception or Assertion failed: (!pthread_mutex_destroy(&m)), function ~recursive_mutex, file /usr/local/include/boost/thread/pthread/recursive_mutex.hpp, line 104. Eliminating the stderr output on crash simulation allows testing with test_runner.py, which reports a test as failed if stderr is produced.
2017-06-26Dont create pcoinsTip until after ReplayBlocks.Matt Corallo
This requires that we not access pcoinsTip in InitBlockIndex's FlushStateToDisk (so we just skip it until later in AppInitMain) and the LoadChainTip in LoadBlockIndex (which there is already one later in AppinitMain, after ReplayBlocks, so skipping it there is fine). Includes some simplifications by Suhas Daftuar and Pieter Wuille.
generated by cgit v1.2.3 (git 2.26.2) at 2024-11-01 17:06:06 +0000