aboutsummaryrefslogtreecommitdiff
path: root/src/wallet/test
AgeCommit message (Collapse)Author
2024-05-21Merge bitcoin/bitcoin#26606: wallet: Implement independent BDB parsermerge-script
d51fbab4b32d56765e8faab6ad01245fb259b0ca wallet, test: Be able to always swap BDB endianness (Ava Chow) 0b753156ce60c29efb2386954ba7555ad8f642f5 test: Test bdb_ro dump of wallet without reset LSNs (Ava Chow) c1984f128284589423b7e0cc06c9a3b23a242d95 test: Test dumping dbs with overflow pages (Ava Chow) fd7b16e391ed320e35255157a28be14c947ef30a test: Test dumps of other endian BDB files (Ava Chow) 6ace3e953f0864bd7818f040c59a1bc70aa47512 bdb: Be able to make byteswapped databases (Ava Chow) d9878903fb34939dee8e1462f079acc68110253d Error if LSNs are not reset (Ava Chow) 4d7a3ae78e55f25868979f1bd920857a4aecb825 Berkeley RO Database fuzz test (TheCharlatan) 3568dce9e93295674cdf5458c5bdf93ff01fd0a2 tests: Add BerkeleyRO to db prefix tests (Ava Chow) 70cfbfdadf16d3b115309c6938f07ef5b96c7cc1 wallettool: Optionally use BERKELEY_RO as format when dumping BDB wallets (Ava Chow) dd57713f6ede3d46e97ee7df87c10001b0bf4c3d Add MakeBerkeleyRODatabase (Ava Chow) 6e50bee67d1d58aecd8a0ce8b7c3f5a7979365f5 Implement handling of other endianness in BerkeleyRODatabase (Ava Chow) cdd61c9cc108df8e13f4e3891ff2c96355b3ee38 wallet: implement independent BDB deserializer in BerkeleyRODatabase (Ava Chow) ecba23097955dad7208baa687fc405c846aee794 wallet: implement BerkeleyRODatabase::Backup (Ava Chow) 0c8e72847603540bb29b8b8aeb80fa3f2e3a2c9a wallet: implement BerkeleyROBatch (Ava Chow) 756ff9b478484b17c4a6e65c171c2e4fecb21ad4 wallet: add dummy BerkeleyRODatabase and BerkeleyROBatch classes (Ava Chow) ca18aea5c4975ace4e307be96c74641d203fa389 Add AutoFile::seek and tell (Ava Chow) Pull request description: Split from #26596 This PR adds `BerkeleyRODatabase` which is an independent implementation of a BDB file parser. It provides read only access to a BDB file, and can therefore be used as a read only database backend for wallets. This will be used for dumping legacy wallet records and migrating legacy wallets without the need for BDB itself. Wallettool's `dump` command is changed to use `BerkeleyRODatabase` instead of `BerkeleyDatabase` (and `CWallet` itself) to demonstrate that this parser works and to test it against the existing wallettool functional tests. ACKs for top commit: josibake: reACK https://github.com/bitcoin/bitcoin/commit/d51fbab4b32d56765e8faab6ad01245fb259b0ca TheCharlatan: Re-ACK d51fbab4b32d56765e8faab6ad01245fb259b0ca furszy: reACK d51fbab4b32d56765e8faab6ad01245fb259b0ca laanwj: re-ACK d51fbab4b32d56765e8faab6ad01245fb259b0ca theStack: ACK d51fbab4b32d56765e8faab6ad01245fb259b0ca Tree-SHA512: 1e7b97edf223b2974eed2e9eac1179fc82bb6359e0a66b7d2a0c8b9fa515eae9ea036f1edf7c76cdab2e75ad994962b134b41056ccfbc33b8d54f0859e86657b
2024-05-16Berkeley RO Database fuzz testTheCharlatan
2024-05-16tests: Add BerkeleyRO to db prefix testsAva Chow
2024-05-13wallet: implement BerkeleyROBatchAva Chow
Implement ReadKey and HasKey of BerkeleyROBatch, and Next of BerkeleyROCursor. Also adds the containers for records to BerkeleyRODatabase so that BerkeleyROBatch will be able to access the records.
2024-05-01scripted-diff: Add IWYU pragma keep to bitcoin-config.h includesMarcoFalke
-BEGIN VERIFY SCRIPT- perl -0777 -pi -e 's/#if defined\(HAVE_CONFIG_H\)\n#include <config\/bitcoin-config.h>.*\n#endif.*\n/#include <config\/bitcoin-config.h> \/\/ IWYU pragma: keep\n/g' $( git grep -l '#include <config/bitcoin-config.h>' ) -END VERIFY SCRIPT-
2024-04-25refactor: Avoid copying util::Result valuesRyan Ofsky
Copying util::Result values is less efficient than moving them because they allocate memory and contain strings. Also this is needed to avoid compile errors in https://github.com/bitcoin/bitcoin/pull/25722 which adds a std::unique_ptr member to util::Result which implicity disables copying.
2024-04-25refactor: Drop util::Result operator=Ryan Ofsky
`util::Result` objects are aggregates that can hold multiple fields with different information. Currently Result objects can only hold a success value of an arbitrary type or a single bilingual_str error message. In followup PR https://github.com/bitcoin/bitcoin/pull/25722, Result objects may be able to hold both success and failure values of different types, plus error and warning messages. Having a Result::operator= assignment operator that completely erases all existing Result information before assigning new information is potentially dangerous in this case. For example, code that looks like it is assigning a warning value could erase previously-assigned success or failure values. Conversely, code that looks like it is just assigning a success or failure value could erase previously assigned error and warning messages. To prevent potential bugs like this, disable Result::operator= assignment operator. It is possible in the future we may want to re-enable operator= in limited cases (such as when implicit conversions are not used) or add a Replace() or Reset() method that mimicks default operator= behavior. Followup PR https://github.com/bitcoin/bitcoin/pull/25722 also adds a Result::Update() method providing another way to update an existing Result object. Co-authored-by: stickies-v <stickies-v@protonmail.com>
2024-03-29Merge bitcoin/bitcoin#29130: wallet: Add `createwalletdescriptor` and ↵Ryan Ofsky
`gethdkeys` RPCs for adding new automatically generated descriptors 746b6d88395607abbd3c13bbdcdd4ca83e9bc9e4 test: Add test for createwalletdescriptor (Ava Chow) 2402b6306215a9ee8d5f4068ea81f4e7f324adeb wallet: Test upgrade of pre-taproot wallet to have tr() descriptors (Ava Chow) 460ae1bf67c0051033c1802d44787d173abb9248 wallet, rpc: Add createwalletdescriptor RPC (Ava Chow) 8e1a475062e62321e58a0624385cc3fa0885aa12 wallet: Be able to retrieve single key from descriptors (Ava Chow) 85b1fb19dd3a3f3c68da1c5e60a6eb911e1119a6 wallet: Add GetActiveHDPubKeys to retrieve xpubs from active descriptors (Ava Chow) 73926f2d31b61ff78d5f0c8f9b5e3130fb1f9620 wallet, descspkm: Refactor wallet descriptor generation to standalone func (Andrew Chow) 54e74f46ea10e479be682750c1279165f29bb2f4 wallet: Refactor function for single DescSPKM setup (Andrew Chow) 3b09d0eb7f2c1d6ebdab73d18db28e5bf7d74f18 tests: Test for gethdkeys (Ava Chow) 5febe28c9e131fb93fac9c35f80c42759654f150 wallet, rpc: Add gethdkeys RPC (Ava Chow) 66632e5c24c1b59afef1e89b562fbd0117ab6ef5 wallet: Add IsActiveScriptPubKeyMan (Ava Chow) fa6a259985b61235ebc21eae2a76014cc9437d5f desc spkm: Add functions to retrieve specific private keys (Ava Chow) fe67841464cc0f970a1c233caba92cb78e9c78dc descriptor: Be able to get the pubkeys involved in a descriptor (Ava Chow) ef6745879d87cdb6f1061337867a689167e965a1 key: Add constructor for CExtKey that takes CExtPubKey and CKey (Ava Chow) Pull request description: This PR adds a `createwalletdescriptor` RPC which allows users to add new automatically generated descriptors to their wallet, e.g. to upgrade a 0.21.x wallet to contain a taproot descriptor. This RPC takes 3 arguments: the output type to create a descriptor for, whether the descriptor will be internal or external, and the HD key to use if the user wishes to use a specific key. The HD key is an optional parameter. If it is not specified, the wallet will use the key shared by the active descriptors, if they are all single key. For most users in the expected upgrade scenario, this should be sufficient. In more advanced cases, the user must specify the HD key to use. Currently, specified HD keys must already exist in the wallet. To make it easier for the user to know, `gethdkeys` is also added to list out the HD keys in use by all of the descriptors in the wallet. This will include all HD keys, whether we have the private key, for it, which descriptors use it and their activeness, and optionally the extended private key. In this way, users with more complex wallets will be still be able to get HD keys from their wallet for use in other scenarios, and if they want to use `createwalletdescriptor`, they can easily get the keys that they can specify to it. See also https://github.com/bitcoin/bitcoin/pull/26728#issuecomment-1866961865 ACKs for top commit: Sjors: re-utACK 746b6d88395607abbd3c13bbdcdd4ca83e9bc9e4 furszy: ACK 746b6d8 ryanofsky: Code review ACK 746b6d88395607abbd3c13bbdcdd4ca83e9bc9e4, and this looks ready to merge. There were various suggested changes since last review where main change seems to be switching `gethdkeys` output to use normalized descriptors (removing hardened path components). Tree-SHA512: f2849101e6fbf1f59cb031eaaaee97af5b1ae92aaab54c5716940d210f08ab4fc952df2725b636596cd5747b8f5beb1a7a533425bc10d09da02659473516fbda
2024-03-08Merge bitcoin/bitcoin#28960: kernel: Remove dependency on CSchedulerAva Chow
d5228efb5391b31a9a0673019e43e7fa2cd4ac07 kernel: Remove dependency on CScheduler (TheCharlatan) 06069b3913dda048f5d640a662b0852f86346ace scripted-diff: Rename MainSignals to ValidationSignals (TheCharlatan) 0d6d2b650d1017691f48c9109a6cd020ab46aa73 scripted-diff: Rename SingleThreadedSchedulerClient to SerialTaskRunner (TheCharlatan) 4abde2c4e3fd9b66394b79874583bdc2a9132c36 [refactor] Make MainSignals RAII styled (TheCharlatan) 84f5c135b8118cbe15b8bfb4db80d61237987f64 refactor: De-globalize g_signals (TheCharlatan) 473dd4b97ae40e43e1a1a97fdbeb40be4855e9bc [refactor] Prepare for g_signals de-globalization (TheCharlatan) 3fba3d5deec6d7bae33823b8da7682f9b03d9deb [refactor] Make signals optional in mempool and chainman (TheCharlatan) Pull request description: By defining a virtual interface class for the scheduler client, users of the kernel can now define their own event consuming infrastructure, without having to spawn threads or rely on the scheduler design. Removing `CScheduler` also allows removing the thread and exception modules from the kernel library. To make the `CMainSignals` class easier to use from a kernel library perspective, remove its global instantiation and adopt RAII practices. Renames `CMainSignals` to `ValidationSignals`, which more accurately describes its purpose and scope. Also make the `ValidationSignals` in the `ChainstateManager` and CTxMemPool` optional. This could be useful in the future for using or testing these classes without having to instantiate any form of signal handling. --- This PR is part of the [libbitcoinkernel project](https://github.com/bitcoin/bitcoin/issues/27587). It improves the kernel API and removes two modules from the kernel library. ACKs for top commit: maflcko: re-ACK d5228efb5391b31a9a0673019e43e7fa2cd4ac07 🌄 ryanofsky: Code review ACK d5228efb5391b31a9a0673019e43e7fa2cd4ac07. Just comment change since last review. vasild: ACK d5228efb5391b31a9a0673019e43e7fa2cd4ac07 furszy: diff ACK d5228ef Tree-SHA512: e93a5f10eb6182effb84bb981859a7ce750e466efd8171045d8d9e7fe46e4065631d9f6f533c5967c4d34c9bb7d7a67e9f4593bd4c5b30cd7b3bbad7be7b331b
2024-02-21[fuzz] Avoid partial negative resultMurch
2024-02-20descriptor: Be able to get the pubkeys involved in a descriptorAva Chow
2024-02-20Merge bitcoin/bitcoin#29404: refactor: bitcoin-config.h includes cleanupfanquake
9d1dbbd4ceb8c04340927f5127195dc306adf3fc scripted-diff: Fix bitcoin_config_h includes (TheCharlatan) Pull request description: As mentioned in https://github.com/bitcoin/bitcoin/pull/26924#issuecomment-1403449932 and https://github.com/bitcoin/bitcoin/pull/29263#issuecomment-1922334399, it is currently not safe to remove `bitcoin-config.h` includes from headers because some unrelated file might be depending on it. See also #26972 for discussion. Solve this by including the file directly everywhere it's required, regardless of whether or not it's already included by another header. There should be no functional change here, but it will allow us to safely remove includes from headers in the future. ~I'm afraid it's a bit tedious to reproduce these commits, but it's reasonably straightforward:~ Edit: See note below ```bash # All commands executed from the src/ subdir. # Collect all tokens from bitcoin-config.h.in # Isolate the tokens and remove blank lines # Replace newlines with | and remove the last trailing one # Collect all files which use these tokens # Filter out subprojects (proper forwarding can be verified from Makefiles) # Filter out .rc files # Save to a text file git grep -E -l `grep undef config/bitcoin-config.h.in | cut -d" " -f2 | grep -v '^$' | tr '\n' '|' | sed 's/|$//'` | grep -v -e "^leveldb/" -e "^secp256k1/" -e "^crc32c/" -e "^minisketch/" -e "^Makefile" -e "\.rc$" > files-with-config-include.txt # Find all files from the above list which don't include bitcoin-config.h git grep -L -E "config/bitcoin-config.h" -- `cat files-with-config-include.txt` # Include them manually with the exception of some files in crypto: # crypto/sha256_arm_shani.cpp crypto/sha256_avx2.cpp crypto/sha256_sse41.cpp crypto/sha256_x86_shani.cpp # These are exceptions which don't use bitcoin-config.h, rather the Makefile.am adds these cppflags manually. # Commit changes. This should match the first commit of this PR. # Use the same search as above to find all files which DON'T use any config tokens git grep -E -L `grep undef config/bitcoin-config.h.in | cut -d" " -f2 | grep -v '^$' | tr '\n' '|' | sed 's/|$//'` | grep -v -e "^leveldb/" -e "^secp256k1/" -e "^crc32c/" -e "^minisketch/" -e "^Makefile" -e "\.rc$" > files-without-config-include.txt # Manually remove the includes and commit changes. This should match the second commit of this PR. ``` Edit: I'll keep this old description for posterity, but the manual approach has been replaced with a scripted diff from TheCharlatan ACKs for top commit: maflcko: ACK 9d1dbbd4ceb8c04340927f5127195dc306adf3f 🚪 TheCharlatan: ACK 9d1dbbd4ceb8c04340927f5127195dc306adf3fc hebasto: ACK 9d1dbbd4ceb8c04340927f5127195dc306adf3fc, I have reviewed the code and it looks OK. fanquake: ACK 9d1dbbd4ceb8c04340927f5127195dc306adf3fc Tree-SHA512: f11ddc4ae6a887f96b954a6b77f310558ddb271088a3fda3edc833669c4251b7f392515224bbb8e5f67eb2c799b4ffed3b07d96454e82ec635c686d0df545872
2024-02-15refactor: De-globalize g_signalsTheCharlatan
2024-02-13scripted-diff: Fix bitcoin_config_h includesTheCharlatan
-BEGIN VERIFY SCRIPT- regex_string='^(?!//).*(AC_APPLE_UNIVERSAL_BUILD|BOOST_PROCESS_USE_STD_FS|CHAR_EQUALS_INT8|CLIENT_VERSION_BUILD|CLIENT_VERSION_IS_RELEASE|CLIENT_VERSION_MAJOR|CLIENT_VERSION_MINOR|COPYRIGHT_HOLDERS|COPYRIGHT_HOLDERS_FINAL|COPYRIGHT_HOLDERS_SUBSTITUTION|COPYRIGHT_YEAR|ENABLE_ARM_SHANI|ENABLE_AVX2|ENABLE_EXTERNAL_SIGNER|ENABLE_SSE41|ENABLE_TRACING|ENABLE_WALLET|ENABLE_X86_SHANI|ENABLE_ZMQ|HAVE_BOOST|HAVE_BUILTIN_CLZL|HAVE_BUILTIN_CLZLL|HAVE_BYTESWAP_H|HAVE_CLMUL|HAVE_CONSENSUS_LIB|HAVE_CXX20|HAVE_DECL_BE16TOH|HAVE_DECL_BE32TOH|HAVE_DECL_BE64TOH|HAVE_DECL_BSWAP_16|HAVE_DECL_BSWAP_32|HAVE_DECL_BSWAP_64|HAVE_DECL_FORK|HAVE_DECL_FREEIFADDRS|HAVE_DECL_GETIFADDRS|HAVE_DECL_HTOBE16|HAVE_DECL_HTOBE32|HAVE_DECL_HTOBE64|HAVE_DECL_HTOLE16|HAVE_DECL_HTOLE32|HAVE_DECL_HTOLE64|HAVE_DECL_LE16TOH|HAVE_DECL_LE32TOH|HAVE_DECL_LE64TOH|HAVE_DECL_PIPE2|HAVE_DECL_SETSID|HAVE_DECL_STRERROR_R|HAVE_DEFAULT_VISIBILITY_ATTRIBUTE|HAVE_DLFCN_H|HAVE_DLLEXPORT_ATTRIBUTE|HAVE_ENDIAN_H|HAVE_EVHTTP_CONNECTION_GET_PEER_CONST_CHAR|HAVE_FDATASYNC|HAVE_GETENTROPY_RAND|HAVE_GETRANDOM|HAVE_GMTIME_R|HAVE_INTTYPES_H|HAVE_LIBADVAPI32|HAVE_LIBCOMCTL32|HAVE_LIBCOMDLG32|HAVE_LIBGDI32|HAVE_LIBIPHLPAPI|HAVE_LIBKERNEL32|HAVE_LIBOLE32|HAVE_LIBOLEAUT32|HAVE_LIBSHELL32|HAVE_LIBSHLWAPI|HAVE_LIBUSER32|HAVE_LIBUUID|HAVE_LIBWINMM|HAVE_LIBWS2_32|HAVE_MALLOC_INFO|HAVE_MALLOPT_ARENA_MAX|HAVE_MINIUPNPC_MINIUPNPC_H|HAVE_MINIUPNPC_UPNPCOMMANDS_H|HAVE_MINIUPNPC_UPNPERRORS_H|HAVE_NATPMP_H|HAVE_O_CLOEXEC|HAVE_POSIX_FALLOCATE|HAVE_PTHREAD|HAVE_PTHREAD_PRIO_INHERIT|HAVE_STDINT_H|HAVE_STDIO_H|HAVE_STDLIB_H|HAVE_STRERROR_R|HAVE_STRINGS_H|HAVE_STRING_H|HAVE_STRONG_GETAUXVAL|HAVE_SYSCTL|HAVE_SYSCTL_ARND|HAVE_SYSTEM|HAVE_SYS_ENDIAN_H|HAVE_SYS_PRCTL_H|HAVE_SYS_RESOURCES_H|HAVE_SYS_SELECT_H|HAVE_SYS_STAT_H|HAVE_SYS_SYSCTL_H|HAVE_SYS_TYPES_H|HAVE_SYS_VMMETER_H|HAVE_THREAD_LOCAL|HAVE_TIMINGSAFE_BCMP|HAVE_UNISTD_H|HAVE_VM_VM_PARAM_H|LT_OBJDIR|PACKAGE_BUGREPORT|PACKAGE_NAME|PACKAGE_STRING|PACKAGE_TARNAME|PACKAGE_URL|PACKAGE_VERSION|PTHREAD_CREATE_JOINABLE|QT_QPA_PLATFORM_ANDROID|QT_QPA_PLATFORM_COCOA|QT_QPA_PLATFORM_MINIMAL|QT_QPA_PLATFORM_WINDOWS|QT_QPA_PLATFORM_XCB|QT_STATICPLUGIN|STDC_HEADERS|STRERROR_R_CHAR_P|USE_ASM|USE_BDB|USE_DBUS|USE_NATPMP|USE_QRCODE|USE_SQLITE|USE_UPNP|_FILE_OFFSET_BITS|_LARGE_FILES)' exclusion_files=":(exclude)src/minisketch :(exclude)src/crc32c :(exclude)src/secp256k1 :(exclude)src/crypto/sha256_arm_shani.cpp :(exclude)src/crypto/sha256_avx2.cpp :(exclude)src/crypto/sha256_sse41.cpp :(exclude)src/crypto/sha256_x86_shani.cpp" git grep --perl-regexp --files-with-matches "$regex_string" -- '*.cpp' $exclusion_files | xargs git grep -L "bitcoin-config.h" | while read -r file; do line_number=$(awk -v my_file="$file" '/\/\/ file COPYING or https?:\/\/www.opensource.org\/licenses\/mit-license.php\./ {line = NR} /^\/\// && NR == line + 1 {while(getline && /^\/\//) line = NR} END {print line+1}' "$file"); sed -i "${line_number}i\\\\n\#if defined(HAVE_CONFIG_H)\\n#include <config/bitcoin-config.h>\\n\#endif" "$file"; done; git grep --perl-regexp --files-with-matches "$regex_string" -- '*.h' $exclusion_files | xargs git grep -L "bitcoin-config.h" | while read -r file; do sed -i "/#define.*_H/a \\\\n\#if defined(HAVE_CONFIG_H)\\n#include <config/bitcoin-config.h>\\n\#endif" "$file"; done; for file in $(git grep --files-with-matches 'bitcoin-config.h' -- '*.cpp' '*.h' $exclusion_files); do if ! grep -q --perl-regexp "$regex_string" $file; then sed -i '/HAVE_CONFIG_H/{N;N;N;d;}' $file; fi; done; -END VERIFY SCRIPT- The first command creates a regular expression for matching all bitcoin-config.h symbols in the following form: ^(?!//).*(AC_APPLE_UNIVERSAL_BUILD|BOOST_PROCESS_USE_STD_FS|...|_LARGE_FILES). It was generated with: ./autogen.sh && printf '^(?!//).*(%s)' $(awk '/^#undef/ {print $2}' src/config/bitcoin-config.h.in | paste -sd "|" -) The second command holds a list of files and directories that should not be processed. These include subtree directories as well as some crypto files that already get their symbols through the makefile. The third command checks for missing bitcoin-config headers in .cpp files and adds the header if it is missing. The fourth command checks for missing bitcoin-config headers in .h files and adds the header if it is missing. The fifth command checks for unneeded bitcoin-config headers in sources files and removes the header if it is unneeded.
2024-02-12wallet: bdb batch 'ErasePrefix', do not create txn internallyfurszy
Transactions are intended to be started on upper layers rather than internally by the bdb batch object. This enables us to consolidate different write operations within a procedure in the same db txn, improving consistency due to the atomic property of the transaction, as well as its performance due to the reduction of disk write operations. Important Note: This approach also ensures that the BerkeleyBatch::ErasePrefix function behaves exactly as the SQLiteBatch::ErasePrefix function, which does not create a db txn internally. Furthermore, since the `BerkeleyBatch::ErasePrefix' implementation erases records one by one (by traversing the db), this change ensures that the function is always called within an active txn context. Without this measure, there's a potential risk to consistency; certain records may be removed while others could persist due to an internal failure during the procedure.
2024-02-12Merge bitcoin/bitcoin#28987: wallet: simplify and batch zap wallet txes processAva Chow
9a3c5c8697659e34d0476103af942a4615818f4e scripted-diff: rename ZapSelectTx to RemoveTxs (furszy) 83b762845f5804f23b63526d403b2f327fe99632 wallet: batch and simplify ZapSelectTx process (furszy) 595d50a1032ad7ffa9945464c86aa57f16665e93 wallet: migration, remove extra NotifyTransactionChanged call (furszy) a2b071f9920c2f4893afcc43a152f593c03966bf wallet: ZapSelectTx, remove db rewrite code (furszy) Pull request description: Work decoupled from #28574. Brother of #28894. Includes two different, yet interconnected, performance and code improvements to the zap wallet transactions process. 1) As the goal of the `ZapSelectTx` function is to erase tx records that match any of the inputted hashes. There is no need to traverse the whole database record by record. We could just check if the tx exist, and remove it directly by calling `EraseTx()`. 2) Instead of performing single write operations per removed tx record, this PR batches them all within a single atomic db txn. Moreover, these changes will enable us to consolidate all individual write operations that take place during the wallet migration process into a single db txn in the future. ACKs for top commit: achow101: ACK 9a3c5c8697659e34d0476103af942a4615818f4e josibake: ACK https://github.com/bitcoin/bitcoin/pull/28987/commits/9a3c5c8697659e34d0476103af942a4615818f4e Tree-SHA512: fb2ecc48224c400ab3b1fbb32e174b5b13bf03794717727f80f01f55fb183883b067a68c0a127b2de8885564da15425d021a96541953bf38a72becc2e9929ccf
2024-02-09Merge bitcoin/bitcoin#27877: wallet: Add CoinGrinder coin selection algorithmAva Chow
13161ecf032b7a850686e5942c12222c8f3d0d52 opt: Skip over barren combinations of tiny UTXOs (Murch) b7672c7cdd87acb105639f475744094b53cc9891 opt: Skip checking max_weight separately (Murch) 1edd2baa37a69ff69c2eaeceaad1028f1968cbab opt: Cut if last addition was minimal weight (Murch) 5248e2a60d243b3d5c77ecd9e4c335daca399a48 opt: Skip heavier UTXOs with same effective value (Murch) 9124c73742287b06dfe6e8a94be56ace25f07b2c opt: Tiebreak UTXOs by weight for CoinGrinder (Murch) 451be19dc10381122f705bbb2127b083f1d598c6 opt: Skip evaluation of equivalent input sets (Murch) 407b1e3432b77511900b77be84d5d7450352f462 opt: Track remaining effective_value in lookahead (Murch) 5f84f3cc043c5fb15072f5072fee752eaa01a2ec opt: Skip branches with worse weight (Murch) d68bc74fb2e3ae4ae775ab544fe5b4ab46025abb fuzz: Test optimality of CoinGrinder (Murch) 67df6c629a2e9878b06c1903e90b67087eaa3688 fuzz: Add CoinGrinder fuzz target (Murch) 1502231229dbc32c94e80a2fc2959275c5d246ef coinselection: Track whether CG completed (Murch) 7488acc64685ae16e20b78e7ad018137f27fe404 test: Add coin_grinder_tests (Murch) 6cc9a46cd0f4ed80d4523bbef1508142e0c80d27 coinselection: Add CoinGrinder algorithm (Murch) 89d09566431f57034d9a7df32547ceb13d79c62c opt: Tie-break UTXO sort by waste for BnB (Murch) aaee65823c6e620bef5cc96d8026567e64d822fe doc: Document max_weight on BnB (Murch) Pull request description: ***Please refer to the [topic on Delving Bitcoin](https://delvingbitcoin.org/t/gutterguard-and-coingrinder-simulation-results/279) discussing Gutter Guard/Coingrinder simulation results.*** Adds a coin selection algorithm that minimizes the weight of the input set while creating change. Motivations --- - At high feerates, using unnecessary inputs can significantly increase the fees - Users are upset when fees are relatively large compared to the amount sent - Some users struggle to maintain a sufficient count of UTXOs in their wallet Approach --- So far, Bitcoin Core has used a balanced approach to coin selection, where it will generate multiple input set candidates using various coin selection algorithms and pick the least wasteful among their results, but not explicitly minimize the input set weight. Under some circumstances, we _do_ want to minimize the weight of the input set. Sometimes changeless solutions require many or heavy inputs, and there is not always a changeless solution for Branch and Bound to find in the first place. This can cause expensive transactions unnecessarily. Given a wallet with sufficient funds, `CoinGrinder` will pick the minimal-waste input set for a transaction with a change output. The current implementation only runs `CoinGrinder` at feerates over 3×long-term-feerate-estimate (by default 30 ṩ/vB), which may be a decent compromise between our goal to reduce costs for the users, but still permit transactions at lower feerates to naturally reduce the wallet’s UTXO pool to curb bloat. Trade-offs --- Simulations for my thesis on coin selection ([see Section 6.3.2.1 [PDF]](https://murch.one/erhardt2016coinselection.pdf)) suggest that minimizing the input set for all transactions tends to grind a wallet’s UTXO pool to dust (pun intended): an approach selecting inputs per coin-age-priority (in effect similar to “largest first selection”) on average produced a UTXO pool with 15× the UTXO count as Bitcoin Core’s Knapsack-based Coin Selection then (in 2016). Therefore, I do not recommend running `CoinGrinder` under all circumstances, but only at extreme feerates or when we have another good reason to minimize the input set for other reasons. In the long-term, we should introduce additional metrics to score different input set candidates, e.g. on basis of their privacy and wallet health impact, to pick from all our coin selection results, but until then, we may want to limit use of `CoinGrinder` in other ways. ACKs for top commit: achow101: ACK 13161ecf032b7a850686e5942c12222c8f3d0d52 sr-gi: ACK [13161ec](https://github.com/bitcoin/bitcoin/pull/27877/commits/13161ecf032b7a850686e5942c12222c8f3d0d52) sipa: ACK 13161ecf032b7a850686e5942c12222c8f3d0d52 Tree-SHA512: 895b08b2ebfd0b71127949b7dba27146a6d10700bf8590402b14f261e7b937f4e2e1b24ca46de440c35f19349043ed2eba4159dc2aa3edae57721384186dae40
2024-02-09scripted-diff: rename ZapSelectTx to RemoveTxsfurszy
-BEGIN VERIFY SCRIPT- sed -i 's/ZapSelectTx/RemoveTxs/g' $(git grep -l 'ZapSelectTx' ./src/wallet) -END VERIFY SCRIPT-
2024-02-09wallet: batch and simplify ZapSelectTx processfurszy
The goal of the function is to erase the wallet transactions that match the inputted hashes. There is no need to traverse the database, reading record by record, to then perform single entry removals for each of them. To ensure consistency and improve performance, this change-set removes all tx records within a single atomic db batch operation, as well as it cleans up code, improves error handling and simplifies the transactions removal process entirely. This optimizes the removal of watch-only transactions during the wallet migration process and the 'removeprunedfunds' RPC command.
2024-02-09opt: Skip over barren combinations of tiny UTXOsMurch
Given a lot of small amount UTXOs it is possible that the lookahead indicates sufficient funds, but any combination of them would push us beyond the current best_weight. We can estimate a lower bound for the minimal necessary weight to reach target from the maximal amount and minimal weight in the tail of the UTXO pool: if adding a number of hypothetical UTXOs of this maximum amount and minimum weight would not be able to beat `best_weight`, we can SHIFT to the omission branch, and CUT if the last selected UTXO is not heavier than the minimum weight of the remainder.
2024-02-09opt: Cut if last addition was minimal weightMurch
In situations where we have UTXO groups of various weight, we can CUT rather than SHIFT when we exceeded the max_weight or the best selection’s weight while the last step was equal to the minimum weight in the lookahead.
2024-02-09opt: Skip evaluation of equivalent input setsMurch
When two successive UTXOs match in effective value and weight, we can skip the second if the prior is not selected: adding it would create an equivalent input set to a previously evaluated. E.g. if we have three UTXOs with effective values {5, 3, 3} of the same weight each, we want to evaluate {5, _, _}, {5, 3, _}, {5, 3, 3}, {_, 3, _}, {_, 3, 3}, but skip {5, _, 3}, and {_, _, 3}, because the first 3 is not selected, and we therefore do not need to evaluate the second 3 at the same position in the input set. If we reach the end of the branch, we must SHIFT the previously selected UTXO group instead.
2024-02-09opt: Track remaining effective_value in lookaheadMurch
Introduces a dedicated data structure to track the total effective_value available in the remaining UTXOs at each index of the UTXO pool. In contrast to the approach in BnB, this allows us to immediately jump to a lower index instead of visiting every UTXO to add back their eff_value to the lookahead.
2024-02-09opt: Skip branches with worse weightMurch
Once we exceed the weight of the current best selection, we can always shift as adding more inputs can never yield a better solution.
2024-02-09fuzz: Test optimality of CoinGrinderMurch
Co-authored-by: Pieter Wuille <pieter@wuille.net>
2024-02-09fuzz: Add CoinGrinder fuzz targetMurch
2024-02-09test: Add coin_grinder_testsMurch
2024-02-09coinselection: Add CoinGrinder algorithmMurch
CoinGrinder is a DFS-based coin selection algorithm that deterministically finds the input set with the lowest weight creating a change output.
2024-02-07Merge bitcoin/bitcoin#29112: sqlite: Disallow writing from multiple ↵Ryan Ofsky
`SQLiteBatch`s cfcb9b1ecf9be5267487713fa1e112ca09a1ae55 test: wallet, coverage for concurrent db transactions (furszy) 548ecd11553bea28bc79e6f9840836283e9c4e99 tests: Test for concurrent writes with db tx (Ava Chow) 395bcd245476d7c83abf2d82e31e90b6d0c432f3 sqlite: Ensure that only one SQLiteBatch is writing to db at a time (Ava Chow) Pull request description: The way that we have configured SQLite to run means that only one database transaction can be open at a time. Typically, each individual read and write operation will be its own transaction that is opened and committed automatically by SQLite. However, sometimes we want these operations to be batched into a multi-statement transaction, so `SQLiteBatch::TxnBegin`, `SQLiteBatch::TxnCommit`, and `SQLiteBatch::TxnAbort` are used to manage the transaction of the database. However, once a db transaction is begun with one `SQLiteBatch`, any operations performed by another `SQLiteBatch` will also occur within the same transaction. Furthermore, those other `SQLiteBatch`s will not be expecting a transaction to be active, and will abort it once the `SQLiteBatch` is destructed. This is problematic as it will prevent some data from being written, and also cause the `SQLiteBatch` that opened the transaction in the first place to be in an unexpected state and throw an error. To avoid this situation, we need to prevent the multiple batches from writing at the same time. To do so, I've implemented added a `CSemaphore` within `SQLiteDatabase` which will be used by any `SQLiteBatch` trying to do a write operation. `wait()` is called by `TxnBegin`, and at the beginning of `WriteKey`, `EraseKey`, and `ErasePrefix`. `post()` is called in `TxnCommit`, `TxnAbort` and at the end of `WriteKey`, `EraseKey`, and `ErasePrefix`. To avoid deadlocking on ` TxnBegin()` followed by a `WriteKey()`, `SQLiteBatch will now also track whether a transaction is in progress so that it knows whether to use the semaphore. This issue is not a problem for BDB wallets since BDB uses WAL and provides transaction objects that must be used if an operation is to occur within a transaction. Specifically, we either pass a transaction pointer, or a nullptr, to all BDB operations, and this allows for concurrent transactions so it doesn't have this problem. Fixes #29110 ACKs for top commit: josibake: ACK https://github.com/bitcoin/bitcoin/pull/29112/commits/cfcb9b1ecf9be5267487713fa1e112ca09a1ae55 furszy: ACK cfcb9b1ecf9be5267487713fa1e112ca09a1ae55 ryanofsky: Code review ACK cfcb9b1ecf9be5267487713fa1e112ca09a1ae55. This looks great and I think it is ready for merge. Just holding off because josibake seemed ready to review https://github.com/bitcoin/bitcoin/pull/29112#issuecomment-1930372190 and might have more feedback. Tree-SHA512: 2dd5a8e76df52451a40e0b8a87c7139d68a0d8e1bf2ebc79168cc313e192dab87cfa4270ff17fea4f7b370060d3bc9b5d294d50f7e07994d9b5a69b40397c927
2024-02-06test: wallet, coverage for concurrent db transactionsfurszy
Verifying that a database handler can't commit/abort changes occurring in a different database handler.
2024-02-05fuzz: remove unused `args` and `context` from `FuzzedWallet`brunoerg
2024-01-31Merge bitcoin/bitcoin#29253: wallet: guard against dangling to-be-reverted ↵Ava Chow
db transactions b298242c8d495c36072415e1b95eaa7bf485a38a test: sqlite, add coverage for dangling to-be-reverted db txns (furszy) fc0e747192e98e779c5f31f2df808f62b3fdd071 sqlite: guard against dangling to-be-reverted db transactions (furszy) 472d2ca98170049e0edec830e2d11c5ef23740a4 sqlite: introduce HasActiveTxn method (furszy) dca874e838c2599bd24433675b291168f8e7b055 sqlite: add ability to interrupt statements (furszy) fdf9f66909a354a95f4b7c5f092f0e9fbe1baa7c test: wallet db, exercise deadlock after write failure (furszy) Pull request description: Discovered while was reviewing #29112, specifically https://github.com/bitcoin/bitcoin/pull/29112#pullrequestreview-1821862931. If the db handler that initiated the database transaction is destroyed, the ongoing transaction cannot be left dangling when the db txn fails to abort. It must be forcefully reverted; otherwise, any subsequent db handler executing a write operation will dump the dangling, to-be-reverted transaction data to disk. This not only breaks the isolation property but also results in the improper storage of incomplete information on disk, impacting the wallet consistency. This PR fixes the issue by resetting the db connection, automatically rolling back the transaction (per https://www.sqlite.org/c3ref/close.html) when the handler object is being destroyed and the txn abortion failed. Testing Notes Can verify the failure by reverting the fix e5217fea and running the test. It will fail without e5217fea and pass with it. ACKs for top commit: achow101: ACK b298242c8d495c36072415e1b95eaa7bf485a38a ryanofsky: Code review ACK b298242c8d495c36072415e1b95eaa7bf485a38a. Just fix for exec result codes and comment update since last review. Tree-SHA512: 44ba0323ab21440e79e9d7791bc1c56a8873c8bd3e8f6a85641b91576e1293011fa8032d8ae5b0580f3fb7a949356f7b9676693d7ceffa617aaad9f6569993eb
2024-01-30test: sqlite, add coverage for dangling to-be-reverted db txnsfurszy
2024-01-19refactor: pass CRecipient to FundTransactionjosibake
Instead turning tx.vout into a vector of `CRecipient`, make `FundTransaction` take a `CRecipient` vector directly. This allows us to remove SFFO logic from the wrapper RPC `FundTransaction` since the `CRecipient` objects have already been created with the correct SFFO values. This also allows us to remove SFFO from both `FundTransaction` function signatures. This sets us up in a future PR to be able to use these RPCs with BIP352 static payment codes.
2024-01-15test: wallet db, exercise deadlock after write failurefurszy
2024-01-04Merge bitcoin/bitcoin#28832: fuzz: rule-out too deep derivation paths in ↵Ava Chow
descriptor parsing targets a44808fb437864878c2d9696b8a96193091446ee fuzz: rule-out too deep derivation paths in descriptor parsing targets (Antoine Poinsot) Pull request description: This fixes the `mocked_descriptor_parse` timeout reported in #28812 and direct the targets more toward what they are intended to fuzz: the descriptor syntax. ACKs for top commit: sipa: utACK a44808fb437864878c2d9696b8a96193091446ee achow101: ACK a44808fb437864878c2d9696b8a96193091446ee dergoegge: ACK a44808fb437864878c2d9696b8a96193091446ee - Not running into timeouts anymore TheCharlatan: ACK a44808fb437864878c2d9696b8a96193091446ee Tree-SHA512: a5dd1dbe9adf8f088bdc435addab88b56f435e6d7d2065bd6d5c6d80a32e3f1f97d3d2323131ab233618cd6dcc477c458abe3c4c865ab569449b8bc176231e93
2024-01-02Merge bitcoin/bitcoin#29076: fuzz: set `m_fallback_fee` and `m_fee_mode` in ↵Ava Chow
`wallet_fees` target e03d6f7ed534f423f58236866f8e83beee1871e1 fuzz: set `m_fallback_fee`/`m_fee_mode` in `wallet_fees` target (brunoerg) Pull request description: `m_fallback_fee` and `m_fee_mode` are used in `GetMinimumFeeRate` but we're not setting any value for them in `wallet_fees` target. That's the reason fuzzing is never reaching the following code: ![Screenshot 2023-12-13 at 15 04 30](https://github.com/bitcoin/bitcoin/assets/19480819/454ddcaa-75ca-452f-ad13-5f142de0bdce) This PR fixes it. ACKs for top commit: maflcko: review ACK e03d6f7ed534f423f58236866f8e83beee1871e1 achow101: ACK e03d6f7ed534f423f58236866f8e83beee1871e1 murchandamus: ACK e03d6f7ed534f423f58236866f8e83beee1871e1 Tree-SHA512: 5d364f5351d65762a3ddf88e3abb7bda401b7e4955285e083031d216fb50082b1ea98e2c065aff75a5a8a3d1bc4c2e5e3ca9f9478d902ee8f8d4347b6cbe53af
2023-12-31fuzz: rule-out too deep derivation paths in descriptor parsing targetsAntoine Poinsot
This fixes the reported timeouts and direct the target cycles toward what it's intended to fuzz: the descriptor syntax.
2023-12-23refactor: share and use `GenerateRandomKey` helperSebastian Falbesoner
Making the `GenerateRandomKey` helper available to other modules via key.{h.cpp} allows us to create random private keys directly at instantiation of CKey, in contrast to the two-step process of creating the instance and then having to call `MakeNewKey(...)`.
2023-12-20Merge bitcoin/bitcoin#28372: fuzz: coinselection, improve ↵Ava Chow
`min_viable_change`/`change_output_size` cd810075eddd8b1a7139559b475b56126f70a93d fuzz: coinselection, improve `min_viable_change`/`change_output_size` (brunoerg) Pull request description: Instead of "randomly" fuzzing `min_viable_change` and `change_output_size`, and since they're correlated, this PR changes the approach to fuzz them according to the logic in `CreateTransactionInternal`. ACKs for top commit: murchandamus: ACK cd810075eddd8b1a7139559b475b56126f70a93d achow101: ACK cd810075eddd8b1a7139559b475b56126f70a93d furszy: Code ACK cd810075eddd Tree-SHA512: 4539b469f00cdf666078d80c07ed062726f804e390400348148cd3092db9cdc178c6d00ead39aef19acf97badfb6576ce23546d8967387e81c5398d52d7f4404
2023-12-15fuzz: coinselection, improve `min_viable_change`/`change_output_size`brunoerg
Change it to use same approach from `CreateTransactionInternal`.
2023-12-13fuzz: set `m_fallback_fee`/`m_fee_mode` in `wallet_fees` targetbrunoerg
2023-12-13Merge bitcoin/bitcoin#29065: bench: wallet, fix change position out of range ↵Ava Chow
error 37c75c58202f89b752500f76c872d7f8caf6bdb3 test: wallet, fix change position out of range error (furszy) Pull request description: Fixes #29061. Only the benchmark is affected. Since #25273, the behavior of 'inserting change at a random position' is instructed by passing ´std::nullopt´ instead of -1. Also, added missing documentation about the meaning of 'change_pos=std::nullopt' inside 'CWallet::CreateTransaction()' ACKs for top commit: achow101: ACK 37c75c58202f89b752500f76c872d7f8caf6bdb3 kevkevinpal: ACK [37c75c5](https://github.com/bitcoin/bitcoin/pull/29065/commits/37c75c58202f89b752500f76c872d7f8caf6bdb3) BrandonOdiwuor: ACK 37c75c58202f89b752500f76c872d7f8caf6bdb3 Tree-SHA512: d9a8d8533540455716a5090fcf407573cad9f0d0018a05f903f89e51620302f9b256318db6f7338b85c047f7fab372d724e916b1721d7ed302dbf3d845b08734
2023-12-12test: wallet, fix change position out of range errorfurszy
Since #25273, the behavior of 'inserting change at a random position' is instructed by passing std::nullopt instead of -1. Also, added missing documentation about the meaning of 'change_pos=std::nullopt' inside 'CWallet::CreateTransaction()'
2023-12-12Merge bitcoin/bitcoin#28994: wallet: skip BnB when SFFO is enabledAndrew Chow
576bee88fd36e207b7288077626947a1fce0fc33 fuzz: disable BnB when SFFO is enabled (furszy) 05e5ff194c7722b4ebc2b9309fc0bf47b3cf1df7 test: add coverage for BnB-SFFO restriction (furszy) 0c5755761c3e544547899ad096121585dffa73df wallet: create tx, log resulting coin selection info (furszy) 5cea25ba795d6eb9ccc721d01560783ae576af34 wallet: skip BnB when SFFO is active (Murch) Pull request description: Solves #28918. Coming from https://github.com/bitcoin/bitcoin/issues/28918#issuecomment-1838626406 discussion. The intention is to decouple only the bugfix relevant commits from #28985, allowing them to be included in the 26.x release. This way, we can avoid disabling the coin selection fuzzing test for an entire release. Note: Have introduced few changes to the bug fix commit so that the unit tests pass without the additional burden introduced in #28985. ACKs for top commit: josibake: ACK https://github.com/bitcoin/bitcoin/commit/576bee88fd36e207b7288077626947a1fce0fc33 murchandamus: ACK https://github.com/bitcoin/bitcoin/commit/576bee88fd36e207b7288077626947a1fce0fc33 achow101: ACK 576bee88fd36e207b7288077626947a1fce0fc33 Tree-SHA512: f5d90eb3f3f524265afe4719495c9bf30f98b9af26cf039f7df5a7db977abae72caa7a3478cdd0ab10cd143bc1662e8fc5286b5bc10fc10f0dd582a45b45c31a
2023-12-12Merge bitcoin/bitcoin#29055: tests, bench: Fix issue with ↵fanquake
CWallet::LoadWallet() being called in the wrong places bd7f5d33e3b9e01cd600afe1682b7afa935f68a3 wallet: Assert that the wallet is not initialized in LoadWallet (Andrew Chow) fb0b6ca4e5d981cf58bf23ae2993117f171608e8 tests, bench: Remove incorrect LoadWallet() calls (Andrew Chow) Pull request description: `CWallet::LoadWallet()` expects to be called after a `CWallet` is constructed, but before any of its member functions called. Doing so invalidates pointers which causes issues with some PRs and branches that I am working on. This was being used incorrectly in a few tests and benchmarks, resulting in segfaults. As a precaution for this kind of issue in the future, I've also added a few asserts to `LoadWallet()` so that developers will notice when it is used incorrectly. As similar issue was fixed in #27666 ACKs for top commit: S3RK: ACK bd7f5d33e3b9e01cd600afe1682b7afa935f68a3 furszy: ACK bd7f5d33 Tree-SHA512: 7664f12b8452994e7fc4d7d4f77697fb5f75edb0dba95ba99a4a23ec03d5b8e0ecbdcb7635547a0e8b4f89f708f98dcb5d039df0559e24b1ae411ed630e16e14
2023-12-11fuzz: disable BnB when SFFO is enabledfurszy
2023-12-11test: add coverage for BnB-SFFO restrictionfurszy
Verify the transaction creation process does not produce a BnB solution when SFFO is enabled. This is currently problematic because it could require a change output. And BnB is specialized on changeless solutions. Co-authored-by: Andrew Chow <achow101@gmail.com> Co-authored-by: Murch <murch@murch.one>
2023-12-11tests, bench: Remove incorrect LoadWallet() callsAndrew Chow
LoadWallet() must only be called immediately after a CWallet is constructed, or not at all. Doing so after any other CWallet member functions have been called may cause pointers and other objects setup by other those functions to become invalidated. Since these tests and benchmarks are using completely new wallets with mock databases, it's not necessary to call LoadWallet() anyways, so these can be dropped.
2023-12-08wallet: return CreatedTransactionResult from FundTransactionAndrew Chow
Instead of using the output parameters, return CreatedTransactionResult from FundTransaction in the same way that CreateTransaction does. Additionally, instead of modifying the original CMutableTransaction, the result from CreateTransactionInternal is used.
generated by cgit v1.2.3 (git 2.26.2) at 2024-08-25 11:39:04 +0000