diff options
author | Ryan Ofsky <ryan@ofsky.org> | 2023-06-27 18:39:27 -0400 |
---|---|---|
committer | Ryan Ofsky <ryan@ofsky.org> | 2023-06-27 19:03:15 -0400 |
commit | d9c7c2fd3ec7b0fcae7e0c9423bff6c6799dd67c (patch) | |
tree | 8709d44fd19e0aa4ff1e5e54d756678a56861b40 /src | |
parent | caff95a0237facddb46fbbdf87e31ff6294b8c70 (diff) | |
parent | 3c83b1d884b419adece95b335b6e956e7459a7ef (diff) |
Merge bitcoin/bitcoin#24914: wallet: Load database records in a particular order
3c83b1d884b419adece95b335b6e956e7459a7ef doc: Add release note for wallet loading changes (Andrew Chow)
2636844f5353797a0b8e40a879652a0d345172ad walletdb: Remove loading code where the database is iterated (Andrew Chow)
cd211b3b9965b5070d68adc1a03043d82d904d5b walletdb: refactor decryption key loading (Andrew Chow)
31c033e5ca3b65f4f5345d5aa17aafedd637ef4f walletdb: refactor defaultkey and wkey loading (Andrew Chow)
c978c6d39cdeb78fc4720767b943d03d6a9a36d8 walletdb: refactor active spkm loading (Andrew Chow)
6fabb7fc99e60584d5f3a2cb01d39f761769a25d walletdb: refactor tx loading (Andrew Chow)
abcc13dd24889bc1c6af7b10da1da96d86aeafed walletdb: refactor address book loading (Andrew Chow)
405b4d914712b5de3b230a0e2960e89f6a0a2b2a walletdb: Refactor descriptor wallet records loading (Andrew Chow)
30ab11c49793d5d55d66c4dedfa576ae8fd6129c walletdb: Refactor legacy wallet record loading into its own function (Andrew Chow)
9e077d9b422ac3c371fe0f63da40e5092171a25e salvage: Remove use of ReadKeyValue in salvage (Andrew Chow)
ad779e9ece9829677c1735d8865f14b23459da80 walletdb: Refactor hd chain loading to its own function (Andrew Chow)
72c2a54ebb99fa3d91d7d15bd8a38a8d16e0ea6c walletdb: Refactor encryption key loading to its own function (Andrew Chow)
3ccde4599b5150577400c4fa9029f4146617f751 walletdb: Refactor crypted key loading to its own function (Andrew Chow)
7be10adff36c0dc49ae56ac571bb033cba7a565b walletdb: Refactor key reading and loading to its own function (Andrew Chow)
52932c5adb29bb9ec5f0bcde9a31b74113a20651 walletdb: Refactor wallet flags loading (Andrew Chow)
01b35b55a119dc7ac915fc621ecebcd5c50ccb55 walletdb: Refactor minversion loading (Andrew Chow)
Pull request description:
Currently when we load a wallet, we just iterate through all of the records in the database and add them completely statelessly. However we have some records which do rely on other records being loaded before they are. To deal with this, we use `CWalletScanState` to hold things temporarily until all of the records have been read and then we load the stateful things.
However this can be slow, and with some future improvements, can cause some pretty drastic slowdowns to retain this pattern. So this PR changes the way we load records by choosing to load the records in a particular order. This lets us do things such as loading a descriptor record, then finding and loading that descriptor's cache and key records. In the future, this will also let us use `IsMine` when loading transactions as then `IsMine` will actually be working as we now always load keys and descriptors before transactions.
In order to get records of a specific type, this PR includes some refactors to how we do database cursors. Functionality is also added to retrieve a cursor that will give us records beginning with a specified prefix.
Lastly, one thing that iterating the entire database let us do was to find unknown records. However even if unknown records were found, we would not do anything with this information except output a number in a log line. With this PR, we would no longer be aware of any unknown records. This does not change functionality as we don't do anything with unknown records, and having unknown records is not an error. Now we would just not be aware that unknown records even exist.
ACKs for top commit:
MarcoFalke:
re-ACK 3c83b1d884b419adece95b335b6e956e7459a7ef 🍤
furszy:
reACK 3c83b1d8
ryanofsky:
Code review ACK 3c83b1d884b419adece95b335b6e956e7459a7ef. Just Marco's suggested error handling fixes since last review
Tree-SHA512: 15fa56332fb2ce4371db468a0c674ee7a3a8889c8cee9f428d06a7d1385d17a9bf54bcb0ba885c87736841fe6a5c934594bcf4476a473616510ee47862ef30b4
Diffstat (limited to 'src')
-rw-r--r-- | src/wallet/salvage.cpp | 28 | ||||
-rw-r--r-- | src/wallet/wallet.cpp | 2 | ||||
-rw-r--r-- | src/wallet/walletdb.cpp | 1301 | ||||
-rw-r--r-- | src/wallet/walletdb.h | 37 |
4 files changed, 800 insertions, 568 deletions
diff --git a/src/wallet/salvage.cpp b/src/wallet/salvage.cpp index e303310273..da16435f04 100644 --- a/src/wallet/salvage.cpp +++ b/src/wallet/salvage.cpp @@ -18,11 +18,6 @@ static const char *HEADER_END = "HEADER=END"; static const char *DATA_END = "DATA=END"; typedef std::pair<std::vector<unsigned char>, std::vector<unsigned char> > KeyValPair; -static bool KeyFilter(const std::string& type) -{ - return WalletBatch::IsKeyType(type) || type == DBKeys::HDCHAIN; -} - class DummyCursor : public DatabaseCursor { Status Next(DataStream& key, DataStream& value) override { return Status::FAIL; } @@ -186,17 +181,24 @@ bool RecoverDatabaseFile(const ArgsManager& args, const fs::path& file_path, bil { /* Filter for only private key type KV pairs to be added to the salvaged wallet */ DataStream ssKey{row.first}; - CDataStream ssValue(row.second, SER_DISK, CLIENT_VERSION); + DataStream ssValue(row.second); std::string strType, strErr; - bool fReadOK; - { - // Required in LoadKeyMetadata(): - LOCK(dummyWallet.cs_wallet); - fReadOK = ReadKeyValue(&dummyWallet, ssKey, ssValue, strType, strErr, KeyFilter); - } - if (!KeyFilter(strType)) { + + // We only care about KEY, MASTER_KEY, CRYPTED_KEY, and HDCHAIN types + ssKey >> strType; + bool fReadOK = false; + if (strType == DBKeys::KEY) { + fReadOK = LoadKey(&dummyWallet, ssKey, ssValue, strErr); + } else if (strType == DBKeys::CRYPTED_KEY) { + fReadOK = LoadCryptedKey(&dummyWallet, ssKey, ssValue, strErr); + } else if (strType == DBKeys::MASTER_KEY) { + fReadOK = LoadEncryptionKey(&dummyWallet, ssKey, ssValue, strErr); + } else if (strType == DBKeys::HDCHAIN) { + fReadOK = LoadHDChain(&dummyWallet, ssValue, strErr); + } else { continue; } + if (!fReadOK) { warnings.push_back(strprintf(Untranslated("WARNING: WalletBatch::Recover skipping %s: %s"), strType, strErr)); diff --git a/src/wallet/wallet.cpp b/src/wallet/wallet.cpp index ba11933b91..a1b26b139e 100644 --- a/src/wallet/wallet.cpp +++ b/src/wallet/wallet.cpp @@ -2929,7 +2929,7 @@ std::shared_ptr<CWallet> CWallet::Create(WalletContext& context, const std::stri else if (nLoadWalletRet == DBErrors::NONCRITICAL_ERROR) { warnings.push_back(strprintf(_("Error reading %s! All keys read correctly, but transaction data" - " or address book entries might be missing or incorrect."), + " or address metadata may be missing or incorrect."), walletFile)); } else if (nLoadWalletRet == DBErrors::TOO_NEW) { diff --git a/src/wallet/walletdb.cpp b/src/wallet/walletdb.cpp index 34fe8ab17f..2aee750ced 100644 --- a/src/wallet/walletdb.cpp +++ b/src/wallet/walletdb.cpp @@ -11,6 +11,7 @@ #include <serialize.h> #include <sync.h> #include <util/bip32.h> +#include <util/check.h> #include <util/fs.h> #include <util/time.h> #include <util/translation.h> @@ -297,426 +298,590 @@ bool WalletBatch::EraseLockedUTXO(const COutPoint& output) return EraseIC(std::make_pair(DBKeys::LOCKED_UTXO, std::make_pair(output.hash, output.n))); } -class CWalletScanState { -public: - unsigned int nKeys{0}; - unsigned int nCKeys{0}; - unsigned int nWatchKeys{0}; - unsigned int nKeyMeta{0}; - unsigned int m_unknown_records{0}; - bool fIsEncrypted{false}; - bool fAnyUnordered{false}; - std::vector<uint256> vWalletUpgrade; - std::map<OutputType, uint256> m_active_external_spks; - std::map<OutputType, uint256> m_active_internal_spks; - std::map<uint256, DescriptorCache> m_descriptor_caches; - std::map<std::pair<uint256, CKeyID>, CKey> m_descriptor_keys; - std::map<std::pair<uint256, CKeyID>, std::pair<CPubKey, std::vector<unsigned char>>> m_descriptor_crypt_keys; - std::map<uint160, CHDChain> m_hd_chains; - bool tx_corrupt{false}; - bool descriptor_unknown{false}; - bool unexpected_legacy_entry{false}; - - CWalletScanState() = default; -}; - -static bool -ReadKeyValue(CWallet* pwallet, DataStream& ssKey, CDataStream& ssValue, - CWalletScanState &wss, std::string& strType, std::string& strErr, const KeyFilterFn& filter_fn = nullptr) EXCLUSIVE_LOCKS_REQUIRED(pwallet->cs_wallet) +bool LoadKey(CWallet* pwallet, DataStream& ssKey, DataStream& ssValue, std::string& strErr) { + LOCK(pwallet->cs_wallet); try { - // Unserialize - // Taking advantage of the fact that pair serialization - // is just the two items serialized one after the other - ssKey >> strType; - // If we have a filter, check if this matches the filter - if (filter_fn && !filter_fn(strType)) { - return true; - } - // Legacy entries in descriptor wallets are not allowed, abort immediately - if (pwallet->IsWalletFlagSet(WALLET_FLAG_DESCRIPTORS) && DBKeys::LEGACY_TYPES.count(strType) > 0) { - wss.unexpected_legacy_entry = true; + CPubKey vchPubKey; + ssKey >> vchPubKey; + if (!vchPubKey.IsValid()) + { + strErr = "Error reading wallet database: CPubKey corrupt"; return false; } - if (strType == DBKeys::NAME) { - std::string strAddress; - ssKey >> strAddress; - std::string label; - ssValue >> label; - pwallet->m_address_book[DecodeDestination(strAddress)].SetLabel(label); - } else if (strType == DBKeys::PURPOSE) { - std::string strAddress; - ssKey >> strAddress; - std::string purpose_str; - ssValue >> purpose_str; - std::optional<AddressPurpose> purpose{PurposeFromString(purpose_str)}; - if (!purpose) { - pwallet->WalletLogPrintf("Warning: nonstandard purpose string '%s' for address '%s'\n", purpose_str, strAddress); - } - pwallet->m_address_book[DecodeDestination(strAddress)].purpose = purpose; - } else if (strType == DBKeys::TX) { - uint256 hash; - ssKey >> hash; - // LoadToWallet call below creates a new CWalletTx that fill_wtx - // callback fills with transaction metadata. - auto fill_wtx = [&](CWalletTx& wtx, bool new_tx) { - if(!new_tx) { - // There's some corruption here since the tx we just tried to load was already in the wallet. - // We don't consider this type of corruption critical, and can fix it by removing tx data and - // rescanning. - wss.tx_corrupt = true; - return false; - } - ssValue >> wtx; - if (wtx.GetHash() != hash) - return false; + CKey key; + CPrivKey pkey; + uint256 hash; + + ssValue >> pkey; + + // Old wallets store keys as DBKeys::KEY [pubkey] => [privkey] + // ... which was slow for wallets with lots of keys, because the public key is re-derived from the private key + // using EC operations as a checksum. + // Newer wallets store keys as DBKeys::KEY [pubkey] => [privkey][hash(pubkey,privkey)], which is much faster while + // remaining backwards-compatible. + try + { + ssValue >> hash; + } + catch (const std::ios_base::failure&) {} - // Undo serialize changes in 31600 - if (31404 <= wtx.fTimeReceivedIsTxTime && wtx.fTimeReceivedIsTxTime <= 31703) - { - if (!ssValue.empty()) - { - uint8_t fTmp; - uint8_t fUnused; - std::string unused_string; - ssValue >> fTmp >> fUnused >> unused_string; - strErr = strprintf("LoadWallet() upgrading tx ver=%d %d %s", - wtx.fTimeReceivedIsTxTime, fTmp, hash.ToString()); - wtx.fTimeReceivedIsTxTime = fTmp; - } - else - { - strErr = strprintf("LoadWallet() repairing tx ver=%d %s", wtx.fTimeReceivedIsTxTime, hash.ToString()); - wtx.fTimeReceivedIsTxTime = 0; - } - wss.vWalletUpgrade.push_back(hash); - } + bool fSkipCheck = false; - if (wtx.nOrderPos == -1) - wss.fAnyUnordered = true; + if (!hash.IsNull()) + { + // hash pubkey/privkey to accelerate wallet load + std::vector<unsigned char> vchKey; + vchKey.reserve(vchPubKey.size() + pkey.size()); + vchKey.insert(vchKey.end(), vchPubKey.begin(), vchPubKey.end()); + vchKey.insert(vchKey.end(), pkey.begin(), pkey.end()); - return true; - }; - if (!pwallet->LoadToWallet(hash, fill_wtx)) { - return false; - } - } else if (strType == DBKeys::WATCHS) { - wss.nWatchKeys++; - CScript script; - ssKey >> script; - uint8_t fYes; - ssValue >> fYes; - if (fYes == '1') { - pwallet->GetOrCreateLegacyScriptPubKeyMan()->LoadWatchOnly(script); - } - } else if (strType == DBKeys::KEY) { - CPubKey vchPubKey; - ssKey >> vchPubKey; - if (!vchPubKey.IsValid()) + if (Hash(vchKey) != hash) { - strErr = "Error reading wallet database: CPubKey corrupt"; + strErr = "Error reading wallet database: CPubKey/CPrivKey corrupt"; return false; } - CKey key; - CPrivKey pkey; - uint256 hash; - wss.nKeys++; - ssValue >> pkey; + fSkipCheck = true; + } - // Old wallets store keys as DBKeys::KEY [pubkey] => [privkey] - // ... which was slow for wallets with lots of keys, because the public key is re-derived from the private key - // using EC operations as a checksum. - // Newer wallets store keys as DBKeys::KEY [pubkey] => [privkey][hash(pubkey,privkey)], which is much faster while - // remaining backwards-compatible. - try - { - ssValue >> hash; + if (!key.Load(pkey, vchPubKey, fSkipCheck)) + { + strErr = "Error reading wallet database: CPrivKey corrupt"; + return false; + } + if (!pwallet->GetOrCreateLegacyScriptPubKeyMan()->LoadKey(key, vchPubKey)) + { + strErr = "Error reading wallet database: LegacyScriptPubKeyMan::LoadKey failed"; + return false; + } + } catch (const std::exception& e) { + if (strErr.empty()) { + strErr = e.what(); + } + return false; + } + return true; +} + +bool LoadCryptedKey(CWallet* pwallet, DataStream& ssKey, DataStream& ssValue, std::string& strErr) +{ + LOCK(pwallet->cs_wallet); + try { + CPubKey vchPubKey; + ssKey >> vchPubKey; + if (!vchPubKey.IsValid()) + { + strErr = "Error reading wallet database: CPubKey corrupt"; + return false; + } + std::vector<unsigned char> vchPrivKey; + ssValue >> vchPrivKey; + + // Get the checksum and check it + bool checksum_valid = false; + if (!ssValue.eof()) { + uint256 checksum; + ssValue >> checksum; + if (!(checksum_valid = Hash(vchPrivKey) == checksum)) { + strErr = "Error reading wallet database: Encrypted key corrupt"; + return false; } - catch (const std::ios_base::failure&) {} + } - bool fSkipCheck = false; + if (!pwallet->GetOrCreateLegacyScriptPubKeyMan()->LoadCryptedKey(vchPubKey, vchPrivKey, checksum_valid)) + { + strErr = "Error reading wallet database: LegacyScriptPubKeyMan::LoadCryptedKey failed"; + return false; + } + } catch (const std::exception& e) { + if (strErr.empty()) { + strErr = e.what(); + } + return false; + } + return true; +} - if (!hash.IsNull()) - { - // hash pubkey/privkey to accelerate wallet load - std::vector<unsigned char> vchKey; - vchKey.reserve(vchPubKey.size() + pkey.size()); - vchKey.insert(vchKey.end(), vchPubKey.begin(), vchPubKey.end()); - vchKey.insert(vchKey.end(), pkey.begin(), pkey.end()); +bool LoadEncryptionKey(CWallet* pwallet, DataStream& ssKey, DataStream& ssValue, std::string& strErr) +{ + LOCK(pwallet->cs_wallet); + try { + // Master encryption key is loaded into only the wallet and not any of the ScriptPubKeyMans. + unsigned int nID; + ssKey >> nID; + CMasterKey kMasterKey; + ssValue >> kMasterKey; + if(pwallet->mapMasterKeys.count(nID) != 0) + { + strErr = strprintf("Error reading wallet database: duplicate CMasterKey id %u", nID); + return false; + } + pwallet->mapMasterKeys[nID] = kMasterKey; + if (pwallet->nMasterKeyMaxID < nID) + pwallet->nMasterKeyMaxID = nID; - if (Hash(vchKey) != hash) - { - strErr = "Error reading wallet database: CPubKey/CPrivKey corrupt"; - return false; - } + } catch (const std::exception& e) { + if (strErr.empty()) { + strErr = e.what(); + } + return false; + } + return true; +} - fSkipCheck = true; - } +bool LoadHDChain(CWallet* pwallet, DataStream& ssValue, std::string& strErr) +{ + LOCK(pwallet->cs_wallet); + try { + CHDChain chain; + ssValue >> chain; + pwallet->GetOrCreateLegacyScriptPubKeyMan()->LoadHDChain(chain); + } catch (const std::exception& e) { + if (strErr.empty()) { + strErr = e.what(); + } + return false; + } + return true; +} - if (!key.Load(pkey, vchPubKey, fSkipCheck)) - { - strErr = "Error reading wallet database: CPrivKey corrupt"; - return false; - } - if (!pwallet->GetOrCreateLegacyScriptPubKeyMan()->LoadKey(key, vchPubKey)) - { - strErr = "Error reading wallet database: LegacyScriptPubKeyMan::LoadKey failed"; - return false; - } - } else if (strType == DBKeys::MASTER_KEY) { - // Master encryption key is loaded into only the wallet and not any of the ScriptPubKeyMans. - unsigned int nID; - ssKey >> nID; - CMasterKey kMasterKey; - ssValue >> kMasterKey; - if(pwallet->mapMasterKeys.count(nID) != 0) - { - strErr = strprintf("Error reading wallet database: duplicate CMasterKey id %u", nID); - return false; - } - pwallet->mapMasterKeys[nID] = kMasterKey; - if (pwallet->nMasterKeyMaxID < nID) - pwallet->nMasterKeyMaxID = nID; - } else if (strType == DBKeys::CRYPTED_KEY) { - CPubKey vchPubKey; - ssKey >> vchPubKey; - if (!vchPubKey.IsValid()) - { - strErr = "Error reading wallet database: CPubKey corrupt"; - return false; +static DBErrors LoadMinVersion(CWallet* pwallet, DatabaseBatch& batch) EXCLUSIVE_LOCKS_REQUIRED(pwallet->cs_wallet) +{ + AssertLockHeld(pwallet->cs_wallet); + int nMinVersion = 0; + if (batch.Read(DBKeys::MINVERSION, nMinVersion)) { + if (nMinVersion > FEATURE_LATEST) + return DBErrors::TOO_NEW; + pwallet->LoadMinVersion(nMinVersion); + } + return DBErrors::LOAD_OK; +} + +static DBErrors LoadWalletFlags(CWallet* pwallet, DatabaseBatch& batch) EXCLUSIVE_LOCKS_REQUIRED(pwallet->cs_wallet) +{ + AssertLockHeld(pwallet->cs_wallet); + uint64_t flags; + if (batch.Read(DBKeys::FLAGS, flags)) { + if (!pwallet->LoadWalletFlags(flags)) { + pwallet->WalletLogPrintf("Error reading wallet database: Unknown non-tolerable wallet flags found\n"); + return DBErrors::TOO_NEW; + } + } + return DBErrors::LOAD_OK; +} + +struct LoadResult +{ + DBErrors m_result{DBErrors::LOAD_OK}; + int m_records{0}; +}; + +using LoadFunc = std::function<DBErrors(CWallet* pwallet, DataStream& key, CDataStream& value, std::string& err)>; +static LoadResult LoadRecords(CWallet* pwallet, DatabaseBatch& batch, const std::string& key, DataStream& prefix, LoadFunc load_func) +{ + LoadResult result; + DataStream ssKey; + CDataStream ssValue(SER_DISK, CLIENT_VERSION); + + Assume(!prefix.empty()); + std::unique_ptr<DatabaseCursor> cursor = batch.GetNewPrefixCursor(prefix); + if (!cursor) { + pwallet->WalletLogPrintf("Error getting database cursor for '%s' records\n", key); + result.m_result = DBErrors::CORRUPT; + return result; + } + + while (true) { + DatabaseCursor::Status status = cursor->Next(ssKey, ssValue); + if (status == DatabaseCursor::Status::DONE) { + break; + } else if (status == DatabaseCursor::Status::FAIL) { + pwallet->WalletLogPrintf("Error reading next '%s' record for wallet database\n", key); + result.m_result = DBErrors::CORRUPT; + return result; + } + std::string type; + ssKey >> type; + assert(type == key); + std::string error; + DBErrors record_res = load_func(pwallet, ssKey, ssValue, error); + if (record_res != DBErrors::LOAD_OK) { + pwallet->WalletLogPrintf("%s\n", error); + } + result.m_result = std::max(result.m_result, record_res); + ++result.m_records; + } + return result; +} + +static LoadResult LoadRecords(CWallet* pwallet, DatabaseBatch& batch, const std::string& key, LoadFunc load_func) +{ + DataStream prefix; + prefix << key; + return LoadRecords(pwallet, batch, key, prefix, load_func); +} + +static DBErrors LoadLegacyWalletRecords(CWallet* pwallet, DatabaseBatch& batch, int last_client) EXCLUSIVE_LOCKS_REQUIRED(pwallet->cs_wallet) +{ + AssertLockHeld(pwallet->cs_wallet); + DBErrors result = DBErrors::LOAD_OK; + + // Make sure descriptor wallets don't have any legacy records + if (pwallet->IsWalletFlagSet(WALLET_FLAG_DESCRIPTORS)) { + for (const auto& type : DBKeys::LEGACY_TYPES) { + DataStream key; + CDataStream value(SER_DISK, CLIENT_VERSION); + + DataStream prefix; + prefix << type; + std::unique_ptr<DatabaseCursor> cursor = batch.GetNewPrefixCursor(prefix); + if (!cursor) { + pwallet->WalletLogPrintf("Error getting database cursor for '%s' records\n", type); + return DBErrors::CORRUPT; } - std::vector<unsigned char> vchPrivKey; - ssValue >> vchPrivKey; - - // Get the checksum and check it - bool checksum_valid = false; - if (!ssValue.eof()) { - uint256 checksum; - ssValue >> checksum; - if (!(checksum_valid = Hash(vchPrivKey) == checksum)) { - strErr = "Error reading wallet database: Encrypted key corrupt"; - return false; - } + + DatabaseCursor::Status status = cursor->Next(key, value); + if (status != DatabaseCursor::Status::DONE) { + pwallet->WalletLogPrintf("Error: Unexpected legacy entry found in descriptor wallet %s. The wallet might have been tampered with or created with malicious intent.\n", pwallet->GetName()); + return DBErrors::UNEXPECTED_LEGACY_ENTRY; } + } - wss.nCKeys++; + return DBErrors::LOAD_OK; + } - if (!pwallet->GetOrCreateLegacyScriptPubKeyMan()->LoadCryptedKey(vchPubKey, vchPrivKey, checksum_valid)) - { - strErr = "Error reading wallet database: LegacyScriptPubKeyMan::LoadCryptedKey failed"; - return false; - } - wss.fIsEncrypted = true; - } else if (strType == DBKeys::KEYMETA) { - CPubKey vchPubKey; - ssKey >> vchPubKey; - CKeyMetadata keyMeta; - ssValue >> keyMeta; - wss.nKeyMeta++; - pwallet->GetOrCreateLegacyScriptPubKeyMan()->LoadKeyMetadata(vchPubKey.GetID(), keyMeta); - - // Extract some CHDChain info from this metadata if it has any - if (keyMeta.nVersion >= CKeyMetadata::VERSION_WITH_HDDATA && !keyMeta.hd_seed_id.IsNull() && keyMeta.hdKeypath.size() > 0) { - // Get the path from the key origin or from the path string - // Not applicable when path is "s" or "m" as those indicate a seed - // See https://github.com/bitcoin/bitcoin/pull/12924 - bool internal = false; - uint32_t index = 0; - if (keyMeta.hdKeypath != "s" && keyMeta.hdKeypath != "m") { - std::vector<uint32_t> path; - if (keyMeta.has_key_origin) { - // We have a key origin, so pull it from its path vector - path = keyMeta.key_origin.path; - } else { - // No key origin, have to parse the string - if (!ParseHDKeypath(keyMeta.hdKeypath, path)) { - strErr = "Error reading wallet database: keymeta with invalid HD keypath"; - return false; - } - } + // Load HD Chain + // Note: There should only be one HDCHAIN record with no data following the type + LoadResult hd_chain_res = LoadRecords(pwallet, batch, DBKeys::HDCHAIN, + [] (CWallet* pwallet, DataStream& key, CDataStream& value, std::string& err) { + return LoadHDChain(pwallet, value, err) ? DBErrors:: LOAD_OK : DBErrors::CORRUPT; + }); + result = std::max(result, hd_chain_res.m_result); + + // Load unencrypted keys + LoadResult key_res = LoadRecords(pwallet, batch, DBKeys::KEY, + [] (CWallet* pwallet, DataStream& key, CDataStream& value, std::string& err) { + return LoadKey(pwallet, key, value, err) ? DBErrors::LOAD_OK : DBErrors::CORRUPT; + }); + result = std::max(result, key_res.m_result); + + // Load encrypted keys + LoadResult ckey_res = LoadRecords(pwallet, batch, DBKeys::CRYPTED_KEY, + [] (CWallet* pwallet, DataStream& key, CDataStream& value, std::string& err) { + return LoadCryptedKey(pwallet, key, value, err) ? DBErrors::LOAD_OK : DBErrors::CORRUPT; + }); + result = std::max(result, ckey_res.m_result); + + // Load scripts + LoadResult script_res = LoadRecords(pwallet, batch, DBKeys::CSCRIPT, + [] (CWallet* pwallet, DataStream& key, CDataStream& value, std::string& strErr) { + uint160 hash; + key >> hash; + CScript script; + value >> script; + if (!pwallet->GetOrCreateLegacyScriptPubKeyMan()->LoadCScript(script)) + { + strErr = "Error reading wallet database: LegacyScriptPubKeyMan::LoadCScript failed"; + return DBErrors::NONCRITICAL_ERROR; + } + return DBErrors::LOAD_OK; + }); + result = std::max(result, script_res.m_result); + + // Check whether rewrite is needed + if (ckey_res.m_records > 0) { + // Rewrite encrypted wallets of versions 0.4.0 and 0.5.0rc: + if (last_client == 40000 || last_client == 50000) result = std::max(result, DBErrors::NEED_REWRITE); + } - // Extract the index and internal from the path - // Path string is m/0'/k'/i' - // Path vector is [0', k', i'] (but as ints OR'd with the hardened bit - // k == 0 for external, 1 for internal. i is the index - if (path.size() != 3) { - strErr = "Error reading wallet database: keymeta found with unexpected path"; - return false; - } - if (path[0] != 0x80000000) { - strErr = strprintf("Unexpected path index of 0x%08x (expected 0x80000000) for the element at index 0", path[0]); - return false; - } - if (path[1] != 0x80000000 && path[1] != (1 | 0x80000000)) { - strErr = strprintf("Unexpected path index of 0x%08x (expected 0x80000000 or 0x80000001) for the element at index 1", path[1]); - return false; - } - if ((path[2] & 0x80000000) == 0) { - strErr = strprintf("Unexpected path index of 0x%08x (expected to be greater than or equal to 0x80000000)", path[2]); - return false; + // Load keymeta + std::map<uint160, CHDChain> hd_chains; + LoadResult keymeta_res = LoadRecords(pwallet, batch, DBKeys::KEYMETA, + [&hd_chains] (CWallet* pwallet, DataStream& key, CDataStream& value, std::string& strErr) { + CPubKey vchPubKey; + key >> vchPubKey; + CKeyMetadata keyMeta; + value >> keyMeta; + pwallet->GetOrCreateLegacyScriptPubKeyMan()->LoadKeyMetadata(vchPubKey.GetID(), keyMeta); + + // Extract some CHDChain info from this metadata if it has any + if (keyMeta.nVersion >= CKeyMetadata::VERSION_WITH_HDDATA && !keyMeta.hd_seed_id.IsNull() && keyMeta.hdKeypath.size() > 0) { + // Get the path from the key origin or from the path string + // Not applicable when path is "s" or "m" as those indicate a seed + // See https://github.com/bitcoin/bitcoin/pull/12924 + bool internal = false; + uint32_t index = 0; + if (keyMeta.hdKeypath != "s" && keyMeta.hdKeypath != "m") { + std::vector<uint32_t> path; + if (keyMeta.has_key_origin) { + // We have a key origin, so pull it from its path vector + path = keyMeta.key_origin.path; + } else { + // No key origin, have to parse the string + if (!ParseHDKeypath(keyMeta.hdKeypath, path)) { + strErr = "Error reading wallet database: keymeta with invalid HD keypath"; + return DBErrors::NONCRITICAL_ERROR; } - internal = path[1] == (1 | 0x80000000); - index = path[2] & ~0x80000000; } - // Insert a new CHDChain, or get the one that already exists - auto ins = wss.m_hd_chains.emplace(keyMeta.hd_seed_id, CHDChain()); - CHDChain& chain = ins.first->second; - if (ins.second) { - // For new chains, we want to default to VERSION_HD_BASE until we see an internal - chain.nVersion = CHDChain::VERSION_HD_BASE; - chain.seed_id = keyMeta.hd_seed_id; + // Extract the index and internal from the path + // Path string is m/0'/k'/i' + // Path vector is [0', k', i'] (but as ints OR'd with the hardened bit + // k == 0 for external, 1 for internal. i is the index + if (path.size() != 3) { + strErr = "Error reading wallet database: keymeta found with unexpected path"; + return DBErrors::NONCRITICAL_ERROR; } - if (internal) { - chain.nVersion = CHDChain::VERSION_HD_CHAIN_SPLIT; - chain.nInternalChainCounter = std::max(chain.nInternalChainCounter, index + 1); - } else { - chain.nExternalChainCounter = std::max(chain.nExternalChainCounter, index + 1); + if (path[0] != 0x80000000) { + strErr = strprintf("Unexpected path index of 0x%08x (expected 0x80000000) for the element at index 0", path[0]); + return DBErrors::NONCRITICAL_ERROR; } + if (path[1] != 0x80000000 && path[1] != (1 | 0x80000000)) { + strErr = strprintf("Unexpected path index of 0x%08x (expected 0x80000000 or 0x80000001) for the element at index 1", path[1]); + return DBErrors::NONCRITICAL_ERROR; + } + if ((path[2] & 0x80000000) == 0) { + strErr = strprintf("Unexpected path index of 0x%08x (expected to be greater than or equal to 0x80000000)", path[2]); + return DBErrors::NONCRITICAL_ERROR; + } + internal = path[1] == (1 | 0x80000000); + index = path[2] & ~0x80000000; } - } else if (strType == DBKeys::WATCHMETA) { - CScript script; - ssKey >> script; - CKeyMetadata keyMeta; - ssValue >> keyMeta; - wss.nKeyMeta++; - pwallet->GetOrCreateLegacyScriptPubKeyMan()->LoadScriptMetadata(CScriptID(script), keyMeta); - } else if (strType == DBKeys::DEFAULTKEY) { - // We don't want or need the default key, but if there is one set, - // we want to make sure that it is valid so that we can detect corruption - CPubKey vchPubKey; - ssValue >> vchPubKey; - if (!vchPubKey.IsValid()) { - strErr = "Error reading wallet database: Default Key corrupt"; - return false; - } - } else if (strType == DBKeys::POOL) { - int64_t nIndex; - ssKey >> nIndex; - CKeyPool keypool; - ssValue >> keypool; - - pwallet->GetOrCreateLegacyScriptPubKeyMan()->LoadKeyPool(nIndex, keypool); - } else if (strType == DBKeys::CSCRIPT) { - uint160 hash; - ssKey >> hash; - CScript script; - ssValue >> script; - if (!pwallet->GetOrCreateLegacyScriptPubKeyMan()->LoadCScript(script)) - { - strErr = "Error reading wallet database: LegacyScriptPubKeyMan::LoadCScript failed"; - return false; + + // Insert a new CHDChain, or get the one that already exists + auto [ins, inserted] = hd_chains.emplace(keyMeta.hd_seed_id, CHDChain()); + CHDChain& chain = ins->second; + if (inserted) { + // For new chains, we want to default to VERSION_HD_BASE until we see an internal + chain.nVersion = CHDChain::VERSION_HD_BASE; + chain.seed_id = keyMeta.hd_seed_id; } - } else if (strType == DBKeys::ORDERPOSNEXT) { - ssValue >> pwallet->nOrderPosNext; - } else if (strType == DBKeys::DESTDATA) { - std::string strAddress, strKey, strValue; - ssKey >> strAddress; - ssKey >> strKey; - ssValue >> strValue; - const CTxDestination& dest{DecodeDestination(strAddress)}; - if (strKey.compare("used") == 0) { - // Load "used" key indicating if an IsMine address has - // previously been spent from with avoid_reuse option enabled. - // The strValue is not used for anything currently, but could - // hold more information in the future. Current values are just - // "1" or "p" for present (which was written prior to - // f5ba424cd44619d9b9be88b8593d69a7ba96db26). - pwallet->LoadAddressPreviouslySpent(dest); - } else if (strKey.compare(0, 2, "rr") == 0) { - // Load "rr##" keys where ## is a decimal number, and strValue - // is a serialized RecentRequestEntry object. - pwallet->LoadAddressReceiveRequest(dest, strKey.substr(2), strValue); + if (internal) { + chain.nVersion = CHDChain::VERSION_HD_CHAIN_SPLIT; + chain.nInternalChainCounter = std::max(chain.nInternalChainCounter, index + 1); + } else { + chain.nExternalChainCounter = std::max(chain.nExternalChainCounter, index + 1); } - } else if (strType == DBKeys::HDCHAIN) { - CHDChain chain; - ssValue >> chain; - pwallet->GetOrCreateLegacyScriptPubKeyMan()->LoadHDChain(chain); - } else if (strType == DBKeys::OLD_KEY) { - strErr = "Found unsupported 'wkey' record, try loading with version 0.18"; - return false; - } else if (strType == DBKeys::ACTIVEEXTERNALSPK || strType == DBKeys::ACTIVEINTERNALSPK) { - uint8_t type; - ssKey >> type; - uint256 id; - ssValue >> id; + } + return DBErrors::LOAD_OK; + }); + result = std::max(result, keymeta_res.m_result); - bool internal = strType == DBKeys::ACTIVEINTERNALSPK; - auto& spk_mans = internal ? wss.m_active_internal_spks : wss.m_active_external_spks; - if (spk_mans.count(static_cast<OutputType>(type)) > 0) { - strErr = "Multiple ScriptPubKeyMans specified for a single type"; - return false; - } - spk_mans[static_cast<OutputType>(type)] = id; - } else if (strType == DBKeys::WALLETDESCRIPTOR) { - uint256 id; - ssKey >> id; - WalletDescriptor desc; - try { - ssValue >> desc; - } catch (const std::ios_base::failure& e) { - strErr = e.what(); - wss.descriptor_unknown = true; - return false; + // Set inactive chains + if (!hd_chains.empty()) { + LegacyScriptPubKeyMan* legacy_spkm = pwallet->GetLegacyScriptPubKeyMan(); + if (legacy_spkm) { + for (const auto& [hd_seed_id, chain] : hd_chains) { + if (hd_seed_id != legacy_spkm->GetHDChain().seed_id) { + legacy_spkm->AddInactiveHDChain(chain); + } } - if (wss.m_descriptor_caches.count(id) == 0) { - wss.m_descriptor_caches[id] = DescriptorCache(); + } else { + pwallet->WalletLogPrintf("Inactive HD Chains found but no Legacy ScriptPubKeyMan\n"); + result = DBErrors::CORRUPT; + } + } + + // Load watchonly scripts + LoadResult watch_script_res = LoadRecords(pwallet, batch, DBKeys::WATCHS, + [] (CWallet* pwallet, DataStream& key, CDataStream& value, std::string& err) { + CScript script; + key >> script; + uint8_t fYes; + value >> fYes; + if (fYes == '1') { + pwallet->GetOrCreateLegacyScriptPubKeyMan()->LoadWatchOnly(script); + } + return DBErrors::LOAD_OK; + }); + result = std::max(result, watch_script_res.m_result); + + // Load watchonly meta + LoadResult watch_meta_res = LoadRecords(pwallet, batch, DBKeys::WATCHMETA, + [] (CWallet* pwallet, DataStream& key, CDataStream& value, std::string& err) { + CScript script; + key >> script; + CKeyMetadata keyMeta; + value >> keyMeta; + pwallet->GetOrCreateLegacyScriptPubKeyMan()->LoadScriptMetadata(CScriptID(script), keyMeta); + return DBErrors::LOAD_OK; + }); + result = std::max(result, watch_meta_res.m_result); + + // Load keypool + LoadResult pool_res = LoadRecords(pwallet, batch, DBKeys::POOL, + [] (CWallet* pwallet, DataStream& key, CDataStream& value, std::string& err) { + int64_t nIndex; + key >> nIndex; + CKeyPool keypool; + value >> keypool; + pwallet->GetOrCreateLegacyScriptPubKeyMan()->LoadKeyPool(nIndex, keypool); + return DBErrors::LOAD_OK; + }); + result = std::max(result, pool_res.m_result); + + // Deal with old "wkey" and "defaultkey" records. + // These are not actually loaded, but we need to check for them + + // We don't want or need the default key, but if there is one set, + // we want to make sure that it is valid so that we can detect corruption + // Note: There should only be one DEFAULTKEY with nothing trailing the type + LoadResult default_key_res = LoadRecords(pwallet, batch, DBKeys::DEFAULTKEY, + [] (CWallet* pwallet, DataStream& key, CDataStream& value, std::string& err) { + CPubKey default_pubkey; + try { + value >> default_pubkey; + } catch (const std::exception& e) { + err = e.what(); + return DBErrors::CORRUPT; + } + if (!default_pubkey.IsValid()) { + err = "Error reading wallet database: Default Key corrupt"; + return DBErrors::CORRUPT; + } + return DBErrors::LOAD_OK; + }); + result = std::max(result, default_key_res.m_result); + + // "wkey" records are unsupported, if we see any, throw an error + LoadResult wkey_res = LoadRecords(pwallet, batch, DBKeys::OLD_KEY, + [] (CWallet* pwallet, DataStream& key, CDataStream& value, std::string& err) { + err = "Found unsupported 'wkey' record, try loading with version 0.18"; + return DBErrors::LOAD_FAIL; + }); + result = std::max(result, wkey_res.m_result); + + if (result <= DBErrors::NONCRITICAL_ERROR) { + // Only do logging and time first key update if there were no critical errors + pwallet->WalletLogPrintf("Legacy Wallet Keys: %u plaintext, %u encrypted, %u w/ metadata, %u total.\n", + key_res.m_records, ckey_res.m_records, keymeta_res.m_records, key_res.m_records + ckey_res.m_records); + + // nTimeFirstKey is only reliable if all keys have metadata + if (pwallet->IsLegacy() && (key_res.m_records + ckey_res.m_records + watch_script_res.m_records) != (keymeta_res.m_records + watch_meta_res.m_records)) { + auto spk_man = pwallet->GetOrCreateLegacyScriptPubKeyMan(); + if (spk_man) { + LOCK(spk_man->cs_KeyStore); + spk_man->UpdateTimeFirstKey(1); } - pwallet->LoadDescriptorScriptPubKeyMan(id, desc); - } else if (strType == DBKeys::WALLETDESCRIPTORCACHE) { + } + } + + return result; +} + +template<typename... Args> +static DataStream PrefixStream(const Args&... args) +{ + DataStream prefix; + SerializeMany(prefix, args...); + return prefix; +} + +static DBErrors LoadDescriptorWalletRecords(CWallet* pwallet, DatabaseBatch& batch, int last_client) EXCLUSIVE_LOCKS_REQUIRED(pwallet->cs_wallet) +{ + AssertLockHeld(pwallet->cs_wallet); + + // Load descriptor record + int num_keys = 0; + int num_ckeys= 0; + LoadResult desc_res = LoadRecords(pwallet, batch, DBKeys::WALLETDESCRIPTOR, + [&batch, &num_keys, &num_ckeys, &last_client] (CWallet* pwallet, DataStream& key, CDataStream& value, std::string& strErr) { + DBErrors result = DBErrors::LOAD_OK; + + uint256 id; + key >> id; + WalletDescriptor desc; + try { + value >> desc; + } catch (const std::ios_base::failure&) { + strErr = strprintf("Error: Unrecognized descriptor found in wallet %s. ", pwallet->GetName()); + strErr += (last_client > CLIENT_VERSION) ? "The wallet might had been created on a newer version. " : + "The database might be corrupted or the software version is not compatible with one of your wallet descriptors. "; + strErr += "Please try running the latest software version"; + return DBErrors::UNKNOWN_DESCRIPTOR; + } + pwallet->LoadDescriptorScriptPubKeyMan(id, desc); + + DescriptorCache cache; + + // Get key cache for this descriptor + DataStream prefix = PrefixStream(DBKeys::WALLETDESCRIPTORCACHE, id); + LoadResult key_cache_res = LoadRecords(pwallet, batch, DBKeys::WALLETDESCRIPTORCACHE, prefix, + [&id, &cache] (CWallet* pwallet, DataStream& key, CDataStream& value, std::string& err) { bool parent = true; uint256 desc_id; uint32_t key_exp_index; uint32_t der_index; - ssKey >> desc_id; - ssKey >> key_exp_index; + key >> desc_id; + assert(desc_id == id); + key >> key_exp_index; // if the der_index exists, it's a derived xpub try { - ssKey >> der_index; + key >> der_index; parent = false; } catch (...) {} std::vector<unsigned char> ser_xpub(BIP32_EXTKEY_SIZE); - ssValue >> ser_xpub; + value >> ser_xpub; CExtPubKey xpub; xpub.Decode(ser_xpub.data()); if (parent) { - wss.m_descriptor_caches[desc_id].CacheParentExtPubKey(key_exp_index, xpub); + cache.CacheParentExtPubKey(key_exp_index, xpub); } else { - wss.m_descriptor_caches[desc_id].CacheDerivedExtPubKey(key_exp_index, der_index, xpub); + cache.CacheDerivedExtPubKey(key_exp_index, der_index, xpub); } - } else if (strType == DBKeys::WALLETDESCRIPTORLHCACHE) { + return DBErrors::LOAD_OK; + }); + result = std::max(result, key_cache_res.m_result); + + // Get last hardened cache for this descriptor + prefix = PrefixStream(DBKeys::WALLETDESCRIPTORLHCACHE, id); + LoadResult lh_cache_res = LoadRecords(pwallet, batch, DBKeys::WALLETDESCRIPTORLHCACHE, prefix, + [&id, &cache] (CWallet* pwallet, DataStream& key, CDataStream& value, std::string& err) { uint256 desc_id; uint32_t key_exp_index; - ssKey >> desc_id; - ssKey >> key_exp_index; + key >> desc_id; + assert(desc_id == id); + key >> key_exp_index; std::vector<unsigned char> ser_xpub(BIP32_EXTKEY_SIZE); - ssValue >> ser_xpub; + value >> ser_xpub; CExtPubKey xpub; xpub.Decode(ser_xpub.data()); - wss.m_descriptor_caches[desc_id].CacheLastHardenedExtPubKey(key_exp_index, xpub); - } else if (strType == DBKeys::WALLETDESCRIPTORKEY) { + cache.CacheLastHardenedExtPubKey(key_exp_index, xpub); + return DBErrors::LOAD_OK; + }); + result = std::max(result, lh_cache_res.m_result); + + // Set the cache for this descriptor + auto spk_man = (DescriptorScriptPubKeyMan*)pwallet->GetScriptPubKeyMan(id); + assert(spk_man); + spk_man->SetCache(cache); + + // Get unencrypted keys + prefix = PrefixStream(DBKeys::WALLETDESCRIPTORKEY, id); + LoadResult key_res = LoadRecords(pwallet, batch, DBKeys::WALLETDESCRIPTORKEY, prefix, + [&id, &spk_man] (CWallet* pwallet, DataStream& key, CDataStream& value, std::string& strErr) { uint256 desc_id; CPubKey pubkey; - ssKey >> desc_id; - ssKey >> pubkey; + key >> desc_id; + assert(desc_id == id); + key >> pubkey; if (!pubkey.IsValid()) { - strErr = "Error reading wallet database: CPubKey corrupt"; - return false; + strErr = "Error reading wallet database: descriptor unencrypted key CPubKey corrupt"; + return DBErrors::CORRUPT; } - CKey key; + CKey privkey; CPrivKey pkey; uint256 hash; - wss.nKeys++; - ssValue >> pkey; - ssValue >> hash; + value >> pkey; + value >> hash; // hash pubkey/privkey to accelerate wallet load std::vector<unsigned char> to_hash; @@ -726,77 +891,254 @@ ReadKeyValue(CWallet* pwallet, DataStream& ssKey, CDataStream& ssValue, if (Hash(to_hash) != hash) { - strErr = "Error reading wallet database: CPubKey/CPrivKey corrupt"; - return false; + strErr = "Error reading wallet database: descriptor unencrypted key CPubKey/CPrivKey corrupt"; + return DBErrors::CORRUPT; } - if (!key.Load(pkey, pubkey, true)) + if (!privkey.Load(pkey, pubkey, true)) { - strErr = "Error reading wallet database: CPrivKey corrupt"; - return false; + strErr = "Error reading wallet database: descriptor unencrypted key CPrivKey corrupt"; + return DBErrors::CORRUPT; } - wss.m_descriptor_keys.insert(std::make_pair(std::make_pair(desc_id, pubkey.GetID()), key)); - } else if (strType == DBKeys::WALLETDESCRIPTORCKEY) { + spk_man->AddKey(pubkey.GetID(), privkey); + return DBErrors::LOAD_OK; + }); + result = std::max(result, key_res.m_result); + num_keys = key_res.m_records; + + // Get encrypted keys + prefix = PrefixStream(DBKeys::WALLETDESCRIPTORCKEY, id); + LoadResult ckey_res = LoadRecords(pwallet, batch, DBKeys::WALLETDESCRIPTORCKEY, prefix, + [&id, &spk_man] (CWallet* pwallet, DataStream& key, CDataStream& value, std::string& err) { uint256 desc_id; CPubKey pubkey; - ssKey >> desc_id; - ssKey >> pubkey; + key >> desc_id; + assert(desc_id == id); + key >> pubkey; if (!pubkey.IsValid()) { - strErr = "Error reading wallet database: CPubKey corrupt"; - return false; + err = "Error reading wallet database: descriptor encrypted key CPubKey corrupt"; + return DBErrors::CORRUPT; } std::vector<unsigned char> privkey; - ssValue >> privkey; - wss.nCKeys++; + value >> privkey; - wss.m_descriptor_crypt_keys.insert(std::make_pair(std::make_pair(desc_id, pubkey.GetID()), std::make_pair(pubkey, privkey))); - wss.fIsEncrypted = true; - } else if (strType == DBKeys::LOCKED_UTXO) { - uint256 hash; - uint32_t n; - ssKey >> hash; - ssKey >> n; - pwallet->LockCoin(COutPoint(hash, n)); - } else if (strType != DBKeys::BESTBLOCK && strType != DBKeys::BESTBLOCK_NOMERKLE && - strType != DBKeys::MINVERSION && strType != DBKeys::ACENTRY && - strType != DBKeys::VERSION && strType != DBKeys::SETTINGS && - strType != DBKeys::FLAGS) { - wss.m_unknown_records++; + spk_man->AddCryptedKey(pubkey.GetID(), pubkey, privkey); + return DBErrors::LOAD_OK; + }); + result = std::max(result, ckey_res.m_result); + num_ckeys = ckey_res.m_records; + + return result; + }); + + if (desc_res.m_result <= DBErrors::NONCRITICAL_ERROR) { + // Only log if there are no critical errors + pwallet->WalletLogPrintf("Descriptors: %u, Descriptor Keys: %u plaintext, %u encrypted, %u total.\n", + desc_res.m_records, num_keys, num_ckeys, num_keys + num_ckeys); + } + + return desc_res.m_result; +} + +static DBErrors LoadAddressBookRecords(CWallet* pwallet, DatabaseBatch& batch) EXCLUSIVE_LOCKS_REQUIRED(pwallet->cs_wallet) +{ + AssertLockHeld(pwallet->cs_wallet); + DBErrors result = DBErrors::LOAD_OK; + + // Load name record + LoadResult name_res = LoadRecords(pwallet, batch, DBKeys::NAME, + [] (CWallet* pwallet, DataStream& key, CDataStream& value, std::string& err) EXCLUSIVE_LOCKS_REQUIRED(pwallet->cs_wallet) { + std::string strAddress; + key >> strAddress; + std::string label; + value >> label; + pwallet->m_address_book[DecodeDestination(strAddress)].SetLabel(label); + return DBErrors::LOAD_OK; + }); + result = std::max(result, name_res.m_result); + + // Load purpose record + LoadResult purpose_res = LoadRecords(pwallet, batch, DBKeys::PURPOSE, + [] (CWallet* pwallet, DataStream& key, CDataStream& value, std::string& err) EXCLUSIVE_LOCKS_REQUIRED(pwallet->cs_wallet) { + std::string strAddress; + key >> strAddress; + std::string purpose_str; + value >> purpose_str; + std::optional<AddressPurpose> purpose{PurposeFromString(purpose_str)}; + if (!purpose) { + pwallet->WalletLogPrintf("Warning: nonstandard purpose string '%s' for address '%s'\n", purpose_str, strAddress); } - } catch (const std::exception& e) { - if (strErr.empty()) { - strErr = e.what(); + pwallet->m_address_book[DecodeDestination(strAddress)].purpose = purpose; + return DBErrors::LOAD_OK; + }); + result = std::max(result, purpose_res.m_result); + + // Load destination data record + LoadResult dest_res = LoadRecords(pwallet, batch, DBKeys::DESTDATA, + [] (CWallet* pwallet, DataStream& key, CDataStream& value, std::string& err) EXCLUSIVE_LOCKS_REQUIRED(pwallet->cs_wallet) { + std::string strAddress, strKey, strValue; + key >> strAddress; + key >> strKey; + value >> strValue; + const CTxDestination& dest{DecodeDestination(strAddress)}; + if (strKey.compare("used") == 0) { + // Load "used" key indicating if an IsMine address has + // previously been spent from with avoid_reuse option enabled. + // The strValue is not used for anything currently, but could + // hold more information in the future. Current values are just + // "1" or "p" for present (which was written prior to + // f5ba424cd44619d9b9be88b8593d69a7ba96db26). + pwallet->LoadAddressPreviouslySpent(dest); + } else if (strKey.compare(0, 2, "rr") == 0) { + // Load "rr##" keys where ## is a decimal number, and strValue + // is a serialized RecentRequestEntry object. + pwallet->LoadAddressReceiveRequest(dest, strKey.substr(2), strValue); } - return false; - } catch (...) { - if (strErr.empty()) { - strErr = "Caught unknown exception in ReadKeyValue"; + return DBErrors::LOAD_OK; + }); + result = std::max(result, dest_res.m_result); + + return result; +} + +static DBErrors LoadTxRecords(CWallet* pwallet, DatabaseBatch& batch, std::vector<uint256>& upgraded_txs, bool& any_unordered) EXCLUSIVE_LOCKS_REQUIRED(pwallet->cs_wallet) +{ + AssertLockHeld(pwallet->cs_wallet); + DBErrors result = DBErrors::LOAD_OK; + + // Load tx record + any_unordered = false; + LoadResult tx_res = LoadRecords(pwallet, batch, DBKeys::TX, + [&any_unordered, &upgraded_txs] (CWallet* pwallet, DataStream& key, CDataStream& value, std::string& err) EXCLUSIVE_LOCKS_REQUIRED(pwallet->cs_wallet) { + DBErrors result = DBErrors::LOAD_OK; + uint256 hash; + key >> hash; + // LoadToWallet call below creates a new CWalletTx that fill_wtx + // callback fills with transaction metadata. + auto fill_wtx = [&](CWalletTx& wtx, bool new_tx) { + if(!new_tx) { + // There's some corruption here since the tx we just tried to load was already in the wallet. + err = "Error: Corrupt transaction found. This can be fixed by removing transactions from wallet and rescanning."; + result = DBErrors::CORRUPT; + return false; + } + value >> wtx; + if (wtx.GetHash() != hash) + return false; + + // Undo serialize changes in 31600 + if (31404 <= wtx.fTimeReceivedIsTxTime && wtx.fTimeReceivedIsTxTime <= 31703) + { + if (!value.empty()) + { + uint8_t fTmp; + uint8_t fUnused; + std::string unused_string; + value >> fTmp >> fUnused >> unused_string; + pwallet->WalletLogPrintf("LoadWallet() upgrading tx ver=%d %d %s\n", + wtx.fTimeReceivedIsTxTime, fTmp, hash.ToString()); + wtx.fTimeReceivedIsTxTime = fTmp; + } + else + { + pwallet->WalletLogPrintf("LoadWallet() repairing tx ver=%d %s\n", wtx.fTimeReceivedIsTxTime, hash.ToString()); + wtx.fTimeReceivedIsTxTime = 0; + } + upgraded_txs.push_back(hash); + } + + if (wtx.nOrderPos == -1) + any_unordered = true; + + return true; + }; + if (!pwallet->LoadToWallet(hash, fill_wtx)) { + // Use std::max as fill_wtx may have already set result to CORRUPT + result = std::max(result, DBErrors::NEED_RESCAN); } - return false; - } - return true; + return result; + }); + result = std::max(result, tx_res.m_result); + + // Load locked utxo record + LoadResult locked_utxo_res = LoadRecords(pwallet, batch, DBKeys::LOCKED_UTXO, + [] (CWallet* pwallet, DataStream& key, CDataStream& value, std::string& err) EXCLUSIVE_LOCKS_REQUIRED(pwallet->cs_wallet) { + uint256 hash; + uint32_t n; + key >> hash; + key >> n; + pwallet->LockCoin(COutPoint(hash, n)); + return DBErrors::LOAD_OK; + }); + result = std::max(result, locked_utxo_res.m_result); + + // Load orderposnext record + // Note: There should only be one ORDERPOSNEXT record with nothing trailing the type + LoadResult order_pos_res = LoadRecords(pwallet, batch, DBKeys::ORDERPOSNEXT, + [] (CWallet* pwallet, DataStream& key, CDataStream& value, std::string& err) EXCLUSIVE_LOCKS_REQUIRED(pwallet->cs_wallet) { + try { + value >> pwallet->nOrderPosNext; + } catch (const std::exception& e) { + err = e.what(); + return DBErrors::NONCRITICAL_ERROR; + } + return DBErrors::LOAD_OK; + }); + result = std::max(result, order_pos_res.m_result); + + return result; } -bool ReadKeyValue(CWallet* pwallet, DataStream& ssKey, CDataStream& ssValue, std::string& strType, std::string& strErr, const KeyFilterFn& filter_fn) +static DBErrors LoadActiveSPKMs(CWallet* pwallet, DatabaseBatch& batch) EXCLUSIVE_LOCKS_REQUIRED(pwallet->cs_wallet) { - CWalletScanState dummy_wss; - LOCK(pwallet->cs_wallet); - return ReadKeyValue(pwallet, ssKey, ssValue, dummy_wss, strType, strErr, filter_fn); + AssertLockHeld(pwallet->cs_wallet); + DBErrors result = DBErrors::LOAD_OK; + + // Load spk records + std::set<std::pair<OutputType, bool>> seen_spks; + for (const auto& spk_key : {DBKeys::ACTIVEEXTERNALSPK, DBKeys::ACTIVEINTERNALSPK}) { + LoadResult spkm_res = LoadRecords(pwallet, batch, spk_key, + [&seen_spks, &spk_key] (CWallet* pwallet, DataStream& key, CDataStream& value, std::string& strErr) { + uint8_t output_type; + key >> output_type; + uint256 id; + value >> id; + + bool internal = spk_key == DBKeys::ACTIVEINTERNALSPK; + auto [it, insert] = seen_spks.emplace(static_cast<OutputType>(output_type), internal); + if (!insert) { + strErr = "Multiple ScriptpubKeyMans specified for a single type"; + return DBErrors::CORRUPT; + } + pwallet->LoadActiveScriptPubKeyMan(id, static_cast<OutputType>(output_type), /*internal=*/internal); + return DBErrors::LOAD_OK; + }); + result = std::max(result, spkm_res.m_result); + } + return result; } -bool WalletBatch::IsKeyType(const std::string& strType) +static DBErrors LoadDecryptionKeys(CWallet* pwallet, DatabaseBatch& batch) EXCLUSIVE_LOCKS_REQUIRED(pwallet->cs_wallet) { - return (strType == DBKeys::KEY || - strType == DBKeys::MASTER_KEY || strType == DBKeys::CRYPTED_KEY); + AssertLockHeld(pwallet->cs_wallet); + + // Load decryption key (mkey) records + LoadResult mkey_res = LoadRecords(pwallet, batch, DBKeys::MASTER_KEY, + [] (CWallet* pwallet, DataStream& key, CDataStream& value, std::string& err) { + if (!LoadEncryptionKey(pwallet, key, value, err)) { + return DBErrors::CORRUPT; + } + return DBErrors::LOAD_OK; + }); + return mkey_res.m_result; } DBErrors WalletBatch::LoadWallet(CWallet* pwallet) { - CWalletScanState wss; - bool fNoncriticalErrors = false; - bool rescan_required = false; DBErrors result = DBErrors::LOAD_OK; + bool any_unordered = false; + std::vector<uint256> upgraded_txs; LOCK(pwallet->cs_wallet); @@ -806,22 +1148,11 @@ DBErrors WalletBatch::LoadWallet(CWallet* pwallet) pwallet->WalletLogPrintf("Wallet file version = %d, last client version = %d\n", pwallet->GetVersion(), last_client); try { - int nMinVersion = 0; - if (m_batch->Read(DBKeys::MINVERSION, nMinVersion)) { - if (nMinVersion > FEATURE_LATEST) - return DBErrors::TOO_NEW; - pwallet->LoadMinVersion(nMinVersion); - } + if ((result = LoadMinVersion(pwallet, *m_batch)) != DBErrors::LOAD_OK) return result; // Load wallet flags, so they are known when processing other records. // The FLAGS key is absent during wallet creation. - uint64_t flags; - if (m_batch->Read(DBKeys::FLAGS, flags)) { - if (!pwallet->LoadWalletFlags(flags)) { - pwallet->WalletLogPrintf("Error reading wallet database: Unknown non-tolerable wallet flags found\n"); - return DBErrors::CORRUPT; - } - } + if ((result = LoadWalletFlags(pwallet, *m_batch)) != DBErrors::LOAD_OK) return result; #ifndef ENABLE_EXTERNAL_SIGNER if (pwallet->IsWalletFlagSet(WALLET_FLAG_EXTERNAL_SIGNER)) { @@ -830,101 +1161,31 @@ DBErrors WalletBatch::LoadWallet(CWallet* pwallet) } #endif - // Get cursor - std::unique_ptr<DatabaseCursor> cursor = m_batch->GetNewCursor(); - if (!cursor) - { - pwallet->WalletLogPrintf("Error getting wallet database cursor\n"); - return DBErrors::CORRUPT; - } + // Load legacy wallet keys + result = std::max(LoadLegacyWalletRecords(pwallet, *m_batch, last_client), result); - while (true) - { - // Read next record - DataStream ssKey{}; - CDataStream ssValue(SER_DISK, CLIENT_VERSION); - DatabaseCursor::Status status = cursor->Next(ssKey, ssValue); - if (status == DatabaseCursor::Status::DONE) { - break; - } else if (status == DatabaseCursor::Status::FAIL) { - cursor.reset(); - pwallet->WalletLogPrintf("Error reading next record from wallet database\n"); - return DBErrors::CORRUPT; - } + // Load descriptors + result = std::max(LoadDescriptorWalletRecords(pwallet, *m_batch, last_client), result); + // Early return if there are unknown descriptors. Later loading of ACTIVEINTERNALSPK and ACTIVEEXTERNALEXPK + // may reference the unknown descriptor's ID which can result in a misleading corruption error + // when in reality the wallet is simply too new. + if (result == DBErrors::UNKNOWN_DESCRIPTOR) return result; - // Try to be tolerant of single corrupt records: - std::string strType, strErr; - if (!ReadKeyValue(pwallet, ssKey, ssValue, wss, strType, strErr)) - { - if (wss.unexpected_legacy_entry) { - strErr = strprintf("Error: Unexpected legacy entry found in descriptor wallet %s. ", pwallet->GetName()); - strErr += "The wallet might have been tampered with or created with malicious intent."; - pwallet->WalletLogPrintf("%s\n", strErr); - return DBErrors::UNEXPECTED_LEGACY_ENTRY; - } - // losing keys is considered a catastrophic error, anything else - // we assume the user can live with: - if (IsKeyType(strType) || strType == DBKeys::DEFAULTKEY) { - result = DBErrors::CORRUPT; - } else if (strType == DBKeys::FLAGS) { - // reading the wallet flags can only fail if unknown flags are present - result = DBErrors::TOO_NEW; - } else if (wss.tx_corrupt) { - pwallet->WalletLogPrintf("Error: Corrupt transaction found. This can be fixed by removing transactions from wallet and rescanning.\n"); - // Set tx_corrupt back to false so that the error is only printed once (per corrupt tx) - wss.tx_corrupt = false; - result = DBErrors::CORRUPT; - } else if (wss.descriptor_unknown) { - strErr = strprintf("Error: Unrecognized descriptor found in wallet %s. ", pwallet->GetName()); - strErr += (last_client > CLIENT_VERSION) ? "The wallet might had been created on a newer version. " : - "The database might be corrupted or the software version is not compatible with one of your wallet descriptors. "; - strErr += "Please try running the latest software version"; - pwallet->WalletLogPrintf("%s\n", strErr); - return DBErrors::UNKNOWN_DESCRIPTOR; - } else { - // Leave other errors alone, if we try to fix them we might make things worse. - fNoncriticalErrors = true; // ... but do warn the user there is something wrong. - if (strType == DBKeys::TX) - // Rescan if there is a bad transaction record: - rescan_required = true; - } - } - if (!strErr.empty()) - pwallet->WalletLogPrintf("%s\n", strErr); - } - } catch (...) { - result = DBErrors::CORRUPT; - } + // Load address book + result = std::max(LoadAddressBookRecords(pwallet, *m_batch), result); - // Set the active ScriptPubKeyMans - for (auto spk_man_pair : wss.m_active_external_spks) { - pwallet->LoadActiveScriptPubKeyMan(spk_man_pair.second, spk_man_pair.first, /*internal=*/false); - } - for (auto spk_man_pair : wss.m_active_internal_spks) { - pwallet->LoadActiveScriptPubKeyMan(spk_man_pair.second, spk_man_pair.first, /*internal=*/true); - } + // Load tx records + result = std::max(LoadTxRecords(pwallet, *m_batch, upgraded_txs, any_unordered), result); - // Set the descriptor caches - for (const auto& desc_cache_pair : wss.m_descriptor_caches) { - auto spk_man = pwallet->GetScriptPubKeyMan(desc_cache_pair.first); - assert(spk_man); - ((DescriptorScriptPubKeyMan*)spk_man)->SetCache(desc_cache_pair.second); - } + // Load SPKMs + result = std::max(LoadActiveSPKMs(pwallet, *m_batch), result); - // Set the descriptor keys - for (const auto& desc_key_pair : wss.m_descriptor_keys) { - auto spk_man = pwallet->GetScriptPubKeyMan(desc_key_pair.first.first); - ((DescriptorScriptPubKeyMan*)spk_man)->AddKey(desc_key_pair.first.second, desc_key_pair.second); - } - for (const auto& desc_key_pair : wss.m_descriptor_crypt_keys) { - auto spk_man = pwallet->GetScriptPubKeyMan(desc_key_pair.first.first); - ((DescriptorScriptPubKeyMan*)spk_man)->AddCryptedKey(desc_key_pair.first.second, desc_key_pair.second.first, desc_key_pair.second.second); - } - - if (rescan_required && result == DBErrors::LOAD_OK) { - result = DBErrors::NEED_RESCAN; - } else if (fNoncriticalErrors && result == DBErrors::LOAD_OK) { - result = DBErrors::NONCRITICAL_ERROR; + // Load decryption keys + result = std::max(LoadDecryptionKeys(pwallet, *m_batch), result); + } catch (...) { + // Exceptions that can be ignored or treated as non-critical are handled by the individual loading functions. + // Any uncaught exceptions will be caught here and treated as critical. + result = DBErrors::CORRUPT; } // Any wallet corruption at all: skip any rewriting or @@ -932,29 +1193,13 @@ DBErrors WalletBatch::LoadWallet(CWallet* pwallet) if (result != DBErrors::LOAD_OK) return result; - pwallet->WalletLogPrintf("Keys: %u plaintext, %u encrypted, %u w/ metadata, %u total. Unknown wallet records: %u\n", - wss.nKeys, wss.nCKeys, wss.nKeyMeta, wss.nKeys + wss.nCKeys, wss.m_unknown_records); - - // nTimeFirstKey is only reliable if all keys have metadata - if (pwallet->IsLegacy() && (wss.nKeys + wss.nCKeys + wss.nWatchKeys) != wss.nKeyMeta) { - auto spk_man = pwallet->GetOrCreateLegacyScriptPubKeyMan(); - if (spk_man) { - LOCK(spk_man->cs_KeyStore); - spk_man->UpdateTimeFirstKey(1); - } - } - - for (const uint256& hash : wss.vWalletUpgrade) + for (const uint256& hash : upgraded_txs) WriteTx(pwallet->mapWallet.at(hash)); - // Rewrite encrypted wallets of versions 0.4.0 and 0.5.0rc: - if (wss.fIsEncrypted && (last_client == 40000 || last_client == 50000)) - return DBErrors::NEED_REWRITE; - if (!has_last_client || last_client != CLIENT_VERSION) // Update m_batch->Write(DBKeys::VERSION, CLIENT_VERSION); - if (wss.fAnyUnordered) + if (any_unordered) result = pwallet->ReorderTransactions(); // Upgrade all of the wallet keymetadata to have the hd master key id @@ -973,20 +1218,6 @@ DBErrors WalletBatch::LoadWallet(CWallet* pwallet) result = DBErrors::CORRUPT; } - // Set the inactive chain - if (wss.m_hd_chains.size() > 0) { - LegacyScriptPubKeyMan* legacy_spkm = pwallet->GetLegacyScriptPubKeyMan(); - if (!legacy_spkm) { - pwallet->WalletLogPrintf("Inactive HD Chains found but no Legacy ScriptPubKeyMan\n"); - return DBErrors::CORRUPT; - } - for (const auto& chain_pair : wss.m_hd_chains) { - if (chain_pair.first != pwallet->GetLegacyScriptPubKeyMan()->GetHDChain().seed_id) { - pwallet->GetLegacyScriptPubKeyMan()->AddInactiveHDChain(chain_pair.second); - } - } - } - return result; } diff --git a/src/wallet/walletdb.h b/src/wallet/walletdb.h index f84a89b23f..8f7c2f030c 100644 --- a/src/wallet/walletdb.h +++ b/src/wallet/walletdb.h @@ -42,19 +42,21 @@ struct WalletContext; static const bool DEFAULT_FLUSHWALLET = true; -/** Error statuses for the wallet database */ -enum class DBErrors +/** Error statuses for the wallet database. + * Values are in order of severity. When multiple errors occur, the most severe (highest value) will be returned. + */ +enum class DBErrors : int { - LOAD_OK, - CORRUPT, - NONCRITICAL_ERROR, - TOO_NEW, - EXTERNAL_SIGNER_SUPPORT_REQUIRED, - LOAD_FAIL, - NEED_REWRITE, - NEED_RESCAN, - UNKNOWN_DESCRIPTOR, - UNEXPECTED_LEGACY_ENTRY + LOAD_OK = 0, + NEED_RESCAN = 1, + NEED_REWRITE = 2, + EXTERNAL_SIGNER_SUPPORT_REQUIRED = 3, + NONCRITICAL_ERROR = 4, + TOO_NEW = 5, + UNKNOWN_DESCRIPTOR = 6, + LOAD_FAIL = 7, + UNEXPECTED_LEGACY_ENTRY = 8, + CORRUPT = 9, }; namespace DBKeys { @@ -276,8 +278,6 @@ public: DBErrors LoadWallet(CWallet* pwallet); DBErrors FindWalletTxHashes(std::vector<uint256>& tx_hashes); DBErrors ZapSelectTx(std::vector<uint256>& vHashIn, std::vector<uint256>& vHashOut); - /* Function to determine if a certain KV/key-type is a key (cryptographical key) type */ - static bool IsKeyType(const std::string& strType); //! write the hdchain model (external chain child index counter) bool WriteHDChain(const CHDChain& chain); @@ -300,11 +300,10 @@ private: //! Compacts BDB state so that wallet.dat is self-contained (if there are changes) void MaybeCompactWalletDB(WalletContext& context); -//! Callback for filtering key types to deserialize in ReadKeyValue -using KeyFilterFn = std::function<bool(const std::string&)>; - -//! Unserialize a given Key-Value pair and load it into the wallet -bool ReadKeyValue(CWallet* pwallet, DataStream& ssKey, CDataStream& ssValue, std::string& strType, std::string& strErr, const KeyFilterFn& filter_fn = nullptr); +bool LoadKey(CWallet* pwallet, DataStream& ssKey, DataStream& ssValue, std::string& strErr); +bool LoadCryptedKey(CWallet* pwallet, DataStream& ssKey, DataStream& ssValue, std::string& strErr); +bool LoadEncryptionKey(CWallet* pwallet, DataStream& ssKey, DataStream& ssValue, std::string& strErr); +bool LoadHDChain(CWallet* pwallet, DataStream& ssValue, std::string& strErr); } // namespace wallet #endif // BITCOIN_WALLET_WALLETDB_H |