Merge #18836: wallet: upgradewallet fixes and additional tests

5f9c0b6360215636cfa62a70d3a70f1feb3977ab wallet: Remove -upgradewallet from dummywallet (MarcoFalke)
a314271f08215feba53ead27096ac7fda34acb3c test: Remove unused wallet.dat (MarcoFalke)
bf7635963c03203e7189ddaa56c6b086a0108cbf tests: Test specific upgradewallet scenarios and that upgrades work (Andrew Chow)
4b418a9decc3e855ee4b0bbf9e61121c8e9904e5 test: Add test_framework/bdb.py module for inspecting bdb files (Andrew Chow)
092fc434854f881330771a93a1280ac67b1d3549 tests: Add a sha256sum_file function to util (Andrew Chow)
0bd995aa19be65b0dd23df1df571c71428c2bc32 wallet: upgrade the CHDChain version number when upgrading to split hd (Andrew Chow)
8e32e1c41c995e832e643f605d35a7aa112837e6 wallet: remove nWalletMaxVersion (Andrew Chow)
bd7398cc6258c258e9f4411c50630ec4a552341b wallet: have ScriptPubKeyMan::Upgrade check against the new version (Andrew Chow)
5f720544f34dedf75b063b962845fa8eca604514 wallet: Add GetClosestWalletFeature function (Andrew Chow)
842ae3842df489f1b8d68e67a234788966218184 wallet: Add utility method for CanSupportFeature (Andrew Chow)

Pull request description:

  This PR cleans up the wallet upgrade mechanism a bit, fixes some probably bugs, and adds more test cases.

  The `nWalletMaxVersion` member variable has been removed as it made `CanSupportFeature` unintuitive and was causing a couple of bugs. The reason this was introduced originally was to allow a wallet upgrade to only occur when the new feature is first used. While this makes sense for the old `-upgradewallet` option, for an RPC, this does not quite make sense. It's more intuitive for an upgrade to occur if possible if the `upgradewallet` RPC is used as that's an explicit request to upgrade a particular wallet to a newer version. `nWalletMaxVersion` was only relevant for upgrades to `FEATURE_WALLETCRYPT` and `FEATURE_COMPRPUBKEY` both of which are incredibly old features. So for such wallets, the behavior of `upgradewallet` will be that the feature is enabled immediately without the wallet needing to be encrypted at that time (note that `FEATURE_WALLETCRYPT` indicates support for encryption, not that the wallet is encrypted) or for a new key to be generated.

  `CanSupportFeature` would previously indicate whether we could upgrade to `nWalletMaxVersion` not just whether the current wallet version supported a feature. While this property was being used to determine whether we should upgrade to HD and HD chain split, it was also causing a few bugs. Determining whether we should upgrade to HD or HD chain split is resolved by passing into `ScriptPubKeyMan::Upgrade` the version we are upgrading to and checking against that. By removing `nWalletMaxVersion` we also fix a bug where you could upgrade to HD chain split without the pre-split keypool.

  `nWalletMaxVersion` was also the version that was being reported by `getwalletinfo` which meant that the version reported was not always consistent across restarts as it depended on whether `upgradewallet` was used. Additionally to make the wallet versions consistent with actually supported versions, instead of just setting the wallet version to whatever is given to `upgradewallet`, we normalize the version number to the closest supported version number. For example, if given 150000, we would store and report 139900.

  Another bug where CHDChain was not being upgraded to the version supporting HD chain split is also fixed by this PR.

  Lastly several more tests have been added. Some refactoring to the test was made to make these tests easier. These tests check specific upgrading scenarios, such as from non-HD (version 60000) to HD to pre-split keypool. Although not specifically related to `upgradewallet`, `UpgradeKeyMetadata` is now being tested too.

  Part of the new tests is checking that the wallet files are identical before and after failed upgrades. To facilitate this, a utility function `sha256sum_file` has been added. Another part of the tests is to examine the wallet file itself to ensure that the records in the wallet.dat file have been correctly modified. So a new `bdb.py` module has been added to deserialize the BDB db of the wallet.dat file. This format isn't explicitly documented anywhere, but the code and comments in BDB's source code in file `dbinc/db_page.h` describe it. This module just dumps all of the fields into a dict.

ACKs for top commit:
  MarcoFalke:
    approach ACK 5f9c0b6360
  laanwj:
    Code review ACK 5f9c0b6360215636cfa62a70d3a70f1feb3977ab
  jonatack:
    ACK 5f9c0b6360215636cfa62a70d3a70f1feb3977ab, approach seems fine, code review, only skimmed the test changes but they look well done, rebased on current master, debug built and verified the `wallet_upgradewallet.py` test runs green both before and after running `test/get_previous_releases.py -b v0.19.1 v0.18.1 v0.17.2 v0.16.3 v0.15.2`

Tree-SHA512: 7c4ebf420850d596a586cb6dd7f2ef39c6477847d12d105fcd362abb07f2a8aa4f7afc5bfd36cbc8b8c72fcdd1de8d2d3f16ad8e8ba736b6f4f31f133fe5feba

15 files changed, 444 insertions, 86 deletions

diff --git a/src/dummywallet.cpp b/src/dummywallet.cpp
index 8d2dcd0279..4543f098a1 100644
--- a/src/dummywallet.cpp
+++ b/src/dummywallet.cpp
@@ -40,7 +40,6 @@ void DummyWalletInit::AddWalletOptions(ArgsManager& argsman) const
         "-salvagewallet",
         "-spendzeroconfchange",
         "-txconfirmtarget=<n>",
-        "-upgradewallet",
         "-wallet=<path>",
         "-walletbroadcast",
         "-walletdir=<dir>",
diff --git a/src/wallet/scriptpubkeyman.cpp b/src/wallet/scriptpubkeyman.cpp
index 188289b010..d2e1be6402 100644
--- a/src/wallet/scriptpubkeyman.cpp
+++ b/src/wallet/scriptpubkeyman.cpp
@@ -438,12 +438,12 @@ bool LegacyScriptPubKeyMan::CanGetAddresses(bool internal) const
     return keypool_has_keys;
 }
 
-bool LegacyScriptPubKeyMan::Upgrade(int prev_version, bilingual_str& error)
+bool LegacyScriptPubKeyMan::Upgrade(int prev_version, int new_version, bilingual_str& error)
 {
     LOCK(cs_KeyStore);
     bool hd_upgrade = false;
     bool split_upgrade = false;
-    if (m_storage.CanSupportFeature(FEATURE_HD) && !IsHDEnabled()) {
+    if (IsFeatureSupported(new_version, FEATURE_HD) && !IsHDEnabled()) {
         WalletLogPrintf("Upgrading wallet to HD\n");
         m_storage.SetMinVersion(FEATURE_HD);
 
@@ -453,10 +453,17 @@ bool LegacyScriptPubKeyMan::Upgrade(int prev_version, bilingual_str& error)
         hd_upgrade = true;
     }
     // Upgrade to HD chain split if necessary
-    if (m_storage.CanSupportFeature(FEATURE_HD_SPLIT)) {
+    if (IsFeatureSupported(new_version, FEATURE_HD_SPLIT)) {
         WalletLogPrintf("Upgrading wallet to use HD chain split\n");
         m_storage.SetMinVersion(FEATURE_PRE_SPLIT_KEYPOOL);
         split_upgrade = FEATURE_HD_SPLIT > prev_version;
+        // Upgrade the HDChain
+        if (m_hd_chain.nVersion < CHDChain::VERSION_HD_CHAIN_SPLIT) {
+            m_hd_chain.nVersion = CHDChain::VERSION_HD_CHAIN_SPLIT;
+            if (!WalletBatch(m_storage.GetDatabase()).WriteHDChain(m_hd_chain)) {
+                throw std::runtime_error(std::string(__func__) + ": writing chain failed");
+            }
+        }
     }
     // Mark all keys currently in the keypool as pre-split
     if (split_upgrade) {
diff --git a/src/wallet/scriptpubkeyman.h b/src/wallet/scriptpubkeyman.h
index 63c10b7a0d..3bf8f78120 100644
--- a/src/wallet/scriptpubkeyman.h
+++ b/src/wallet/scriptpubkeyman.h
@@ -37,7 +37,7 @@ public:
     virtual bool IsWalletFlagSet(uint64_t) const = 0;
     virtual void UnsetBlankWalletFlag(WalletBatch&) = 0;
     virtual bool CanSupportFeature(enum WalletFeature) const = 0;
-    virtual void SetMinVersion(enum WalletFeature, WalletBatch* = nullptr, bool = false) = 0;
+    virtual void SetMinVersion(enum WalletFeature, WalletBatch* = nullptr) = 0;
     virtual const CKeyingMaterial& GetEncryptionKey() const = 0;
     virtual bool HasEncryptionKeys() const = 0;
     virtual bool IsLocked() const = 0;
@@ -206,7 +206,7 @@ public:
     virtual bool CanGetAddresses(bool internal = false) const { return false; }
 
     /** Upgrades the wallet to the specified version */
-    virtual bool Upgrade(int prev_version, bilingual_str& error) { return false; }
+    virtual bool Upgrade(int prev_version, int new_version, bilingual_str& error) { return false; }
 
     virtual bool HavePrivateKeys() const { return false; }
 
@@ -371,7 +371,7 @@ public:
 
     bool SetupGeneration(bool force = false) override;
 
-    bool Upgrade(int prev_version, bilingual_str& error) override;
+    bool Upgrade(int prev_version, int new_version, bilingual_str& error) override;
 
     bool HavePrivateKeys() const override;
 
diff --git a/src/wallet/wallet.cpp b/src/wallet/wallet.cpp
index e1f34fbcf9..d414555511 100644
--- a/src/wallet/wallet.cpp
+++ b/src/wallet/wallet.cpp
@@ -436,21 +436,13 @@ void CWallet::chainStateFlushed(const CBlockLocator& loc)
     batch.WriteBestBlock(loc);
 }
 
-void CWallet::SetMinVersion(enum WalletFeature nVersion, WalletBatch* batch_in, bool fExplicit)
+void CWallet::SetMinVersion(enum WalletFeature nVersion, WalletBatch* batch_in)
 {
     LOCK(cs_wallet);
     if (nWalletVersion >= nVersion)
         return;
-
-    // when doing an explicit upgrade, if we pass the max version permitted, upgrade all the way
-    if (fExplicit && nVersion > nWalletMaxVersion)
-            nVersion = FEATURE_LATEST;
-
     nWalletVersion = nVersion;
 
-    if (nVersion > nWalletMaxVersion)
-        nWalletMaxVersion = nVersion;
-
     {
         WalletBatch* batch = batch_in ? batch_in : new WalletBatch(*database);
         if (nWalletVersion > 40000)
@@ -460,18 +452,6 @@ void CWallet::SetMinVersion(enum WalletFeature nVersion, WalletBatch* batch_in,
     }
 }
 
-bool CWallet::SetMaxVersion(int nVersion)
-{
-    LOCK(cs_wallet);
-    // cannot downgrade below current version
-    if (nWalletVersion > nVersion)
-        return false;
-
-    nWalletMaxVersion = nVersion;
-
-    return true;
-}
-
 std::set<uint256> CWallet::GetConflicts(const uint256& txid) const
 {
     std::set<uint256> result;
@@ -656,7 +636,7 @@ bool CWallet::EncryptWallet(const SecureString& strWalletPassphrase)
         }
 
         // Encryption was introduced in version 0.4.0
-        SetMinVersion(FEATURE_WALLETCRYPT, encrypted_batch, true);
+        SetMinVersion(FEATURE_WALLETCRYPT, encrypted_batch);
 
         if (!encrypted_batch->TxnCommit()) {
             delete encrypted_batch;
@@ -4125,33 +4105,31 @@ const CAddressBookData* CWallet::FindAddressBookEntry(const CTxDestination& dest
 bool CWallet::UpgradeWallet(int version, bilingual_str& error, std::vector<bilingual_str>& warnings)
 {
     int prev_version = GetVersion();
-    int nMaxVersion = version;
-    if (nMaxVersion == 0) // the -upgradewallet without argument case
-    {
+    if (version == 0) {
         WalletLogPrintf("Performing wallet upgrade to %i\n", FEATURE_LATEST);
-        nMaxVersion = FEATURE_LATEST;
-        SetMinVersion(FEATURE_LATEST); // permanently upgrade the wallet immediately
+        version = FEATURE_LATEST;
     } else {
-        WalletLogPrintf("Allowing wallet upgrade up to %i\n", nMaxVersion);
+        WalletLogPrintf("Allowing wallet upgrade up to %i\n", version);
     }
-    if (nMaxVersion < GetVersion())
+    if (version < prev_version)
     {
         error = _("Cannot downgrade wallet");
         return false;
     }
-    SetMaxVersion(nMaxVersion);
 
     LOCK(cs_wallet);
 
     // Do not upgrade versions to any version between HD_SPLIT and FEATURE_PRE_SPLIT_KEYPOOL unless already supporting HD_SPLIT
-    int max_version = GetVersion();
-    if (!CanSupportFeature(FEATURE_HD_SPLIT) && max_version >= FEATURE_HD_SPLIT && max_version < FEATURE_PRE_SPLIT_KEYPOOL) {
+    if (!CanSupportFeature(FEATURE_HD_SPLIT) && version >= FEATURE_HD_SPLIT && version < FEATURE_PRE_SPLIT_KEYPOOL) {
         error = _("Cannot upgrade a non HD split wallet without upgrading to support pre split keypool. Please use version 169900 or no version specified.");
         return false;
     }
 
+    // Permanently upgrade to the version
+    SetMinVersion(GetClosestWalletFeature(version));
+
     for (auto spk_man : GetActiveScriptPubKeyMans()) {
-        if (!spk_man->Upgrade(prev_version, error)) {
+        if (!spk_man->Upgrade(prev_version, version, error)) {
             return false;
         }
     }
diff --git a/src/wallet/wallet.h b/src/wallet/wallet.h
index 00e0e3c84d..0934213fc7 100644
--- a/src/wallet/wallet.h
+++ b/src/wallet/wallet.h
@@ -636,9 +636,6 @@ private:
     //! the current wallet version: clients below this version are not able to load the wallet
     int nWalletVersion GUARDED_BY(cs_wallet){FEATURE_BASE};
 
-    //! the maximum wallet format version: memory-only variable that specifies to what version this wallet may be upgraded
-    int nWalletMaxVersion GUARDED_BY(cs_wallet) = FEATURE_BASE;
-
     int64_t nNextResend = 0;
     bool fBroadcastTransactions = false;
     // Local time that the tip block was received. Used to schedule wallet rebroadcasts.
@@ -800,8 +797,8 @@ public:
     const CWalletTx* GetWalletTx(const uint256& hash) const EXCLUSIVE_LOCKS_REQUIRED(cs_wallet);
     bool IsTrusted(const CWalletTx& wtx, std::set<uint256>& trusted_parents) const EXCLUSIVE_LOCKS_REQUIRED(cs_wallet);
 
-    //! check whether we are allowed to upgrade (or already support) to the named feature
-    bool CanSupportFeature(enum WalletFeature wf) const override EXCLUSIVE_LOCKS_REQUIRED(cs_wallet) { AssertLockHeld(cs_wallet); return nWalletMaxVersion >= wf; }
+    //! check whether we support the named feature
+    bool CanSupportFeature(enum WalletFeature wf) const override EXCLUSIVE_LOCKS_REQUIRED(cs_wallet) { AssertLockHeld(cs_wallet); return IsFeatureSupported(nWalletVersion, wf); }
 
     /**
      * populate vCoins with vector of available COutputs.
@@ -853,7 +850,7 @@ public:
     //! Upgrade stored CKeyMetadata objects to store key origin info as KeyOriginInfo
     void UpgradeKeyMetadata() EXCLUSIVE_LOCKS_REQUIRED(cs_wallet);
 
-    bool LoadMinVersion(int nVersion) EXCLUSIVE_LOCKS_REQUIRED(cs_wallet) { AssertLockHeld(cs_wallet); nWalletVersion = nVersion; nWalletMaxVersion = std::max(nWalletMaxVersion, nVersion); return true; }
+    bool LoadMinVersion(int nVersion) EXCLUSIVE_LOCKS_REQUIRED(cs_wallet) { AssertLockHeld(cs_wallet); nWalletVersion = nVersion; return true; }
 
     /**
      * Adds a destination data tuple to the store, and saves it to disk
@@ -1076,11 +1073,8 @@ public:
 
     unsigned int GetKeyPoolSize() const EXCLUSIVE_LOCKS_REQUIRED(cs_wallet);
 
-    //! signify that a particular wallet feature is now used. this may change nWalletVersion and nWalletMaxVersion if those are lower
-    void SetMinVersion(enum WalletFeature, WalletBatch* batch_in = nullptr, bool fExplicit = false) override;
-
-    //! change which version we're allowed to upgrade to (note that this does not immediately imply upgrading to that format)
-    bool SetMaxVersion(int nVersion);
+    //! signify that a particular wallet feature is now used.
+    void SetMinVersion(enum WalletFeature, WalletBatch* batch_in = nullptr) override;
 
     //! get the current wallet format (the oldest client version guaranteed to understand this wallet)
     int GetVersion() const { LOCK(cs_wallet); return nWalletVersion; }
diff --git a/src/wallet/walletutil.cpp b/src/wallet/walletutil.cpp
index 6563c45134..702293e6c7 100644
--- a/src/wallet/walletutil.cpp
+++ b/src/wallet/walletutil.cpp
@@ -79,3 +79,21 @@ std::vector<fs::path> ListWalletDir()
 
     return paths;
 }
+
+bool IsFeatureSupported(int wallet_version, int feature_version)
+{
+    return wallet_version >= feature_version;
+}
+
+WalletFeature GetClosestWalletFeature(int version)
+{
+    if (version >= FEATURE_LATEST) return FEATURE_LATEST;
+    if (version >= FEATURE_PRE_SPLIT_KEYPOOL) return FEATURE_PRE_SPLIT_KEYPOOL;
+    if (version >= FEATURE_NO_DEFAULT_KEY) return FEATURE_NO_DEFAULT_KEY;
+    if (version >= FEATURE_HD_SPLIT) return FEATURE_HD_SPLIT;
+    if (version >= FEATURE_HD) return FEATURE_HD;
+    if (version >= FEATURE_COMPRPUBKEY) return FEATURE_COMPRPUBKEY;
+    if (version >= FEATURE_WALLETCRYPT) return FEATURE_WALLETCRYPT;
+    if (version >= FEATURE_BASE) return FEATURE_BASE;
+    return static_cast<WalletFeature>(0);
+}
diff --git a/src/wallet/walletutil.h b/src/wallet/walletutil.h
index afdcb2e18a..27521abd81 100644
--- a/src/wallet/walletutil.h
+++ b/src/wallet/walletutil.h
@@ -29,7 +29,8 @@ enum WalletFeature
     FEATURE_LATEST = FEATURE_PRE_SPLIT_KEYPOOL
 };
 
-
+bool IsFeatureSupported(int wallet_version, int feature_version);
+WalletFeature GetClosestWalletFeature(int version);
 
 enum WalletFlags : uint64_t {
     // wallet flags in the upper section (> 1 << 31) will lead to not opening the wallet if flag is unknown
diff --git a/test/functional/data/wallets/high_minversion/.walletlock b/test/functional/data/wallets/high_minversion/.walletlock
deleted file mode 100644
index e69de29bb2..0000000000
--- a/test/functional/data/wallets/high_minversion/.walletlock
+++ /dev/null
diff --git a/test/functional/data/wallets/high_minversion/GENERATE.md b/test/functional/data/wallets/high_minversion/GENERATE.md
deleted file mode 100644
index e55c4557ca..0000000000
--- a/test/functional/data/wallets/high_minversion/GENERATE.md
+++ /dev/null
@@ -1,8 +0,0 @@
-The wallet has been created by starting Bitcoin Core with the options
-`-regtest -datadir=/tmp -nowallet -walletdir=$(pwd)/test/functional/data/wallets/`.
-
-In the source code, `WalletFeature::FEATURE_LATEST` has been modified to be large, so that the minversion is too high
-for a current build of the wallet.
-
-The wallet has then been created with the RPC `createwallet high_minversion true true`, so that a blank wallet with
-private keys disabled is created.
diff --git a/test/functional/data/wallets/high_minversion/db.log b/test/functional/data/wallets/high_minversion/db.log
deleted file mode 100644
index e69de29bb2..0000000000
--- a/test/functional/data/wallets/high_minversion/db.log
+++ /dev/null
diff --git a/test/functional/data/wallets/high_minversion/wallet.dat b/test/functional/data/wallets/high_minversion/wallet.dat
deleted file mode 100644
index 99ab809263..0000000000
--- a/test/functional/data/wallets/high_minversion/wallet.dat
+++ /dev/null
diff --git a/test/functional/test_framework/bdb.py b/test/functional/test_framework/bdb.py
new file mode 100644
index 0000000000..9de358aa0a
--- /dev/null
+++ b/test/functional/test_framework/bdb.py
@@ -0,0 +1,152 @@
+#!/usr/bin/env python3
+# Copyright (c) 2020 The Bitcoin Core developers
+# Distributed under the MIT software license, see the accompanying
+# file COPYING or http://www.opensource.org/licenses/mit-license.php.
+"""
+Utilities for working directly with the wallet's BDB database file
+
+This is specific to the configuration of BDB used in this project:
+    - pagesize: 4096 bytes
+    - Outer database contains single subdatabase named 'main'
+    - btree
+    - btree leaf pages
+
+Each key-value pair is two entries in a btree leaf. The first is the key, the one that follows
+is the value. And so on. Note that the entry data is itself not in the correct order. Instead
+entry offsets are stored in the correct order and those offsets are needed to then retrieve
+the data itself.
+
+Page format can be found in BDB source code dbinc/db_page.h
+This only implements the deserialization of btree metadata pages and normal btree pages. Overflow
+pages are not implemented but may be needed in the future if dealing with wallets with large
+transactions.
+
+`db_dump -da wallet.dat` is useful to see the data in a wallet.dat BDB file
+"""
+
+import binascii
+import struct
+
+# Important constants
+PAGESIZE = 4096
+OUTER_META_PAGE = 0
+INNER_META_PAGE = 2
+
+# Page type values
+BTREE_INTERNAL = 3
+BTREE_LEAF = 5
+BTREE_META = 9
+
+# Some magic numbers for sanity checking
+BTREE_MAGIC = 0x053162
+DB_VERSION = 9
+
+# Deserializes a leaf page into a dict.
+# Btree internal pages have the same header, for those, return None.
+# For the btree leaf pages, deserialize them and put all the data into a dict
+def dump_leaf_page(data):
+    page_info = {}
+    page_header = data[0:26]
+    _, pgno, prev_pgno, next_pgno, entries, hf_offset, level, pg_type = struct.unpack('QIIIHHBB', page_header)
+    page_info['pgno'] = pgno
+    page_info['prev_pgno'] = prev_pgno
+    page_info['next_pgno'] = next_pgno
+    page_info['entries'] = entries
+    page_info['hf_offset'] = hf_offset
+    page_info['level'] = level
+    page_info['pg_type'] = pg_type
+    page_info['entry_offsets'] = struct.unpack('{}H'.format(entries), data[26:26 + entries * 2])
+    page_info['entries'] = []
+
+    if pg_type == BTREE_INTERNAL:
+        # Skip internal pages. These are the internal nodes of the btree and don't contain anything relevant to us
+        return None
+
+    assert pg_type == BTREE_LEAF, 'A non-btree leaf page has been encountered while dumping leaves'
+
+    for i in range(0, entries):
+        offset = page_info['entry_offsets'][i]
+        entry = {'offset': offset}
+        page_data_header = data[offset:offset + 3]
+        e_len, pg_type = struct.unpack('HB', page_data_header)
+        entry['len'] = e_len
+        entry['pg_type'] = pg_type
+        entry['data'] = data[offset + 3:offset + 3 + e_len]
+        page_info['entries'].append(entry)
+
+    return page_info
+
+# Deserializes a btree metadata page into a dict.
+# Does a simple sanity check on the magic value, type, and version
+def dump_meta_page(page):
+    # metadata page
+    # general metadata
+    metadata = {}
+    meta_page = page[0:72]
+    _, pgno, magic, version, pagesize, encrypt_alg, pg_type, metaflags, _, free, last_pgno, nparts, key_count, record_count, flags, uid = struct.unpack('QIIIIBBBBIIIIII20s', meta_page)
+    metadata['pgno'] = pgno
+    metadata['magic'] = magic
+    metadata['version'] = version
+    metadata['pagesize'] = pagesize
+    metadata['encrypt_alg'] = encrypt_alg
+    metadata['pg_type'] = pg_type
+    metadata['metaflags'] = metaflags
+    metadata['free'] = free
+    metadata['last_pgno'] = last_pgno
+    metadata['nparts'] = nparts
+    metadata['key_count'] = key_count
+    metadata['record_count'] = record_count
+    metadata['flags'] = flags
+    metadata['uid'] = binascii.hexlify(uid)
+
+    assert magic == BTREE_MAGIC, 'bdb magic does not match bdb btree magic'
+    assert pg_type == BTREE_META, 'Metadata page is not a btree metadata page'
+    assert version == DB_VERSION, 'Database too new'
+
+    # btree metadata
+    btree_meta_page = page[72:512]
+    _, minkey, re_len, re_pad, root, _, crypto_magic, _, iv, chksum = struct.unpack('IIIII368sI12s16s20s', btree_meta_page)
+    metadata['minkey'] = minkey
+    metadata['re_len'] = re_len
+    metadata['re_pad'] = re_pad
+    metadata['root'] = root
+    metadata['crypto_magic'] = crypto_magic
+    metadata['iv'] = binascii.hexlify(iv)
+    metadata['chksum'] = binascii.hexlify(chksum)
+    return metadata
+
+# Given the dict from dump_leaf_page, get the key-value pairs and put them into a dict
+def extract_kv_pairs(page_data):
+    out = {}
+    last_key = None
+    for i, entry in enumerate(page_data['entries']):
+        # By virtue of these all being pairs, even number entries are keys, and odd are values
+        if i % 2 == 0:
+            out[entry['data']] = b''
+            last_key = entry['data']
+        else:
+            out[last_key] = entry['data']
+    return out
+
+# Extract the key-value pairs of the BDB file given in filename
+def dump_bdb_kv(filename):
+    # Read in the BDB file and start deserializing it
+    pages = []
+    with open(filename, 'rb') as f:
+        data = f.read(PAGESIZE)
+        while len(data) > 0:
+            pages.append(data)
+            data = f.read(PAGESIZE)
+
+    # Sanity check the meta pages
+    dump_meta_page(pages[OUTER_META_PAGE])
+    dump_meta_page(pages[INNER_META_PAGE])
+
+    # Fetch the kv pairs from the leaf pages
+    kv = {}
+    for i in range(3, len(pages)):
+        info = dump_leaf_page(pages[i])
+        if info is not None:
+            info_kv = extract_kv_pairs(info)
+            kv = {**kv, **info_kv}
+    return kv
diff --git a/test/functional/test_framework/util.py b/test/functional/test_framework/util.py
index 3356f1ab10..62ff5c6e33 100644
--- a/test/functional/test_framework/util.py
+++ b/test/functional/test_framework/util.py
@@ -8,6 +8,7 @@ from base64 import b64encode
 from binascii import unhexlify
 from decimal import Decimal, ROUND_DOWN
 from subprocess import CalledProcessError
+import hashlib
 import inspect
 import json
 import logging
@@ -260,6 +261,14 @@ def wait_until_helper(predicate, *, attempts=float('inf'), timeout=float('inf'),
         raise AssertionError("Predicate {} not true after {} seconds".format(predicate_source, timeout))
     raise RuntimeError('Unreachable')
 
+def sha256sum_file(filename):
+    h = hashlib.sha256()
+    with open(filename, 'rb') as f:
+        d = f.read(4096)
+        while len(d) > 0:
+            h.update(d)
+            d = f.read(4096)
+    return h.digest()
 
 # RPC/P2P connection constants and functions
 ############################################
diff --git a/test/functional/wallet_multiwallet.py b/test/functional/wallet_multiwallet.py
index cf55b28afb..df16ec741f 100755
--- a/test/functional/wallet_multiwallet.py
+++ b/test/functional/wallet_multiwallet.py
@@ -171,6 +171,9 @@ class MultiWalletTest(BitcoinTestFramework):
         open(not_a_dir, 'a', encoding="utf8").close()
         self.nodes[0].assert_start_raises_init_error(['-walletdir=' + not_a_dir], 'Error: Specified -walletdir "' + not_a_dir + '" is not a directory')
 
+        self.log.info("Do not allow -upgradewallet with multiwallet")
+        self.nodes[0].assert_start_raises_init_error(['-upgradewallet'], "Error: Error parsing command line arguments: Invalid parameter -upgradewallet")
+
         # if wallets/ doesn't exist, datadir should be the default wallet dir
         wallet_dir2 = data_dir('walletdir')
         os.rename(wallet_dir(), wallet_dir2)
diff --git a/test/functional/wallet_upgradewallet.py b/test/functional/wallet_upgradewallet.py
index 15d9b109c5..8ab4b3f76c 100755
--- a/test/functional/wallet_upgradewallet.py
+++ b/test/functional/wallet_upgradewallet.py
@@ -13,23 +13,47 @@ Only v0.15.2 and v0.16.3 are required by this test. The others are used in featu
 
 import os
 import shutil
+import struct
 
+from io import BytesIO
+
+from test_framework.bdb import dump_bdb_kv
+from test_framework.messages import deser_compact_size, deser_string
 from test_framework.test_framework import BitcoinTestFramework
 from test_framework.util import (
     assert_equal,
     assert_greater_than,
     assert_is_hex_string,
+    assert_raises_rpc_error,
+    sha256sum_file,
 )
 
 
+UPGRADED_KEYMETA_VERSION = 12
+
+def deser_keymeta(f):
+    ver, create_time = struct.unpack('<Iq', f.read(12))
+    kp_str = deser_string(f)
+    seed_id = f.read(20)
+    fpr = f.read(4)
+    path_len = 0
+    path = []
+    has_key_orig = False
+    if ver == UPGRADED_KEYMETA_VERSION:
+        path_len = deser_compact_size(f)
+        for i in range(0, path_len):
+            path.append(struct.unpack('<I', f.read(4))[0])
+        has_key_orig = bool(f.read(1))
+    return ver, create_time, kp_str, seed_id, fpr, path_len, path, has_key_orig
+
 class UpgradeWalletTest(BitcoinTestFramework):
     def set_test_params(self):
         self.setup_clean_chain = True
         self.num_nodes = 3
         self.extra_args = [
-            ["-addresstype=bech32"], # current wallet version
-            ["-usehd=1"],            # v0.16.3 wallet
-            ["-usehd=0"]             # v0.15.2 wallet
+            ["-addresstype=bech32", "-keypool=2"], # current wallet version
+            ["-usehd=1", "-keypool=2"],            # v0.16.3 wallet
+            ["-usehd=0", "-keypool=2"]             # v0.15.2 wallet
         ]
         self.wallet_names = [self.default_wallet_name, None, None]
 
@@ -87,22 +111,53 @@ class UpgradeWalletTest(BitcoinTestFramework):
 
         self.log.info("Test upgradewallet RPC...")
         # Prepare for copying of the older wallet
-        node_master_wallet_dir = os.path.join(node_master.datadir, "regtest/wallets")
+        node_master_wallet_dir = os.path.join(node_master.datadir, "regtest/wallets", self.default_wallet_name)
+        node_master_wallet = os.path.join(node_master_wallet_dir, self.default_wallet_name, self.wallet_data_filename)
         v16_3_wallet       = os.path.join(v16_3_node.datadir, "regtest/wallets/wallet.dat")
         v15_2_wallet       = os.path.join(v15_2_node.datadir, "regtest/wallet.dat")
+        split_hd_wallet    = os.path.join(v15_2_node.datadir, "regtest/splithd")
         self.stop_nodes()
 
-        # Copy the 0.16.3 wallet to the last Bitcoin Core version and open it:
-        shutil.rmtree(node_master_wallet_dir)
-        os.mkdir(node_master_wallet_dir)
-        shutil.copy(
-            v16_3_wallet,
-            node_master_wallet_dir
-        )
-        self.restart_node(0, ['-nowallet'])
-        node_master.loadwallet('')
+        # Make split hd wallet
+        self.start_node(2, ['-usehd=1', '-keypool=2', '-wallet=splithd'])
+        self.stop_node(2)
+
+        def copy_v16():
+            node_master.get_wallet_rpc(self.default_wallet_name).unloadwallet()
+            # Copy the 0.16.3 wallet to the last Bitcoin Core version and open it:
+            shutil.rmtree(node_master_wallet_dir)
+            os.mkdir(node_master_wallet_dir)
+            shutil.copy(
+                v16_3_wallet,
+                node_master_wallet_dir
+            )
+            node_master.loadwallet(self.default_wallet_name)
+
+        def copy_non_hd():
+            node_master.get_wallet_rpc(self.default_wallet_name).unloadwallet()
+            # Copy the 0.15.2 non hd wallet to the last Bitcoin Core version and open it:
+            shutil.rmtree(node_master_wallet_dir)
+            os.mkdir(node_master_wallet_dir)
+            shutil.copy(
+                v15_2_wallet,
+                node_master_wallet_dir
+            )
+            node_master.loadwallet(self.default_wallet_name)
 
-        wallet = node_master.get_wallet_rpc('')
+        def copy_split_hd():
+            node_master.get_wallet_rpc(self.default_wallet_name).unloadwallet()
+            # Copy the 0.15.2 split hd wallet to the last Bitcoin Core version and open it:
+            shutil.rmtree(node_master_wallet_dir)
+            os.mkdir(node_master_wallet_dir)
+            shutil.copy(
+                split_hd_wallet,
+                os.path.join(node_master_wallet_dir, 'wallet.dat')
+            )
+            node_master.loadwallet(self.default_wallet_name)
+
+        self.restart_node(0)
+        copy_v16()
+        wallet = node_master.get_wallet_rpc(self.default_wallet_name)
         old_version = wallet.getwalletinfo()["walletversion"]
 
         # calling upgradewallet without version arguments
@@ -114,18 +169,8 @@ class UpgradeWalletTest(BitcoinTestFramework):
         # wallet should still contain the same balance
         assert_equal(wallet.getbalance(), v16_3_balance)
 
-        self.stop_node(0)
-        # Copy the 0.15.2 wallet to the last Bitcoin Core version and open it:
-        shutil.rmtree(node_master_wallet_dir)
-        os.mkdir(node_master_wallet_dir)
-        shutil.copy(
-            v15_2_wallet,
-            node_master_wallet_dir
-        )
-        self.restart_node(0, ['-nowallet'])
-        node_master.loadwallet('')
-
-        wallet = node_master.get_wallet_rpc('')
+        copy_non_hd()
+        wallet = node_master.get_wallet_rpc(self.default_wallet_name)
         # should have no master key hash before conversion
         assert_equal('hdseedid' in wallet.getwalletinfo(), False)
         # calling upgradewallet with explicit version number
@@ -137,5 +182,165 @@ class UpgradeWalletTest(BitcoinTestFramework):
         # after conversion master key hash should be present
         assert_is_hex_string(wallet.getwalletinfo()['hdseedid'])
 
+        self.log.info('Intermediary versions don\'t effect anything')
+        copy_non_hd()
+        # Wallet starts with 60000
+        assert_equal(60000, wallet.getwalletinfo()['walletversion'])
+        wallet.unloadwallet()
+        before_checksum = sha256sum_file(node_master_wallet)
+        node_master.loadwallet('')
+        # Can "upgrade" to 129999 which should have no effect on the wallet
+        wallet.upgradewallet(129999)
+        assert_equal(60000, wallet.getwalletinfo()['walletversion'])
+        wallet.unloadwallet()
+        assert_equal(before_checksum, sha256sum_file(node_master_wallet))
+        node_master.loadwallet('')
+
+        self.log.info('Wallets cannot be downgraded')
+        copy_non_hd()
+        assert_raises_rpc_error(-4, 'Cannot downgrade wallet', wallet.upgradewallet, 40000)
+        wallet.unloadwallet()
+        assert_equal(before_checksum, sha256sum_file(node_master_wallet))
+        node_master.loadwallet('')
+
+        self.log.info('Can upgrade to HD')
+        # Inspect the old wallet and make sure there is no hdchain
+        orig_kvs = dump_bdb_kv(node_master_wallet)
+        assert b'\x07hdchain' not in orig_kvs
+        # Upgrade to HD, no split
+        wallet.upgradewallet(130000)
+        assert_equal(130000, wallet.getwalletinfo()['walletversion'])
+        # Check that there is now a hd chain and it is version 1, no internal chain counter
+        new_kvs = dump_bdb_kv(node_master_wallet)
+        assert b'\x07hdchain' in new_kvs
+        hd_chain = new_kvs[b'\x07hdchain']
+        assert_equal(28, len(hd_chain))
+        hd_chain_version, external_counter, seed_id = struct.unpack('<iI20s', hd_chain)
+        assert_equal(1, hd_chain_version)
+        seed_id = bytearray(seed_id)
+        seed_id.reverse()
+        old_kvs = new_kvs
+        # First 2 keys should still be non-HD
+        for i in range(0, 2):
+            info = wallet.getaddressinfo(wallet.getnewaddress())
+            assert 'hdkeypath' not in info
+            assert 'hdseedid' not in info
+        # Next key should be HD
+        info = wallet.getaddressinfo(wallet.getnewaddress())
+        assert_equal(seed_id.hex(), info['hdseedid'])
+        assert_equal('m/0\'/0\'/0\'', info['hdkeypath'])
+        prev_seed_id = info['hdseedid']
+        # Change key should be the same keypool
+        info = wallet.getaddressinfo(wallet.getrawchangeaddress())
+        assert_equal(prev_seed_id, info['hdseedid'])
+        assert_equal('m/0\'/0\'/1\'', info['hdkeypath'])
+
+        self.log.info('Cannot upgrade to HD Split, needs Pre Split Keypool')
+        assert_raises_rpc_error(-4, 'Cannot upgrade a non HD split wallet without upgrading to support pre split keypool', wallet.upgradewallet, 139900)
+        assert_equal(130000, wallet.getwalletinfo()['walletversion'])
+        assert_raises_rpc_error(-4, 'Cannot upgrade a non HD split wallet without upgrading to support pre split keypool', wallet.upgradewallet, 159900)
+        assert_equal(130000, wallet.getwalletinfo()['walletversion'])
+        assert_raises_rpc_error(-4, 'Cannot upgrade a non HD split wallet without upgrading to support pre split keypool', wallet.upgradewallet, 169899)
+        assert_equal(130000, wallet.getwalletinfo()['walletversion'])
+
+        self.log.info('Upgrade HD to HD chain split')
+        wallet.upgradewallet(169900)
+        assert_equal(169900, wallet.getwalletinfo()['walletversion'])
+        # Check that the hdchain updated correctly
+        new_kvs = dump_bdb_kv(node_master_wallet)
+        hd_chain = new_kvs[b'\x07hdchain']
+        assert_equal(32, len(hd_chain))
+        hd_chain_version, external_counter, seed_id, internal_counter = struct.unpack('<iI20sI', hd_chain)
+        assert_equal(2, hd_chain_version)
+        assert_equal(0, internal_counter)
+        seed_id = bytearray(seed_id)
+        seed_id.reverse()
+        assert_equal(seed_id.hex(), prev_seed_id)
+        # Next change address is the same keypool
+        info = wallet.getaddressinfo(wallet.getrawchangeaddress())
+        assert_equal(prev_seed_id, info['hdseedid'])
+        assert_equal('m/0\'/0\'/2\'', info['hdkeypath'])
+        # Next change address is the new keypool
+        info = wallet.getaddressinfo(wallet.getrawchangeaddress())
+        assert_equal(prev_seed_id, info['hdseedid'])
+        assert_equal('m/0\'/1\'/0\'', info['hdkeypath'])
+        # External addresses use the same keypool
+        info = wallet.getaddressinfo(wallet.getnewaddress())
+        assert_equal(prev_seed_id, info['hdseedid'])
+        assert_equal('m/0\'/0\'/3\'', info['hdkeypath'])
+
+        self.log.info('Upgrade non-HD to HD chain split')
+        copy_non_hd()
+        wallet.upgradewallet(169900)
+        assert_equal(169900, wallet.getwalletinfo()['walletversion'])
+        # Check that the hdchain updated correctly
+        new_kvs = dump_bdb_kv(node_master_wallet)
+        hd_chain = new_kvs[b'\x07hdchain']
+        assert_equal(32, len(hd_chain))
+        hd_chain_version, external_counter, seed_id, internal_counter = struct.unpack('<iI20sI', hd_chain)
+        assert_equal(2, hd_chain_version)
+        assert_equal(2, internal_counter)
+        # Drain the keypool by fetching one external key and one change key. Should still be the same keypool
+        info = wallet.getaddressinfo(wallet.getnewaddress())
+        assert 'hdseedid' not in info
+        assert 'hdkeypath' not in info
+        info = wallet.getaddressinfo(wallet.getrawchangeaddress())
+        assert 'hdseedid' not in info
+        assert 'hdkeypath' not in info
+        # The next addresses are HD and should be on different HD chains
+        info = wallet.getaddressinfo(wallet.getnewaddress())
+        ext_id = info['hdseedid']
+        assert_equal('m/0\'/0\'/0\'', info['hdkeypath'])
+        info = wallet.getaddressinfo(wallet.getrawchangeaddress())
+        assert_equal(ext_id, info['hdseedid'])
+        assert_equal('m/0\'/1\'/0\'', info['hdkeypath'])
+
+        self.log.info('KeyMetadata should upgrade when loading into master')
+        copy_v16()
+        old_kvs = dump_bdb_kv(v16_3_wallet)
+        new_kvs = dump_bdb_kv(node_master_wallet)
+        for k, old_v in old_kvs.items():
+            if k.startswith(b'\x07keymeta'):
+                new_ver, new_create_time, new_kp_str, new_seed_id, new_fpr, new_path_len, new_path, new_has_key_orig = deser_keymeta(BytesIO(new_kvs[k]))
+                old_ver, old_create_time, old_kp_str, old_seed_id, old_fpr, old_path_len, old_path, old_has_key_orig = deser_keymeta(BytesIO(old_v))
+                assert_equal(10, old_ver)
+                if old_kp_str == b"": # imported things that don't have keymeta (i.e. imported coinbase privkeys) won't be upgraded
+                    assert_equal(new_kvs[k], old_v)
+                    continue
+                assert_equal(12, new_ver)
+                assert_equal(new_create_time, old_create_time)
+                assert_equal(new_kp_str, old_kp_str)
+                assert_equal(new_seed_id, old_seed_id)
+                assert_equal(0, old_path_len)
+                assert_equal(new_path_len, len(new_path))
+                assert_equal([], old_path)
+                assert_equal(False, old_has_key_orig)
+                assert_equal(True, new_has_key_orig)
+
+                # Check that the path is right
+                built_path = []
+                for s in new_kp_str.decode().split('/')[1:]:
+                    h = 0
+                    if s[-1] == '\'':
+                        s = s[:-1]
+                        h = 0x80000000
+                    p = int(s) | h
+                    built_path.append(p)
+                assert_equal(new_path, built_path)
+
+        self.log.info('Upgrading to NO_DEFAULT_KEY should not remove the defaultkey')
+        copy_split_hd()
+        # Check the wallet has a default key initially
+        old_kvs = dump_bdb_kv(node_master_wallet)
+        defaultkey = old_kvs[b'\x0adefaultkey']
+        # Upgrade the wallet. Should still have the same default key
+        wallet.upgradewallet(159900)
+        new_kvs = dump_bdb_kv(node_master_wallet)
+        up_defaultkey = new_kvs[b'\x0adefaultkey']
+        assert_equal(defaultkey, up_defaultkey)
+        # 0.16.3 doesn't have a default key
+        v16_3_kvs = dump_bdb_kv(v16_3_wallet)
+        assert b'\x0adefaultkey' not in v16_3_kvs
+
 if __name__ == '__main__':
     UpgradeWalletTest().main()