diff options
Diffstat (limited to 'test')
142 files changed, 2843 insertions, 717 deletions
diff --git a/test/README.md b/test/README.md index 26fd525064..e1dab92a06 100644 --- a/test/README.md +++ b/test/README.md @@ -88,7 +88,7 @@ By default, up to 4 tests will be run in parallel by test_runner. To specify how many jobs to run, append `--jobs=n` The individual tests and the test_runner harness have many command-line -options. Run `test_runner.py -h` to see them all. +options. Run `test/functional/test_runner.py -h` to see them all. #### Troubleshooting and debugging test failures @@ -101,7 +101,7 @@ killed all its bitcoind nodes), then there may be a port conflict which will cause the test to fail. It is recommended that you run the tests on a system where no other bitcoind processes are running. -On linux, the test_framework will warn if there is another +On linux, the test framework will warn if there is another bitcoind process running when the tests are started. If there are zombie bitcoind processes after test failure, you can kill them @@ -130,7 +130,7 @@ tests will fail. If this happens, remove the cache directory (and make sure bitcoind processes are stopped as above): ```bash -rm -rf cache +rm -rf test/cache killall bitcoind ``` @@ -145,10 +145,19 @@ levels using the logger included in the test_framework, e.g. `test_framework.log` and no logs are output to the console. - when run directly, *all* logs are written to `test_framework.log` and INFO level and above are output to the console. -- when run on Travis, no logs are output to the console. However, if a test +- when run by [our CI (Continuous Integration)](/ci/README.md), no logs are output to the console. However, if a test fails, the `test_framework.log` and bitcoind `debug.log`s will all be dumped to the console to help troubleshooting. +These log files can be located under the test data directory (which is always +printed in the first line of test output): + - `<test data directory>/test_framework.log` + - `<test data directory>/node<node number>/regtest/debug.log`. + +The node number identifies the relevant test node, starting from `node0`, which +corresponds to its position in the nodes list of the specific test, +e.g. `self.nodes[0]`. + To change the level of logs output to the console, use the `-l` command line argument. @@ -157,7 +166,7 @@ aggregate log by running the `combine_logs.py` script. The output can be plain text, colorized text or html. For example: ``` -combine_logs.py -c <test data directory> | less -r +test/functional/combine_logs.py -c <test data directory> | less -r ``` will pipe the colorized logs from the test into less. @@ -245,7 +254,14 @@ Use the `-v` option for verbose output. #### Dependencies -The lint tests require codespell and flake8. To install: `pip3 install codespell flake8`. +| Lint test | Dependency | Version [used by CI](../ci/lint/04_install.sh) | Installation +|-----------|:----------:|:-------------------------------------------:|-------------- +| [`lint-python.sh`](lint/lint-python.sh) | [flake8](https://gitlab.com/pycqa/flake8) | [3.7.8](https://github.com/bitcoin/bitcoin/pull/15257) | `pip3 install flake8==3.7.8` +| [`lint-shell.sh`](lint/lint-shell.sh) | [ShellCheck](https://github.com/koalaman/shellcheck) | [0.6.0](https://github.com/bitcoin/bitcoin/pull/15166) | [details...](https://github.com/koalaman/shellcheck#installing) +| [`lint-shell.sh`](lint/lint-shell.sh) | [yq](https://github.com/kislyuk/yq) | default | `pip3 install yq` +| [`lint-spelling.sh`](lint/lint-spelling.sh) | [codespell](https://github.com/codespell-project/codespell) | [1.15.0](https://github.com/bitcoin/bitcoin/pull/16186) | `pip3 install codespell==1.15.0` + +Please be aware that on Linux distributions all dependencies are usually available as packages, but could be outdated. #### Running the tests diff --git a/test/config.ini.in b/test/config.ini.in index 060c553da2..9687206ee1 100644 --- a/test/config.ini.in +++ b/test/config.ini.in @@ -16,6 +16,7 @@ RPCAUTH=@abs_top_srcdir@/share/rpcauth/rpcauth.py # Which components are enabled. These are commented out by `configure` if they were disabled when running config. @ENABLE_WALLET_TRUE@ENABLE_WALLET=true @BUILD_BITCOIN_CLI_TRUE@ENABLE_CLI=true +@BUILD_BITCOIN_WALLET_TRUE@ENABLE_WALLET_TOOL=true @BUILD_BITCOIND_TRUE@ENABLE_BITCOIND=true @ENABLE_FUZZ_TRUE@ENABLE_FUZZ=true @ENABLE_ZMQ_TRUE@ENABLE_ZMQ=true diff --git a/test/functional/README.md b/test/functional/README.md index 197c2afbe4..6582c1cbcd 100644 --- a/test/functional/README.md +++ b/test/functional/README.md @@ -4,13 +4,13 @@ #### Example test -The [example_test.py](example_test.py) is a heavily commented example of a test case that uses both -the RPC and P2P interfaces. If you are writing your first test, copy that file -and modify to fit your needs. +The file [test/functional/example_test.py](example_test.py) is a heavily commented example +of a test case that uses both the RPC and P2P interfaces. If you are writing your first test, copy +that file and modify to fit your needs. #### Coverage -Running `test_runner.py` with the `--coverage` argument tracks which RPCs are +Running `test/functional/test_runner.py` with the `--coverage` argument tracks which RPCs are called by the tests and prints a report of uncovered RPCs in the summary. This can be used (along with the `--extended` argument) to find out which RPCs we don't have test cases for. @@ -51,10 +51,13 @@ don't have test cases for. #### General test-writing advice +- Instead of inline comments or no test documentation at all, log the comments to the test log, e.g. + `self.log.info('Create enough transactions to fill a block')`. Logs make the test code easier to read and the test + logic easier [to debug](/test/README.md#test-logging). - Set `self.num_nodes` to the minimum number of nodes necessary for the test. Having additional unrequired nodes adds to the execution time of the test as well as memory/CPU/disk requirements (which is important when running tests in - parallel or on Travis). + parallel). - Avoid stop-starting the nodes multiple times during the test if possible. A stop-start takes several seconds, so doing it several times blows up the runtime of the test. @@ -82,7 +85,7 @@ P2P messages. These can be found in the following source files: #### Using the P2P interface -- `messages.py` contains all the definitions for objects that pass +- [messages.py](test_framework/messages.py) contains all the definitions for objects that pass over the network (`CBlock`, `CTransaction`, etc, along with the network-level wrappers for them, `msg_block`, `msg_tx`, etc). @@ -96,32 +99,45 @@ the Bitcoin Core node application logic. For custom behaviour, subclass the P2PInterface object and override the callback methods. - Can be used to write tests where specific P2P protocol behavior is tested. -Examples tests are `p2p_unrequested_blocks.py`, `p2p_compactblocks.py`. +Examples tests are [p2p_unrequested_blocks.py](p2p_unrequested_blocks.py), +[p2p_compactblocks.py](p2p_compactblocks.py). -### test-framework modules +#### Prototyping tests -#### [test_framework/authproxy.py](test_framework/authproxy.py) +The [`TestShell`](test-shell.md) class exposes the BitcoinTestFramework +functionality to interactive Python3 environments and can be used to prototype +tests. This may be especially useful in a REPL environment with session logging +utilities, such as +[IPython](https://ipython.readthedocs.io/en/stable/interactive/reference.html#session-logging-and-restoring). +The logs of such interactive sessions can later be adapted into permanent test +cases. + +### Test framework modules +The following are useful modules for test developers. They are located in +[test/functional/test_framework/](test_framework). + +#### [authproxy.py](test_framework/authproxy.py) Taken from the [python-bitcoinrpc repository](https://github.com/jgarzik/python-bitcoinrpc). -#### [test_framework/test_framework.py](test_framework/test_framework.py) +#### [test_framework.py](test_framework/test_framework.py) Base class for functional tests. -#### [test_framework/util.py](test_framework/util.py) +#### [util.py](test_framework/util.py) Generally useful functions. -#### [test_framework/mininode.py](test_framework/mininode.py) +#### [mininode.py](test_framework/mininode.py) Basic code to support P2P connectivity to a bitcoind. -#### [test_framework/script.py](test_framework/script.py) +#### [script.py](test_framework/script.py) Utilities for manipulating transaction scripts (originally from python-bitcoinlib) -#### [test_framework/key.py](test_framework/key.py) +#### [key.py](test_framework/key.py) Test-only secp256k1 elliptic curve implementation -#### [test_framework/bignum.py](test_framework/bignum.py) +#### [bignum.py](test_framework/bignum.py) Helpers for script.py -#### [test_framework/blocktools.py](test_framework/blocktools.py) +#### [blocktools.py](test_framework/blocktools.py) Helper functions for creating blocks and transactions. ### Benchmarking with perf diff --git a/test/functional/combine_logs.py b/test/functional/combine_logs.py index 367d0f6916..00f2833f55 100755 --- a/test/functional/combine_logs.py +++ b/test/functional/combine_logs.py @@ -1,4 +1,7 @@ #!/usr/bin/env python3 +# Copyright (c) 2017-2019 The Bitcoin Core developers +# Distributed under the MIT software license, see the accompanying +# file COPYING or http://www.opensource.org/licenses/mit-license.php. """Combine logs from multiple bitcoin nodes as well as the test_framework log. This streams the combined log output to stdout. Use combine_logs.py > outputfile @@ -8,7 +11,6 @@ If no argument is provided, the most recent test directory will be used.""" import argparse from collections import defaultdict, namedtuple -import glob import heapq import itertools import os @@ -78,10 +80,11 @@ def read_logs(tmp_dir): for each of the input log files.""" # Find out what the folder is called that holds the debug.log file - chain = glob.glob("{}/node0/*/debug.log".format(tmp_dir)) - if chain: - chain = chain[0] # pick the first one if more than one chain was found (should never happen) - chain = re.search(r'node0/(.+?)/debug\.log$', chain).group(1) # extract the chain name + glob = pathlib.Path(tmp_dir).glob('node0/**/debug.log') + path = next(glob, None) + if path: + assert next(glob, None) is None # more than one debug.log, should never happen + chain = re.search(r'node0/(.+?)/debug\.log$', path.as_posix()).group(1) # extract the chain name else: chain = 'regtest' # fallback to regtest (should only happen when none exists) diff --git a/test/functional/create_cache.py b/test/functional/create_cache.py index edf16fa47e..1108a8e354 100755 --- a/test/functional/create_cache.py +++ b/test/functional/create_cache.py @@ -1,5 +1,5 @@ #!/usr/bin/env python3 -# Copyright (c) 2016-2018 The Bitcoin Core developers +# Copyright (c) 2016-2019 The Bitcoin Core developers # Distributed under the MIT software license, see the accompanying # file COPYING or http://www.opensource.org/licenses/mit-license.php. """Create a blockchain cache. @@ -16,7 +16,6 @@ class CreateCache(BitcoinTestFramework): def set_test_params(self): self.num_nodes = 0 - self.supports_cli = True def setup_network(self): pass diff --git a/test/functional/data/invalid_txs.py b/test/functional/data/invalid_txs.py index fd69bbd2c7..ce14998fd1 100644 --- a/test/functional/data/invalid_txs.py +++ b/test/functional/data/invalid_txs.py @@ -1,5 +1,5 @@ #!/usr/bin/env python3 -# Copyright (c) 2015-2018 The Bitcoin Core developers +# Copyright (c) 2015-2019 The Bitcoin Core developers # Distributed under the MIT software license, see the accompanying # file COPYING or http://www.opensource.org/licenses/mit-license.php. """ @@ -21,7 +21,13 @@ Invalid tx cases not covered here can be found by running: """ import abc -from test_framework.messages import CTransaction, CTxIn, CTxOut, COutPoint +from test_framework.messages import ( + COutPoint, + CTransaction, + CTxIn, + CTxOut, + MAX_MONEY, +) from test_framework import script as sc from test_framework.blocktools import create_tx_with_script, MAX_BLOCK_SIGOPS from test_framework.script import ( @@ -166,7 +172,7 @@ class SpendTooMuch(BadTxTemplate): self.spend_tx, 0, script_pub_key=basic_p2sh, amount=(self.spend_avail + 1)) -class SpendNegative(BadTxTemplate): +class CreateNegative(BadTxTemplate): reject_reason = 'bad-txns-vout-negative' expect_disconnect = True @@ -174,6 +180,25 @@ class SpendNegative(BadTxTemplate): return create_tx_with_script(self.spend_tx, 0, amount=-1) +class CreateTooLarge(BadTxTemplate): + reject_reason = 'bad-txns-vout-toolarge' + expect_disconnect = True + + def get_tx(self): + return create_tx_with_script(self.spend_tx, 0, amount=MAX_MONEY + 1) + + +class CreateSumTooLarge(BadTxTemplate): + reject_reason = 'bad-txns-txouttotal-toolarge' + expect_disconnect = True + + def get_tx(self): + tx = create_tx_with_script(self.spend_tx, 0, amount=MAX_MONEY) + tx.vout = [tx.vout[0]] * 2 + tx.calc_sha256() + return tx + + class InvalidOPIFConstruction(BadTxTemplate): reject_reason = "mandatory-script-verify-flag-failed (Invalid OP_IF construction)" expect_disconnect = True @@ -237,4 +262,3 @@ DisabledOpcodeTemplates = [getDisabledOpcodeTemplate(opcode) for opcode in [ def iter_all_templates(): """Iterate through all bad transaction template types.""" return BadTxTemplate.__subclasses__() - diff --git a/test/functional/data/rpc_psbt.json b/test/functional/data/rpc_psbt.json index 57f2608ee9..0f6cd97fd8 100644 --- a/test/functional/data/rpc_psbt.json +++ b/test/functional/data/rpc_psbt.json @@ -18,7 +18,15 @@ "cHNidP8BAHMCAAAAATAa6YblFqHsisW0vGVz0y+DtGXiOtdhZ9aLOOcwtNvbAAAAAAD/////AnR7AQAAAAAAF6kUA6oXrogrXQ1Usl1jEE5P/s57nqKHYEOZOwAAAAAXqRS5IbG6b3IuS/qDtlV6MTmYakLsg4cAAAAAAAEBHwDKmjsAAAAAFgAU0tlLZK4IWH7vyO6xh8YB6Tn5A3wCAwABAAAAAAEAFgAUYunpgv/zTdgjlhAxawkM0qO3R8sAAQAiACCHa62DLx0WgBXtQSMqnqZaGBXZ7xPA74dZ9ktbKyeKZQEBJVEhA7fOI6AcW0vwCmQlN836uzFbZoMyhnR471EwnSvVf4qHUa4A", "cHNidP8BAHMCAAAAATAa6YblFqHsisW0vGVz0y+DtGXiOtdhZ9aLOOcwtNvbAAAAAAD/////AnR7AQAAAAAAF6kUA6oXrogrXQ1Usl1jEE5P/s57nqKHYEOZOwAAAAAXqRS5IbG6b3IuS/qDtlV6MTmYakLsg4cAAAAAAAEBHwDKmjsAAAAAFgAU0tlLZK4IWH7vyO6xh8YB6Tn5A3wAAgAAFgAUYunpgv/zTdgjlhAxawkM0qO3R8sAAQAiACCHa62DLx0WgBXtQSMqnqZaGBXZ7xPA74dZ9ktbKyeKZQEBJVEhA7fOI6AcW0vwCmQlN836uzFbZoMyhnR471EwnSvVf4qHUa4A", "cHNidP8BAHMCAAAAATAa6YblFqHsisW0vGVz0y+DtGXiOtdhZ9aLOOcwtNvbAAAAAAD/////AnR7AQAAAAAAF6kUA6oXrogrXQ1Usl1jEE5P/s57nqKHYEOZOwAAAAAXqRS5IbG6b3IuS/qDtlV6MTmYakLsg4cAAAAAAAEBHwDKmjsAAAAAFgAU0tlLZK4IWH7vyO6xh8YB6Tn5A3wAAQAWABRi6emC//NN2COWEDFrCQzSo7dHywABACIAIIdrrYMvHRaAFe1BIyqeploYFdnvE8Dvh1n2S1srJ4plIQEAJVEhA7fOI6AcW0vwCmQlN836uzFbZoMyhnR471EwnSvVf4qHUa4A", - "cHNidP8BAHMCAAAAAbiWoY6pOQepFsEGhUPXaulX9rvye2NH+NrdlAHg+WgpAQAAAAD/////AkBLTAAAAAAAF6kUqWwXCcLM5BN2zoNqMNT5qMlIi7+HQEtMAAAAAAAXqRSVF/in2XNxAlN1OSxkyp0z+Wtg2YcAAAAAAAEBIBNssgAAAAAAF6kUamsvautR8hRlMRY6OKNTx03DK96HAQcXFgAUo8u1LWpHprjt/uENAwBpGZD0UH0BCGsCRzBEAiAONfH3DYiw67ZbylrsxCF/XXpVwyWBRgofyRbPslzvwgIgIKCsWw5sHSIPh1icNvcVLZLHWj6NA7Dk+4Os2pOnMbQBIQPGStfYHPtyhpV7zIWtn0Q4GXv5gK1zy/tnJ+cBXu4iiwABABYAFMwmJQEz+HDpBEEabxJ5PogPsqZRAAEAFgAUyCrGc3h3FYCmiIspbv2pSTKZ5jU" + "cHNidP8BAHMCAAAAAbiWoY6pOQepFsEGhUPXaulX9rvye2NH+NrdlAHg+WgpAQAAAAD/////AkBLTAAAAAAAF6kUqWwXCcLM5BN2zoNqMNT5qMlIi7+HQEtMAAAAAAAXqRSVF/in2XNxAlN1OSxkyp0z+Wtg2YcAAAAAAAEBIBNssgAAAAAAF6kUamsvautR8hRlMRY6OKNTx03DK96HAQcXFgAUo8u1LWpHprjt/uENAwBpGZD0UH0BCGsCRzBEAiAONfH3DYiw67ZbylrsxCF/XXpVwyWBRgofyRbPslzvwgIgIKCsWw5sHSIPh1icNvcVLZLHWj6NA7Dk+4Os2pOnMbQBIQPGStfYHPtyhpV7zIWtn0Q4GXv5gK1zy/tnJ+cBXu4iiwABABYAFMwmJQEz+HDpBEEabxJ5PogPsqZRAAEAFgAUyCrGc3h3FYCmiIspbv2pSTKZ5jU", + "cHNidP8BACoCAAAAAAFAQg8AAAAAABepFG6Rty1Vk+fUOR4v9E6R6YXDFkHwhwAAAAAAAQEAAQEBagA=", + "cHNidP8BACoCAAAAAAFAQg8AAAAAABepFG6Rty1Vk+fUOR4v9E6R6YXDFkHwhwAAAAAAAQAAAQABagA=", + "cHNidP8BADMBAAAAAREREREREREREREREREREREREfrK3hERERERERERERERfwAAAAD/////AAAAAAAAAQEJ//////////8AAQEJAADK/gAAAAAAAA==", + "cHNidP8BADMBAAAAAREREREREREREREREREREREREfrK3hERERERERERERERfwAAAAD/////AAAAAAAAAQMErd7f7gEDBAEAAAAA", + "cHNidP8BADMBAAAAAREREREREREREREREREREREREfrK3hERERERERERERERfwAAAAD/////AAAAAAAAAQQAAQQBagA=", + "cHNidP8BADMBAAAAAREREREREREREREREREREREREfrK3hERERERERERERERfwAAAAD/////AAAAAAAAAQEJAOH1BQAAAAAAAQUAAQUBUQA=", + "cHNidP8BADMBAAAAAREREREREREREREREREREREREfrK3hERERERERERERERfwAAAAD/////AAAAAAAAAQcAAQcBUQA=", + "cHNidP8BADMBAAAAAREREREREREREREREREREREREfrK3hERERERERERERERfwAAAAD/////AAAAAAAAAQEJAOH1BQAAAAAAAQgBAAEIAwEBUQA=" ], "valid" : [ "cHNidP8BAHUCAAAAASaBcTce3/KF6Tet7qSze3gADAVmy7OtZGQXE8pCFxv2AAAAAAD+////AtPf9QUAAAAAGXapFNDFmQPFusKGh2DpD9UhpGZap2UgiKwA4fUFAAAAABepFDVF5uM7gyxHBQ8k0+65PJwDlIvHh7MuEwAAAQD9pQEBAAAAAAECiaPHHqtNIOA3G7ukzGmPopXJRjr6Ljl/hTPMti+VZ+UBAAAAFxYAFL4Y0VKpsBIDna89p95PUzSe7LmF/////4b4qkOnHf8USIk6UwpyN+9rRgi7st0tAXHmOuxqSJC0AQAAABcWABT+Pp7xp0XpdNkCxDVZQ6vLNL1TU/////8CAMLrCwAAAAAZdqkUhc/xCX/Z4Ai7NK9wnGIZeziXikiIrHL++E4sAAAAF6kUM5cluiHv1irHU6m80GfWx6ajnQWHAkcwRAIgJxK+IuAnDzlPVoMR3HyppolwuAJf3TskAinwf4pfOiQCIAGLONfc0xTnNMkna9b7QPZzMlvEuqFEyADS8vAtsnZcASED0uFWdJQbrUqZY3LLh+GFbTZSYG2YVi/jnF6efkE/IQUCSDBFAiEA0SuFLYXc2WHS9fSrZgZU327tzHlMDDPOXMMJ/7X85Y0CIGczio4OFyXBl/saiK9Z9R5E5CVbIBZ8hoQDHAXR8lkqASECI7cr7vCWXRC+B3jv7NYfysb3mk6haTkzgHNEZPhPKrMAAAAAAAAA", diff --git a/test/functional/feature_abortnode.py b/test/functional/feature_abortnode.py index 62c3eca07d..9b878e8bf8 100755 --- a/test/functional/feature_abortnode.py +++ b/test/functional/feature_abortnode.py @@ -29,7 +29,7 @@ class AbortNodeTest(BitcoinTestFramework): datadir = get_datadir_path(self.options.tmpdir, 0) # Deleting the undo file will result in reorg failure - os.unlink(os.path.join(datadir, 'regtest', 'blocks', 'rev00000.dat')) + os.unlink(os.path.join(datadir, self.chain, 'blocks', 'rev00000.dat')) # Connecting to a node with a more work chain will trigger a reorg # attempt. @@ -40,7 +40,7 @@ class AbortNodeTest(BitcoinTestFramework): # Check that node0 aborted self.log.info("Waiting for crash") - wait_until(lambda: self.nodes[0].is_node_stopped(), timeout=60) + wait_until(lambda: self.nodes[0].is_node_stopped(), timeout=200) self.log.info("Node crashed - now verifying restart fails") self.nodes[0].assert_start_raises_init_error() diff --git a/test/functional/feature_asmap.py b/test/functional/feature_asmap.py new file mode 100755 index 0000000000..2c6553fbe2 --- /dev/null +++ b/test/functional/feature_asmap.py @@ -0,0 +1,106 @@ +#!/usr/bin/env python3 +# Copyright (c) 2020 The Bitcoin Core developers +# Distributed under the MIT software license, see the accompanying +# file COPYING or http://www.opensource.org/licenses/mit-license.php. +"""Test asmap config argument for ASN-based IP bucketing. + +Verify node behaviour and debug log when launching bitcoind in these cases: + +1. `bitcoind` with no -asmap arg, using /16 prefix for IP bucketing + +2. `bitcoind -asmap=<absolute path>`, using the unit test skeleton asmap + +3. `bitcoind -asmap=<relative path>`, using the unit test skeleton asmap + +4. `bitcoind -asmap/-asmap=` with no file specified, using the default asmap + +5. `bitcoind -asmap` with no file specified and a missing default asmap file + +6. `bitcoind -asmap` with an empty (unparsable) default asmap file + +The tests are order-independent. + +""" +import os +import shutil + +from test_framework.test_framework import BitcoinTestFramework + +DEFAULT_ASMAP_FILENAME = 'ip_asn.map' # defined in src/init.cpp +ASMAP = '../../src/test/data/asmap.raw' # path to unit test skeleton asmap +VERSION = 'fec61fa21a9f46f3b17bdcd660d7f4cd90b966aad3aec593c99b35f0aca15853' + +def expected_messages(filename): + return ['Opened asmap file "{}" (59 bytes) from disk'.format(filename), + 'Using asmap version {} for IP bucketing'.format(VERSION)] + +class AsmapTest(BitcoinTestFramework): + def set_test_params(self): + self.setup_clean_chain = False + self.num_nodes = 1 + + def test_without_asmap_arg(self): + self.log.info('Test bitcoind with no -asmap arg passed') + self.stop_node(0) + with self.node.assert_debug_log(['Using /16 prefix for IP bucketing']): + self.start_node(0) + + def test_asmap_with_absolute_path(self): + self.log.info('Test bitcoind -asmap=<absolute path>') + self.stop_node(0) + filename = os.path.join(self.datadir, 'my-map-file.map') + shutil.copyfile(self.asmap_raw, filename) + with self.node.assert_debug_log(expected_messages(filename)): + self.start_node(0, ['-asmap={}'.format(filename)]) + os.remove(filename) + + def test_asmap_with_relative_path(self): + self.log.info('Test bitcoind -asmap=<relative path>') + self.stop_node(0) + name = 'ASN_map' + filename = os.path.join(self.datadir, name) + shutil.copyfile(self.asmap_raw, filename) + with self.node.assert_debug_log(expected_messages(filename)): + self.start_node(0, ['-asmap={}'.format(name)]) + os.remove(filename) + + def test_default_asmap(self): + shutil.copyfile(self.asmap_raw, self.default_asmap) + for arg in ['-asmap', '-asmap=']: + self.log.info('Test bitcoind {} (using default map file)'.format(arg)) + self.stop_node(0) + with self.node.assert_debug_log(expected_messages(self.default_asmap)): + self.start_node(0, [arg]) + os.remove(self.default_asmap) + + def test_default_asmap_with_missing_file(self): + self.log.info('Test bitcoind -asmap with missing default map file') + self.stop_node(0) + msg = "Error: Could not find asmap file \"{}\"".format(self.default_asmap) + self.node.assert_start_raises_init_error(extra_args=['-asmap'], expected_msg=msg) + + def test_empty_asmap(self): + self.log.info('Test bitcoind -asmap with empty map file') + self.stop_node(0) + with open(self.default_asmap, "w", encoding="utf-8") as f: + f.write("") + msg = "Error: Could not parse asmap file \"{}\"".format(self.default_asmap) + self.node.assert_start_raises_init_error(extra_args=['-asmap'], expected_msg=msg) + os.remove(self.default_asmap) + + def run_test(self): + self.node = self.nodes[0] + self.datadir = os.path.join(self.node.datadir, self.chain) + self.default_asmap = os.path.join(self.datadir, DEFAULT_ASMAP_FILENAME) + self.asmap_raw = os.path.join(os.path.dirname(os.path.realpath(__file__)), ASMAP) + + self.test_without_asmap_arg() + self.test_asmap_with_absolute_path() + self.test_asmap_with_relative_path() + self.test_default_asmap() + self.test_default_asmap_with_missing_file() + self.test_empty_asmap() + + +if __name__ == '__main__': + AsmapTest().main() diff --git a/test/functional/feature_backwards_compatibility.py b/test/functional/feature_backwards_compatibility.py new file mode 100755 index 0000000000..0db74432e2 --- /dev/null +++ b/test/functional/feature_backwards_compatibility.py @@ -0,0 +1,347 @@ +#!/usr/bin/env python3 +# Copyright (c) 2018-2019 The Bitcoin Core developers +# Distributed under the MIT software license, see the accompanying +# file COPYING or http://www.opensource.org/licenses/mit-license.php. +"""Backwards compatibility functional test + +Test various backwards compatibility scenarios. Download the previous node binaries: + +contrib/devtools/previous_release.sh -b v0.19.0.1 v0.18.1 v0.17.1 + +Due to RPC changes introduced in various versions the below tests +won't work for older versions without some patches or workarounds. + +Use only the latest patch version of each release, unless a test specifically +needs an older patch version. +""" + +import os +import shutil + +from test_framework.test_framework import BitcoinTestFramework, SkipTest +from test_framework.descriptors import descsum_create + +from test_framework.util import ( + assert_equal, + sync_blocks, + sync_mempools +) + +class BackwardsCompatibilityTest(BitcoinTestFramework): + def set_test_params(self): + self.setup_clean_chain = True + self.num_nodes = 5 + # Add new version after each release: + self.extra_args = [ + ["-addresstype=bech32"], # Pre-release: use to mine blocks + ["-nowallet", "-walletrbf=1", "-addresstype=bech32"], # Pre-release: use to receive coins, swap wallets, etc + ["-nowallet", "-walletrbf=1", "-addresstype=bech32"], # v0.19.0.1 + ["-nowallet", "-walletrbf=1", "-addresstype=bech32"], # v0.18.1 + ["-nowallet", "-walletrbf=1", "-addresstype=bech32"] # v0.17.1 + ] + + def setup_nodes(self): + if os.getenv("TEST_PREVIOUS_RELEASES") == "false": + raise SkipTest("backwards compatibility tests") + + releases_path = os.getenv("PREVIOUS_RELEASES_DIR") or os.getcwd() + "/releases" + if not os.path.isdir(releases_path): + if os.getenv("TEST_PREVIOUS_RELEASES") == "true": + raise AssertionError("TEST_PREVIOUS_RELEASES=1 but releases missing: " + releases_path) + raise SkipTest("This test requires binaries for previous releases") + + self.add_nodes(self.num_nodes, extra_args=self.extra_args, versions=[ + None, + None, + 190000, + 180100, + 170100 + ], binary=[ + self.options.bitcoind, + self.options.bitcoind, + releases_path + "/v0.19.0.1/bin/bitcoind", + releases_path + "/v0.18.1/bin/bitcoind", + releases_path + "/v0.17.1/bin/bitcoind" + ], binary_cli=[ + self.options.bitcoincli, + self.options.bitcoincli, + releases_path + "/v0.19.0.1/bin/bitcoin-cli", + releases_path + "/v0.18.1/bin/bitcoin-cli", + releases_path + "/v0.17.1/bin/bitcoin-cli" + ]) + + self.start_nodes() + + def run_test(self): + self.nodes[0].generatetoaddress(101, self.nodes[0].getnewaddress()) + + sync_blocks(self.nodes) + + # Sanity check the test framework: + res = self.nodes[self.num_nodes - 1].getblockchaininfo() + assert_equal(res['blocks'], 101) + + node_master = self.nodes[self.num_nodes - 4] + node_v19 = self.nodes[self.num_nodes - 3] + node_v18 = self.nodes[self.num_nodes - 2] + node_v17 = self.nodes[self.num_nodes - 1] + + self.log.info("Test wallet backwards compatibility...") + # Create a number of wallets and open them in older versions: + + # w1: regular wallet, created on master: update this test when default + # wallets can no longer be opened by older versions. + node_master.createwallet(wallet_name="w1") + wallet = node_master.get_wallet_rpc("w1") + info = wallet.getwalletinfo() + assert info['private_keys_enabled'] + assert info['keypoolsize'] > 0 + # Create a confirmed transaction, receiving coins + address = wallet.getnewaddress() + self.nodes[0].sendtoaddress(address, 10) + sync_mempools(self.nodes) + self.nodes[0].generate(1) + sync_blocks(self.nodes) + # Create a conflicting transaction using RBF + return_address = self.nodes[0].getnewaddress() + tx1_id = self.nodes[1].sendtoaddress(return_address, 1) + tx2_id = self.nodes[1].bumpfee(tx1_id)["txid"] + # Confirm the transaction + sync_mempools(self.nodes) + self.nodes[0].generate(1) + sync_blocks(self.nodes) + # Create another conflicting transaction using RBF + tx3_id = self.nodes[1].sendtoaddress(return_address, 1) + tx4_id = self.nodes[1].bumpfee(tx3_id)["txid"] + # Abandon transaction, but don't confirm + self.nodes[1].abandontransaction(tx3_id) + + # w1_v19: regular wallet, created with v0.19 + node_v19.createwallet(wallet_name="w1_v19") + wallet = node_v19.get_wallet_rpc("w1_v19") + info = wallet.getwalletinfo() + assert info['private_keys_enabled'] + assert info['keypoolsize'] > 0 + # Use addmultisigaddress (see #18075) + address_18075 = wallet.addmultisigaddress(1, ["0296b538e853519c726a2c91e61ec11600ae1390813a627c66fb8be7947be63c52", "037211a824f55b505228e4c3d5194c1fcfaa15a456abdf37f9b9d97a4040afc073"], "", "legacy")["address"] + assert wallet.getaddressinfo(address_18075)["solvable"] + + # w1_v18: regular wallet, created with v0.18 + node_v18.createwallet(wallet_name="w1_v18") + wallet = node_v18.get_wallet_rpc("w1_v18") + info = wallet.getwalletinfo() + assert info['private_keys_enabled'] + assert info['keypoolsize'] > 0 + + # w2: wallet with private keys disabled, created on master: update this + # test when default wallets private keys disabled can no longer be + # opened by older versions. + node_master.createwallet(wallet_name="w2", disable_private_keys=True) + wallet = node_master.get_wallet_rpc("w2") + info = wallet.getwalletinfo() + assert info['private_keys_enabled'] == False + assert info['keypoolsize'] == 0 + + # w2_v19: wallet with private keys disabled, created with v0.19 + node_v19.createwallet(wallet_name="w2_v19", disable_private_keys=True) + wallet = node_v19.get_wallet_rpc("w2_v19") + info = wallet.getwalletinfo() + assert info['private_keys_enabled'] == False + assert info['keypoolsize'] == 0 + + # w2_v18: wallet with private keys disabled, created with v0.18 + node_v18.createwallet(wallet_name="w2_v18", disable_private_keys=True) + wallet = node_v18.get_wallet_rpc("w2_v18") + info = wallet.getwalletinfo() + assert info['private_keys_enabled'] == False + assert info['keypoolsize'] == 0 + + # w3: blank wallet, created on master: update this + # test when default blank wallets can no longer be opened by older versions. + node_master.createwallet(wallet_name="w3", blank=True) + wallet = node_master.get_wallet_rpc("w3") + info = wallet.getwalletinfo() + assert info['private_keys_enabled'] + assert info['keypoolsize'] == 0 + + # w3_v19: blank wallet, created with v0.19 + node_v19.createwallet(wallet_name="w3_v19", blank=True) + wallet = node_v19.get_wallet_rpc("w3_v19") + info = wallet.getwalletinfo() + assert info['private_keys_enabled'] + assert info['keypoolsize'] == 0 + + # w3_v18: blank wallet, created with v0.18 + node_v18.createwallet(wallet_name="w3_v18", blank=True) + wallet = node_v18.get_wallet_rpc("w3_v18") + info = wallet.getwalletinfo() + assert info['private_keys_enabled'] + assert info['keypoolsize'] == 0 + + # Copy the wallets to older nodes: + node_master_wallets_dir = os.path.join(node_master.datadir, "regtest/wallets") + node_v19_wallets_dir = os.path.join(node_v19.datadir, "regtest/wallets") + node_v18_wallets_dir = os.path.join(node_v18.datadir, "regtest/wallets") + node_v17_wallets_dir = os.path.join(node_v17.datadir, "regtest/wallets") + node_master.unloadwallet("w1") + node_master.unloadwallet("w2") + node_v19.unloadwallet("w1_v19") + node_v19.unloadwallet("w2_v19") + node_v18.unloadwallet("w1_v18") + node_v18.unloadwallet("w2_v18") + + # Copy wallets to v0.17 + for wallet in os.listdir(node_master_wallets_dir): + shutil.copytree( + os.path.join(node_master_wallets_dir, wallet), + os.path.join(node_v17_wallets_dir, wallet) + ) + for wallet in os.listdir(node_v18_wallets_dir): + shutil.copytree( + os.path.join(node_v18_wallets_dir, wallet), + os.path.join(node_v17_wallets_dir, wallet) + ) + + # Copy wallets to v0.18 + for wallet in os.listdir(node_master_wallets_dir): + shutil.copytree( + os.path.join(node_master_wallets_dir, wallet), + os.path.join(node_v18_wallets_dir, wallet) + ) + + # Copy wallets to v0.19 + for wallet in os.listdir(node_master_wallets_dir): + shutil.copytree( + os.path.join(node_master_wallets_dir, wallet), + os.path.join(node_v19_wallets_dir, wallet) + ) + + # Open the wallets in v0.19 + node_v19.loadwallet("w1") + wallet = node_v19.get_wallet_rpc("w1") + info = wallet.getwalletinfo() + assert info['private_keys_enabled'] + assert info['keypoolsize'] > 0 + txs = wallet.listtransactions() + assert_equal(len(txs), 5) + assert_equal(txs[1]["txid"], tx1_id) + assert_equal(txs[2]["walletconflicts"], [tx1_id]) + assert_equal(txs[1]["replaced_by_txid"], tx2_id) + assert not(txs[1]["abandoned"]) + assert_equal(txs[1]["confirmations"], -1) + assert_equal(txs[2]["blockindex"], 1) + assert txs[3]["abandoned"] + assert_equal(txs[4]["walletconflicts"], [tx3_id]) + assert_equal(txs[3]["replaced_by_txid"], tx4_id) + assert not(hasattr(txs[3], "blockindex")) + + node_v19.loadwallet("w2") + wallet = node_v19.get_wallet_rpc("w2") + info = wallet.getwalletinfo() + assert info['private_keys_enabled'] == False + assert info['keypoolsize'] == 0 + + node_v19.loadwallet("w3") + wallet = node_v19.get_wallet_rpc("w3") + info = wallet.getwalletinfo() + assert info['private_keys_enabled'] + assert info['keypoolsize'] == 0 + + # Open the wallets in v0.18 + node_v18.loadwallet("w1") + wallet = node_v18.get_wallet_rpc("w1") + info = wallet.getwalletinfo() + assert info['private_keys_enabled'] + assert info['keypoolsize'] > 0 + txs = wallet.listtransactions() + assert_equal(len(txs), 5) + assert_equal(txs[1]["txid"], tx1_id) + assert_equal(txs[2]["walletconflicts"], [tx1_id]) + assert_equal(txs[1]["replaced_by_txid"], tx2_id) + assert not(txs[1]["abandoned"]) + assert_equal(txs[1]["confirmations"], -1) + assert_equal(txs[2]["blockindex"], 1) + assert txs[3]["abandoned"] + assert_equal(txs[4]["walletconflicts"], [tx3_id]) + assert_equal(txs[3]["replaced_by_txid"], tx4_id) + assert not(hasattr(txs[3], "blockindex")) + + node_v18.loadwallet("w2") + wallet = node_v18.get_wallet_rpc("w2") + info = wallet.getwalletinfo() + assert info['private_keys_enabled'] == False + assert info['keypoolsize'] == 0 + + node_v18.loadwallet("w3") + wallet = node_v18.get_wallet_rpc("w3") + info = wallet.getwalletinfo() + assert info['private_keys_enabled'] + assert info['keypoolsize'] == 0 + + # Open the wallets in v0.17 + node_v17.loadwallet("w1_v18") + wallet = node_v17.get_wallet_rpc("w1_v18") + info = wallet.getwalletinfo() + assert info['private_keys_enabled'] + assert info['keypoolsize'] > 0 + + node_v17.loadwallet("w1") + wallet = node_v17.get_wallet_rpc("w1") + info = wallet.getwalletinfo() + assert info['private_keys_enabled'] + assert info['keypoolsize'] > 0 + + node_v17.loadwallet("w2_v18") + wallet = node_v17.get_wallet_rpc("w2_v18") + info = wallet.getwalletinfo() + assert info['private_keys_enabled'] == False + assert info['keypoolsize'] == 0 + + node_v17.loadwallet("w2") + wallet = node_v17.get_wallet_rpc("w2") + info = wallet.getwalletinfo() + assert info['private_keys_enabled'] == False + assert info['keypoolsize'] == 0 + + # RPC loadwallet failure causes bitcoind to exit, in addition to the RPC + # call failure, so the following test won't work: + # assert_raises_rpc_error(-4, "Wallet loading failed.", node_v17.loadwallet, 'w3_v18') + + # Instead, we stop node and try to launch it with the wallet: + self.stop_node(self.num_nodes - 1) + node_v17.assert_start_raises_init_error(["-wallet=w3_v18"], "Error: Error loading w3_v18: Wallet requires newer version of Bitcoin Core") + node_v17.assert_start_raises_init_error(["-wallet=w3"], "Error: Error loading w3: Wallet requires newer version of Bitcoin Core") + self.start_node(self.num_nodes - 1) + + self.log.info("Test wallet upgrade path...") + # u1: regular wallet, created with v0.17 + node_v17.createwallet(wallet_name="u1_v17") + wallet = node_v17.get_wallet_rpc("u1_v17") + address = wallet.getnewaddress("bech32") + info = wallet.getaddressinfo(address) + hdkeypath = info["hdkeypath"] + pubkey = info["pubkey"] + + # Copy the 0.17 wallet to the last Bitcoin Core version and open it: + node_v17.unloadwallet("u1_v17") + shutil.copytree( + os.path.join(node_v17_wallets_dir, "u1_v17"), + os.path.join(node_master_wallets_dir, "u1_v17") + ) + node_master.loadwallet("u1_v17") + wallet = node_master.get_wallet_rpc("u1_v17") + info = wallet.getaddressinfo(address) + descriptor = "wpkh([" + info["hdmasterfingerprint"] + hdkeypath[1:] + "]" + pubkey + ")" + assert_equal(info["desc"], descsum_create(descriptor)) + + # Copy the 0.19 wallet to the last Bitcoin Core version and open it: + shutil.copytree( + os.path.join(node_v19_wallets_dir, "w1_v19"), + os.path.join(node_master_wallets_dir, "w1_v19") + ) + node_master.loadwallet("w1_v19") + wallet = node_master.get_wallet_rpc("w1_v19") + assert wallet.getaddressinfo(address_18075)["solvable"] + +if __name__ == '__main__': + BackwardsCompatibilityTest().main() diff --git a/test/functional/feature_bip68_sequence.py b/test/functional/feature_bip68_sequence.py index 677362756c..fe45ed34b5 100755 --- a/test/functional/feature_bip68_sequence.py +++ b/test/functional/feature_bip68_sequence.py @@ -8,7 +8,6 @@ import time from test_framework.blocktools import create_block, create_coinbase, add_witness_commitment from test_framework.messages import COIN, COutPoint, CTransaction, CTxIn, CTxOut, FromHex, ToHex -from test_framework.script import CScript from test_framework.test_framework import BitcoinTestFramework from test_framework.util import ( assert_equal, @@ -17,6 +16,7 @@ from test_framework.util import ( satoshi_round, softfork_active, ) +from test_framework.script_util import DUMMY_P2WPKH_SCRIPT SEQUENCE_LOCKTIME_DISABLE_FLAG = (1<<31) SEQUENCE_LOCKTIME_TYPE_FLAG = (1<<22) # this means use time (0 means height) @@ -24,15 +24,14 @@ SEQUENCE_LOCKTIME_GRANULARITY = 9 # this is a bit-shift SEQUENCE_LOCKTIME_MASK = 0x0000ffff # RPC error for non-BIP68 final transactions -NOT_FINAL_ERROR = "non-BIP68-final (code 64)" +NOT_FINAL_ERROR = "non-BIP68-final" class BIP68Test(BitcoinTestFramework): def set_test_params(self): self.num_nodes = 2 - # TODO remove output type argument and fix resulting "tx-size-small" errors self.extra_args = [ - ["-acceptnonstdtxn=1", "-addresstype=p2sh-segwit"], - ["-acceptnonstdtxn=0", "-addresstype=p2sh-segwit"], + ["-acceptnonstdtxn=1"], + ["-acceptnonstdtxn=0"], ] def skip_test_if_missing_module(self): @@ -85,7 +84,7 @@ class BIP68Test(BitcoinTestFramework): # input to mature. sequence_value = SEQUENCE_LOCKTIME_DISABLE_FLAG | 1 tx1.vin = [CTxIn(COutPoint(int(utxo["txid"], 16), utxo["vout"]), nSequence=sequence_value)] - tx1.vout = [CTxOut(value, CScript([b'a']))] + tx1.vout = [CTxOut(value, DUMMY_P2WPKH_SCRIPT)] tx1_signed = self.nodes[0].signrawtransactionwithwallet(ToHex(tx1))["hex"] tx1_id = self.nodes[0].sendrawtransaction(tx1_signed) @@ -97,7 +96,7 @@ class BIP68Test(BitcoinTestFramework): tx2.nVersion = 2 sequence_value = sequence_value & 0x7fffffff tx2.vin = [CTxIn(COutPoint(tx1_id, 0), nSequence=sequence_value)] - tx2.vout = [CTxOut(int(value - self.relayfee * COIN), CScript([b'a' * 35]))] + tx2.vout = [CTxOut(int(value - self.relayfee * COIN), DUMMY_P2WPKH_SCRIPT)] tx2.rehash() assert_raises_rpc_error(-26, NOT_FINAL_ERROR, self.nodes[0].sendrawtransaction, ToHex(tx2)) @@ -192,7 +191,7 @@ class BIP68Test(BitcoinTestFramework): value += utxos[j]["amount"]*COIN # Overestimate the size of the tx - signatures should be less than 120 bytes, and leave 50 for the output tx_size = len(ToHex(tx))//2 + 120*num_inputs + 50 - tx.vout.append(CTxOut(int(value-self.relayfee*tx_size*COIN/1000), CScript([b'a']))) + tx.vout.append(CTxOut(int(value-self.relayfee*tx_size*COIN/1000), DUMMY_P2WPKH_SCRIPT)) rawtx = self.nodes[0].signrawtransactionwithwallet(ToHex(tx))["hex"] if (using_sequence_locks and not should_pass): @@ -221,7 +220,7 @@ class BIP68Test(BitcoinTestFramework): tx2 = CTransaction() tx2.nVersion = 2 tx2.vin = [CTxIn(COutPoint(tx1.sha256, 0), nSequence=0)] - tx2.vout = [CTxOut(int(tx1.vout[0].nValue - self.relayfee*COIN), CScript([b'a']))] + tx2.vout = [CTxOut(int(tx1.vout[0].nValue - self.relayfee*COIN), DUMMY_P2WPKH_SCRIPT)] tx2_raw = self.nodes[0].signrawtransactionwithwallet(ToHex(tx2))["hex"] tx2 = FromHex(tx2, tx2_raw) tx2.rehash() @@ -239,7 +238,7 @@ class BIP68Test(BitcoinTestFramework): tx = CTransaction() tx.nVersion = 2 tx.vin = [CTxIn(COutPoint(orig_tx.sha256, 0), nSequence=sequence_value)] - tx.vout = [CTxOut(int(orig_tx.vout[0].nValue - relayfee * COIN), CScript([b'a' * 35]))] + tx.vout = [CTxOut(int(orig_tx.vout[0].nValue - relayfee * COIN), DUMMY_P2WPKH_SCRIPT)] tx.rehash() if (orig_tx.hash in node.getrawmempool()): @@ -352,7 +351,7 @@ class BIP68Test(BitcoinTestFramework): tx2 = CTransaction() tx2.nVersion = 1 tx2.vin = [CTxIn(COutPoint(tx1.sha256, 0), nSequence=0)] - tx2.vout = [CTxOut(int(tx1.vout[0].nValue - self.relayfee*COIN), CScript([b'a']))] + tx2.vout = [CTxOut(int(tx1.vout[0].nValue - self.relayfee*COIN), DUMMY_P2WPKH_SCRIPT)] # sign tx2 tx2_raw = self.nodes[0].signrawtransactionwithwallet(ToHex(tx2))["hex"] @@ -367,7 +366,7 @@ class BIP68Test(BitcoinTestFramework): tx3 = CTransaction() tx3.nVersion = 2 tx3.vin = [CTxIn(COutPoint(tx2.sha256, 0), nSequence=sequence_value)] - tx3.vout = [CTxOut(int(tx2.vout[0].nValue - self.relayfee * COIN), CScript([b'a' * 35]))] + tx3.vout = [CTxOut(int(tx2.vout[0].nValue - self.relayfee * COIN), DUMMY_P2WPKH_SCRIPT)] tx3.rehash() assert_raises_rpc_error(-26, NOT_FINAL_ERROR, self.nodes[0].sendrawtransaction, ToHex(tx3)) diff --git a/test/functional/feature_block.py b/test/functional/feature_block.py index c74270febc..38bf2faf89 100755 --- a/test/functional/feature_block.py +++ b/test/functional/feature_block.py @@ -46,7 +46,7 @@ from test_framework.script import ( OP_RETURN, OP_TRUE, SIGHASH_ALL, - SignatureHash, + LegacySignatureHash, hash160, ) from test_framework.test_framework import BitcoinTestFramework @@ -532,7 +532,7 @@ class FullBlockTest(BitcoinTestFramework): # second input is corresponding P2SH output from b39 tx.vin.append(CTxIn(COutPoint(b39.vtx[i].sha256, 0), b'')) # Note: must pass the redeem_script (not p2sh_script) to the signature hash function - (sighash, err) = SignatureHash(redeem_script, tx, 1, SIGHASH_ALL) + (sighash, err) = LegacySignatureHash(redeem_script, tx, 1, SIGHASH_ALL) sig = self.coinbase_key.sign_ecdsa(sighash) + bytes(bytearray([SIGHASH_ALL])) scriptSig = CScript([sig, redeem_script]) @@ -1261,7 +1261,7 @@ class FullBlockTest(BitcoinTestFramework): self.save_spendable_output() spend = self.get_spendable_output() - self.send_blocks(blocks, True, timeout=960) + self.send_blocks(blocks, True, timeout=1920) chain1_tip = i # now create alt chain of same length @@ -1273,14 +1273,14 @@ class FullBlockTest(BitcoinTestFramework): # extend alt chain to trigger re-org block = self.next_block("alt" + str(chain1_tip + 1), version=4) - self.send_blocks([block], True, timeout=960) + self.send_blocks([block], True, timeout=1920) # ... and re-org back to the first chain self.move_tip(chain1_tip) block = self.next_block(chain1_tip + 1, version=4) self.send_blocks([block], False, force_send=True) block = self.next_block(chain1_tip + 2, version=4) - self.send_blocks([block], True, timeout=960) + self.send_blocks([block], True, timeout=1920) self.log.info("Reject a block with an invalid block header version") b_v1 = self.next_block('b_v1', version=1) @@ -1312,7 +1312,7 @@ class FullBlockTest(BitcoinTestFramework): if (scriptPubKey[0] == OP_TRUE): # an anyone-can-spend tx.vin[0].scriptSig = CScript() return - (sighash, err) = SignatureHash(spend_tx.vout[0].scriptPubKey, tx, 0, SIGHASH_ALL) + (sighash, err) = LegacySignatureHash(spend_tx.vout[0].scriptPubKey, tx, 0, SIGHASH_ALL) tx.vin[0].scriptSig = CScript([self.coinbase_key.sign_ecdsa(sighash) + bytes(bytearray([SIGHASH_ALL]))]) def create_and_sign_transaction(self, spend_tx, value, script=CScript([OP_TRUE])): @@ -1401,7 +1401,7 @@ class FullBlockTest(BitcoinTestFramework): self.nodes[0].disconnect_p2ps() self.bootstrap_p2p(timeout=timeout) - def send_blocks(self, blocks, success=True, reject_reason=None, force_send=False, reconnect=False, timeout=60): + def send_blocks(self, blocks, success=True, reject_reason=None, force_send=False, reconnect=False, timeout=960): """Sends blocks to test node. Syncs and verifies that tip has advanced to most recent block. Call with success = False if the tip shouldn't advance to the most recent block.""" diff --git a/test/functional/feature_blocksdir.py b/test/functional/feature_blocksdir.py index 6f01f97ea2..7bfad52c24 100755 --- a/test/functional/feature_blocksdir.py +++ b/test/functional/feature_blocksdir.py @@ -1,5 +1,5 @@ #!/usr/bin/env python3 -# Copyright (c) 2018 The Bitcoin Core developers +# Copyright (c) 2018-2019 The Bitcoin Core developers # Distributed under the MIT software license, see the accompanying # file COPYING or http://www.opensource.org/licenses/mit-license.php. """Test the blocksdir option. diff --git a/test/functional/feature_cltv.py b/test/functional/feature_cltv.py index 13f1c9216a..073ed8d7c7 100755 --- a/test/functional/feature_cltv.py +++ b/test/functional/feature_cltv.py @@ -55,12 +55,12 @@ class BIP65Test(BitcoinTestFramework): def set_test_params(self): self.num_nodes = 1 self.extra_args = [[ - '-whitelist=127.0.0.1', + '-whitelist=noban@127.0.0.1', '-par=1', # Use only one script thread to get the exact reject reason for testing '-acceptnonstdtxn=1', # cltv_invalidate is nonstandard ]] self.setup_clean_chain = True - self.rpc_timeout = 120 + self.rpc_timeout = 480 def skip_test_if_missing_module(self): self.skip_if_no_wallet() @@ -126,7 +126,7 @@ class BIP65Test(BitcoinTestFramework): # First we show that this tx is valid except for CLTV by getting it # rejected from the mempool for exactly that reason. assert_equal( - [{'txid': spendtx.hash, 'allowed': False, 'reject-reason': '64: non-mandatory-script-verify-flag (Negative locktime)'}], + [{'txid': spendtx.hash, 'allowed': False, 'reject-reason': 'non-mandatory-script-verify-flag (Negative locktime)'}], self.nodes[0].testmempoolaccept(rawtxs=[spendtx.serialize().hex()], maxfeerate=0) ) @@ -135,7 +135,7 @@ class BIP65Test(BitcoinTestFramework): block.hashMerkleRoot = block.calc_merkle_root() block.solve() - with self.nodes[0].assert_debug_log(expected_msgs=['CheckInputs on {} failed with non-mandatory-script-verify-flag (Negative locktime)'.format(block.vtx[-1].hash)]): + with self.nodes[0].assert_debug_log(expected_msgs=['CheckInputScripts on {} failed with non-mandatory-script-verify-flag (Negative locktime)'.format(block.vtx[-1].hash)]): self.nodes[0].p2p.send_and_ping(msg_block(block)) assert_equal(int(self.nodes[0].getbestblockhash(), 16), tip) self.nodes[0].p2p.sync_with_ping() diff --git a/test/functional/feature_config_args.py b/test/functional/feature_config_args.py index b997c76025..1a7c656274 100755 --- a/test/functional/feature_config_args.py +++ b/test/functional/feature_config_args.py @@ -1,5 +1,5 @@ #!/usr/bin/env python3 -# Copyright (c) 2017-2019 The Bitcoin Core developers +# Copyright (c) 2017-2020 The Bitcoin Core developers # Distributed under the MIT software license, see the accompanying # file COPYING or http://www.opensource.org/licenses/mit-license.php. """Test various command line arguments and configuration file parameters.""" @@ -13,6 +13,7 @@ class ConfArgsTest(BitcoinTestFramework): def set_test_params(self): self.setup_clean_chain = True self.num_nodes = 1 + self.supports_cli = False def test_config_file_parser(self): # Assume node is stopped @@ -22,7 +23,7 @@ class ConfArgsTest(BitcoinTestFramework): conf.write('includeconf={}\n'.format(inc_conf_file_path)) self.nodes[0].assert_start_raises_init_error( - expected_msg='Error: Error parsing command line arguments: Invalid parameter -dash_cli', + expected_msg='Error: Error parsing command line arguments: Invalid parameter -dash_cli=1', extra_args=['-dash_cli=1'], ) with open(inc_conf_file_path, 'w', encoding='utf-8') as conf: @@ -38,7 +39,7 @@ class ConfArgsTest(BitcoinTestFramework): if self.is_wallet_compiled(): with open(inc_conf_file_path, 'w', encoding='utf8') as conf: conf.write("wallet=foo\n") - self.nodes[0].assert_start_raises_init_error(expected_msg='Error: Config setting for -wallet only applied on regtest network when in [regtest] section.') + self.nodes[0].assert_start_raises_init_error(expected_msg='Error: Config setting for -wallet only applied on %s network when in [%s] section.' % (self.chain, self.chain)) with open(inc_conf_file_path, 'w', encoding='utf-8') as conf: conf.write('regtest=0\n') # mainnet @@ -78,14 +79,44 @@ class ConfArgsTest(BitcoinTestFramework): conf.write('') # clear def test_log_buffer(self): - with self.nodes[0].assert_debug_log(expected_msgs=['Warning: parsed potentially confusing double-negative -connect=0']): + with self.nodes[0].assert_debug_log(expected_msgs=['Warning: parsed potentially confusing double-negative -connect=0\n']): self.start_node(0, extra_args=['-noconnect=0']) self.stop_node(0) + def test_args_log(self): + self.log.info('Test config args logging') + with self.nodes[0].assert_debug_log( + expected_msgs=[ + 'Command-line arg: addnode="some.node"', + 'Command-line arg: rpcauth=****', + 'Command-line arg: rpcbind=****', + 'Command-line arg: rpcpassword=****', + 'Command-line arg: rpcuser=****', + 'Command-line arg: torpassword=****', + 'Config file arg: %s="1"' % self.chain, + 'Config file arg: [%s] server="1"' % self.chain, + ], + unexpected_msgs=[ + 'alice:f7efda5c189b999524f151318c0c86$d5b51b3beffbc0', + '127.1.1.1', + 'secret-rpcuser', + 'secret-torpassword', + ]): + self.start_node(0, extra_args=[ + '-addnode=some.node', + '-rpcauth=alice:f7efda5c189b999524f151318c0c86$d5b51b3beffbc0', + '-rpcbind=127.1.1.1', + '-rpcpassword=', + '-rpcuser=secret-rpcuser', + '-torpassword=secret-torpassword', + ]) + self.stop_node(0) + def run_test(self): self.stop_node(0) self.test_log_buffer() + self.test_args_log() self.test_config_file_parser() @@ -103,7 +134,7 @@ class ConfArgsTest(BitcoinTestFramework): # Check that using non-existent datadir in conf file fails conf_file = os.path.join(default_data_dir, "bitcoin.conf") - # datadir needs to be set before [regtest] section + # datadir needs to be set before [chain] section conf_file_contents = open(conf_file, encoding='utf8').read() with open(conf_file, 'w', encoding='utf8') as f: f.write("datadir=" + new_data_dir + "\n") @@ -115,17 +146,17 @@ class ConfArgsTest(BitcoinTestFramework): os.mkdir(new_data_dir) self.start_node(0, ['-conf='+conf_file, '-wallet=w1']) self.stop_node(0) - assert os.path.exists(os.path.join(new_data_dir, 'regtest', 'blocks')) + assert os.path.exists(os.path.join(new_data_dir, self.chain, 'blocks')) if self.is_wallet_compiled(): - assert os.path.exists(os.path.join(new_data_dir, 'regtest', 'wallets', 'w1')) + assert os.path.exists(os.path.join(new_data_dir, self.chain, 'wallets', 'w1')) # Ensure command line argument overrides datadir in conf os.mkdir(new_data_dir_2) self.nodes[0].datadir = new_data_dir_2 self.start_node(0, ['-datadir='+new_data_dir_2, '-conf='+conf_file, '-wallet=w2']) - assert os.path.exists(os.path.join(new_data_dir_2, 'regtest', 'blocks')) + assert os.path.exists(os.path.join(new_data_dir_2, self.chain, 'blocks')) if self.is_wallet_compiled(): - assert os.path.exists(os.path.join(new_data_dir_2, 'regtest', 'wallets', 'w2')) + assert os.path.exists(os.path.join(new_data_dir_2, self.chain, 'wallets', 'w2')) if __name__ == '__main__': diff --git a/test/functional/feature_csv_activation.py b/test/functional/feature_csv_activation.py index 6bd321992a..a98480a6dd 100755 --- a/test/functional/feature_csv_activation.py +++ b/test/functional/feature_csv_activation.py @@ -35,6 +35,7 @@ bip112txs_vary_nSequence_9 - 16 txs with nSequence relative_locktimes of 9 evalu bip112txs_vary_OP_CSV - 16 txs with nSequence = 10 evaluated against varying {relative_locktimes of 10} OP_CSV OP_DROP bip112txs_vary_OP_CSV_9 - 16 txs with nSequence = 9 evaluated against varying {relative_locktimes of 10} OP_CSV OP_DROP bip112tx_special - test negative argument to OP_CSV +bip112tx_emptystack - test empty stack (= no argument) OP_CSV """ from decimal import Decimal from itertools import product @@ -56,6 +57,8 @@ from test_framework.util import ( softfork_active, ) +TESTING_TX_COUNT = 83 # Number of testing transactions: 1 BIP113 tx, 16 BIP68 txs, 66 BIP112 txs (see comments above) +COINBASE_BLOCK_COUNT = TESTING_TX_COUNT # Number of coinbase blocks we need to generate as inputs for our txs BASE_RELATIVE_LOCKTIME = 10 CSV_ACTIVATION_HEIGHT = 432 SEQ_DISABLE_FLAG = 1 << 31 @@ -95,6 +98,13 @@ def create_bip112special(node, input, txversion, address): signtx.vin[0].scriptSig = CScript([-1, OP_CHECKSEQUENCEVERIFY, OP_DROP] + list(CScript(signtx.vin[0].scriptSig))) return signtx +def create_bip112emptystack(node, input, txversion, address): + tx = create_transaction(node, input, address, amount=Decimal("49.98")) + tx.nVersion = txversion + signtx = sign_transaction(node, tx) + signtx.vin[0].scriptSig = CScript([OP_CHECKSEQUENCEVERIFY] + list(CScript(signtx.vin[0].scriptSig))) + return signtx + def send_generic_input_tx(node, coinbases, address): return node.sendrawtransaction(ToHex(sign_transaction(node, create_transaction(node, node.getblock(coinbases.pop())['tx'][0], address, amount=Decimal("49.99"))))) @@ -138,7 +148,13 @@ class BIP68_112_113Test(BitcoinTestFramework): def set_test_params(self): self.num_nodes = 1 self.setup_clean_chain = True - self.extra_args = [['-whitelist=127.0.0.1', '-blockversion=4', '-addresstype=legacy']] + self.extra_args = [[ + '-whitelist=noban@127.0.0.1', + '-blockversion=4', + '-addresstype=legacy', + '-par=1', # Use only one script thread to get the exact reject reason for testing + ]] + self.supports_cli = False def skip_test_if_missing_module(self): self.skip_if_no_wallet() @@ -162,11 +178,11 @@ class BIP68_112_113Test(BitcoinTestFramework): block.solve() return block - def send_blocks(self, blocks, success=True): + def send_blocks(self, blocks, success=True, reject_reason=None): """Sends blocks to test node. Syncs and verifies that tip has advanced to most recent block. Call with success = False if the tip shouldn't advance to the most recent block.""" - self.nodes[0].p2p.send_blocks_and_test(blocks, self.nodes[0], success=success) + self.nodes[0].p2p.send_blocks_and_test(blocks, self.nodes[0], success=success, reject_reason=reject_reason) def run_test(self): self.nodes[0].add_p2p_connection(P2PDataStore()) @@ -174,15 +190,16 @@ class BIP68_112_113Test(BitcoinTestFramework): self.log.info("Generate blocks in the past for coinbase outputs.") long_past_time = int(time.time()) - 600 * 1000 # enough to build up to 1000 blocks 10 minutes apart without worrying about getting into the future self.nodes[0].setmocktime(long_past_time - 100) # enough so that the generated blocks will still all be before long_past_time - self.coinbase_blocks = self.nodes[0].generate(1 + 16 + 2 * 32 + 1) # 82 blocks generated for inputs + self.coinbase_blocks = self.nodes[0].generate(COINBASE_BLOCK_COUNT) # blocks generated for inputs self.nodes[0].setmocktime(0) # set time back to present so yielded blocks aren't in the future as we advance last_block_time - self.tipheight = 82 # height of the next block to build + self.tipheight = COINBASE_BLOCK_COUNT # height of the next block to build self.last_block_time = long_past_time self.tip = int(self.nodes[0].getbestblockhash(), 16) self.nodeaddress = self.nodes[0].getnewaddress() # Activation height is hardcoded - test_blocks = self.generate_blocks(345) + # We advance to block height five below BIP112 activation for the following tests + test_blocks = self.generate_blocks(CSV_ACTIVATION_HEIGHT-5 - COINBASE_BLOCK_COUNT) self.send_blocks(test_blocks) assert not softfork_active(self.nodes[0], 'csv') @@ -213,6 +230,8 @@ class BIP68_112_113Test(BitcoinTestFramework): # 1 special input with -1 OP_CSV OP_DROP (actually will be prepended to spending scriptSig) bip112specialinput = send_generic_input_tx(self.nodes[0], self.coinbase_blocks, self.nodeaddress) + # 1 special input with (empty stack) OP_CSV (actually will be prepended to spending scriptSig) + bip112emptystackinput = send_generic_input_tx(self.nodes[0],self.coinbase_blocks, self.nodeaddress) # 1 normal input bip113input = send_generic_input_tx(self.nodes[0], self.coinbase_blocks, self.nodeaddress) @@ -223,7 +242,7 @@ class BIP68_112_113Test(BitcoinTestFramework): self.tip = int(inputblockhash, 16) self.tipheight += 1 self.last_block_time += 600 - assert_equal(len(self.nodes[0].getblock(inputblockhash, True)["tx"]), 82 + 1) + assert_equal(len(self.nodes[0].getblock(inputblockhash, True)["tx"]), TESTING_TX_COUNT + 1) # 2 more version 4 blocks test_blocks = self.generate_blocks(2) @@ -262,6 +281,9 @@ class BIP68_112_113Test(BitcoinTestFramework): # -1 OP_CSV OP_DROP input bip112tx_special_v1 = create_bip112special(self.nodes[0], bip112specialinput, 1, self.nodeaddress) bip112tx_special_v2 = create_bip112special(self.nodes[0], bip112specialinput, 2, self.nodeaddress) + # (empty stack) OP_CSV input + bip112tx_emptystack_v1 = create_bip112emptystack(self.nodes[0], bip112emptystackinput, 1, self.nodeaddress) + bip112tx_emptystack_v2 = create_bip112emptystack(self.nodes[0], bip112emptystackinput, 2, self.nodeaddress) self.log.info("TESTING") @@ -269,11 +291,12 @@ class BIP68_112_113Test(BitcoinTestFramework): self.log.info("Test version 1 txs") success_txs = [] - # add BIP113 tx and -1 CSV tx + # BIP113 tx, -1 CSV tx and empty stack CSV tx should succeed bip113tx_v1.nLockTime = self.last_block_time - 600 * 5 # = MTP of prior block (not <) but < time put on current block bip113signed1 = sign_transaction(self.nodes[0], bip113tx_v1) success_txs.append(bip113signed1) success_txs.append(bip112tx_special_v1) + success_txs.append(bip112tx_emptystack_v1) # add BIP 68 txs success_txs.extend(all_rlt_txs(bip68txs_v1)) # add BIP 112 with seq=10 txs @@ -288,11 +311,12 @@ class BIP68_112_113Test(BitcoinTestFramework): self.log.info("Test version 2 txs") success_txs = [] - # add BIP113 tx and -1 CSV tx + # BIP113 tx, -1 CSV tx and empty stack CSV tx should succeed bip113tx_v2.nLockTime = self.last_block_time - 600 * 5 # = MTP of prior block (not <) but < time put on current block bip113signed2 = sign_transaction(self.nodes[0], bip113tx_v2) success_txs.append(bip113signed2) success_txs.append(bip112tx_special_v2) + success_txs.append(bip112tx_emptystack_v2) # add BIP 68 txs success_txs.extend(all_rlt_txs(bip68txs_v2)) # add BIP 112 with seq=10 txs @@ -319,7 +343,7 @@ class BIP68_112_113Test(BitcoinTestFramework): bip113tx_v2.nLockTime = self.last_block_time - 600 * 5 # = MTP of prior block (not <) but < time put on current block bip113signed2 = sign_transaction(self.nodes[0], bip113tx_v2) for bip113tx in [bip113signed1, bip113signed2]: - self.send_blocks([self.create_test_block([bip113tx])], success=False) + self.send_blocks([self.create_test_block([bip113tx])], success=False, reject_reason='bad-txns-nonfinal') # BIP 113 tests should now pass if the locktime is < MTP bip113tx_v1.nLockTime = self.last_block_time - 600 * 5 - 1 # < MTP of prior block bip113signed1 = sign_transaction(self.nodes[0], bip113tx_v1) @@ -351,11 +375,11 @@ class BIP68_112_113Test(BitcoinTestFramework): # All txs without flag fail as we are at delta height = 8 < 10 and delta time = 8 * 600 < 10 * 512 bip68timetxs = [tx['tx'] for tx in bip68txs_v2 if not tx['sdf'] and tx['stf']] for tx in bip68timetxs: - self.send_blocks([self.create_test_block([tx])], success=False) + self.send_blocks([self.create_test_block([tx])], success=False, reject_reason='bad-txns-nonfinal') bip68heighttxs = [tx['tx'] for tx in bip68txs_v2 if not tx['sdf'] and not tx['stf']] for tx in bip68heighttxs: - self.send_blocks([self.create_test_block([tx])], success=False) + self.send_blocks([self.create_test_block([tx])], success=False, reject_reason='bad-txns-nonfinal') # Advance one block to 438 test_blocks = self.generate_blocks(1) @@ -366,7 +390,7 @@ class BIP68_112_113Test(BitcoinTestFramework): self.send_blocks([self.create_test_block(bip68success_txs)]) self.nodes[0].invalidateblock(self.nodes[0].getbestblockhash()) for tx in bip68heighttxs: - self.send_blocks([self.create_test_block([tx])], success=False) + self.send_blocks([self.create_test_block([tx])], success=False, reject_reason='bad-txns-nonfinal') # Advance one block to 439 test_blocks = self.generate_blocks(1) @@ -380,8 +404,11 @@ class BIP68_112_113Test(BitcoinTestFramework): self.log.info("BIP 112 tests") self.log.info("Test version 1 txs") - # -1 OP_CSV tx should fail - self.send_blocks([self.create_test_block([bip112tx_special_v1])], success=False) + # -1 OP_CSV tx and (empty stack) OP_CSV tx should fail + self.send_blocks([self.create_test_block([bip112tx_special_v1])], success=False, + reject_reason='non-mandatory-script-verify-flag (Negative locktime)') + self.send_blocks([self.create_test_block([bip112tx_emptystack_v1])], success=False, + reject_reason='non-mandatory-script-verify-flag (Operation not valid with the current stack size)') # If SEQUENCE_LOCKTIME_DISABLE_FLAG is set in argument to OP_CSV, version 1 txs should still pass success_txs = [tx['tx'] for tx in bip112txs_vary_OP_CSV_v1 if tx['sdf']] @@ -392,15 +419,19 @@ class BIP68_112_113Test(BitcoinTestFramework): # If SEQUENCE_LOCKTIME_DISABLE_FLAG is unset in argument to OP_CSV, version 1 txs should now fail fail_txs = all_rlt_txs(bip112txs_vary_nSequence_v1) fail_txs += all_rlt_txs(bip112txs_vary_nSequence_9_v1) - fail_txs += [tx['tx'] for tx in bip112txs_vary_OP_CSV_9_v1 if not tx['sdf']] + fail_txs += [tx['tx'] for tx in bip112txs_vary_OP_CSV_v1 if not tx['sdf']] fail_txs += [tx['tx'] for tx in bip112txs_vary_OP_CSV_9_v1 if not tx['sdf']] for tx in fail_txs: - self.send_blocks([self.create_test_block([tx])], success=False) + self.send_blocks([self.create_test_block([tx])], success=False, + reject_reason='non-mandatory-script-verify-flag (Locktime requirement not satisfied)') self.log.info("Test version 2 txs") - # -1 OP_CSV tx should fail - self.send_blocks([self.create_test_block([bip112tx_special_v2])], success=False) + # -1 OP_CSV tx and (empty stack) OP_CSV tx should fail + self.send_blocks([self.create_test_block([bip112tx_special_v2])], success=False, + reject_reason='non-mandatory-script-verify-flag (Negative locktime)') + self.send_blocks([self.create_test_block([bip112tx_emptystack_v2])], success=False, + reject_reason='non-mandatory-script-verify-flag (Operation not valid with the current stack size)') # If SEQUENCE_LOCKTIME_DISABLE_FLAG is set in argument to OP_CSV, version 2 txs should pass (all sequence locks are met) success_txs = [tx['tx'] for tx in bip112txs_vary_OP_CSV_v2 if tx['sdf']] @@ -415,18 +446,21 @@ class BIP68_112_113Test(BitcoinTestFramework): fail_txs = all_rlt_txs(bip112txs_vary_nSequence_9_v2) fail_txs += [tx['tx'] for tx in bip112txs_vary_OP_CSV_9_v2 if not tx['sdf']] for tx in fail_txs: - self.send_blocks([self.create_test_block([tx])], success=False) + self.send_blocks([self.create_test_block([tx])], success=False, + reject_reason='non-mandatory-script-verify-flag (Locktime requirement not satisfied)') # If SEQUENCE_LOCKTIME_DISABLE_FLAG is set in nSequence, tx should fail fail_txs = [tx['tx'] for tx in bip112txs_vary_nSequence_v2 if tx['sdf']] for tx in fail_txs: - self.send_blocks([self.create_test_block([tx])], success=False) + self.send_blocks([self.create_test_block([tx])], success=False, + reject_reason='non-mandatory-script-verify-flag (Locktime requirement not satisfied)') # If sequencelock types mismatch, tx should fail fail_txs = [tx['tx'] for tx in bip112txs_vary_nSequence_v2 if not tx['sdf'] and tx['stf']] fail_txs += [tx['tx'] for tx in bip112txs_vary_OP_CSV_v2 if not tx['sdf'] and tx['stf']] for tx in fail_txs: - self.send_blocks([self.create_test_block([tx])], success=False) + self.send_blocks([self.create_test_block([tx])], success=False, + reject_reason='non-mandatory-script-verify-flag (Locktime requirement not satisfied)') # Remaining txs should pass, just test masking works properly success_txs = [tx['tx'] for tx in bip112txs_vary_nSequence_v2 if not tx['sdf'] and not tx['stf']] @@ -444,7 +478,5 @@ class BIP68_112_113Test(BitcoinTestFramework): self.send_blocks([self.create_test_block(time_txs)]) self.nodes[0].invalidateblock(self.nodes[0].getbestblockhash()) - # TODO: Test empty stack fails - if __name__ == '__main__': BIP68_112_113Test().main() diff --git a/test/functional/feature_dbcrash.py b/test/functional/feature_dbcrash.py index 6bd6bb5b8c..5bbdb8cda1 100755 --- a/test/functional/feature_dbcrash.py +++ b/test/functional/feature_dbcrash.py @@ -51,6 +51,7 @@ class ChainstateWriteCrashTest(BitcoinTestFramework): self.num_nodes = 4 self.setup_clean_chain = False self.rpc_timeout = 480 + self.supports_cli = False # Set -maxmempool=0 to turn off mempool memory sharing with dbcache # Set -rpcservertimeout=900 to reduce socket disconnects in this diff --git a/test/functional/feature_dersig.py b/test/functional/feature_dersig.py index 92f6eea5b2..9628945a80 100755 --- a/test/functional/feature_dersig.py +++ b/test/functional/feature_dersig.py @@ -40,7 +40,10 @@ def unDERify(tx): class BIP66Test(BitcoinTestFramework): def set_test_params(self): self.num_nodes = 1 - self.extra_args = [['-whitelist=127.0.0.1', '-par=1']] # Use only one script thread to get the exact log msg for testing + self.extra_args = [[ + '-whitelist=noban@127.0.0.1', + '-par=1', # Use only one script thread to get the exact log msg for testing + ]] self.setup_clean_chain = True self.rpc_timeout = 120 @@ -110,7 +113,7 @@ class BIP66Test(BitcoinTestFramework): # First we show that this tx is valid except for DERSIG by getting it # rejected from the mempool for exactly that reason. assert_equal( - [{'txid': spendtx.hash, 'allowed': False, 'reject-reason': '64: non-mandatory-script-verify-flag (Non-canonical DER signature)'}], + [{'txid': spendtx.hash, 'allowed': False, 'reject-reason': 'non-mandatory-script-verify-flag (Non-canonical DER signature)'}], self.nodes[0].testmempoolaccept(rawtxs=[spendtx.serialize().hex()], maxfeerate=0) ) @@ -120,7 +123,7 @@ class BIP66Test(BitcoinTestFramework): block.rehash() block.solve() - with self.nodes[0].assert_debug_log(expected_msgs=['CheckInputs on {} failed with non-mandatory-script-verify-flag (Non-canonical DER signature)'.format(block.vtx[-1].hash)]): + with self.nodes[0].assert_debug_log(expected_msgs=['CheckInputScripts on {} failed with non-mandatory-script-verify-flag (Non-canonical DER signature)'.format(block.vtx[-1].hash)]): self.nodes[0].p2p.send_and_ping(msg_block(block)) assert_equal(int(self.nodes[0].getbestblockhash(), 16), tip) self.nodes[0].p2p.sync_with_ping() diff --git a/test/functional/feature_fee_estimation.py b/test/functional/feature_fee_estimation.py index ec402fbdb6..5128485ec0 100755 --- a/test/functional/feature_fee_estimation.py +++ b/test/functional/feature_fee_estimation.py @@ -143,11 +143,11 @@ class EstimateFeeTest(BitcoinTestFramework): def set_test_params(self): self.num_nodes = 3 # mine non-standard txs (e.g. txs with "dust" outputs) - # Force fSendTrickle to true (via whitelist) + # Force fSendTrickle to true (via whitelist.noban) self.extra_args = [ - ["-acceptnonstdtxn", "-whitelist=127.0.0.1"], - ["-acceptnonstdtxn", "-whitelist=127.0.0.1", "-blockmaxweight=68000"], - ["-acceptnonstdtxn", "-whitelist=127.0.0.1", "-blockmaxweight=32000"], + ["-acceptnonstdtxn", "-whitelist=noban@127.0.0.1"], + ["-acceptnonstdtxn", "-whitelist=noban@127.0.0.1", "-blockmaxweight=68000"], + ["-acceptnonstdtxn", "-whitelist=noban@127.0.0.1", "-blockmaxweight=32000"], ] def skip_test_if_missing_module(self): diff --git a/test/functional/feature_filelock.py b/test/functional/feature_filelock.py index ba116e41f5..be1c044aa5 100755 --- a/test/functional/feature_filelock.py +++ b/test/functional/feature_filelock.py @@ -1,5 +1,5 @@ #!/usr/bin/env python3 -# Copyright (c) 2018 The Bitcoin Core developers +# Copyright (c) 2018-2019 The Bitcoin Core developers # Distributed under the MIT software license, see the accompanying # file COPYING or http://www.opensource.org/licenses/mit-license.php. """Check that it's not possible to start a second bitcoind instance using the same datadir or wallet.""" @@ -19,7 +19,7 @@ class FilelockTest(BitcoinTestFramework): self.nodes[0].wait_for_rpc_connection() def run_test(self): - datadir = os.path.join(self.nodes[0].datadir, 'regtest') + datadir = os.path.join(self.nodes[0].datadir, self.chain) self.log.info("Using datadir {}".format(datadir)) self.log.info("Check that we can't start a second bitcoind instance using the same datadir") diff --git a/test/functional/feature_help.py b/test/functional/feature_help.py index ed1d25c0d6..e3e2456183 100755 --- a/test/functional/feature_help.py +++ b/test/functional/feature_help.py @@ -17,7 +17,7 @@ class HelpTest(BitcoinTestFramework): # Don't start the node def get_node_output(self, *, ret_code_expected): - ret_code = self.nodes[0].process.wait(timeout=5) + ret_code = self.nodes[0].process.wait(timeout=60) assert_equal(ret_code, ret_code_expected) self.nodes[0].stdout.seek(0) self.nodes[0].stderr.seek(0) diff --git a/test/functional/feature_includeconf.py b/test/functional/feature_includeconf.py index 2cd6a05d08..6f1a0cd348 100755 --- a/test/functional/feature_includeconf.py +++ b/test/functional/feature_includeconf.py @@ -1,5 +1,5 @@ #!/usr/bin/env python3 -# Copyright (c) 2018 The Bitcoin Core developers +# Copyright (c) 2018-2019 The Bitcoin Core developers # Distributed under the MIT software license, see the accompanying # file COPYING or http://www.opensource.org/licenses/mit-license.php. """Tests the includeconf argument diff --git a/test/functional/feature_loadblock.py b/test/functional/feature_loadblock.py new file mode 100755 index 0000000000..7f1e8f5bae --- /dev/null +++ b/test/functional/feature_loadblock.py @@ -0,0 +1,85 @@ +#!/usr/bin/env python3 +# Copyright (c) 2017-2019 The Bitcoin Core developers +# Distributed under the MIT software license, see the accompanying +# file COPYING or http://www.opensource.org/licenses/mit-license.php. +"""Test loadblock option + +Test the option to start a node with the option loadblock which loads +a serialized blockchain from a file (usually called bootstrap.dat). +To generate that file this test uses the helper scripts available +in contrib/linearize. +""" + +import os +import subprocess +import sys +import tempfile +import urllib + +from test_framework.test_framework import ( + BitcoinTestFramework, +) +from test_framework.util import assert_equal, wait_until + + +class LoadblockTest(BitcoinTestFramework): + def set_test_params(self): + self.setup_clean_chain = True + self.num_nodes = 2 + self.supports_cli = False + + def run_test(self): + self.nodes[1].setnetworkactive(state=False) + self.nodes[0].generate(100) + + # Parsing the url of our node to get settings for config file + data_dir = self.nodes[0].datadir + node_url = urllib.parse.urlparse(self.nodes[0].url) + cfg_file = os.path.join(data_dir, "linearize.cfg") + bootstrap_file = os.path.join(self.options.tmpdir, "bootstrap.dat") + genesis_block = self.nodes[0].getblockhash(0) + blocks_dir = os.path.join(data_dir, self.chain, "blocks") + hash_list = tempfile.NamedTemporaryFile(dir=data_dir, + mode='w', + delete=False, + encoding="utf-8") + + self.log.info("Create linearization config file") + with open(cfg_file, "a", encoding="utf-8") as cfg: + cfg.write("datadir={}\n".format(data_dir)) + cfg.write("rpcuser={}\n".format(node_url.username)) + cfg.write("rpcpassword={}\n".format(node_url.password)) + cfg.write("port={}\n".format(node_url.port)) + cfg.write("host={}\n".format(node_url.hostname)) + cfg.write("output_file={}\n".format(bootstrap_file)) + cfg.write("max_height=100\n") + cfg.write("netmagic=fabfb5da\n") + cfg.write("input={}\n".format(blocks_dir)) + cfg.write("genesis={}\n".format(genesis_block)) + cfg.write("hashlist={}\n".format(hash_list.name)) + + base_dir = self.config["environment"]["SRCDIR"] + linearize_dir = os.path.join(base_dir, "contrib", "linearize") + + self.log.info("Run linearization of block hashes") + linearize_hashes_file = os.path.join(linearize_dir, "linearize-hashes.py") + subprocess.run([sys.executable, linearize_hashes_file, cfg_file], + stdout=hash_list, + check=True) + + self.log.info("Run linearization of block data") + linearize_data_file = os.path.join(linearize_dir, "linearize-data.py") + subprocess.run([sys.executable, linearize_data_file, cfg_file], + check=True) + + self.log.info("Restart second, unsynced node with bootstrap file") + self.stop_node(1) + self.start_node(1, ["-loadblock=" + bootstrap_file]) + wait_until(lambda: self.nodes[1].getblockcount() == 100) + + assert_equal(self.nodes[1].getblockchaininfo()['blocks'], 100) + assert_equal(self.nodes[0].getbestblockhash(), self.nodes[1].getbestblockhash()) + + +if __name__ == '__main__': + LoadblockTest().main() diff --git a/test/functional/feature_logging.py b/test/functional/feature_logging.py index e6ff21ee9c..1bae29b302 100755 --- a/test/functional/feature_logging.py +++ b/test/functional/feature_logging.py @@ -1,5 +1,5 @@ #!/usr/bin/env python3 -# Copyright (c) 2017-2018 The Bitcoin Core developers +# Copyright (c) 2017-2019 The Bitcoin Core developers # Distributed under the MIT software license, see the accompanying # file COPYING or http://www.opensource.org/licenses/mit-license.php. """Test debug logging.""" @@ -16,7 +16,7 @@ class LoggingTest(BitcoinTestFramework): self.setup_clean_chain = True def relative_log_path(self, name): - return os.path.join(self.nodes[0].datadir, "regtest", name) + return os.path.join(self.nodes[0].datadir, self.chain, name) def run_test(self): # test default log file name diff --git a/test/functional/feature_maxuploadtarget.py b/test/functional/feature_maxuploadtarget.py index 180ea0e51d..974388d798 100755 --- a/test/functional/feature_maxuploadtarget.py +++ b/test/functional/feature_maxuploadtarget.py @@ -1,5 +1,5 @@ #!/usr/bin/env python3 -# Copyright (c) 2015-2018 The Bitcoin Core developers +# Copyright (c) 2015-2019 The Bitcoin Core developers # Distributed under the MIT software license, see the accompanying # file COPYING or http://www.opensource.org/licenses/mit-license.php. """Test behavior of -maxuploadtarget. @@ -36,6 +36,7 @@ class MaxUploadTest(BitcoinTestFramework): self.setup_clean_chain = True self.num_nodes = 1 self.extra_args = [["-maxuploadtarget=800", "-acceptnonstdtxn=1"]] + self.supports_cli = False # Cache for utxos, as the listunspent may take a long time later in the test self.utxo_cache = [] @@ -139,10 +140,9 @@ class MaxUploadTest(BitcoinTestFramework): self.nodes[0].disconnect_p2ps() - #stop and start node 0 with 1MB maxuploadtarget, whitelist 127.0.0.1 - self.log.info("Restarting nodes with -whitelist=127.0.0.1") + self.log.info("Restarting node 0 with noban permission and 1MB maxuploadtarget") self.stop_node(0) - self.start_node(0, ["-whitelist=127.0.0.1", "-maxuploadtarget=1"]) + self.start_node(0, ["-whitelist=noban@127.0.0.1", "-maxuploadtarget=1"]) # Reconnect to self.nodes[0] self.nodes[0].add_p2p_connection(TestP2PConn()) diff --git a/test/functional/feature_notifications.py b/test/functional/feature_notifications.py index da00b773ad..b110a559c0 100755 --- a/test/functional/feature_notifications.py +++ b/test/functional/feature_notifications.py @@ -13,6 +13,16 @@ from test_framework.util import ( connect_nodes, ) +# Linux allow all characters other than \x00 +# Windows disallow control characters (0-31) and /\?%:|"<> +FILE_CHAR_START = 32 if os.name == 'nt' else 1 +FILE_CHAR_END = 128 +FILE_CHAR_BLACKLIST = '/\\?%*:|"<>' if os.name == 'nt' else '/' + + +def notify_outputname(walletname, txid): + return txid if os.name == 'nt' else '{}_{}'.format(walletname, txid) + class NotificationsTest(BitcoinTestFramework): def set_test_params(self): @@ -20,6 +30,7 @@ class NotificationsTest(BitcoinTestFramework): self.setup_clean_chain = True def setup_network(self): + self.wallet = ''.join(chr(i) for i in range(FILE_CHAR_START, FILE_CHAR_END) if chr(i) not in FILE_CHAR_BLACKLIST) self.alertnotify_dir = os.path.join(self.options.tmpdir, "alertnotify") self.blocknotify_dir = os.path.join(self.options.tmpdir, "blocknotify") self.walletnotify_dir = os.path.join(self.options.tmpdir, "walletnotify") @@ -33,7 +44,8 @@ class NotificationsTest(BitcoinTestFramework): "-blocknotify=echo > {}".format(os.path.join(self.blocknotify_dir, '%s'))], ["-blockversion=211", "-rescan", - "-walletnotify=echo > {}".format(os.path.join(self.walletnotify_dir, '%s'))]] + "-wallet={}".format(self.wallet), + "-walletnotify=echo > {}".format(os.path.join(self.walletnotify_dir, notify_outputname('%w', '%s')))]] super().setup_network() def run_test(self): @@ -53,7 +65,7 @@ class NotificationsTest(BitcoinTestFramework): wait_until(lambda: len(os.listdir(self.walletnotify_dir)) == block_count, timeout=10) # directory content should equal the generated transaction hashes - txids_rpc = list(map(lambda t: t['txid'], self.nodes[1].listtransactions("*", block_count))) + txids_rpc = list(map(lambda t: notify_outputname(self.wallet, t['txid']), self.nodes[1].listtransactions("*", block_count))) assert_equal(sorted(txids_rpc), sorted(os.listdir(self.walletnotify_dir))) self.stop_node(1) for tx_file in os.listdir(self.walletnotify_dir): @@ -67,7 +79,7 @@ class NotificationsTest(BitcoinTestFramework): wait_until(lambda: len(os.listdir(self.walletnotify_dir)) == block_count, timeout=10) # directory content should equal the generated transaction hashes - txids_rpc = list(map(lambda t: t['txid'], self.nodes[1].listtransactions("*", block_count))) + txids_rpc = list(map(lambda t: notify_outputname(self.wallet, t['txid']), self.nodes[1].listtransactions("*", block_count))) assert_equal(sorted(txids_rpc), sorted(os.listdir(self.walletnotify_dir))) # TODO: add test for `-alertnotify` large fork notifications diff --git a/test/functional/feature_nulldummy.py b/test/functional/feature_nulldummy.py index 250dee1528..9c92ee7f90 100755 --- a/test/functional/feature_nulldummy.py +++ b/test/functional/feature_nulldummy.py @@ -20,7 +20,7 @@ from test_framework.script import CScript from test_framework.test_framework import BitcoinTestFramework from test_framework.util import assert_equal, assert_raises_rpc_error -NULLDUMMY_ERROR = "non-mandatory-script-verify-flag (Dummy CHECKMULTISIG argument must be zero) (code 64)" +NULLDUMMY_ERROR = "non-mandatory-script-verify-flag (Dummy CHECKMULTISIG argument must be zero)" def trueDummy(tx): scriptSig = CScript(tx.vin[0].scriptSig) @@ -41,7 +41,10 @@ class NULLDUMMYTest(BitcoinTestFramework): self.setup_clean_chain = True # This script tests NULLDUMMY activation, which is part of the 'segwit' deployment, so we go through # normal segwit activation here (and don't use the default always-on behaviour). - self.extra_args = [['-whitelist=127.0.0.1', '-segwitheight=432', '-addresstype=legacy']] + self.extra_args = [[ + '-segwitheight=432', + '-addresstype=legacy', + ]] def skip_test_if_missing_module(self): self.skip_if_no_wallet() diff --git a/test/functional/feature_pruning.py b/test/functional/feature_pruning.py index 51523f13e7..21b6b299f6 100755 --- a/test/functional/feature_pruning.py +++ b/test/functional/feature_pruning.py @@ -78,6 +78,7 @@ class PruneTest(BitcoinTestFramework): def set_test_params(self): self.setup_clean_chain = True self.num_nodes = 6 + self.supports_cli = False # Create nodes 0 and 1 to mine. # Create node 2 to test pruning. @@ -100,7 +101,7 @@ class PruneTest(BitcoinTestFramework): def setup_network(self): self.setup_nodes() - self.prunedir = os.path.join(self.nodes[2].datadir, 'regtest', 'blocks', '') + self.prunedir = os.path.join(self.nodes[2].datadir, self.chain, 'blocks', '') connect_nodes(self.nodes[0], 1) connect_nodes(self.nodes[1], 2) @@ -278,7 +279,7 @@ class PruneTest(BitcoinTestFramework): assert_equal(ret, node.getblockchaininfo()['pruneheight']) def has_block(index): - return os.path.isfile(os.path.join(self.nodes[node_number].datadir, "regtest", "blocks", "blk{:05}.dat".format(index))) + return os.path.isfile(os.path.join(self.nodes[node_number].datadir, self.chain, "blocks", "blk{:05}.dat".format(index))) # should not prune because chain tip of node 3 (995) < PruneAfterHeight (1000) assert_raises_rpc_error(-1, "Blockchain is too short for pruning", node.pruneblockchain, height(500)) diff --git a/test/functional/feature_rbf.py b/test/functional/feature_rbf.py index a1d4ce4c73..9e578f0026 100755 --- a/test/functional/feature_rbf.py +++ b/test/functional/feature_rbf.py @@ -10,13 +10,14 @@ from test_framework.messages import COIN, COutPoint, CTransaction, CTxIn, CTxOut from test_framework.script import CScript, OP_DROP from test_framework.test_framework import BitcoinTestFramework from test_framework.util import assert_equal, assert_raises_rpc_error, satoshi_round +from test_framework.script_util import DUMMY_P2WPKH_SCRIPT MAX_REPLACEMENT_LIMIT = 100 def txToHex(tx): return tx.serialize().hex() -def make_utxo(node, amount, confirmed=True, scriptPubKey=CScript([1])): +def make_utxo(node, amount, confirmed=True, scriptPubKey=DUMMY_P2WPKH_SCRIPT): """Create a txout with a given amount and scriptPubKey Mines coins as needed. @@ -65,7 +66,6 @@ def make_utxo(node, amount, confirmed=True, scriptPubKey=CScript([1])): class ReplaceByFeeTest(BitcoinTestFramework): def set_test_params(self): self.num_nodes = 1 - # TODO remove output type argument and fix resulting "tx-size-small" errors self.extra_args = [ [ "-acceptnonstdtxn=1", @@ -74,9 +74,9 @@ class ReplaceByFeeTest(BitcoinTestFramework): "-limitancestorsize=101", "-limitdescendantcount=200", "-limitdescendantsize=101", - "-addresstype=p2sh-segwit", ], ] + self.supports_cli = False def skip_test_if_missing_module(self): self.skip_if_no_wallet() @@ -133,7 +133,7 @@ class ReplaceByFeeTest(BitcoinTestFramework): tx1a = CTransaction() tx1a.vin = [CTxIn(tx0_outpoint, nSequence=0)] - tx1a.vout = [CTxOut(1 * COIN, CScript([b'a' * 35]))] + tx1a.vout = [CTxOut(1 * COIN, DUMMY_P2WPKH_SCRIPT)] tx1a_hex = txToHex(tx1a) tx1a_txid = self.nodes[0].sendrawtransaction(tx1a_hex, 0) @@ -142,7 +142,7 @@ class ReplaceByFeeTest(BitcoinTestFramework): # Should fail because we haven't changed the fee tx1b = CTransaction() tx1b.vin = [CTxIn(tx0_outpoint, nSequence=0)] - tx1b.vout = [CTxOut(1 * COIN, CScript([b'b' * 35]))] + tx1b.vout = [CTxOut(1 * COIN, DUMMY_P2WPKH_SCRIPT + b'a')] tx1b_hex = txToHex(tx1b) # This will raise an exception due to insufficient fee @@ -151,7 +151,7 @@ class ReplaceByFeeTest(BitcoinTestFramework): # Extra 0.1 BTC fee tx1b = CTransaction() tx1b.vin = [CTxIn(tx0_outpoint, nSequence=0)] - tx1b.vout = [CTxOut(int(0.9 * COIN), CScript([b'b' * 35]))] + tx1b.vout = [CTxOut(int(0.9 * COIN), DUMMY_P2WPKH_SCRIPT)] tx1b_hex = txToHex(tx1b) # Works when enabled tx1b_txid = self.nodes[0].sendrawtransaction(tx1b_hex, 0) @@ -186,7 +186,7 @@ class ReplaceByFeeTest(BitcoinTestFramework): # child fees - 40 BTC - so this attempt is rejected. dbl_tx = CTransaction() dbl_tx.vin = [CTxIn(tx0_outpoint, nSequence=0)] - dbl_tx.vout = [CTxOut(initial_nValue - 30 * COIN, CScript([1] * 35))] + dbl_tx.vout = [CTxOut(initial_nValue - 30 * COIN, DUMMY_P2WPKH_SCRIPT)] dbl_tx_hex = txToHex(dbl_tx) # This will raise an exception due to insufficient fee @@ -195,7 +195,7 @@ class ReplaceByFeeTest(BitcoinTestFramework): # Accepted with sufficient fee dbl_tx = CTransaction() dbl_tx.vin = [CTxIn(tx0_outpoint, nSequence=0)] - dbl_tx.vout = [CTxOut(1 * COIN, CScript([1] * 35))] + dbl_tx.vout = [CTxOut(1 * COIN, DUMMY_P2WPKH_SCRIPT)] dbl_tx_hex = txToHex(dbl_tx) self.nodes[0].sendrawtransaction(dbl_tx_hex, 0) @@ -248,7 +248,7 @@ class ReplaceByFeeTest(BitcoinTestFramework): # Attempt double-spend, will fail because too little fee paid dbl_tx = CTransaction() dbl_tx.vin = [CTxIn(tx0_outpoint, nSequence=0)] - dbl_tx.vout = [CTxOut(initial_nValue - fee * n, CScript([1] * 35))] + dbl_tx.vout = [CTxOut(initial_nValue - fee * n, DUMMY_P2WPKH_SCRIPT)] dbl_tx_hex = txToHex(dbl_tx) # This will raise an exception due to insufficient fee assert_raises_rpc_error(-26, "insufficient fee", self.nodes[0].sendrawtransaction, dbl_tx_hex, 0) @@ -256,7 +256,7 @@ class ReplaceByFeeTest(BitcoinTestFramework): # 1 BTC fee is enough dbl_tx = CTransaction() dbl_tx.vin = [CTxIn(tx0_outpoint, nSequence=0)] - dbl_tx.vout = [CTxOut(initial_nValue - fee * n - 1 * COIN, CScript([1] * 35))] + dbl_tx.vout = [CTxOut(initial_nValue - fee * n - 1 * COIN, DUMMY_P2WPKH_SCRIPT)] dbl_tx_hex = txToHex(dbl_tx) self.nodes[0].sendrawtransaction(dbl_tx_hex, 0) @@ -276,7 +276,7 @@ class ReplaceByFeeTest(BitcoinTestFramework): dbl_tx = CTransaction() dbl_tx.vin = [CTxIn(tx0_outpoint, nSequence=0)] - dbl_tx.vout = [CTxOut(initial_nValue - 2 * fee * n, CScript([1] * 35))] + dbl_tx.vout = [CTxOut(initial_nValue - 2 * fee * n, DUMMY_P2WPKH_SCRIPT)] dbl_tx_hex = txToHex(dbl_tx) # This will raise an exception assert_raises_rpc_error(-26, "too many potential replacements", self.nodes[0].sendrawtransaction, dbl_tx_hex, 0) @@ -291,7 +291,7 @@ class ReplaceByFeeTest(BitcoinTestFramework): tx1a = CTransaction() tx1a.vin = [CTxIn(tx0_outpoint, nSequence=0)] - tx1a.vout = [CTxOut(1 * COIN, CScript([b'a' * 35]))] + tx1a.vout = [CTxOut(1 * COIN, DUMMY_P2WPKH_SCRIPT)] tx1a_hex = txToHex(tx1a) self.nodes[0].sendrawtransaction(tx1a_hex, 0) @@ -312,7 +312,7 @@ class ReplaceByFeeTest(BitcoinTestFramework): tx1a = CTransaction() tx1a.vin = [CTxIn(utxo1, nSequence=0)] - tx1a.vout = [CTxOut(int(1.1 * COIN), CScript([b'a' * 35]))] + tx1a.vout = [CTxOut(int(1.1 * COIN), DUMMY_P2WPKH_SCRIPT)] tx1a_hex = txToHex(tx1a) tx1a_txid = self.nodes[0].sendrawtransaction(tx1a_hex, 0) @@ -331,7 +331,7 @@ class ReplaceByFeeTest(BitcoinTestFramework): # Spend tx1a's output to test the indirect case. tx1b = CTransaction() tx1b.vin = [CTxIn(COutPoint(tx1a_txid, 0), nSequence=0)] - tx1b.vout = [CTxOut(1 * COIN, CScript([b'a' * 35]))] + tx1b.vout = [CTxOut(1 * COIN, DUMMY_P2WPKH_SCRIPT)] tx1b_hex = txToHex(tx1b) tx1b_txid = self.nodes[0].sendrawtransaction(tx1b_hex, 0) tx1b_txid = int(tx1b_txid, 16) @@ -352,7 +352,7 @@ class ReplaceByFeeTest(BitcoinTestFramework): tx1 = CTransaction() tx1.vin = [CTxIn(confirmed_utxo)] - tx1.vout = [CTxOut(1 * COIN, CScript([b'a' * 35]))] + tx1.vout = [CTxOut(1 * COIN, DUMMY_P2WPKH_SCRIPT)] tx1_hex = txToHex(tx1) self.nodes[0].sendrawtransaction(tx1_hex, 0) @@ -391,7 +391,7 @@ class ReplaceByFeeTest(BitcoinTestFramework): for i in range(MAX_REPLACEMENT_LIMIT+1): tx_i = CTransaction() tx_i.vin = [CTxIn(COutPoint(txid, i), nSequence=0)] - tx_i.vout = [CTxOut(split_value - fee, CScript([b'a' * 35]))] + tx_i.vout = [CTxOut(split_value - fee, DUMMY_P2WPKH_SCRIPT)] tx_i_hex = txToHex(tx_i) self.nodes[0].sendrawtransaction(tx_i_hex, 0) @@ -424,7 +424,7 @@ class ReplaceByFeeTest(BitcoinTestFramework): # Create a non-opting in transaction tx1a = CTransaction() tx1a.vin = [CTxIn(tx0_outpoint, nSequence=0xffffffff)] - tx1a.vout = [CTxOut(1 * COIN, CScript([b'a' * 35]))] + tx1a.vout = [CTxOut(1 * COIN, DUMMY_P2WPKH_SCRIPT)] tx1a_hex = txToHex(tx1a) tx1a_txid = self.nodes[0].sendrawtransaction(tx1a_hex, 0) @@ -434,7 +434,7 @@ class ReplaceByFeeTest(BitcoinTestFramework): # Shouldn't be able to double-spend tx1b = CTransaction() tx1b.vin = [CTxIn(tx0_outpoint, nSequence=0)] - tx1b.vout = [CTxOut(int(0.9 * COIN), CScript([b'b' * 35]))] + tx1b.vout = [CTxOut(int(0.9 * COIN), DUMMY_P2WPKH_SCRIPT)] tx1b_hex = txToHex(tx1b) # This will raise an exception @@ -445,14 +445,14 @@ class ReplaceByFeeTest(BitcoinTestFramework): # Create a different non-opting in transaction tx2a = CTransaction() tx2a.vin = [CTxIn(tx1_outpoint, nSequence=0xfffffffe)] - tx2a.vout = [CTxOut(1 * COIN, CScript([b'a' * 35]))] + tx2a.vout = [CTxOut(1 * COIN, DUMMY_P2WPKH_SCRIPT)] tx2a_hex = txToHex(tx2a) tx2a_txid = self.nodes[0].sendrawtransaction(tx2a_hex, 0) # Still shouldn't be able to double-spend tx2b = CTransaction() tx2b.vin = [CTxIn(tx1_outpoint, nSequence=0)] - tx2b.vout = [CTxOut(int(0.9 * COIN), CScript([b'b' * 35]))] + tx2b.vout = [CTxOut(int(0.9 * COIN), DUMMY_P2WPKH_SCRIPT)] tx2b_hex = txToHex(tx2b) # This will raise an exception @@ -478,12 +478,12 @@ class ReplaceByFeeTest(BitcoinTestFramework): tx3b = CTransaction() tx3b.vin = [CTxIn(COutPoint(tx1a_txid, 0), nSequence=0)] - tx3b.vout = [CTxOut(int(0.5 * COIN), CScript([b'e' * 35]))] + tx3b.vout = [CTxOut(int(0.5 * COIN), DUMMY_P2WPKH_SCRIPT)] tx3b_hex = txToHex(tx3b) tx3c = CTransaction() tx3c.vin = [CTxIn(COutPoint(tx2a_txid, 0), nSequence=0)] - tx3c.vout = [CTxOut(int(0.5 * COIN), CScript([b'f' * 35]))] + tx3c.vout = [CTxOut(int(0.5 * COIN), DUMMY_P2WPKH_SCRIPT)] tx3c_hex = txToHex(tx3c) self.nodes[0].sendrawtransaction(tx3b_hex, 0) @@ -500,7 +500,7 @@ class ReplaceByFeeTest(BitcoinTestFramework): tx1a = CTransaction() tx1a.vin = [CTxIn(tx0_outpoint, nSequence=0)] - tx1a.vout = [CTxOut(1 * COIN, CScript([b'a' * 35]))] + tx1a.vout = [CTxOut(1 * COIN, DUMMY_P2WPKH_SCRIPT)] tx1a_hex = txToHex(tx1a) tx1a_txid = self.nodes[0].sendrawtransaction(tx1a_hex, 0) @@ -526,14 +526,14 @@ class ReplaceByFeeTest(BitcoinTestFramework): tx2a = CTransaction() tx2a.vin = [CTxIn(tx1_outpoint, nSequence=0)] - tx2a.vout = [CTxOut(1 * COIN, CScript([b'a' * 35]))] + tx2a.vout = [CTxOut(1 * COIN, DUMMY_P2WPKH_SCRIPT)] tx2a_hex = txToHex(tx2a) self.nodes[0].sendrawtransaction(tx2a_hex, 0) # Lower fee, but we'll prioritise it tx2b = CTransaction() tx2b.vin = [CTxIn(tx1_outpoint, nSequence=0)] - tx2b.vout = [CTxOut(int(1.01 * COIN), CScript([b'a' * 35]))] + tx2b.vout = [CTxOut(int(1.01 * COIN), DUMMY_P2WPKH_SCRIPT)] tx2b.rehash() tx2b_hex = txToHex(tx2b) diff --git a/test/functional/feature_segwit.py b/test/functional/feature_segwit.py index c69c7f90e8..909a43c8d9 100755 --- a/test/functional/feature_segwit.py +++ b/test/functional/feature_segwit.py @@ -72,6 +72,7 @@ class SegWitTest(BitcoinTestFramework): "-addresstype=legacy", ], ] + self.rpc_timeout = 120 def skip_test_if_missing_module(self): self.skip_if_no_wallet() @@ -193,10 +194,10 @@ class SegWitTest(BitcoinTestFramework): assert self.nodes[0].getrawtransaction(tx_id, False, blockhash) == tx.serialize_without_witness().hex() self.log.info("Verify witness txs without witness data are invalid after the fork") - self.fail_accept(self.nodes[2], 'non-mandatory-script-verify-flag (Witness program hash mismatch) (code 64)', wit_ids[NODE_2][WIT_V0][2], sign=False) - self.fail_accept(self.nodes[2], 'non-mandatory-script-verify-flag (Witness program was passed an empty witness) (code 64)', wit_ids[NODE_2][WIT_V1][2], sign=False) - self.fail_accept(self.nodes[2], 'non-mandatory-script-verify-flag (Witness program hash mismatch) (code 64)', p2sh_ids[NODE_2][WIT_V0][2], sign=False, redeem_script=witness_script(False, self.pubkey[2])) - self.fail_accept(self.nodes[2], 'non-mandatory-script-verify-flag (Witness program was passed an empty witness) (code 64)', p2sh_ids[NODE_2][WIT_V1][2], sign=False, redeem_script=witness_script(True, self.pubkey[2])) + self.fail_accept(self.nodes[2], 'non-mandatory-script-verify-flag (Witness program hash mismatch)', wit_ids[NODE_2][WIT_V0][2], sign=False) + self.fail_accept(self.nodes[2], 'non-mandatory-script-verify-flag (Witness program was passed an empty witness)', wit_ids[NODE_2][WIT_V1][2], sign=False) + self.fail_accept(self.nodes[2], 'non-mandatory-script-verify-flag (Witness program hash mismatch)', p2sh_ids[NODE_2][WIT_V0][2], sign=False, redeem_script=witness_script(False, self.pubkey[2])) + self.fail_accept(self.nodes[2], 'non-mandatory-script-verify-flag (Witness program was passed an empty witness)', p2sh_ids[NODE_2][WIT_V1][2], sign=False, redeem_script=witness_script(True, self.pubkey[2])) self.log.info("Verify default node can now use witness txs") self.success_mine(self.nodes[0], wit_ids[NODE_0][WIT_V0][0], True) # block 432 diff --git a/test/functional/feature_shutdown.py b/test/functional/feature_shutdown.py index 5084cb1322..d782d3b1d8 100755 --- a/test/functional/feature_shutdown.py +++ b/test/functional/feature_shutdown.py @@ -1,5 +1,5 @@ #!/usr/bin/env python3 -# Copyright (c) 2018 The Bitcoin Core developers +# Copyright (c) 2018-2019 The Bitcoin Core developers # Distributed under the MIT software license, see the accompanying # file COPYING or http://www.opensource.org/licenses/mit-license.php. """Test bitcoind shutdown.""" @@ -17,6 +17,7 @@ class ShutdownTest(BitcoinTestFramework): def set_test_params(self): self.setup_clean_chain = True self.num_nodes = 1 + self.supports_cli = False def run_test(self): node = get_rpc_proxy(self.nodes[0].url, 1, timeout=600, coveragedir=self.nodes[0].coverage_dir) diff --git a/test/functional/feature_uacomment.py b/test/functional/feature_uacomment.py index 85c250173f..4720f6dea3 100755 --- a/test/functional/feature_uacomment.py +++ b/test/functional/feature_uacomment.py @@ -1,5 +1,5 @@ #!/usr/bin/env python3 -# Copyright (c) 2017-2018 The Bitcoin Core developers +# Copyright (c) 2017-2019 The Bitcoin Core developers # Distributed under the MIT software license, see the accompanying # file COPYING or http://www.opensource.org/licenses/mit-license.php. """Test the -uacomment option.""" diff --git a/test/functional/interface_bitcoin_cli.py b/test/functional/interface_bitcoin_cli.py index 0a378c5ef5..f04a58cd19 100755 --- a/test/functional/interface_bitcoin_cli.py +++ b/test/functional/interface_bitcoin_cli.py @@ -12,6 +12,9 @@ class TestBitcoinCli(BitcoinTestFramework): self.setup_clean_chain = True self.num_nodes = 1 + def skip_test_if_missing_module(self): + self.skip_if_no_cli() + def run_test(self): """Main test logic""" @@ -56,7 +59,6 @@ class TestBitcoinCli(BitcoinTestFramework): blockchain_info = self.nodes[0].getblockchaininfo() assert_equal(cli_get_info['version'], network_info['version']) - assert_equal(cli_get_info['protocolversion'], network_info['protocolversion']) assert_equal(cli_get_info['blocks'], blockchain_info['blocks']) assert_equal(cli_get_info['timeoffset'], network_info['timeoffset']) assert_equal(cli_get_info['connections'], network_info['connections']) @@ -64,9 +66,7 @@ class TestBitcoinCli(BitcoinTestFramework): assert_equal(cli_get_info['difficulty'], blockchain_info['difficulty']) assert_equal(cli_get_info['chain'], blockchain_info['chain']) if self.is_wallet_compiled(): - assert_equal(cli_get_info['walletversion'], wallet_info['walletversion']) assert_equal(cli_get_info['balance'], wallet_info['balance']) - assert_equal(cli_get_info['keypoololdest'], wallet_info['keypoololdest']) assert_equal(cli_get_info['keypoolsize'], wallet_info['keypoolsize']) assert_equal(cli_get_info['paytxfee'], wallet_info['paytxfee']) assert_equal(cli_get_info['relayfee'], network_info['relayfee']) diff --git a/test/functional/interface_http.py b/test/functional/interface_http.py index bb868d7115..d007490f80 100755 --- a/test/functional/interface_http.py +++ b/test/functional/interface_http.py @@ -13,6 +13,7 @@ import urllib.parse class HTTPBasicsTest (BitcoinTestFramework): def set_test_params(self): self.num_nodes = 3 + self.supports_cli = False def setup_network(self): self.setup_nodes() diff --git a/test/functional/interface_rest.py b/test/functional/interface_rest.py index a036dfc790..e73ec90819 100755 --- a/test/functional/interface_rest.py +++ b/test/functional/interface_rest.py @@ -44,6 +44,7 @@ class RESTTest (BitcoinTestFramework): self.setup_clean_chain = True self.num_nodes = 2 self.extra_args = [["-rest"], []] + self.supports_cli = False def skip_test_if_missing_module(self): self.skip_if_no_wallet() @@ -151,7 +152,7 @@ class RESTTest (BitcoinTestFramework): bin_response = self.test_rest_request("/getutxos", http_method='POST', req_type=ReqType.BIN, body=bin_request, ret_type=RetType.BYTES) output = BytesIO(bin_response) - chain_height, = unpack("i", output.read(4)) + chain_height, = unpack("<i", output.read(4)) response_hash = output.read(32)[::-1].hex() assert_equal(bb_hash, response_hash) # check if getutxo's chaintip during calculation was fine diff --git a/test/functional/interface_rpc.py b/test/functional/interface_rpc.py index e99fa22646..3b3b5bb0c1 100755 --- a/test/functional/interface_rpc.py +++ b/test/functional/interface_rpc.py @@ -22,6 +22,7 @@ class RPCInterfaceTest(BitcoinTestFramework): def set_test_params(self): self.num_nodes = 1 self.setup_clean_chain = True + self.supports_cli = False def test_getrpcinfo(self): self.log.info("Testing getrpcinfo...") @@ -32,7 +33,7 @@ class RPCInterfaceTest(BitcoinTestFramework): command = info['active_commands'][0] assert_equal(command['method'], 'getrpcinfo') assert_greater_than_or_equal(command['duration'], 0) - assert_equal(info['logpath'], os.path.join(self.nodes[0].datadir, 'regtest', 'debug.log')) + assert_equal(info['logpath'], os.path.join(self.nodes[0].datadir, self.chain, 'debug.log')) def test_batch_request(self): self.log.info("Testing basic JSON-RPC batch request...") diff --git a/test/functional/interface_zmq.py b/test/functional/interface_zmq.py index 5aea10fbce..89c55f31f3 100755 --- a/test/functional/interface_zmq.py +++ b/test/functional/interface_zmq.py @@ -59,6 +59,10 @@ class ZMQTest (BitcoinTestFramework): # Note that the publishing order is not defined in the documentation and # is subject to change. import zmq + + # Invalid zmq arguments don't take down the node, see #17185. + self.restart_node(0, ["-zmqpubrawtx=foo", "-zmqpubhashtx=bar"]) + address = 'tcp://127.0.0.1:28332' socket = self.ctx.socket(zmq.SUB) socket.set(zmq.RCVTIMEO, 60000) diff --git a/test/functional/mempool_accept.py b/test/functional/mempool_accept.py index dee7a04516..9ade22a7eb 100755 --- a/test/functional/mempool_accept.py +++ b/test/functional/mempool_accept.py @@ -8,6 +8,7 @@ from io import BytesIO import math from test_framework.test_framework import BitcoinTestFramework +from test_framework.key import ECKey from test_framework.messages import ( BIP125_SEQUENCE_NUMBER, COIN, @@ -15,11 +16,15 @@ from test_framework.messages import ( CTransaction, CTxOut, MAX_BLOCK_BASE_SIZE, + MAX_MONEY, ) from test_framework.script import ( hash160, CScript, OP_0, + OP_2, + OP_3, + OP_CHECKMULTISIG, OP_EQUAL, OP_HASH160, OP_RETURN, @@ -35,8 +40,9 @@ class MempoolAcceptanceTest(BitcoinTestFramework): def set_test_params(self): self.num_nodes = 1 self.extra_args = [[ - '-txindex', + '-txindex','-permitbaremultisig=0', ]] * self.num_nodes + self.supports_cli = False def skip_test_if_missing_module(self): self.skip_if_no_wallet() @@ -71,7 +77,7 @@ class MempoolAcceptanceTest(BitcoinTestFramework): node.generate(1) self.mempool_size = 0 self.check_mempool_result( - result_expected=[{'txid': txid_in_block, 'allowed': False, 'reject-reason': '18: txn-already-known'}], + result_expected=[{'txid': txid_in_block, 'allowed': False, 'reject-reason': 'txn-already-known'}], rawtxs=[raw_tx_in_block], ) @@ -109,7 +115,7 @@ class MempoolAcceptanceTest(BitcoinTestFramework): node.sendrawtransaction(hexstring=raw_tx_0) self.mempool_size += 1 self.check_mempool_result( - result_expected=[{'txid': txid_0, 'allowed': False, 'reject-reason': '18: txn-already-in-mempool'}], + result_expected=[{'txid': txid_0, 'allowed': False, 'reject-reason': 'txn-already-in-mempool'}], rawtxs=[raw_tx_0], ) @@ -133,7 +139,7 @@ class MempoolAcceptanceTest(BitcoinTestFramework): tx.vout[0].nValue -= int(4 * fee * COIN) # Set more fee # skip re-signing the tx self.check_mempool_result( - result_expected=[{'txid': tx.rehash(), 'allowed': False, 'reject-reason': '18: txn-mempool-conflict'}], + result_expected=[{'txid': tx.rehash(), 'allowed': False, 'reject-reason': 'txn-mempool-conflict'}], rawtxs=[tx.serialize().hex()], maxfeerate=0, ) @@ -192,7 +198,7 @@ class MempoolAcceptanceTest(BitcoinTestFramework): # Skip re-signing the transaction for context independent checks from now on # tx.deserialize(BytesIO(hex_str_to_bytes(node.signrawtransactionwithwallet(tx.serialize().hex())['hex']))) self.check_mempool_result( - result_expected=[{'txid': tx.rehash(), 'allowed': False, 'reject-reason': '16: bad-txns-vout-empty'}], + result_expected=[{'txid': tx.rehash(), 'allowed': False, 'reject-reason': 'bad-txns-vout-empty'}], rawtxs=[tx.serialize().hex()], ) @@ -200,7 +206,7 @@ class MempoolAcceptanceTest(BitcoinTestFramework): tx.deserialize(BytesIO(hex_str_to_bytes(raw_tx_reference))) tx.vin = [tx.vin[0]] * math.ceil(MAX_BLOCK_BASE_SIZE / len(tx.vin[0].serialize())) self.check_mempool_result( - result_expected=[{'txid': tx.rehash(), 'allowed': False, 'reject-reason': '16: bad-txns-oversize'}], + result_expected=[{'txid': tx.rehash(), 'allowed': False, 'reject-reason': 'bad-txns-oversize'}], rawtxs=[tx.serialize().hex()], ) @@ -208,25 +214,25 @@ class MempoolAcceptanceTest(BitcoinTestFramework): tx.deserialize(BytesIO(hex_str_to_bytes(raw_tx_reference))) tx.vout[0].nValue *= -1 self.check_mempool_result( - result_expected=[{'txid': tx.rehash(), 'allowed': False, 'reject-reason': '16: bad-txns-vout-negative'}], + result_expected=[{'txid': tx.rehash(), 'allowed': False, 'reject-reason': 'bad-txns-vout-negative'}], rawtxs=[tx.serialize().hex()], ) # The following two validations prevent overflow of the output amounts (see CVE-2010-5139). self.log.info('A transaction with too large output value') tx.deserialize(BytesIO(hex_str_to_bytes(raw_tx_reference))) - tx.vout[0].nValue = 21000000 * COIN + 1 + tx.vout[0].nValue = MAX_MONEY + 1 self.check_mempool_result( - result_expected=[{'txid': tx.rehash(), 'allowed': False, 'reject-reason': '16: bad-txns-vout-toolarge'}], + result_expected=[{'txid': tx.rehash(), 'allowed': False, 'reject-reason': 'bad-txns-vout-toolarge'}], rawtxs=[tx.serialize().hex()], ) self.log.info('A transaction with too large sum of output values') tx.deserialize(BytesIO(hex_str_to_bytes(raw_tx_reference))) tx.vout = [tx.vout[0]] * 2 - tx.vout[0].nValue = 21000000 * COIN + tx.vout[0].nValue = MAX_MONEY self.check_mempool_result( - result_expected=[{'txid': tx.rehash(), 'allowed': False, 'reject-reason': '16: bad-txns-txouttotal-toolarge'}], + result_expected=[{'txid': tx.rehash(), 'allowed': False, 'reject-reason': 'bad-txns-txouttotal-toolarge'}], rawtxs=[tx.serialize().hex()], ) @@ -234,7 +240,7 @@ class MempoolAcceptanceTest(BitcoinTestFramework): tx.deserialize(BytesIO(hex_str_to_bytes(raw_tx_reference))) tx.vin = [tx.vin[0]] * 2 self.check_mempool_result( - result_expected=[{'txid': tx.rehash(), 'allowed': False, 'reject-reason': '16: bad-txns-inputs-duplicate'}], + result_expected=[{'txid': tx.rehash(), 'allowed': False, 'reject-reason': 'bad-txns-inputs-duplicate'}], rawtxs=[tx.serialize().hex()], ) @@ -243,7 +249,7 @@ class MempoolAcceptanceTest(BitcoinTestFramework): raw_tx_coinbase_spent = node.getrawtransaction(txid=node.decoderawtransaction(hexstring=raw_tx_in_block)['vin'][0]['txid']) tx.deserialize(BytesIO(hex_str_to_bytes(raw_tx_coinbase_spent))) self.check_mempool_result( - result_expected=[{'txid': tx.rehash(), 'allowed': False, 'reject-reason': '16: coinbase'}], + result_expected=[{'txid': tx.rehash(), 'allowed': False, 'reject-reason': 'coinbase'}], rawtxs=[tx.serialize().hex()], ) @@ -251,19 +257,34 @@ class MempoolAcceptanceTest(BitcoinTestFramework): tx.deserialize(BytesIO(hex_str_to_bytes(raw_tx_reference))) tx.nVersion = 3 # A version currently non-standard self.check_mempool_result( - result_expected=[{'txid': tx.rehash(), 'allowed': False, 'reject-reason': '64: version'}], + result_expected=[{'txid': tx.rehash(), 'allowed': False, 'reject-reason': 'version'}], rawtxs=[tx.serialize().hex()], ) tx.deserialize(BytesIO(hex_str_to_bytes(raw_tx_reference))) tx.vout[0].scriptPubKey = CScript([OP_0]) # Some non-standard script self.check_mempool_result( - result_expected=[{'txid': tx.rehash(), 'allowed': False, 'reject-reason': '64: scriptpubkey'}], + result_expected=[{'txid': tx.rehash(), 'allowed': False, 'reject-reason': 'scriptpubkey'}], + rawtxs=[tx.serialize().hex()], + ) + tx.deserialize(BytesIO(hex_str_to_bytes(raw_tx_reference))) + key = ECKey() + key.generate() + pubkey = key.get_pubkey().get_bytes() + tx.vout[0].scriptPubKey = CScript([OP_2, pubkey, pubkey, pubkey, OP_3, OP_CHECKMULTISIG]) # Some bare multisig script (2-of-3) + self.check_mempool_result( + result_expected=[{'txid': tx.rehash(), 'allowed': False, 'reject-reason': 'bare-multisig'}], rawtxs=[tx.serialize().hex()], ) tx.deserialize(BytesIO(hex_str_to_bytes(raw_tx_reference))) tx.vin[0].scriptSig = CScript([OP_HASH160]) # Some not-pushonly scriptSig self.check_mempool_result( - result_expected=[{'txid': tx.rehash(), 'allowed': False, 'reject-reason': '64: scriptsig-not-pushonly'}], + result_expected=[{'txid': tx.rehash(), 'allowed': False, 'reject-reason': 'scriptsig-not-pushonly'}], + rawtxs=[tx.serialize().hex()], + ) + tx.deserialize(BytesIO(hex_str_to_bytes(raw_tx_reference))) + tx.vin[0].scriptSig = CScript([b'a' * 1648]) # Some too large scriptSig (>1650 bytes) + self.check_mempool_result( + result_expected=[{'txid': tx.rehash(), 'allowed': False, 'reject-reason': 'scriptsig-size'}], rawtxs=[tx.serialize().hex()], ) tx.deserialize(BytesIO(hex_str_to_bytes(raw_tx_reference))) @@ -271,21 +292,21 @@ class MempoolAcceptanceTest(BitcoinTestFramework): num_scripts = 100000 // len(output_p2sh_burn.serialize()) # Use enough outputs to make the tx too large for our policy tx.vout = [output_p2sh_burn] * num_scripts self.check_mempool_result( - result_expected=[{'txid': tx.rehash(), 'allowed': False, 'reject-reason': '64: tx-size'}], + result_expected=[{'txid': tx.rehash(), 'allowed': False, 'reject-reason': 'tx-size'}], rawtxs=[tx.serialize().hex()], ) tx.deserialize(BytesIO(hex_str_to_bytes(raw_tx_reference))) tx.vout[0] = output_p2sh_burn tx.vout[0].nValue -= 1 # Make output smaller, such that it is dust for our policy self.check_mempool_result( - result_expected=[{'txid': tx.rehash(), 'allowed': False, 'reject-reason': '64: dust'}], + result_expected=[{'txid': tx.rehash(), 'allowed': False, 'reject-reason': 'dust'}], rawtxs=[tx.serialize().hex()], ) tx.deserialize(BytesIO(hex_str_to_bytes(raw_tx_reference))) tx.vout[0].scriptPubKey = CScript([OP_RETURN, b'\xff']) tx.vout = [tx.vout[0]] * 2 self.check_mempool_result( - result_expected=[{'txid': tx.rehash(), 'allowed': False, 'reject-reason': '64: multi-op-return'}], + result_expected=[{'txid': tx.rehash(), 'allowed': False, 'reject-reason': 'multi-op-return'}], rawtxs=[tx.serialize().hex()], ) @@ -294,7 +315,7 @@ class MempoolAcceptanceTest(BitcoinTestFramework): tx.vin[0].nSequence -= 1 # Should be non-max, so locktime is not ignored tx.nLockTime = node.getblockcount() + 1 self.check_mempool_result( - result_expected=[{'txid': tx.rehash(), 'allowed': False, 'reject-reason': '64: non-final'}], + result_expected=[{'txid': tx.rehash(), 'allowed': False, 'reject-reason': 'non-final'}], rawtxs=[tx.serialize().hex()], ) @@ -303,7 +324,7 @@ class MempoolAcceptanceTest(BitcoinTestFramework): tx.vin[0].nSequence = 2 # We could include it in the second block mined from now, but not the very next one # Can skip re-signing the tx because of early rejection self.check_mempool_result( - result_expected=[{'txid': tx.rehash(), 'allowed': False, 'reject-reason': '64: non-BIP68-final'}], + result_expected=[{'txid': tx.rehash(), 'allowed': False, 'reject-reason': 'non-BIP68-final'}], rawtxs=[tx.serialize().hex()], maxfeerate=0, ) diff --git a/test/functional/mempool_expiry.py b/test/functional/mempool_expiry.py new file mode 100755 index 0000000000..8b9b7b155a --- /dev/null +++ b/test/functional/mempool_expiry.py @@ -0,0 +1,100 @@ +#!/usr/bin/env python3 +# Copyright (c) 2020 The Bitcoin Core developers +# Distributed under the MIT software license, see the accompanying +# file COPYING or http://www.opensource.org/licenses/mit-license.php. +"""Tests that a mempool transaction expires after a given timeout and that its +children are removed as well. + +Both the default expiry timeout defined by DEFAULT_MEMPOOL_EXPIRY and a user +definable expiry timeout via the '-mempoolexpiry=<n>' command line argument +(<n> is the timeout in hours) are tested. +""" + +from datetime import timedelta + +from test_framework.test_framework import BitcoinTestFramework +from test_framework.util import ( + assert_equal, + assert_raises_rpc_error, + find_vout_for_address, +) + +DEFAULT_MEMPOOL_EXPIRY = 336 # hours +CUSTOM_MEMPOOL_EXPIRY = 10 # hours + + +class MempoolExpiryTest(BitcoinTestFramework): + def set_test_params(self): + self.num_nodes = 1 + + def skip_test_if_missing_module(self): + self.skip_if_no_wallet() + + def test_transaction_expiry(self, timeout): + """Tests that a transaction expires after the expiry timeout and its + children are removed as well.""" + node = self.nodes[0] + + # Send a parent transaction that will expire. + parent_address = node.getnewaddress() + parent_txid = node.sendtoaddress(parent_address, 1.0) + + # Set the mocktime to the arrival time of the parent transaction. + entry_time = node.getmempoolentry(parent_txid)['time'] + node.setmocktime(entry_time) + + # Create child transaction spending the parent transaction + vout = find_vout_for_address(node, parent_txid, parent_address) + inputs = [{'txid': parent_txid, 'vout': vout}] + outputs = {node.getnewaddress(): 0.99} + child_raw = node.createrawtransaction(inputs, outputs) + child_signed = node.signrawtransactionwithwallet(child_raw)['hex'] + + # Let half of the timeout elapse and broadcast the child transaction. + half_expiry_time = entry_time + int(60 * 60 * timeout/2) + node.setmocktime(half_expiry_time) + child_txid = node.sendrawtransaction(child_signed) + self.log.info('Broadcast child transaction after {} hours.'.format( + timedelta(seconds=(half_expiry_time-entry_time)))) + + # Let most of the timeout elapse and check that the parent tx is still + # in the mempool. + nearly_expiry_time = entry_time + 60 * 60 * timeout - 5 + node.setmocktime(nearly_expiry_time) + # Expiry of mempool transactions is only checked when a new transaction + # is added to the to the mempool. + node.sendtoaddress(node.getnewaddress(), 1.0) + self.log.info('Test parent tx not expired after {} hours.'.format( + timedelta(seconds=(nearly_expiry_time-entry_time)))) + assert_equal(entry_time, node.getmempoolentry(parent_txid)['time']) + + # Transaction should be evicted from the mempool after the expiry time + # has passed. + expiry_time = entry_time + 60 * 60 * timeout + 5 + node.setmocktime(expiry_time) + # Expiry of mempool transactions is only checked when a new transaction + # is added to the to the mempool. + node.sendtoaddress(node.getnewaddress(), 1.0) + self.log.info('Test parent tx expiry after {} hours.'.format( + timedelta(seconds=(expiry_time-entry_time)))) + assert_raises_rpc_error(-5, 'Transaction not in mempool', + node.getmempoolentry, parent_txid) + + # The child transaction should be removed from the mempool as well. + self.log.info('Test child tx is evicted as well.') + assert_raises_rpc_error(-5, 'Transaction not in mempool', + node.getmempoolentry, child_txid) + + def run_test(self): + self.log.info('Test default mempool expiry timeout of %d hours.' % + DEFAULT_MEMPOOL_EXPIRY) + self.test_transaction_expiry(DEFAULT_MEMPOOL_EXPIRY) + + self.log.info('Test custom mempool expiry timeout of %d hours.' % + CUSTOM_MEMPOOL_EXPIRY) + self.restart_node(0, ['-mempoolexpiry=%d' % CUSTOM_MEMPOOL_EXPIRY]) + self.test_transaction_expiry(CUSTOM_MEMPOOL_EXPIRY) + + +if __name__ == '__main__': + MempoolExpiryTest().main() diff --git a/test/functional/mempool_limit.py b/test/functional/mempool_limit.py index edf2069933..39035f7cb1 100755 --- a/test/functional/mempool_limit.py +++ b/test/functional/mempool_limit.py @@ -18,6 +18,7 @@ class MempoolLimitTest(BitcoinTestFramework): "-maxmempool=5", "-spendzeroconfchange=0", ]] + self.supports_cli = False def skip_test_if_missing_module(self): self.skip_if_no_wallet() diff --git a/test/functional/mempool_packages.py b/test/functional/mempool_packages.py index c7d241503a..a07dad18d6 100755 --- a/test/functional/mempool_packages.py +++ b/test/functional/mempool_packages.py @@ -12,15 +12,31 @@ from test_framework.util import ( assert_equal, assert_raises_rpc_error, satoshi_round, + wait_until, ) +# default limits MAX_ANCESTORS = 25 MAX_DESCENDANTS = 25 +# custom limits for node1 +MAX_ANCESTORS_CUSTOM = 5 +MAX_DESCENDANTS_CUSTOM = 10 +assert MAX_DESCENDANTS_CUSTOM >= MAX_ANCESTORS_CUSTOM class MempoolPackagesTest(BitcoinTestFramework): def set_test_params(self): self.num_nodes = 2 - self.extra_args = [["-maxorphantx=1000"], ["-maxorphantx=1000", "-limitancestorcount=5"]] + self.extra_args = [ + [ + "-maxorphantx=1000", + "-whitelist=noban@127.0.0.1", # immediate tx relay + ], + [ + "-maxorphantx=1000", + "-limitancestorcount={}".format(MAX_ANCESTORS_CUSTOM), + "-limitdescendantcount={}".format(MAX_DESCENDANTS_CUSTOM), + ], + ] def skip_test_if_missing_module(self): self.skip_if_no_wallet() @@ -188,7 +204,14 @@ class MempoolPackagesTest(BitcoinTestFramework): assert_equal(mempool[x]['descendantfees'], descendant_fees * COIN + 2000) assert_equal(mempool[x]['fees']['descendant'], descendant_fees+satoshi_round(0.00002)) - # TODO: check that node1's mempool is as expected + # Check that node1's mempool is as expected (-> custom ancestor limit) + mempool0 = self.nodes[0].getrawmempool(False) + mempool1 = self.nodes[1].getrawmempool(False) + assert_equal(len(mempool1), MAX_ANCESTORS_CUSTOM) + assert set(mempool1).issubset(set(mempool0)) + for tx in chain[:MAX_ANCESTORS_CUSTOM]: + assert tx in mempool1 + # TODO: more detailed check of node1's mempool (fees etc.) # TODO: test ancestor size limits @@ -206,9 +229,11 @@ class MempoolPackagesTest(BitcoinTestFramework): transaction_package.append({'txid': txid, 'vout': i, 'amount': sent_value}) # Sign and send up to MAX_DESCENDANT transactions chained off the parent tx + chain = [] # save sent txs for the purpose of checking node1's mempool later (see below) for i in range(MAX_DESCENDANTS - 1): utxo = transaction_package.pop(0) (txid, sent_value) = self.chain_transaction(self.nodes[0], utxo['txid'], utxo['vout'], utxo['amount'], fee, 10) + chain.append(txid) if utxo['txid'] is parent_transaction: tx_children.append(txid) for j in range(10): @@ -225,7 +250,21 @@ class MempoolPackagesTest(BitcoinTestFramework): utxo = transaction_package.pop(0) assert_raises_rpc_error(-26, "too-long-mempool-chain", self.chain_transaction, self.nodes[0], utxo['txid'], utxo['vout'], utxo['amount'], fee, 10) - # TODO: check that node1's mempool is as expected + # Check that node1's mempool is as expected, containing: + # - txs from previous ancestor test (-> custom ancestor limit) + # - parent tx for descendant test + # - txs chained off parent tx (-> custom descendant limit) + wait_until(lambda: len(self.nodes[1].getrawmempool(False)) == + MAX_ANCESTORS_CUSTOM + 1 + MAX_DESCENDANTS_CUSTOM, timeout=10) + mempool0 = self.nodes[0].getrawmempool(False) + mempool1 = self.nodes[1].getrawmempool(False) + assert set(mempool1).issubset(set(mempool0)) + assert parent_transaction in mempool1 + for tx in chain[:MAX_DESCENDANTS_CUSTOM]: + assert tx in mempool1 + for tx in chain[MAX_DESCENDANTS_CUSTOM:]: + assert tx not in mempool1 + # TODO: more detailed check of node1's mempool (fees etc.) # TODO: test descendant size limits diff --git a/test/functional/mempool_persist.py b/test/functional/mempool_persist.py index 398fdeb326..d3690b245e 100755 --- a/test/functional/mempool_persist.py +++ b/test/functional/mempool_persist.py @@ -1,5 +1,5 @@ #!/usr/bin/env python3 -# Copyright (c) 2014-2018 The Bitcoin Core developers +# Copyright (c) 2014-2019 The Bitcoin Core developers # Distributed under the MIT software license, see the accompanying # file COPYING or http://www.opensource.org/licenses/mit-license.php. """Test mempool persistence. @@ -117,8 +117,8 @@ class MempoolPersistTest(BitcoinTestFramework): wait_until(lambda: self.nodes[0].getmempoolinfo()["loaded"]) assert_equal(len(self.nodes[0].getrawmempool()), 5) - mempooldat0 = os.path.join(self.nodes[0].datadir, 'regtest', 'mempool.dat') - mempooldat1 = os.path.join(self.nodes[1].datadir, 'regtest', 'mempool.dat') + mempooldat0 = os.path.join(self.nodes[0].datadir, self.chain, 'mempool.dat') + mempooldat1 = os.path.join(self.nodes[1].datadir, self.chain, 'mempool.dat') self.log.debug("Remove the mempool.dat file. Verify that savemempool to disk via RPC re-creates it") os.remove(mempooldat0) self.nodes[0].savemempool() diff --git a/test/functional/mempool_reorg.py b/test/functional/mempool_reorg.py index 123f0b4c28..3b148d5cf0 100755 --- a/test/functional/mempool_reorg.py +++ b/test/functional/mempool_reorg.py @@ -76,7 +76,7 @@ class MempoolCoinbaseTest(BitcoinTestFramework): spend_101_id = self.nodes[0].sendrawtransaction(spend_101_raw) spend_102_1_id = self.nodes[0].sendrawtransaction(spend_102_1_raw) - self.sync_all() + self.sync_all(timeout=360) assert_equal(set(self.nodes[0].getrawmempool()), {spend_101_id, spend_102_1_id, timelock_tx_id}) @@ -91,7 +91,7 @@ class MempoolCoinbaseTest(BitcoinTestFramework): for node in self.nodes: node.invalidateblock(new_blocks[0]) - self.sync_all() + self.sync_all(timeout=360) # mempool should be empty. assert_equal(set(self.nodes[0].getrawmempool()), set()) diff --git a/test/functional/mining_basic.py b/test/functional/mining_basic.py index f9231614ce..8262e30592 100755 --- a/test/functional/mining_basic.py +++ b/test/functional/mining_basic.py @@ -43,6 +43,7 @@ class MiningTest(BitcoinTestFramework): def set_test_params(self): self.num_nodes = 2 self.setup_clean_chain = True + self.supports_cli = False def mine_chain(self): self.log.info('Create some old blocks') @@ -69,7 +70,7 @@ class MiningTest(BitcoinTestFramework): self.log.info('getmininginfo') mining_info = node.getmininginfo() assert_equal(mining_info['blocks'], 200) - assert_equal(mining_info['chain'], 'regtest') + assert_equal(mining_info['chain'], self.chain) assert 'currentblocktx' not in mining_info assert 'currentblockweight' not in mining_info assert_equal(mining_info['difficulty'], Decimal('4.656542373906925E-10')) diff --git a/test/functional/mining_getblocktemplate_longpoll.py b/test/functional/mining_getblocktemplate_longpoll.py index 445ec124ce..6d0b241e57 100755 --- a/test/functional/mining_getblocktemplate_longpoll.py +++ b/test/functional/mining_getblocktemplate_longpoll.py @@ -27,6 +27,7 @@ class LongpollThread(threading.Thread): class GetBlockTemplateLPTest(BitcoinTestFramework): def set_test_params(self): self.num_nodes = 2 + self.supports_cli = False def skip_test_if_missing_module(self): self.skip_if_no_wallet() diff --git a/test/functional/mining_prioritisetransaction.py b/test/functional/mining_prioritisetransaction.py index 7e05a8e6c8..1426fdaacb 100755 --- a/test/functional/mining_prioritisetransaction.py +++ b/test/functional/mining_prioritisetransaction.py @@ -18,6 +18,7 @@ class PrioritiseTransactionTest(BitcoinTestFramework): "-printpriority=1", "-acceptnonstdtxn=1", ]] * self.num_nodes + self.supports_cli = False def skip_test_if_missing_module(self): self.skip_if_no_wallet() diff --git a/test/functional/p2p_disconnect_ban.py b/test/functional/p2p_disconnect_ban.py index 23dea4b729..9047fc6828 100755 --- a/test/functional/p2p_disconnect_ban.py +++ b/test/functional/p2p_disconnect_ban.py @@ -1,5 +1,5 @@ #!/usr/bin/env python3 -# Copyright (c) 2014-2018 The Bitcoin Core developers +# Copyright (c) 2014-2019 The Bitcoin Core developers # Distributed under the MIT software license, see the accompanying # file COPYING or http://www.opensource.org/licenses/mit-license.php. """Test node disconnect and ban behavior""" @@ -16,6 +16,7 @@ from test_framework.util import ( class DisconnectBanTest(BitcoinTestFramework): def set_test_params(self): self.num_nodes = 2 + self.supports_cli = False def run_test(self): self.log.info("Connect nodes both way") diff --git a/test/functional/p2p_dos_header_tree.py b/test/functional/p2p_dos_header_tree.py index 7d386c47f6..6676b84e54 100755 --- a/test/functional/p2p_dos_header_tree.py +++ b/test/functional/p2p_dos_header_tree.py @@ -57,7 +57,7 @@ class RejectLowDifficultyHeadersTest(BitcoinTestFramework): } in self.nodes[0].getchaintips() self.log.info("Feed all fork headers (fails due to checkpoint)") - with self.nodes[0].assert_debug_log(['bad-fork-prior-to-checkpoint (code 67)']): + with self.nodes[0].assert_debug_log(['bad-fork-prior-to-checkpoint']): self.nodes[0].p2p.send_message(msg_headers(self.headers_fork)) self.nodes[0].p2p.wait_for_disconnect() diff --git a/test/functional/p2p_invalid_block.py b/test/functional/p2p_invalid_block.py index 905534b862..801407757f 100755 --- a/test/functional/p2p_invalid_block.py +++ b/test/functional/p2p_invalid_block.py @@ -22,7 +22,7 @@ class InvalidBlockRequestTest(BitcoinTestFramework): def set_test_params(self): self.num_nodes = 1 self.setup_clean_chain = True - self.extra_args = [["-whitelist=127.0.0.1"]] + self.extra_args = [["-whitelist=noban@127.0.0.1"]] def run_test(self): # Add p2p connection to node0 diff --git a/test/functional/p2p_invalid_messages.py b/test/functional/p2p_invalid_messages.py index 58c72f89d8..9876d749ff 100755 --- a/test/functional/p2p_invalid_messages.py +++ b/test/functional/p2p_invalid_messages.py @@ -4,7 +4,6 @@ # file COPYING or http://www.opensource.org/licenses/mit-license.php. """Test node responses to invalid network messages.""" import asyncio -import os import struct import sys @@ -66,27 +65,21 @@ class InvalidMessagesTest(BitcoinTestFramework): msg_at_size = msg_unrecognized(str_data="b" * valid_data_limit) assert len(msg_at_size.serialize()) == msg_limit - increase_allowed = 0.5 - if [s for s in os.environ.get("BITCOIN_CONFIG", "").split(" ") if "--with-sanitizers" in s and "address" in s]: - increase_allowed = 3.5 - with node.assert_memory_usage_stable(increase_allowed=increase_allowed): - self.log.info( - "Sending a bunch of large, junk messages to test " - "memory exhaustion. May take a bit...") + self.log.info("Sending a bunch of large, junk messages to test memory exhaustion. May take a bit...") - # Run a bunch of times to test for memory exhaustion. - for _ in range(80): - node.p2p.send_message(msg_at_size) + # Run a bunch of times to test for memory exhaustion. + for _ in range(80): + node.p2p.send_message(msg_at_size) - # Check that, even though the node is being hammered by nonsense from one - # connection, it can still service other peers in a timely way. - for _ in range(20): - conn2.sync_with_ping(timeout=2) + # Check that, even though the node is being hammered by nonsense from one + # connection, it can still service other peers in a timely way. + for _ in range(20): + conn2.sync_with_ping(timeout=2) - # Peer 1, despite serving up a bunch of nonsense, should still be connected. - self.log.info("Waiting for node to drop junk messages.") - node.p2p.sync_with_ping(timeout=320) - assert node.p2p.is_connected + # Peer 1, despite serving up a bunch of nonsense, should still be connected. + self.log.info("Waiting for node to drop junk messages.") + node.p2p.sync_with_ping(timeout=400) + assert node.p2p.is_connected # # 1. @@ -101,11 +94,10 @@ class InvalidMessagesTest(BitcoinTestFramework): msg_over_size = msg_unrecognized(str_data="b" * (valid_data_limit + 1)) assert len(msg_over_size.serialize()) == (msg_limit + 1) - with node.assert_debug_log(["Oversized message from peer=4, disconnecting"]): - # An unknown message type (or *any* message type) over - # MAX_PROTOCOL_MESSAGE_LENGTH should result in a disconnect. - node.p2p.send_message(msg_over_size) - node.p2p.wait_for_disconnect(timeout=4) + # An unknown message type (or *any* message type) over + # MAX_PROTOCOL_MESSAGE_LENGTH should result in a disconnect. + node.p2p.send_message(msg_over_size) + node.p2p.wait_for_disconnect(timeout=4) node.disconnect_p2ps() conn = node.add_p2p_connection(P2PDataStore()) @@ -153,13 +145,13 @@ class InvalidMessagesTest(BitcoinTestFramework): def test_magic_bytes(self): conn = self.nodes[0].add_p2p_connection(P2PDataStore()) - def swap_magic_bytes(): + async def swap_magic_bytes(): conn._on_data = lambda: None # Need to ignore all incoming messages from now, since they come with "invalid" magic bytes conn.magic_bytes = b'\x00\x11\x22\x32' # Call .result() to block until the atomic swap is complete, otherwise # we might run into races later on - asyncio.run_coroutine_threadsafe(asyncio.coroutine(swap_magic_bytes)(), NetworkThread.network_event_loop).result() + asyncio.run_coroutine_threadsafe(swap_magic_bytes(), NetworkThread.network_event_loop).result() with self.nodes[0].assert_debug_log(['PROCESSMESSAGE: INVALID MESSAGESTART ping']): conn.send_message(messages.msg_ping(nonce=0xff)) @@ -168,7 +160,7 @@ class InvalidMessagesTest(BitcoinTestFramework): def test_checksum(self): conn = self.nodes[0].add_p2p_connection(P2PDataStore()) - with self.nodes[0].assert_debug_log(['ProcessMessages(badmsg, 2 bytes): CHECKSUM ERROR expected 78df0a04 was ffffffff']): + with self.nodes[0].assert_debug_log(['CHECKSUM ERROR (badmsg, 2 bytes), expected 78df0a04 was ffffffff']): msg = conn.build_message(msg_unrecognized(str_data="d")) cut_len = ( 4 + # magic diff --git a/test/functional/p2p_invalid_tx.py b/test/functional/p2p_invalid_tx.py index 3cca2d78db..5975a52b2a 100755 --- a/test/functional/p2p_invalid_tx.py +++ b/test/functional/p2p_invalid_tx.py @@ -1,5 +1,5 @@ #!/usr/bin/env python3 -# Copyright (c) 2015-2018 The Bitcoin Core developers +# Copyright (c) 2015-2019 The Bitcoin Core developers # Distributed under the MIT software license, see the accompanying # file COPYING or http://www.opensource.org/licenses/mit-license.php. """Test node responses to invalid transactions. diff --git a/test/functional/p2p_permissions.py b/test/functional/p2p_permissions.py index 40b28d7533..3a7bf4bfc3 100755 --- a/test/functional/p2p_permissions.py +++ b/test/functional/p2p_permissions.py @@ -1,5 +1,5 @@ #!/usr/bin/env python3 -# Copyright (c) 2015-2018 The Bitcoin Core developers +# Copyright (c) 2015-2019 The Bitcoin Core developers # Distributed under the MIT software license, see the accompanying # file COPYING or http://www.opensource.org/licenses/mit-license.php. """Test p2p permission message. @@ -7,21 +7,35 @@ Test that permissions are correctly calculated and applied """ +from test_framework.address import ADDRESS_BCRT1_P2WSH_OP_TRUE +from test_framework.messages import ( + CTransaction, + CTxInWitness, + FromHex, +) +from test_framework.mininode import P2PDataStore +from test_framework.script import ( + CScript, + OP_TRUE, +) from test_framework.test_node import ErrorMatch from test_framework.test_framework import BitcoinTestFramework from test_framework.util import ( assert_equal, connect_nodes, p2p_port, + wait_until, ) + class P2PPermissionsTests(BitcoinTestFramework): def set_test_params(self): self.num_nodes = 2 self.setup_clean_chain = True - self.extra_args = [[],[]] def run_test(self): + self.check_tx_relay() + self.checkpermission( # default permissions (no specific permissions) ["-whitelist=127.0.0.1"], @@ -48,9 +62,9 @@ class P2PPermissionsTests(BitcoinTestFramework): ip_port = "127.0.0.1:{}".format(p2p_port(1)) self.replaceinconfig(1, "bind=127.0.0.1", "whitebind=bloomfilter,forcerelay@" + ip_port) self.checkpermission( - ["-whitelist=noban@127.0.0.1" ], + ["-whitelist=noban@127.0.0.1"], # Check parameter interaction forcerelay should activate relay - ["noban", "bloomfilter", "forcerelay", "relay" ], + ["noban", "bloomfilter", "forcerelay", "relay"], False) self.replaceinconfig(1, "whitebind=bloomfilter,forcerelay@" + ip_port, "bind=127.0.0.1") @@ -83,6 +97,51 @@ class P2PPermissionsTests(BitcoinTestFramework): self.nodes[1].assert_start_raises_init_error(["-whitelist=noban@127.0.0.1:230"], "Invalid netmask specified in", match=ErrorMatch.PARTIAL_REGEX) self.nodes[1].assert_start_raises_init_error(["-whitebind=noban@127.0.0.1/10"], "Cannot resolve -whitebind address", match=ErrorMatch.PARTIAL_REGEX) + def check_tx_relay(self): + block_op_true = self.nodes[0].getblock(self.nodes[0].generatetoaddress(100, ADDRESS_BCRT1_P2WSH_OP_TRUE)[0]) + self.sync_all() + + self.log.debug("Create a connection from a whitelisted wallet that rebroadcasts raw txs") + # A python mininode is needed to send the raw transaction directly. If a full node was used, it could only + # rebroadcast via the inv-getdata mechanism. However, even for whitelisted connections, a full node would + # currently not request a txid that is already in the mempool. + self.restart_node(1, extra_args=["-whitelist=forcerelay@127.0.0.1"]) + p2p_rebroadcast_wallet = self.nodes[1].add_p2p_connection(P2PDataStore()) + + self.log.debug("Send a tx from the wallet initially") + tx = FromHex( + CTransaction(), + self.nodes[0].createrawtransaction( + inputs=[{ + 'txid': block_op_true['tx'][0], + 'vout': 0, + }], outputs=[{ + ADDRESS_BCRT1_P2WSH_OP_TRUE: 5, + }]), + ) + tx.wit.vtxinwit = [CTxInWitness()] + tx.wit.vtxinwit[0].scriptWitness.stack = [CScript([OP_TRUE])] + txid = tx.rehash() + + self.log.debug("Wait until tx is in node[1]'s mempool") + p2p_rebroadcast_wallet.send_txs_and_test([tx], self.nodes[1]) + + self.log.debug("Check that node[1] will send the tx to node[0] even though it is already in the mempool") + connect_nodes(self.nodes[1], 0) + with self.nodes[1].assert_debug_log(["Force relaying tx {} from whitelisted peer=0".format(txid)]): + p2p_rebroadcast_wallet.send_txs_and_test([tx], self.nodes[1]) + wait_until(lambda: txid in self.nodes[0].getrawmempool()) + + self.log.debug("Check that node[1] will not send an invalid tx to node[0]") + tx.vout[0].nValue += 1 + txid = tx.rehash() + p2p_rebroadcast_wallet.send_txs_and_test( + [tx], + self.nodes[1], + success=False, + reject_reason='Not relaying non-mempool transaction {} from whitelisted peer=0'.format(txid), + ) + def checkpermission(self, args, expectedPermissions, whitelisted): self.restart_node(1, args) connect_nodes(self.nodes[0], 1) @@ -95,9 +154,10 @@ class P2PPermissionsTests(BitcoinTestFramework): def replaceinconfig(self, nodeid, old, new): with open(self.nodes[nodeid].bitcoinconf, encoding="utf8") as f: - newText=f.read().replace(old, new) + newText = f.read().replace(old, new) with open(self.nodes[nodeid].bitcoinconf, 'w', encoding="utf8") as f: f.write(newText) + if __name__ == '__main__': P2PPermissionsTests().main() diff --git a/test/functional/p2p_segwit.py b/test/functional/p2p_segwit.py index 98f6b1d71d..ad5a124680 100755 --- a/test/functional/p2p_segwit.py +++ b/test/functional/p2p_segwit.py @@ -31,7 +31,7 @@ from test_framework.messages import ( msg_inv, msg_tx, msg_block, - msg_witness_tx, + msg_no_witness_tx, ser_uint256, ser_vector, sha256, @@ -67,8 +67,8 @@ from test_framework.script import ( SIGHASH_ANYONECANPAY, SIGHASH_NONE, SIGHASH_SINGLE, - SegwitVersion1SignatureHash, - SignatureHash, + SegwitV0SignatureHash, + LegacySignatureHash, hash160, ) from test_framework.test_framework import BitcoinTestFramework @@ -103,7 +103,7 @@ def get_p2pkh_script(pubkeyhash): def sign_p2pk_witness_input(script, tx_to, in_idx, hashtype, value, key): """Add signature for a P2PK witness program.""" - tx_hash = SegwitVersion1SignatureHash(script, tx_to, in_idx, hashtype, value) + tx_hash = SegwitV0SignatureHash(script, tx_to, in_idx, hashtype, value) signature = key.sign_ecdsa(tx_hash) + chr(hashtype).encode('latin-1') tx_to.wit.vtxinwit[in_idx].scriptWitness.stack = [signature, script] tx_to.rehash() @@ -125,10 +125,11 @@ def test_transaction_acceptance(node, p2p, tx, with_witness, accepted, reason=No - use the getrawmempool rpc to check for acceptance.""" reason = [reason] if reason else [] with node.assert_debug_log(expected_msgs=reason): - p2p.send_message(msg_witness_tx(tx) if with_witness else msg_tx(tx)) + p2p.send_message(msg_tx(tx) if with_witness else msg_no_witness_tx(tx)) p2p.sync_with_ping() assert_equal(tx.hash in node.getrawmempool(), accepted) + def test_witness_block(node, p2p, block, accepted, with_witness=True, reason=None): """Send a block to the node and check that it's accepted @@ -187,10 +188,11 @@ class SegWitTest(BitcoinTestFramework): self.num_nodes = 3 # This test tests SegWit both pre and post-activation, so use the normal BIP9 activation. self.extra_args = [ - ["-whitelist=127.0.0.1", "-acceptnonstdtxn=1", "-segwitheight={}".format(SEGWIT_HEIGHT)], - ["-whitelist=127.0.0.1", "-acceptnonstdtxn=0", "-segwitheight={}".format(SEGWIT_HEIGHT)], - ["-whitelist=127.0.0.1", "-acceptnonstdtxn=1", "-segwitheight=-1"] + ["-acceptnonstdtxn=1", "-segwitheight={}".format(SEGWIT_HEIGHT), "-whitelist=noban@127.0.0.1"], + ["-acceptnonstdtxn=0", "-segwitheight={}".format(SEGWIT_HEIGHT)], + ["-acceptnonstdtxn=1", "-segwitheight=-1"], ] + self.supports_cli = False def skip_test_if_missing_module(self): self.skip_if_no_wallet() @@ -310,9 +312,9 @@ class SegWitTest(BitcoinTestFramework): # Check that serializing it with or without witness is the same # This is a sanity check of our testing framework. - assert_equal(msg_tx(tx).serialize(), msg_witness_tx(tx).serialize()) + assert_equal(msg_no_witness_tx(tx).serialize(), msg_tx(tx).serialize()) - self.test_node.send_message(msg_witness_tx(tx)) + self.test_node.send_message(msg_tx(tx)) self.test_node.sync_with_ping() # make sure the tx was processed assert tx.hash in self.nodes[0].getrawmempool() # Save this transaction for later @@ -1116,7 +1118,7 @@ class SegWitTest(BitcoinTestFramework): MAX_PROGRAM_LENGTH = 10000 # This program is 19 max pushes (9937 bytes), then 64 more opcode-bytes. - long_witness_program = CScript([b'a' * 520] * 19 + [OP_DROP] * 63 + [OP_TRUE]) + long_witness_program = CScript([b'a' * MAX_SCRIPT_ELEMENT_SIZE] * 19 + [OP_DROP] * 63 + [OP_TRUE]) assert len(long_witness_program) == MAX_PROGRAM_LENGTH + 1 long_witness_hash = sha256(long_witness_program) long_script_pubkey = CScript([OP_0, long_witness_hash]) @@ -1140,7 +1142,7 @@ class SegWitTest(BitcoinTestFramework): test_witness_block(self.nodes[0], self.test_node, block, accepted=False) # Try again with one less byte in the witness program - witness_program = CScript([b'a' * 520] * 19 + [OP_DROP] * 62 + [OP_TRUE]) + witness_program = CScript([b'a' * MAX_SCRIPT_ELEMENT_SIZE] * 19 + [OP_DROP] * 62 + [OP_TRUE]) assert len(witness_program) == MAX_PROGRAM_LENGTH witness_hash = sha256(witness_program) script_pubkey = CScript([OP_0, witness_hash]) @@ -1489,7 +1491,7 @@ class SegWitTest(BitcoinTestFramework): tx2.vin.append(CTxIn(COutPoint(tx.sha256, 0), b"")) tx2.vout.append(CTxOut(tx.vout[0].nValue - 1000, script_wsh)) script = get_p2pkh_script(pubkeyhash) - sig_hash = SegwitVersion1SignatureHash(script, tx2, 0, SIGHASH_ALL, tx.vout[0].nValue) + sig_hash = SegwitV0SignatureHash(script, tx2, 0, SIGHASH_ALL, tx.vout[0].nValue) signature = key.sign_ecdsa(sig_hash) + b'\x01' # 0x1 is SIGHASH_ALL tx2.wit.vtxinwit.append(CTxInWitness()) tx2.wit.vtxinwit[0].scriptWitness.stack = [signature, pubkey] @@ -1543,7 +1545,7 @@ class SegWitTest(BitcoinTestFramework): tx5 = CTransaction() tx5.vin.append(CTxIn(COutPoint(tx4.sha256, 0), b"")) tx5.vout.append(CTxOut(tx4.vout[0].nValue - 1000, CScript([OP_TRUE]))) - (sig_hash, err) = SignatureHash(script_pubkey, tx5, 0, SIGHASH_ALL) + (sig_hash, err) = LegacySignatureHash(script_pubkey, tx5, 0, SIGHASH_ALL) signature = key.sign_ecdsa(sig_hash) + b'\x01' # 0x1 is SIGHASH_ALL tx5.vin[0].scriptSig = CScript([signature, pubkey]) tx5.rehash() @@ -1693,7 +1695,7 @@ class SegWitTest(BitcoinTestFramework): tx2.vout.append(CTxOut(tx.vout[0].nValue, CScript([OP_TRUE]))) script = get_p2pkh_script(pubkeyhash) - sig_hash = SegwitVersion1SignatureHash(script, tx2, 0, SIGHASH_ALL, tx.vout[0].nValue) + sig_hash = SegwitV0SignatureHash(script, tx2, 0, SIGHASH_ALL, tx.vout[0].nValue) signature = key.sign_ecdsa(sig_hash) + b'\x01' # 0x1 is SIGHASH_ALL # Check that we can't have a scriptSig diff --git a/test/functional/rpc_bind.py b/test/functional/rpc_bind.py index 8979251a26..664a15a5ec 100755 --- a/test/functional/rpc_bind.py +++ b/test/functional/rpc_bind.py @@ -15,6 +15,7 @@ class RPCBindTest(BitcoinTestFramework): self.setup_clean_chain = True self.bind_to_localhost_only = False self.num_nodes = 1 + self.supports_cli = False def setup_network(self): self.add_nodes(self.num_nodes, None) diff --git a/test/functional/rpc_blockchain.py b/test/functional/rpc_blockchain.py index 278ce6d911..adf6f1ca4f 100755 --- a/test/functional/rpc_blockchain.py +++ b/test/functional/rpc_blockchain.py @@ -49,6 +49,7 @@ class BlockchainTest(BitcoinTestFramework): def set_test_params(self): self.setup_clean_chain = True self.num_nodes = 1 + self.supports_cli = False def run_test(self): self.mine_chain() diff --git a/test/functional/rpc_createmultisig.py b/test/functional/rpc_createmultisig.py index 2a64a29967..a983716177 100755 --- a/test/functional/rpc_createmultisig.py +++ b/test/functional/rpc_createmultisig.py @@ -4,7 +4,7 @@ # file COPYING or http://www.opensource.org/licenses/mit-license.php. """Test multisig RPCs""" -from test_framework.descriptors import descsum_create +from test_framework.descriptors import descsum_create, drop_origins from test_framework.test_framework import BitcoinTestFramework from test_framework.util import ( assert_raises_rpc_error, @@ -22,6 +22,7 @@ class RpcCreateMultiSigTest(BitcoinTestFramework): def set_test_params(self): self.setup_clean_chain = True self.num_nodes = 3 + self.supports_cli = False def skip_test_if_missing_module(self): self.skip_if_no_wallet() @@ -115,9 +116,20 @@ class RpcCreateMultiSigTest(BitcoinTestFramework): def do_multisig(self): node0, node1, node2 = self.nodes + # Construct the expected descriptor + desc = 'multi({},{})'.format(self.nsigs, ','.join(self.pub)) + if self.output_type == 'legacy': + desc = 'sh({})'.format(desc) + elif self.output_type == 'p2sh-segwit': + desc = 'sh(wsh({}))'.format(desc) + elif self.output_type == 'bech32': + desc = 'wsh({})'.format(desc) + desc = descsum_create(desc) + msig = node2.createmultisig(self.nsigs, self.pub, self.output_type) madd = msig["address"] mredeem = msig["redeemScript"] + assert_equal(desc, msig['descriptor']) if self.output_type == 'bech32': assert madd[0:4] == "bcrt" # actually a bech32 address @@ -125,6 +137,7 @@ class RpcCreateMultiSigTest(BitcoinTestFramework): msigw = node1.addmultisigaddress(self.nsigs, self.pub, None, self.output_type) maddw = msigw["address"] mredeemw = msigw["redeemScript"] + assert_equal(desc, drop_origins(msigw['descriptor'])) # addmultisigiaddress and createmultisig work the same assert maddw == madd assert mredeemw == mredeem diff --git a/test/functional/rpc_deriveaddresses.py b/test/functional/rpc_deriveaddresses.py index 42128d5767..42d7d59d56 100755 --- a/test/functional/rpc_deriveaddresses.py +++ b/test/functional/rpc_deriveaddresses.py @@ -10,7 +10,6 @@ from test_framework.util import assert_equal, assert_raises_rpc_error class DeriveaddressesTest(BitcoinTestFramework): def set_test_params(self): self.num_nodes = 1 - self.supports_cli = 1 def run_test(self): assert_raises_rpc_error(-5, "Missing checksum", self.nodes[0].deriveaddresses, "a") diff --git a/test/functional/rpc_dumptxoutset.py b/test/functional/rpc_dumptxoutset.py new file mode 100755 index 0000000000..438e7f68e0 --- /dev/null +++ b/test/functional/rpc_dumptxoutset.py @@ -0,0 +1,51 @@ +#!/usr/bin/env python3 +# Copyright (c) 2019 The Bitcoin Core developers +# Distributed under the MIT software license, see the accompanying +# file COPYING or http://www.opensource.org/licenses/mit-license.php. +"""Test the generation of UTXO snapshots using `dumptxoutset`. +""" +from test_framework.test_framework import BitcoinTestFramework +from test_framework.util import assert_equal, assert_raises_rpc_error + +import hashlib +from pathlib import Path + + +class DumptxoutsetTest(BitcoinTestFramework): + def set_test_params(self): + self.setup_clean_chain = True + self.num_nodes = 1 + + def run_test(self): + """Test a trivial usage of the dumptxoutset RPC command.""" + node = self.nodes[0] + mocktime = node.getblockheader(node.getblockhash(0))['time'] + 1 + node.setmocktime(mocktime) + node.generate(100) + + FILENAME = 'txoutset.dat' + out = node.dumptxoutset(FILENAME) + expected_path = Path(node.datadir) / self.chain / FILENAME + + assert expected_path.is_file() + + assert_equal(out['coins_written'], 100) + assert_equal(out['base_height'], 100) + assert_equal(out['path'], str(expected_path)) + # Blockhash should be deterministic based on mocked time. + assert_equal( + out['base_hash'], + '6fd417acba2a8738b06fee43330c50d58e6a725046c3d843c8dd7e51d46d1ed6') + + with open(str(expected_path), 'rb') as f: + digest = hashlib.sha256(f.read()).hexdigest() + # UTXO snapshot hash should be deterministic based on mocked time. + assert_equal( + digest, 'be032e5f248264ba08e11099ac09dbd001f6f87ffc68bf0f87043d8146d50664') + + # Specifying a path to an existing file will fail. + assert_raises_rpc_error( + -8, '{} already exists'.format(FILENAME), node.dumptxoutset, FILENAME) + +if __name__ == '__main__': + DumptxoutsetTest().main() diff --git a/test/functional/rpc_fundrawtransaction.py b/test/functional/rpc_fundrawtransaction.py index c956af1cbe..c435ef24ce 100755 --- a/test/functional/rpc_fundrawtransaction.py +++ b/test/functional/rpc_fundrawtransaction.py @@ -28,6 +28,9 @@ class RawTransactionsTest(BitcoinTestFramework): def set_test_params(self): self.num_nodes = 4 self.setup_clean_chain = True + # This test isn't testing tx relay. Set whitelist on the peers for + # instant tx relay. + self.extra_args = [['-whitelist=noban@127.0.0.1']] * self.num_nodes def skip_test_if_missing_module(self): self.skip_if_no_wallet() @@ -41,6 +44,7 @@ class RawTransactionsTest(BitcoinTestFramework): connect_nodes(self.nodes[0], 3) def run_test(self): + self.log.info("Connect nodes, set fees, generate blocks, and sync") self.min_relay_tx_fee = self.nodes[0].getnetworkinfo()['relayfee'] # This test is not meant to test fee estimation and we'd like # to be sure all txs are sent at a consistent desired feerate @@ -88,9 +92,11 @@ class RawTransactionsTest(BitcoinTestFramework): self.test_option_feerate() self.test_address_reuse() self.test_option_subtract_fee_from_outputs() + self.test_subtract_fee_with_presets() def test_change_position(self): - # ensure that setting changePosition in fundraw with an exact match is handled properly + """Ensure setting changePosition in fundraw with an exact match is handled properly.""" + self.log.info("Test fundrawtxn changePosition option") rawmatch = self.nodes[2].createrawtransaction([], {self.nodes[2].getnewaddress():50}) rawmatch = self.nodes[2].fundrawtransaction(rawmatch, {"changePosition":1, "subtractFeeFromOutputs":[0]}) assert_equal(rawmatch["changepos"], -1) @@ -115,9 +121,7 @@ class RawTransactionsTest(BitcoinTestFramework): self.sync_all() def test_simple(self): - ############### - # simple test # - ############### + self.log.info("Test fundrawtxn") inputs = [ ] outputs = { self.nodes[0].getnewaddress() : 1.0 } rawtx = self.nodes[2].createrawtransaction(inputs, outputs) @@ -127,9 +131,7 @@ class RawTransactionsTest(BitcoinTestFramework): assert len(dec_tx['vin']) > 0 #test that we have enough inputs def test_simple_two_coins(self): - ############################## - # simple test with two coins # - ############################## + self.log.info("Test fundrawtxn with 2 coins") inputs = [ ] outputs = { self.nodes[0].getnewaddress() : 2.2 } rawtx = self.nodes[2].createrawtransaction(inputs, outputs) @@ -141,9 +143,8 @@ class RawTransactionsTest(BitcoinTestFramework): assert_equal(dec_tx['vin'][0]['scriptSig']['hex'], '') def test_simple_two_outputs(self): - ################################ - # simple test with two outputs # - ################################ + self.log.info("Test fundrawtxn with 2 outputs") + inputs = [ ] outputs = { self.nodes[0].getnewaddress() : 2.6, self.nodes[1].getnewaddress() : 2.5 } rawtx = self.nodes[2].createrawtransaction(inputs, outputs) @@ -159,9 +160,7 @@ class RawTransactionsTest(BitcoinTestFramework): assert_equal(dec_tx['vin'][0]['scriptSig']['hex'], '') def test_change(self): - ######################################################################### - # test a fundrawtransaction with a VIN greater than the required amount # - ######################################################################### + self.log.info("Test fundrawtxn with a vin > required amount") utx = get_unspent(self.nodes[2].listunspent(), 5) inputs = [ {'txid' : utx['txid'], 'vout' : utx['vout']}] @@ -181,9 +180,7 @@ class RawTransactionsTest(BitcoinTestFramework): assert_equal(fee + totalOut, utx['amount']) #compare vin total and totalout+fee def test_no_change(self): - ##################################################################### - # test a fundrawtransaction with which will not get a change output # - ##################################################################### + self.log.info("Test fundrawtxn not having a change output") utx = get_unspent(self.nodes[2].listunspent(), 5) inputs = [ {'txid' : utx['txid'], 'vout' : utx['vout']}] @@ -203,9 +200,7 @@ class RawTransactionsTest(BitcoinTestFramework): assert_equal(fee + totalOut, utx['amount']) #compare vin total and totalout+fee def test_invalid_option(self): - #################################################### - # test a fundrawtransaction with an invalid option # - #################################################### + self.log.info("Test fundrawtxn with an invalid option") utx = get_unspent(self.nodes[2].listunspent(), 5) inputs = [ {'txid' : utx['txid'], 'vout' : utx['vout']} ] @@ -220,9 +215,7 @@ class RawTransactionsTest(BitcoinTestFramework): assert_raises_rpc_error(-3, "Unexpected key reserveChangeKey", lambda: self.nodes[2].fundrawtransaction(hexstring=rawtx, options={'reserveChangeKey': True})) def test_invalid_change_address(self): - ############################################################ - # test a fundrawtransaction with an invalid change address # - ############################################################ + self.log.info("Test fundrawtxn with an invalid change address") utx = get_unspent(self.nodes[2].listunspent(), 5) inputs = [ {'txid' : utx['txid'], 'vout' : utx['vout']} ] @@ -234,9 +227,7 @@ class RawTransactionsTest(BitcoinTestFramework): assert_raises_rpc_error(-5, "changeAddress must be a valid bitcoin address", self.nodes[2].fundrawtransaction, rawtx, {'changeAddress':'foobar'}) def test_valid_change_address(self): - ############################################################ - # test a fundrawtransaction with a provided change address # - ############################################################ + self.log.info("Test fundrawtxn with a provided change address") utx = get_unspent(self.nodes[2].listunspent(), 5) inputs = [ {'txid' : utx['txid'], 'vout' : utx['vout']} ] @@ -253,9 +244,7 @@ class RawTransactionsTest(BitcoinTestFramework): assert_equal(change, out['scriptPubKey']['addresses'][0]) def test_change_type(self): - ######################################################### - # test a fundrawtransaction with a provided change type # - ######################################################### + self.log.info("Test fundrawtxn with a provided change type") utx = get_unspent(self.nodes[2].listunspent(), 5) inputs = [ {'txid' : utx['txid'], 'vout' : utx['vout']} ] @@ -268,9 +257,7 @@ class RawTransactionsTest(BitcoinTestFramework): assert_equal('witness_v0_keyhash', dec_tx['vout'][rawtx['changepos']]['scriptPubKey']['type']) def test_coin_selection(self): - ######################################################################### - # test a fundrawtransaction with a VIN smaller than the required amount # - ######################################################################### + self.log.info("Test fundrawtxn with a vin < required amount") utx = get_unspent(self.nodes[2].listunspent(), 1) inputs = [ {'txid' : utx['txid'], 'vout' : utx['vout']}] @@ -302,9 +289,7 @@ class RawTransactionsTest(BitcoinTestFramework): assert_equal(len(dec_tx['vout']), 2) def test_two_vin(self): - ########################################### - # test a fundrawtransaction with two VINs # - ########################################### + self.log.info("Test fundrawtxn with 2 vins") utx = get_unspent(self.nodes[2].listunspent(), 1) utx2 = get_unspent(self.nodes[2].listunspent(), 5) @@ -335,9 +320,7 @@ class RawTransactionsTest(BitcoinTestFramework): assert_equal(matchingIns, 2) #we now must see two vins identical to vins given as params def test_two_vin_two_vout(self): - ######################################################### - # test a fundrawtransaction with two VINs and two vOUTs # - ######################################################### + self.log.info("Test fundrawtxn with 2 vins and 2 vouts") utx = get_unspent(self.nodes[2].listunspent(), 1) utx2 = get_unspent(self.nodes[2].listunspent(), 5) @@ -360,52 +343,54 @@ class RawTransactionsTest(BitcoinTestFramework): assert_equal(len(dec_tx['vout']), 3) def test_invalid_input(self): - ############################################## - # test a fundrawtransaction with invalid vin # - ############################################## + self.log.info("Test fundrawtxn with an invalid vin") inputs = [ {'txid' : "1c7f966dab21119bac53213a2bc7532bff1fa844c124fd750a7d0b1332440bd1", 'vout' : 0} ] #invalid vin! outputs = { self.nodes[0].getnewaddress() : 1.0} rawtx = self.nodes[2].createrawtransaction(inputs, outputs) assert_raises_rpc_error(-4, "Insufficient funds", self.nodes[2].fundrawtransaction, rawtx) def test_fee_p2pkh(self): - ############################################################ - #compare fee of a standard pubkeyhash transaction + """Compare fee of a standard pubkeyhash transaction.""" + self.log.info("Test fundrawtxn p2pkh fee") inputs = [] outputs = {self.nodes[1].getnewaddress():1.1} rawtx = self.nodes[0].createrawtransaction(inputs, outputs) fundedTx = self.nodes[0].fundrawtransaction(rawtx) - #create same transaction over sendtoaddress + # Create same transaction over sendtoaddress. txId = self.nodes[0].sendtoaddress(self.nodes[1].getnewaddress(), 1.1) signedFee = self.nodes[0].getrawmempool(True)[txId]['fee'] - #compare fee + # Compare fee. feeDelta = Decimal(fundedTx['fee']) - Decimal(signedFee) assert feeDelta >= 0 and feeDelta <= self.fee_tolerance - ############################################################ def test_fee_p2pkh_multi_out(self): - ############################################################ - #compare fee of a standard pubkeyhash transaction with multiple outputs + """Compare fee of a standard pubkeyhash transaction with multiple outputs.""" + self.log.info("Test fundrawtxn p2pkh fee with multiple outputs") inputs = [] - outputs = {self.nodes[1].getnewaddress():1.1,self.nodes[1].getnewaddress():1.2,self.nodes[1].getnewaddress():0.1,self.nodes[1].getnewaddress():1.3,self.nodes[1].getnewaddress():0.2,self.nodes[1].getnewaddress():0.3} + outputs = { + self.nodes[1].getnewaddress():1.1, + self.nodes[1].getnewaddress():1.2, + self.nodes[1].getnewaddress():0.1, + self.nodes[1].getnewaddress():1.3, + self.nodes[1].getnewaddress():0.2, + self.nodes[1].getnewaddress():0.3, + } rawtx = self.nodes[0].createrawtransaction(inputs, outputs) fundedTx = self.nodes[0].fundrawtransaction(rawtx) - #create same transaction over sendtoaddress + + # Create same transaction over sendtoaddress. txId = self.nodes[0].sendmany("", outputs) signedFee = self.nodes[0].getrawmempool(True)[txId]['fee'] - #compare fee + # Compare fee. feeDelta = Decimal(fundedTx['fee']) - Decimal(signedFee) assert feeDelta >= 0 and feeDelta <= self.fee_tolerance - ############################################################ def test_fee_p2sh(self): - ############################################################ - #compare fee of a 2of2 multisig p2sh transaction - - # create 2of2 addr + """Compare fee of a 2-of-2 multisig p2sh transaction.""" + # Create 2-of-2 addr. addr1 = self.nodes[1].getnewaddress() addr2 = self.nodes[1].getnewaddress() @@ -419,20 +404,19 @@ class RawTransactionsTest(BitcoinTestFramework): rawtx = self.nodes[0].createrawtransaction(inputs, outputs) fundedTx = self.nodes[0].fundrawtransaction(rawtx) - #create same transaction over sendtoaddress + # Create same transaction over sendtoaddress. txId = self.nodes[0].sendtoaddress(mSigObj, 1.1) signedFee = self.nodes[0].getrawmempool(True)[txId]['fee'] - #compare fee + # Compare fee. feeDelta = Decimal(fundedTx['fee']) - Decimal(signedFee) assert feeDelta >= 0 and feeDelta <= self.fee_tolerance - ############################################################ def test_fee_4of5(self): - ############################################################ - #compare fee of a standard pubkeyhash transaction + """Compare fee of a standard pubkeyhash transaction.""" + self.log.info("Test fundrawtxn fee with 4-of-5 addresses") - # create 4of5 addr + # Create 4-of-5 addr. addr1 = self.nodes[1].getnewaddress() addr2 = self.nodes[1].getnewaddress() addr3 = self.nodes[1].getnewaddress() @@ -445,40 +429,52 @@ class RawTransactionsTest(BitcoinTestFramework): addr4Obj = self.nodes[1].getaddressinfo(addr4) addr5Obj = self.nodes[1].getaddressinfo(addr5) - mSigObj = self.nodes[1].addmultisigaddress(4, [addr1Obj['pubkey'], addr2Obj['pubkey'], addr3Obj['pubkey'], addr4Obj['pubkey'], addr5Obj['pubkey']])['address'] + mSigObj = self.nodes[1].addmultisigaddress( + 4, + [ + addr1Obj['pubkey'], + addr2Obj['pubkey'], + addr3Obj['pubkey'], + addr4Obj['pubkey'], + addr5Obj['pubkey'], + ] + )['address'] inputs = [] outputs = {mSigObj:1.1} rawtx = self.nodes[0].createrawtransaction(inputs, outputs) fundedTx = self.nodes[0].fundrawtransaction(rawtx) - #create same transaction over sendtoaddress + # Create same transaction over sendtoaddress. txId = self.nodes[0].sendtoaddress(mSigObj, 1.1) signedFee = self.nodes[0].getrawmempool(True)[txId]['fee'] - #compare fee + # Compare fee. feeDelta = Decimal(fundedTx['fee']) - Decimal(signedFee) assert feeDelta >= 0 and feeDelta <= self.fee_tolerance - ############################################################ def test_spend_2of2(self): - ############################################################ - # spend a 2of2 multisig transaction over fundraw + """Spend a 2-of-2 multisig transaction over fundraw.""" + self.log.info("Test fundrawtxn spending 2-of-2 multisig") - # create 2of2 addr + # Create 2-of-2 addr. addr1 = self.nodes[2].getnewaddress() addr2 = self.nodes[2].getnewaddress() addr1Obj = self.nodes[2].getaddressinfo(addr1) addr2Obj = self.nodes[2].getaddressinfo(addr2) - mSigObj = self.nodes[2].addmultisigaddress(2, [addr1Obj['pubkey'], addr2Obj['pubkey']])['address'] + mSigObj = self.nodes[2].addmultisigaddress( + 2, + [ + addr1Obj['pubkey'], + addr2Obj['pubkey'], + ] + )['address'] - - # send 1.2 BTC to msig addr + # Send 1.2 BTC to msig addr. self.nodes[0].sendtoaddress(mSigObj, 1.2) - self.sync_all() - self.nodes[1].generate(1) + self.nodes[0].generate(1) self.sync_all() oldBalance = self.nodes[1].getbalance() @@ -489,35 +485,18 @@ class RawTransactionsTest(BitcoinTestFramework): signedTx = self.nodes[2].signrawtransactionwithwallet(fundedTx['hex']) self.nodes[2].sendrawtransaction(signedTx['hex']) - self.sync_all() - self.nodes[1].generate(1) + self.nodes[2].generate(1) self.sync_all() - # make sure funds are received at node1 + # Make sure funds are received at node1. assert_equal(oldBalance+Decimal('1.10000000'), self.nodes[1].getbalance()) def test_locked_wallet(self): - ############################################################ - # locked wallet test - self.nodes[1].encryptwallet("test") - self.stop_nodes() + self.log.info("Test fundrawtxn with locked wallet") - self.start_nodes() - # This test is not meant to test fee estimation and we'd like - # to be sure all txs are sent at a consistent desired feerate - for node in self.nodes: - node.settxfee(self.min_relay_tx_fee) - - connect_nodes(self.nodes[0], 1) - connect_nodes(self.nodes[1], 2) - connect_nodes(self.nodes[0], 2) - connect_nodes(self.nodes[0], 3) - # Again lock the watchonly UTXO or nodes[0] may spend it, because - # lockunspent is memory-only and thus lost on restart - self.nodes[0].lockunspent(False, [{"txid": self.watchonly_txid, "vout": self.watchonly_vout}]) - self.sync_all() + self.nodes[1].encryptwallet("test") - # drain the keypool + # Drain the keypool. self.nodes[1].getnewaddress() self.nodes[1].getrawchangeaddress() inputs = [] @@ -527,7 +506,7 @@ class RawTransactionsTest(BitcoinTestFramework): # creating the key must be impossible because the wallet is locked assert_raises_rpc_error(-4, "Keypool ran out, please call keypoolrefill first", self.nodes[1].fundrawtransaction, rawtx) - #refill the keypool + # Refill the keypool. self.nodes[1].walletpassphrase("test", 100) self.nodes[1].keypoolrefill(8) #need to refill the keypool to get an internal change address self.nodes[1].walletlock() @@ -541,25 +520,23 @@ class RawTransactionsTest(BitcoinTestFramework): rawtx = self.nodes[1].createrawtransaction(inputs, outputs) fundedTx = self.nodes[1].fundrawtransaction(rawtx) - #now we need to unlock + # Now we need to unlock. self.nodes[1].walletpassphrase("test", 600) signedTx = self.nodes[1].signrawtransactionwithwallet(fundedTx['hex']) self.nodes[1].sendrawtransaction(signedTx['hex']) self.nodes[1].generate(1) self.sync_all() - # make sure funds are received at node1 + # Make sure funds are received at node1. assert_equal(oldBalance+Decimal('51.10000000'), self.nodes[0].getbalance()) def test_many_inputs_fee(self): - ############################################### - # multiple (~19) inputs tx test | Compare fee # - ############################################### + """Multiple (~19) inputs tx test | Compare fee.""" + self.log.info("Test fundrawtxn fee with many inputs") - #empty node1, send some small coins from node0 to node1 + # Empty node1, send some small coins from node0 to node1. self.nodes[1].sendtoaddress(self.nodes[0].getnewaddress(), self.nodes[1].getbalance(), "", "", True) - self.sync_all() - self.nodes[0].generate(1) + self.nodes[1].generate(1) self.sync_all() for i in range(0,20): @@ -567,29 +544,27 @@ class RawTransactionsTest(BitcoinTestFramework): self.nodes[0].generate(1) self.sync_all() - #fund a tx with ~20 small inputs + # Fund a tx with ~20 small inputs. inputs = [] outputs = {self.nodes[0].getnewaddress():0.15,self.nodes[0].getnewaddress():0.04} rawtx = self.nodes[1].createrawtransaction(inputs, outputs) fundedTx = self.nodes[1].fundrawtransaction(rawtx) - #create same transaction over sendtoaddress + # Create same transaction over sendtoaddress. txId = self.nodes[1].sendmany("", outputs) signedFee = self.nodes[1].getrawmempool(True)[txId]['fee'] - #compare fee + # Compare fee. feeDelta = Decimal(fundedTx['fee']) - Decimal(signedFee) assert feeDelta >= 0 and feeDelta <= self.fee_tolerance * 19 #~19 inputs def test_many_inputs_send(self): - ############################################# - # multiple (~19) inputs tx test | sign/send # - ############################################# + """Multiple (~19) inputs tx test | sign/send.""" + self.log.info("Test fundrawtxn sign+send with many inputs") - #again, empty node1, send some small coins from node0 to node1 + # Again, empty node1, send some small coins from node0 to node1. self.nodes[1].sendtoaddress(self.nodes[0].getnewaddress(), self.nodes[1].getbalance(), "", "", True) - self.sync_all() - self.nodes[0].generate(1) + self.nodes[1].generate(1) self.sync_all() for i in range(0,20): @@ -597,7 +572,7 @@ class RawTransactionsTest(BitcoinTestFramework): self.nodes[0].generate(1) self.sync_all() - #fund a tx with ~20 small inputs + # Fund a tx with ~20 small inputs. oldBalance = self.nodes[0].getbalance() inputs = [] @@ -606,15 +581,12 @@ class RawTransactionsTest(BitcoinTestFramework): fundedTx = self.nodes[1].fundrawtransaction(rawtx) fundedAndSignedTx = self.nodes[1].signrawtransactionwithwallet(fundedTx['hex']) self.nodes[1].sendrawtransaction(fundedAndSignedTx['hex']) - self.sync_all() - self.nodes[0].generate(1) + self.nodes[1].generate(1) self.sync_all() assert_equal(oldBalance+Decimal('50.19000000'), self.nodes[0].getbalance()) #0.19+block reward def test_op_return(self): - ##################################################### - # test fundrawtransaction with OP_RETURN and no vin # - ##################################################### + self.log.info("Test fundrawtxn with OP_RETURN and no vin") rawtx = "0100000000010000000000000000066a047465737400000000" dec_tx = self.nodes[2].decoderawtransaction(rawtx) @@ -629,9 +601,7 @@ class RawTransactionsTest(BitcoinTestFramework): assert_equal(len(dec_tx['vout']), 2) # one change output added def test_watchonly(self): - ################################################## - # test a fundrawtransaction using only watchonly # - ################################################## + self.log.info("Test fundrawtxn using only watchonly") inputs = [] outputs = {self.nodes[2].getnewaddress(): self.watchonly_amount / 2} @@ -646,15 +616,13 @@ class RawTransactionsTest(BitcoinTestFramework): assert_greater_than(result["changepos"], -1) def test_all_watched_funds(self): - ############################################################### - # test fundrawtransaction using the entirety of watched funds # - ############################################################### + self.log.info("Test fundrawtxn using entirety of watched funds") inputs = [] outputs = {self.nodes[2].getnewaddress(): self.watchonly_amount} rawtx = self.nodes[3].createrawtransaction(inputs, outputs) - # Backward compatibility test (2nd param is includeWatching) + # Backward compatibility test (2nd param is includeWatching). result = self.nodes[3].fundrawtransaction(rawtx, True) res_dec = self.nodes[0].decoderawtransaction(result["hex"]) assert_equal(len(res_dec["vin"]), 2) @@ -673,11 +641,9 @@ class RawTransactionsTest(BitcoinTestFramework): self.sync_all() def test_option_feerate(self): - ####################### - # Test feeRate option # - ####################### + self.log.info("Test fundrawtxn feeRate option") - # Make sure there is exactly one input so coin selection can't skew the result + # Make sure there is exactly one input so coin selection can't skew the result. assert_equal(len(self.nodes[3].listunspent(1)), 1) inputs = [] @@ -692,9 +658,8 @@ class RawTransactionsTest(BitcoinTestFramework): assert_fee_amount(result3['fee'], count_bytes(result3['hex']), 10 * result_fee_rate) def test_address_reuse(self): - ################################ - # Test no address reuse occurs # - ################################ + """Test no address reuse occurs.""" + self.log.info("Test fundrawtxn does not reuse addresses") rawtx = self.nodes[3].createrawtransaction(inputs=[], outputs={self.nodes[3].getnewaddress(): 1}) result3 = self.nodes[3].fundrawtransaction(rawtx) @@ -705,15 +670,13 @@ class RawTransactionsTest(BitcoinTestFramework): changeaddress += out['scriptPubKey']['addresses'][0] assert changeaddress != "" nextaddr = self.nodes[3].getnewaddress() - # Now the change address key should be removed from the keypool + # Now the change address key should be removed from the keypool. assert changeaddress != nextaddr def test_option_subtract_fee_from_outputs(self): - ###################################### - # Test subtractFeeFromOutputs option # - ###################################### + self.log.info("Test fundrawtxn subtractFeeFromOutputs option") - # Make sure there is exactly one input so coin selection can't skew the result + # Make sure there is exactly one input so coin selection can't skew the result. assert_equal(len(self.nodes[3].listunspent(1)), 1) inputs = [] @@ -744,39 +707,52 @@ class RawTransactionsTest(BitcoinTestFramework): rawtx = self.nodes[3].createrawtransaction(inputs, outputs) result = [self.nodes[3].fundrawtransaction(rawtx), - # split the fee between outputs 0, 2, and 3, but not output 1 + # Split the fee between outputs 0, 2, and 3, but not output 1. self.nodes[3].fundrawtransaction(rawtx, {"subtractFeeFromOutputs": [0, 2, 3]})] dec_tx = [self.nodes[3].decoderawtransaction(result[0]['hex']), self.nodes[3].decoderawtransaction(result[1]['hex'])] - # Nested list of non-change output amounts for each transaction + # Nested list of non-change output amounts for each transaction. output = [[out['value'] for i, out in enumerate(d['vout']) if i != r['changepos']] for d, r in zip(dec_tx, result)] - # List of differences in output amounts between normal and subtractFee transactions + # List of differences in output amounts between normal and subtractFee transactions. share = [o0 - o1 for o0, o1 in zip(output[0], output[1])] - # output 1 is the same in both transactions + # Output 1 is the same in both transactions. assert_equal(share[1], 0) - # the other 3 outputs are smaller as a result of subtractFeeFromOutputs + # The other 3 outputs are smaller as a result of subtractFeeFromOutputs. assert_greater_than(share[0], 0) assert_greater_than(share[2], 0) assert_greater_than(share[3], 0) - # outputs 2 and 3 take the same share of the fee + # Outputs 2 and 3 take the same share of the fee. assert_equal(share[2], share[3]) - # output 0 takes at least as much share of the fee, and no more than 2 satoshis more, than outputs 2 and 3 + # Output 0 takes at least as much share of the fee, and no more than 2 + # satoshis more, than outputs 2 and 3. assert_greater_than_or_equal(share[0], share[2]) assert_greater_than_or_equal(share[2] + Decimal(2e-8), share[0]) - # the fee is the same in both transactions + # The fee is the same in both transactions. assert_equal(result[0]['fee'], result[1]['fee']) - # the total subtracted from the outputs is equal to the fee + # The total subtracted from the outputs is equal to the fee. assert_equal(share[0] + share[2] + share[3], result[0]['fee']) + def test_subtract_fee_with_presets(self): + self.log.info("Test fundrawtxn subtract fee from outputs with preset inputs that are sufficient") + + addr = self.nodes[0].getnewaddress() + txid = self.nodes[0].sendtoaddress(addr, 10) + vout = find_vout_for_address(self.nodes[0], txid, addr) + + rawtx = self.nodes[0].createrawtransaction([{'txid': txid, 'vout': vout}], [{self.nodes[0].getnewaddress(): 5}]) + fundedtx = self.nodes[0].fundrawtransaction(rawtx, {'subtractFeeFromOutputs': [0]}) + signedtx = self.nodes[0].signrawtransactionwithwallet(fundedtx['hex']) + self.nodes[0].sendrawtransaction(signedtx['hex']) + if __name__ == '__main__': RawTransactionsTest().main() diff --git a/test/functional/rpc_getaddressinfo_label_deprecation.py b/test/functional/rpc_getaddressinfo_label_deprecation.py new file mode 100755 index 0000000000..5e739ebede --- /dev/null +++ b/test/functional/rpc_getaddressinfo_label_deprecation.py @@ -0,0 +1,43 @@ +#!/usr/bin/env python3 +# Copyright (c) 2020 The Bitcoin Core developers +# Distributed under the MIT software license, see the accompanying +# file COPYING or http://www.opensource.org/licenses/mit-license.php. +""" +Test deprecation of the RPC getaddressinfo `label` field. It has been +superceded by the `labels` field. + +""" +from test_framework.test_framework import BitcoinTestFramework + +class GetAddressInfoLabelDeprecationTest(BitcoinTestFramework): + def set_test_params(self): + self.num_nodes = 2 + self.setup_clean_chain = False + # Start node[0] with -deprecatedrpc=label, and node[1] without. + self.extra_args = [["-deprecatedrpc=label"], []] + + def skip_test_if_missing_module(self): + self.skip_if_no_wallet() + + def test_label_with_deprecatedrpc_flag(self): + self.log.info("Test getaddressinfo label with -deprecatedrpc flag") + node = self.nodes[0] + address = node.getnewaddress() + info = node.getaddressinfo(address) + assert "label" in info + + def test_label_without_deprecatedrpc_flag(self): + self.log.info("Test getaddressinfo label without -deprecatedrpc flag") + node = self.nodes[1] + address = node.getnewaddress() + info = node.getaddressinfo(address) + assert "label" not in info + + def run_test(self): + """Test getaddressinfo label with and without -deprecatedrpc flag.""" + self.test_label_with_deprecatedrpc_flag() + self.test_label_without_deprecatedrpc_flag() + + +if __name__ == '__main__': + GetAddressInfoLabelDeprecationTest().main() diff --git a/test/functional/rpc_getaddressinfo_labels_purpose_deprecation.py b/test/functional/rpc_getaddressinfo_labels_purpose_deprecation.py new file mode 100755 index 0000000000..3f2e8dee18 --- /dev/null +++ b/test/functional/rpc_getaddressinfo_labels_purpose_deprecation.py @@ -0,0 +1,48 @@ +#!/usr/bin/env python3 +# Copyright (c) 2020-2019 The Bitcoin Core developers +# Distributed under the MIT software license, see the accompanying +# file COPYING or http://www.opensource.org/licenses/mit-license.php. +""" +Test deprecation of RPC getaddressinfo `labels` returning an array +containing a JSON object of `name` and purpose` key-value pairs. It now +returns an array containing only the label name. + +""" +from test_framework.test_framework import BitcoinTestFramework +from test_framework.util import assert_equal + +LABELS_TO_TEST = frozenset({"" , "New 𝅘𝅥𝅯 $<#>&!рыба Label"}) + +class GetAddressInfoLabelsPurposeDeprecationTest(BitcoinTestFramework): + def set_test_params(self): + self.num_nodes = 2 + self.setup_clean_chain = False + # Start node[0] with -deprecatedrpc=labelspurpose and node[1] without. + self.extra_args = [["-deprecatedrpc=labelspurpose"], []] + + def skip_test_if_missing_module(self): + self.skip_if_no_wallet() + + def test_labels(self, node_num, label_name, expected_value): + node = self.nodes[node_num] + address = node.getnewaddress() + if label_name != "": + node.setlabel(address, label_name) + self.log.info(" set label to {}".format(label_name)) + labels = node.getaddressinfo(address)["labels"] + self.log.info(" labels = {}".format(labels)) + assert_equal(labels, expected_value) + + def run_test(self): + """Test getaddressinfo labels with and without -deprecatedrpc flag.""" + self.log.info("Test getaddressinfo labels with -deprecatedrpc flag") + for label in LABELS_TO_TEST: + self.test_labels(node_num=0, label_name=label, expected_value=[{"name": label, "purpose": "receive"}]) + + self.log.info("Test getaddressinfo labels without -deprecatedrpc flag") + for label in LABELS_TO_TEST: + self.test_labels(node_num=1, label_name=label, expected_value=[label]) + + +if __name__ == '__main__': + GetAddressInfoLabelsPurposeDeprecationTest().main() diff --git a/test/functional/rpc_getblockstats.py b/test/functional/rpc_getblockstats.py index efab69ac26..57794ae973 100755 --- a/test/functional/rpc_getblockstats.py +++ b/test/functional/rpc_getblockstats.py @@ -33,6 +33,7 @@ class GetblockstatsTest(BitcoinTestFramework): def set_test_params(self): self.num_nodes = 1 self.setup_clean_chain = True + self.supports_cli = False def get_stats(self): return [self.nodes[0].getblockstats(hash_or_height=self.start_height + i) for i in range(self.max_stat_pos+1)] diff --git a/test/functional/rpc_getdescriptorinfo.py b/test/functional/rpc_getdescriptorinfo.py new file mode 100755 index 0000000000..977dc805ef --- /dev/null +++ b/test/functional/rpc_getdescriptorinfo.py @@ -0,0 +1,65 @@ +#!/usr/bin/env python3 +# Copyright (c) 2019 The Bitcoin Core developers +# Distributed under the MIT software license, see the accompanying +# file COPYING or http://www.opensource.org/licenses/mit-license.php. +"""Test getdescriptorinfo RPC. +""" + +from test_framework.test_framework import BitcoinTestFramework +from test_framework.descriptors import descsum_create +from test_framework.util import ( + assert_equal, + assert_raises_rpc_error, +) + + +class DescriptorTest(BitcoinTestFramework): + def set_test_params(self): + self.num_nodes = 1 + self.extra_args = [["-disablewallet"]] + + def test_desc(self, desc, isrange, issolvable, hasprivatekeys): + info = self.nodes[0].getdescriptorinfo(desc) + assert_equal(info, self.nodes[0].getdescriptorinfo(descsum_create(desc))) + assert_equal(info['descriptor'], descsum_create(desc)) + assert_equal(info['isrange'], isrange) + assert_equal(info['issolvable'], issolvable) + assert_equal(info['hasprivatekeys'], hasprivatekeys) + + def run_test(self): + assert_raises_rpc_error(-1, 'getdescriptorinfo', self.nodes[0].getdescriptorinfo) + assert_raises_rpc_error(-3, 'Expected type string', self.nodes[0].getdescriptorinfo, 1) + assert_raises_rpc_error(-5, 'is not a valid descriptor function', self.nodes[0].getdescriptorinfo, '') + + # P2PK output with the specified public key. + self.test_desc('pk(0279be667ef9dcbbac55a06295ce870b07029bfcdb2dce28d959f2815b16f81798)', isrange=False, issolvable=True, hasprivatekeys=False) + # P2PKH output with the specified public key. + self.test_desc('pkh(02c6047f9441ed7d6d3045406e95c07cd85c778e4b8cef3ca7abac09b95c709ee5)', isrange=False, issolvable=True, hasprivatekeys=False) + # P2WPKH output with the specified public key. + self.test_desc('wpkh(02f9308a019258c31049344f85f89d5229b531c845836f99b08601f113bce036f9)', isrange=False, issolvable=True, hasprivatekeys=False) + # P2SH-P2WPKH output with the specified public key. + self.test_desc('sh(wpkh(03fff97bd5755eeea420453a14355235d382f6472f8568a18b2f057a1460297556))', isrange=False, issolvable=True, hasprivatekeys=False) + # Any P2PK, P2PKH, P2WPKH, or P2SH-P2WPKH output with the specified public key. + self.test_desc('combo(0279be667ef9dcbbac55a06295ce870b07029bfcdb2dce28d959f2815b16f81798)', isrange=False, issolvable=True, hasprivatekeys=False) + # An (overly complicated) P2SH-P2WSH-P2PKH output with the specified public key. + self.test_desc('sh(wsh(pkh(02e493dbf1c10d80f3581e4904930b1404cc6c13900ee0758474fa94abe8c4cd13)))', isrange=False, issolvable=True, hasprivatekeys=False) + # A bare *1-of-2* multisig output with keys in the specified order. + self.test_desc('multi(1,022f8bde4d1a07209355b4a7250a5c5128e88b84bddc619ab7cba8d569b240efe4,025cbdf0646e5db4eaa398f365f2ea7a0e3d419b7e0330e39ce92bddedcac4f9bc)', isrange=False, issolvable=True, hasprivatekeys=False) + # A P2SH *2-of-2* multisig output with keys in the specified order. + self.test_desc('sh(multi(2,022f01e5e15cca351daff3843fb70f3c2f0a1bdd05e5af888a67784ef3e10a2a01,03acd484e2f0c7f65309ad178a9f559abde09796974c57e714c35f110dfc27ccbe))', isrange=False, issolvable=True, hasprivatekeys=False) + # A P2WSH *2-of-3* multisig output with keys in the specified order. + self.test_desc('wsh(multi(2,03a0434d9e47f3c86235477c7b1ae6ae5d3442d49b1943c2b752a68e2a47e247c7,03774ae7f858a9411e5ef4246b70c65aac5649980be5c17891bbec17895da008cb,03d01115d548e7561b15c38f004d734633687cf4419620095bc5b0f47070afe85a))', isrange=False, issolvable=True, hasprivatekeys=False) + # A P2SH-P2WSH *1-of-3* multisig output with keys in the specified order. + self.test_desc('sh(wsh(multi(1,03f28773c2d975288bc7d1d205c3748651b075fbc6610e58cddeeddf8f19405aa8,03499fdf9e895e719cfd64e67f07d38e3226aa7b63678949e6e49b241a60e823e4,02d7924d4f7d43ea965a465ae3095ff41131e5946f3c85f79e44adbcf8e27e080e)))', isrange=False, issolvable=True, hasprivatekeys=False) + # A P2PK output with the public key of the specified xpub. + self.test_desc('pk(tpubD6NzVbkrYhZ4WaWSyoBvQwbpLkojyoTZPRsgXELWz3Popb3qkjcJyJUGLnL4qHHoQvao8ESaAstxYSnhyswJ76uZPStJRJCTKvosUCJZL5B)', isrange=False, issolvable=True, hasprivatekeys=False) + # A P2PKH output with child key *1'/2* of the specified xpub. + self.test_desc("pkh(tpubD6NzVbkrYhZ4WaWSyoBvQwbpLkojyoTZPRsgXELWz3Popb3qkjcJyJUGLnL4qHHoQvao8ESaAstxYSnhyswJ76uZPStJRJCTKvosUCJZL5B/1'/2)", isrange=False, issolvable=True, hasprivatekeys=False) + # A set of P2PKH outputs, but additionally specifies that the specified xpub is a child of a master with fingerprint `d34db33f`, and derived using path `44'/0'/0'`. + self.test_desc("pkh([d34db33f/44'/0'/0']tpubD6NzVbkrYhZ4WaWSyoBvQwbpLkojyoTZPRsgXELWz3Popb3qkjcJyJUGLnL4qHHoQvao8ESaAstxYSnhyswJ76uZPStJRJCTKvosUCJZL5B/1/*)", isrange=True, issolvable=True, hasprivatekeys=False) + # A set of *1-of-2* P2WSH multisig outputs where the first multisig key is the *1/0/`i`* child of the first specified xpub and the second multisig key is the *0/0/`i`* child of the second specified xpub, and `i` is any number in a configurable range (`0-1000` by default). + self.test_desc("wsh(multi(1,tpubD6NzVbkrYhZ4WaWSyoBvQwbpLkojyoTZPRsgXELWz3Popb3qkjcJyJUGLnL4qHHoQvao8ESaAstxYSnhyswJ76uZPStJRJCTKvosUCJZL5B/1/0/*,tpubD6NzVbkrYhZ4WaWSyoBvQwbpLkojyoTZPRsgXELWz3Popb3qkjcJyJUGLnL4qHHoQvao8ESaAstxYSnhyswJ76uZPStJRJCTKvosUCJZL5B/0/0/*))", isrange=True, issolvable=True, hasprivatekeys=False) + + +if __name__ == '__main__': + DescriptorTest().main() diff --git a/test/functional/rpc_help.py b/test/functional/rpc_help.py index 78d6e78aed..027ae368e7 100755 --- a/test/functional/rpc_help.py +++ b/test/functional/rpc_help.py @@ -1,5 +1,5 @@ #!/usr/bin/env python3 -# Copyright (c) 2018 The Bitcoin Core developers +# Copyright (c) 2018-2019 The Bitcoin Core developers # Distributed under the MIT software license, see the accompanying # file COPYING or http://www.opensource.org/licenses/mit-license.php. """Test RPC help output.""" @@ -13,6 +13,7 @@ import os class HelpRpcTest(BitcoinTestFramework): def set_test_params(self): self.num_nodes = 1 + self.supports_cli = False def run_test(self): self.test_categories() diff --git a/test/functional/rpc_invalidateblock.py b/test/functional/rpc_invalidateblock.py index 595b40f7cb..1fdc134f97 100755 --- a/test/functional/rpc_invalidateblock.py +++ b/test/functional/rpc_invalidateblock.py @@ -5,7 +5,7 @@ """Test the invalidateblock RPC.""" from test_framework.test_framework import BitcoinTestFramework -from test_framework.address import ADDRESS_BCRT1_UNSPENDABLE +from test_framework.address import ADDRESS_BCRT1_UNSPENDABLE_DESCRIPTOR from test_framework.util import ( assert_equal, connect_nodes, @@ -62,7 +62,7 @@ class InvalidateTest(BitcoinTestFramework): wait_until(lambda: self.nodes[1].getblockcount() == 4, timeout=5) self.log.info("Verify that we reconsider all ancestors as well") - blocks = self.nodes[1].generatetoaddress(10, ADDRESS_BCRT1_UNSPENDABLE) + blocks = self.nodes[1].generatetodescriptor(10, ADDRESS_BCRT1_UNSPENDABLE_DESCRIPTOR) assert_equal(self.nodes[1].getbestblockhash(), blocks[-1]) # Invalidate the two blocks at the tip self.nodes[1].invalidateblock(blocks[-1]) @@ -74,7 +74,7 @@ class InvalidateTest(BitcoinTestFramework): assert_equal(self.nodes[1].getbestblockhash(), blocks[-1]) self.log.info("Verify that we reconsider all descendants") - blocks = self.nodes[1].generatetoaddress(10, ADDRESS_BCRT1_UNSPENDABLE) + blocks = self.nodes[1].generatetodescriptor(10, ADDRESS_BCRT1_UNSPENDABLE_DESCRIPTOR) assert_equal(self.nodes[1].getbestblockhash(), blocks[-1]) # Invalidate the two blocks at the tip self.nodes[1].invalidateblock(blocks[-2]) diff --git a/test/functional/rpc_misc.py b/test/functional/rpc_misc.py index 8a3f8c6f06..c8517d719e 100755 --- a/test/functional/rpc_misc.py +++ b/test/functional/rpc_misc.py @@ -19,10 +19,18 @@ from test_framework.authproxy import JSONRPCException class RpcMiscTest(BitcoinTestFramework): def set_test_params(self): self.num_nodes = 1 + self.supports_cli = False def run_test(self): node = self.nodes[0] + self.log.info("test CHECK_NONFATAL") + assert_raises_rpc_error( + -1, + "Internal bug detected: 'request.params.size() != 100'", + lambda: node.echo(*[0] * 100), + ) + self.log.info("test getmemoryinfo") memory = node.getmemoryinfo()['locked'] assert_greater_than(memory['used'], 0) diff --git a/test/functional/rpc_named_arguments.py b/test/functional/rpc_named_arguments.py index ecac9c2f82..41b9312969 100755 --- a/test/functional/rpc_named_arguments.py +++ b/test/functional/rpc_named_arguments.py @@ -13,6 +13,7 @@ from test_framework.util import ( class NamedArgumentTest(BitcoinTestFramework): def set_test_params(self): self.num_nodes = 1 + self.supports_cli = False def run_test(self): node = self.nodes[0] diff --git a/test/functional/rpc_net.py b/test/functional/rpc_net.py index 615f9abbe0..376bb35f07 100755 --- a/test/functional/rpc_net.py +++ b/test/functional/rpc_net.py @@ -1,5 +1,5 @@ #!/usr/bin/env python3 -# Copyright (c) 2017-2018 The Bitcoin Core developers +# Copyright (c) 2017-2019 The Bitcoin Core developers # Distributed under the MIT software license, see the accompanying # file COPYING or http://www.opensource.org/licenses/mit-license.php. """Test RPC calls related to net. @@ -45,6 +45,7 @@ class NetTest(BitcoinTestFramework): self.setup_clean_chain = True self.num_nodes = 2 self.extra_args = [["-minrelaytxfee=0.00001000"],["-minrelaytxfee=0.00000500"]] + self.supports_cli = False def run_test(self): self.log.info('Connect nodes both way') diff --git a/test/functional/rpc_preciousblock.py b/test/functional/rpc_preciousblock.py index 0663ffdf5b..8386e47411 100755 --- a/test/functional/rpc_preciousblock.py +++ b/test/functional/rpc_preciousblock.py @@ -36,6 +36,7 @@ class PreciousTest(BitcoinTestFramework): def set_test_params(self): self.setup_clean_chain = True self.num_nodes = 3 + self.supports_cli = False def setup_network(self): self.setup_nodes() diff --git a/test/functional/rpc_psbt.py b/test/functional/rpc_psbt.py index 61572654e0..3a63377545 100755 --- a/test/functional/rpc_psbt.py +++ b/test/functional/rpc_psbt.py @@ -32,6 +32,7 @@ class PSBTTest(BitcoinTestFramework): ["-walletrbf=0"], [] ] + self.supports_cli = False def skip_test_if_missing_module(self): self.skip_if_no_wallet() @@ -192,12 +193,20 @@ class PSBTTest(BitcoinTestFramework): psbt_orig = self.nodes[0].createpsbt([{"txid":txid1, "vout":vout1}, {"txid":txid2, "vout":vout2}], {self.nodes[0].getnewaddress():25.999}) # Update psbts, should only have data for one input and not the other - psbt1 = self.nodes[1].walletprocesspsbt(psbt_orig)['psbt'] + psbt1 = self.nodes[1].walletprocesspsbt(psbt_orig, False, "ALL")['psbt'] psbt1_decoded = self.nodes[0].decodepsbt(psbt1) assert psbt1_decoded['inputs'][0] and not psbt1_decoded['inputs'][1] - psbt2 = self.nodes[2].walletprocesspsbt(psbt_orig)['psbt'] + # Check that BIP32 path was added + assert "bip32_derivs" in psbt1_decoded['inputs'][0] + psbt2 = self.nodes[2].walletprocesspsbt(psbt_orig, False, "ALL", False)['psbt'] psbt2_decoded = self.nodes[0].decodepsbt(psbt2) assert not psbt2_decoded['inputs'][0] and psbt2_decoded['inputs'][1] + # Check that BIP32 paths were not added + assert "bip32_derivs" not in psbt2_decoded['inputs'][1] + + # Sign PSBTs (workaround issue #18039) + psbt1 = self.nodes[1].walletprocesspsbt(psbt_orig)['psbt'] + psbt2 = self.nodes[2].walletprocesspsbt(psbt_orig)['psbt'] # Combine, finalize, and send the psbts combined = self.nodes[0].combinepsbt([psbt1, psbt2]) @@ -230,16 +239,18 @@ class PSBTTest(BitcoinTestFramework): # Same construction without optional arguments psbtx_info = self.nodes[0].walletcreatefundedpsbt([{"txid":unspent["txid"], "vout":unspent["vout"]}], [{self.nodes[2].getnewaddress():unspent["amount"]+1}]) decoded_psbt = self.nodes[0].decodepsbt(psbtx_info["psbt"]) - for tx_in in decoded_psbt["tx"]["vin"]: + for tx_in, psbt_in in zip(decoded_psbt["tx"]["vin"], decoded_psbt["inputs"]): assert_equal(tx_in["sequence"], MAX_BIP125_RBF_SEQUENCE) + assert "bip32_derivs" in psbt_in assert_equal(decoded_psbt["tx"]["locktime"], 0) # Same construction without optional arguments, for a node with -walletrbf=0 unspent1 = self.nodes[1].listunspent()[0] psbtx_info = self.nodes[1].walletcreatefundedpsbt([{"txid":unspent1["txid"], "vout":unspent1["vout"]}], [{self.nodes[2].getnewaddress():unspent1["amount"]+1}], block_height) decoded_psbt = self.nodes[1].decodepsbt(psbtx_info["psbt"]) - for tx_in in decoded_psbt["tx"]["vin"]: + for tx_in, psbt_in in zip(decoded_psbt["tx"]["vin"], decoded_psbt["inputs"]): assert_greater_than(tx_in["sequence"], MAX_BIP125_RBF_SEQUENCE) + assert "bip32_derivs" in psbt_in # Make sure change address wallet does not have P2SH innerscript access to results in success # when attempting BnB coin selection @@ -416,5 +427,29 @@ class PSBTTest(BitcoinTestFramework): analyzed = self.nodes[0].analyzepsbt(signed) assert analyzed['inputs'][0]['has_utxo'] and analyzed['inputs'][0]['is_final'] and analyzed['next'] == 'extractor' + self.log.info("PSBT spending unspendable outputs should have error message and Creator as next") + analysis = self.nodes[0].analyzepsbt('cHNidP8BAJoCAAAAAljoeiG1ba8MI76OcHBFbDNvfLqlyHV5JPVFiHuyq911AAAAAAD/////g40EJ9DsZQpoqka7CwmK6kQiwHGyyng1Kgd5WdB86h0BAAAAAP////8CcKrwCAAAAAAWAEHYXCtx0AYLCcmIauuBXlCZHdoSTQDh9QUAAAAAFv8/wADXYP/7//////8JxOh0LR2HAI8AAAAAAAEBIADC6wsAAAAAF2oUt/X69ELjeX2nTof+fZ10l+OyAokDAQcJAwEHEAABAACAAAEBIADC6wsAAAAAF2oUt/X69ELjeX2nTof+fZ10l+OyAokDAQcJAwEHENkMak8AAAAA') + assert_equal(analysis['next'], 'creator') + assert_equal(analysis['error'], 'PSBT is not valid. Input 0 spends unspendable output') + + self.log.info("PSBT with invalid values should have error message and Creator as next") + analysis = self.nodes[0].analyzepsbt('cHNidP8BAHECAAAAAfA00BFgAm6tp86RowwH6BMImQNL5zXUcTT97XoLGz0BAAAAAAD/////AgD5ApUAAAAAFgAUKNw0x8HRctAgmvoevm4u1SbN7XL87QKVAAAAABYAFPck4gF7iL4NL4wtfRAKgQbghiTUAAAAAAABAR8AgIFq49AHABYAFJUDtxf2PHo641HEOBOAIvFMNTr2AAAA') + assert_equal(analysis['next'], 'creator') + assert_equal(analysis['error'], 'PSBT is not valid. Input 0 has invalid value') + + self.log.info("PSBT with signed, but not finalized, inputs should have Finalizer as next") + analysis = self.nodes[0].analyzepsbt('cHNidP8BAHECAAAAAZYezcxdnbXoQCmrD79t/LzDgtUo9ERqixk8wgioAobrAAAAAAD9////AlDDAAAAAAAAFgAUy/UxxZuzZswcmFnN/E9DGSiHLUsuGPUFAAAAABYAFLsH5o0R38wXx+X2cCosTMCZnQ4baAAAAAABAR8A4fUFAAAAABYAFOBI2h5thf3+Lflb2LGCsVSZwsltIgIC/i4dtVARCRWtROG0HHoGcaVklzJUcwo5homgGkSNAnJHMEQCIGx7zKcMIGr7cEES9BR4Kdt/pzPTK3fKWcGyCJXb7MVnAiALOBgqlMH4GbC1HDh/HmylmO54fyEy4lKde7/BT/PWxwEBAwQBAAAAIgYC/i4dtVARCRWtROG0HHoGcaVklzJUcwo5homgGkSNAnIYDwVpQ1QAAIABAACAAAAAgAAAAAAAAAAAAAAiAgL+CIiB59NSCssOJRGiMYQK1chahgAaaJpIXE41Cyir+xgPBWlDVAAAgAEAAIAAAACAAQAAAAAAAAAA') + assert_equal(analysis['next'], 'finalizer') + + analysis = self.nodes[0].analyzepsbt('cHNidP8BAHECAAAAAfA00BFgAm6tp86RowwH6BMImQNL5zXUcTT97XoLGz0BAAAAAAD/////AgCAgWrj0AcAFgAUKNw0x8HRctAgmvoevm4u1SbN7XL87QKVAAAAABYAFPck4gF7iL4NL4wtfRAKgQbghiTUAAAAAAABAR8A8gUqAQAAABYAFJUDtxf2PHo641HEOBOAIvFMNTr2AAAA') + assert_equal(analysis['next'], 'creator') + assert_equal(analysis['error'], 'PSBT is not valid. Output amount invalid') + + analysis = self.nodes[0].analyzepsbt('cHNidP8BAJoCAAAAAkvEW8NnDtdNtDpsmze+Ht2LH35IJcKv00jKAlUs21RrAwAAAAD/////S8Rbw2cO1020OmybN74e3Ysffkglwq/TSMoCVSzbVGsBAAAAAP7///8CwLYClQAAAAAWABSNJKzjaUb3uOxixsvh1GGE3fW7zQD5ApUAAAAAFgAUKNw0x8HRctAgmvoevm4u1SbN7XIAAAAAAAEAnQIAAAACczMa321tVHuN4GKWKRncycI22aX3uXgwSFUKM2orjRsBAAAAAP7///9zMxrfbW1Ue43gYpYpGdzJwjbZpfe5eDBIVQozaiuNGwAAAAAA/v///wIA+QKVAAAAABl2qRT9zXUVA8Ls5iVqynLHe5/vSe1XyYisQM0ClQAAAAAWABRmWQUcjSjghQ8/uH4Bn/zkakwLtAAAAAAAAQEfQM0ClQAAAAAWABRmWQUcjSjghQ8/uH4Bn/zkakwLtAAAAA==') + assert_equal(analysis['next'], 'creator') + assert_equal(analysis['error'], 'PSBT is not valid. Input 0 specifies invalid prevout') + + assert_raises_rpc_error(-25, 'Missing inputs', self.nodes[0].walletprocesspsbt, 'cHNidP8BAJoCAAAAAkvEW8NnDtdNtDpsmze+Ht2LH35IJcKv00jKAlUs21RrAwAAAAD/////S8Rbw2cO1020OmybN74e3Ysffkglwq/TSMoCVSzbVGsBAAAAAP7///8CwLYClQAAAAAWABSNJKzjaUb3uOxixsvh1GGE3fW7zQD5ApUAAAAAFgAUKNw0x8HRctAgmvoevm4u1SbN7XIAAAAAAAEAnQIAAAACczMa321tVHuN4GKWKRncycI22aX3uXgwSFUKM2orjRsBAAAAAP7///9zMxrfbW1Ue43gYpYpGdzJwjbZpfe5eDBIVQozaiuNGwAAAAAA/v///wIA+QKVAAAAABl2qRT9zXUVA8Ls5iVqynLHe5/vSe1XyYisQM0ClQAAAAAWABRmWQUcjSjghQ8/uH4Bn/zkakwLtAAAAAAAAQEfQM0ClQAAAAAWABRmWQUcjSjghQ8/uH4Bn/zkakwLtAAAAA==') + if __name__ == '__main__': PSBTTest().main() diff --git a/test/functional/rpc_rawtransaction.py b/test/functional/rpc_rawtransaction.py index ca0d47a5f8..14cad3d1b8 100755 --- a/test/functional/rpc_rawtransaction.py +++ b/test/functional/rpc_rawtransaction.py @@ -53,6 +53,7 @@ class RawTransactionsTest(BitcoinTestFramework): ["-txindex"], ["-txindex"], ] + self.supports_cli = False def skip_test_if_missing_module(self): self.skip_if_no_wallet() @@ -209,7 +210,7 @@ class RawTransactionsTest(BitcoinTestFramework): rawtx = self.nodes[2].signrawtransactionwithwallet(rawtx) # This will raise an exception since there are missing inputs - assert_raises_rpc_error(-25, "Missing inputs", self.nodes[2].sendrawtransaction, rawtx['hex']) + assert_raises_rpc_error(-25, "bad-txns-inputs-missingorspent", self.nodes[2].sendrawtransaction, rawtx['hex']) ##################################### # getrawtransaction with block hash # @@ -454,7 +455,7 @@ class RawTransactionsTest(BitcoinTestFramework): # Thus, testmempoolaccept should reject testres = self.nodes[2].testmempoolaccept([rawTxSigned['hex']], 0.00001000)[0] assert_equal(testres['allowed'], False) - assert_equal(testres['reject-reason'], '256: absurdly-high-fee') + assert_equal(testres['reject-reason'], 'absurdly-high-fee') # and sendrawtransaction should throw assert_raises_rpc_error(-26, "absurdly-high-fee", self.nodes[2].sendrawtransaction, rawTxSigned['hex'], 0.00001000) # and the following calls should both succeed @@ -478,7 +479,7 @@ class RawTransactionsTest(BitcoinTestFramework): # Thus, testmempoolaccept should reject testres = self.nodes[2].testmempoolaccept([rawTxSigned['hex']])[0] assert_equal(testres['allowed'], False) - assert_equal(testres['reject-reason'], '256: absurdly-high-fee') + assert_equal(testres['reject-reason'], 'absurdly-high-fee') # and sendrawtransaction should throw assert_raises_rpc_error(-26, "absurdly-high-fee", self.nodes[2].sendrawtransaction, rawTxSigned['hex']) # and the following calls should both succeed diff --git a/test/functional/rpc_scantxoutset.py b/test/functional/rpc_scantxoutset.py index 9f94d11a93..c3d34be0dd 100755 --- a/test/functional/rpc_scantxoutset.py +++ b/test/functional/rpc_scantxoutset.py @@ -54,7 +54,7 @@ class ScantxoutsetTest(BitcoinTestFramework): self.log.info("Stop node, remove wallet, mine again some blocks...") self.stop_node(0) - shutil.rmtree(os.path.join(self.nodes[0].datadir, "regtest", 'wallets')) + shutil.rmtree(os.path.join(self.nodes[0].datadir, self.chain, 'wallets')) self.start_node(0) self.nodes[0].generate(110) @@ -116,5 +116,12 @@ class ScantxoutsetTest(BitcoinTestFramework): assert_equal(descriptors(self.nodes[0].scantxoutset("start", [ "combo(tprv8ZgxMBicQKsPd7Uf69XL1XwhmjHopUGep8GuEiJDZmbQz6o58LninorQAfcKZWARbtRtfnLcJ5MQ2AtHcQJCCRUcMRvmDUjyEmNUWwx8UbK/1/1/0)"])), ["pkh([0c5f9a1e/1/1/0]03e1c5b6e650966971d7e71ef2674f80222752740fc1dfd63bbbd220d2da9bd0fb)#cxmct4w8"]) assert_equal(descriptors(self.nodes[0].scantxoutset("start", [ {"desc": "combo(tpubD6NzVbkrYhZ4WaWSyoBvQwbpLkojyoTZPRsgXELWz3Popb3qkjcJyJUGLnL4qHHoQvao8ESaAstxYSnhyswJ76uZPStJRJCTKvosUCJZL5B/1/1/*)", "range": 1500}])), ['pkh([0c5f9a1e/1/1/0]03e1c5b6e650966971d7e71ef2674f80222752740fc1dfd63bbbd220d2da9bd0fb)#cxmct4w8', 'pkh([0c5f9a1e/1/1/1500]03832901c250025da2aebae2bfb38d5c703a57ab66ad477f9c578bfbcd78abca6f)#vchwd07g', 'pkh([0c5f9a1e/1/1/1]030d820fc9e8211c4169be8530efbc632775d8286167afd178caaf1089b77daba7)#z2t3ypsa']) + # Check that status and abort don't need second arg + assert_equal(self.nodes[0].scantxoutset("status"), None) + assert_equal(self.nodes[0].scantxoutset("abort"), False) + + # Check that second arg is needed for start + assert_raises_rpc_error(-1, "scanobjects argument is required for the start action", self.nodes[0].scantxoutset, "start") + if __name__ == '__main__': ScantxoutsetTest().main() diff --git a/test/functional/rpc_setban.py b/test/functional/rpc_setban.py index b1d2b6f431..1cc1fb164b 100755 --- a/test/functional/rpc_setban.py +++ b/test/functional/rpc_setban.py @@ -26,7 +26,7 @@ class SetBanTests(BitcoinTestFramework): self.nodes[1].setban("127.0.0.1", "add") # Node 0 should not be able to reconnect - with self.nodes[1].assert_debug_log(expected_msgs=['dropped (banned)\n'], timeout=5): + with self.nodes[1].assert_debug_log(expected_msgs=['dropped (banned)\n'], timeout=50): self.restart_node(1, []) self.nodes[0].addnode("127.0.0.1:" + str(p2p_port(1)), "onetry") diff --git a/test/functional/rpc_txoutproof.py b/test/functional/rpc_txoutproof.py index 8913b8698d..ca8be42d3b 100755 --- a/test/functional/rpc_txoutproof.py +++ b/test/functional/rpc_txoutproof.py @@ -1,5 +1,5 @@ #!/usr/bin/env python3 -# Copyright (c) 2014-2018 The Bitcoin Core developers +# Copyright (c) 2014-2019 The Bitcoin Core developers # Distributed under the MIT software license, see the accompanying # file COPYING or http://www.opensource.org/licenses/mit-license.php. """Test gettxoutproof and verifytxoutproof RPCs.""" diff --git a/test/functional/rpc_users.py b/test/functional/rpc_users.py index 8bbb3c04fa..b75ce15f2e 100755 --- a/test/functional/rpc_users.py +++ b/test/functional/rpc_users.py @@ -1,5 +1,5 @@ #!/usr/bin/env python3 -# Copyright (c) 2015-2018 The Bitcoin Core developers +# Copyright (c) 2015-2019 The Bitcoin Core developers # Distributed under the MIT software license, see the accompanying # file COPYING or http://www.opensource.org/licenses/mit-license.php. """Test multiple RPC users.""" @@ -35,6 +35,7 @@ def call_with_auth(node, user, password): class HTTPBasicsTest(BitcoinTestFramework): def set_test_params(self): self.num_nodes = 2 + self.supports_cli = False def setup_chain(self): super().setup_chain() diff --git a/test/functional/rpc_whitelist.py b/test/functional/rpc_whitelist.py new file mode 100755 index 0000000000..219132410b --- /dev/null +++ b/test/functional/rpc_whitelist.py @@ -0,0 +1,100 @@ +#!/usr/bin/env python3 +# Copyright (c) 2017-2019 The Bitcoin Core developers +# Distributed under the MIT software license, see the accompanying +# file COPYING or http://www.opensource.org/licenses/mit-license.php. +""" +A test for RPC users with restricted permissions +""" +from test_framework.test_framework import BitcoinTestFramework +import os +from test_framework.util import ( + get_datadir_path, + assert_equal, + str_to_b64str +) +import http.client +import urllib.parse + +def rpccall(node, user, method): + url = urllib.parse.urlparse(node.url) + headers = {"Authorization": "Basic " + str_to_b64str('{}:{}'.format(user[0], user[3]))} + conn = http.client.HTTPConnection(url.hostname, url.port) + conn.connect() + conn.request('POST', '/', '{"method": "' + method + '"}', headers) + resp = conn.getresponse() + conn.close() + return resp + + +class RPCWhitelistTest(BitcoinTestFramework): + def set_test_params(self): + self.num_nodes = 1 + + def setup_chain(self): + super().setup_chain() + # 0 => Username + # 1 => Password (Hashed) + # 2 => Permissions + # 3 => Password Plaintext + self.users = [ + ["user1", "50358aa884c841648e0700b073c32b2e$b73e95fff0748cc0b517859d2ca47d9bac1aa78231f3e48fa9222b612bd2083e", "getbestblockhash,getblockcount,", "12345"], + ["user2", "8650ba41296f62092377a38547f361de$4620db7ba063ef4e2f7249853e9f3c5c3592a9619a759e3e6f1c63f2e22f1d21", "getblockcount", "54321"] + ] + # For exceptions + self.strange_users = [ + # Test empty + ["strangedude", "62d67dffec03836edd698314f1b2be62$c2fb4be29bb0e3646298661123cf2d8629640979cabc268ef05ea613ab54068d", ":", "s7R4nG3R7H1nGZ"], + ["strangedude2", "575c012c7fe4b1e83b9d809412da3ef7$09f448d0acfc19924dd62ecb96004d3c2d4b91f471030dfe43c6ea64a8f658c1", "", "s7R4nG3R7H1nGZ"], + # Test trailing comma + ["strangedude3", "23189c561b5975a56f4cf94030495d61$3a2f6aac26351e2257428550a553c4c1979594e36675bbd3db692442387728c0", ":getblockcount,", "s7R4nG3R7H1nGZ"], + # Test overwrite + ["strangedude4", "990c895760a70df83949e8278665e19a$8f0906f20431ff24cb9e7f5b5041e4943bdf2a5c02a19ef4960dcf45e72cde1c", ":getblockcount, getbestblockhash", "s7R4nG3R7H1nGZ"], + ["strangedude4", "990c895760a70df83949e8278665e19a$8f0906f20431ff24cb9e7f5b5041e4943bdf2a5c02a19ef4960dcf45e72cde1c", ":getblockcount", "s7R4nG3R7H1nGZ"], + # Testing the same permission twice + ["strangedude5", "d12c6e962d47a454f962eb41225e6ec8$2dd39635b155536d3c1a2e95d05feff87d5ba55f2d5ff975e6e997a836b717c9", ":getblockcount,getblockcount", "s7R4nG3R7H1nGZ"] + ] + # These commands shouldn't be allowed for any user to test failures + self.never_allowed = ["getnetworkinfo"] + with open(os.path.join(get_datadir_path(self.options.tmpdir, 0), "bitcoin.conf"), 'a', encoding='utf8') as f: + f.write("\nrpcwhitelistdefault=0\n") + for user in self.users: + f.write("rpcauth=" + user[0] + ":" + user[1] + "\n") + f.write("rpcwhitelist=" + user[0] + ":" + user[2] + "\n") + # Special cases + for strangedude in self.strange_users: + f.write("rpcauth=" + strangedude[0] + ":" + strangedude[1] + "\n") + f.write("rpcwhitelist=" + strangedude[0] + strangedude[2] + "\n") + + + def run_test(self): + for user in self.users: + permissions = user[2].replace(" ", "").split(",") + # Pop all empty items + i = 0 + while i < len(permissions): + if permissions[i] == '': + permissions.pop(i) + + i += 1 + for permission in permissions: + self.log.info("[" + user[0] + "]: Testing a permitted permission (" + permission + ")") + assert_equal(200, rpccall(self.nodes[0], user, permission).status) + for permission in self.never_allowed: + self.log.info("[" + user[0] + "]: Testing a non permitted permission (" + permission + ")") + assert_equal(403, rpccall(self.nodes[0], user, permission).status) + # Now test the strange users + for permission in self.never_allowed: + self.log.info("Strange test 1") + assert_equal(403, rpccall(self.nodes[0], self.strange_users[0], permission).status) + for permission in self.never_allowed: + self.log.info("Strange test 2") + assert_equal(403, rpccall(self.nodes[0], self.strange_users[1], permission).status) + self.log.info("Strange test 3") + assert_equal(200, rpccall(self.nodes[0], self.strange_users[2], "getblockcount").status) + self.log.info("Strange test 4") + assert_equal(403, rpccall(self.nodes[0], self.strange_users[3], "getbestblockhash").status) + self.log.info("Strange test 5") + assert_equal(200, rpccall(self.nodes[0], self.strange_users[4], "getblockcount").status) + +if __name__ == "__main__": + RPCWhitelistTest().main() diff --git a/test/functional/test-shell.md b/test/functional/test-shell.md new file mode 100644 index 0000000000..f6ea9ef682 --- /dev/null +++ b/test/functional/test-shell.md @@ -0,0 +1,186 @@ +Test Shell for Interactive Environments +========================================= + +This document describes how to use the `TestShell` submodule in the functional +test suite. + +The `TestShell` submodule extends the `BitcoinTestFramework` functionality to +external interactive environments for prototyping and educational purposes. Just +like `BitcoinTestFramework`, the `TestShell` allows the user to: + +* Manage regtest bitcoind subprocesses. +* Access RPC interfaces of the underlying bitcoind instances. +* Log events to the functional test logging utility. + +The `TestShell` can be useful in interactive environments where it is necessary +to extend the object lifetime of the underlying `BitcoinTestFramework` between +user inputs. Such environments include the Python3 command line interpreter or +[Jupyter](https://jupyter.org/) notebooks running a Python3 kernel. + +## 1. Requirements + +* Python3 +* `bitcoind` built in the same repository as the `TestShell`. + +## 2. Importing `TestShell` from the Bitcoin Core repository + +We can import the `TestShell` by adding the path of the Bitcoin Core +`test_framework` module to the beginning of the PATH variable, and then +importing the `TestShell` class from the `test_shell` sub-package. + +``` +>>> import sys +>>> sys.path.insert(0, "/path/to/bitcoin/test/functional") +>>> from test_framework.test_shell import TestShell +``` + +The following `TestShell` methods manage the lifetime of the underlying bitcoind +processes and logging utilities. + +* `TestShell.setup()` +* `TestShell.shutdown()` + +The `TestShell` inherits all `BitcoinTestFramework` members and methods, such +as: +* `TestShell.nodes[index].rpc_method()` +* `TestShell.log.info("Custom log message")` + +The following sections demonstrate how to initialize, run, and shut down a +`TestShell` object. + +## 3. Initializing a `TestShell` object + +``` +>>> test = TestShell().setup(num_nodes=2, setup_clean_chain=True) +20XX-XX-XXTXX:XX:XX.XXXXXXX TestFramework (INFO): Initializing test directory /path/to/bitcoin_func_test_XXXXXXX +``` +The `TestShell` forwards all functional test parameters of the parent +`BitcoinTestFramework` object. The full set of argument keywords which can be +used to initialize the `TestShell` can be found in [section +#6](#custom-testshell-parameters) of this document. + +**Note: Running multiple instances of `TestShell` is not allowed.** Running a +single process also ensures that logging remains consolidated in the same +temporary folder. If you need more bitcoind nodes than set by default (1), +simply increase the `num_nodes` parameter during setup. + +``` +>>> test2 = TestShell().setup() +TestShell is already running! +``` + +## 4. Interacting with the `TestShell` + +Unlike the `BitcoinTestFramework` class, the `TestShell` keeps the underlying +Bitcoind subprocesses (nodes) and logging utilities running until the user +explicitly shuts down the `TestShell` object. + +During the time between the `setup` and `shutdown` calls, all `bitcoind` node +processes and `BitcoinTestFramework` convenience methods can be accessed +interactively. + +**Example: Mining a regtest chain** + +By default, the `TestShell` nodes are initialized with a clean chain. This means +that each node of the `TestShell` is initialized with a block height of 0. + +``` +>>> test.nodes[0].getblockchaininfo()["blocks"] +0 +``` + +We now let the first node generate 101 regtest blocks, and direct the coinbase +rewards to a wallet address owned by the mining node. + +``` +>>> address = test.nodes[0].getnewaddress() +>>> test.nodes[0].generatetoaddress(101, address) +['2b98dd0044aae6f1cca7f88a0acf366a4bfe053c7f7b00da3c0d115f03d67efb', ... +``` +Since the two nodes are both initialized by default to establish an outbound +connection to each other during `setup`, the second node's chain will include +the mined blocks as soon as they propagate. + +``` +>>> test.nodes[1].getblockchaininfo()["blocks"] +101 +``` +The block rewards from the first block are now spendable by the wallet of the +first node. + +``` +>>> test.nodes[0].getbalance() +Decimal('50.00000000') +``` + +We can also log custom events to the logger. + +``` +>>> test.nodes[0].log.info("Successfully mined regtest chain!") +20XX-XX-XXTXX:XX:XX.XXXXXXX TestFramework.node0 (INFO): Successfully mined regtest chain! +``` + +**Note: Please also consider the functional test +[readme](../test/functional/README.md), which provides an overview of the +test-framework**. Modules such as +[key.py](../test/functional/test_framework/key.py), +[script.py](../test/functional/test_framework/script.py) and +[messages.py](../test/functional/test_framework/messages.py) are particularly +useful in constructing objects which can be passed to the bitcoind nodes managed +by a running `TestShell` object. + +## 5. Shutting the `TestShell` down + +Shutting down the `TestShell` will safely tear down all running bitcoind +instances and remove all temporary data and logging directories. + +``` +>>> test.shutdown() +20XX-XX-XXTXX:XX:XX.XXXXXXX TestFramework (INFO): Stopping nodes +20XX-XX-XXTXX:XX:XX.XXXXXXX TestFramework (INFO): Cleaning up /path/to/bitcoin_func_test_XXXXXXX on exit +20XX-XX-XXTXX:XX:XX.XXXXXXX TestFramework (INFO): Tests successful +``` +To prevent the logs from being removed after a shutdown, simply set the +`TestShell.options.nocleanup` member to `True`. +``` +>>> test.options.nocleanup = True +>>> test.shutdown() +20XX-XX-XXTXX:XX:XX.XXXXXXX TestFramework (INFO): Stopping nodes +20XX-XX-XXTXX:XX:XX.XXXXXXX TestFramework (INFO): Not cleaning up dir /path/to/bitcoin_func_test_XXXXXXX on exit +20XX-XX-XXTXX:XX:XX.XXXXXXX TestFramework (INFO): Tests successful +``` + +The following utility consolidates logs from the bitcoind nodes and the +underlying `BitcoinTestFramework`: + +* `/path/to/bitcoin/test/functional/combine_logs.py + '/path/to/bitcoin_func_test_XXXXXXX'` + +## 6. Custom `TestShell` parameters + +The `TestShell` object initializes with the default settings inherited from the +`BitcoinTestFramework` class. The user can override these in +`TestShell.setup(key=value)`. + +**Note:** `TestShell.reset()` will reset test parameters to default values and +can be called after the TestShell is shut down. + +| Test parameter key | Default Value | Description | +|---|---|---| +| `bind_to_localhost_only` | `True` | Binds bitcoind RPC services to `127.0.0.1` if set to `True`.| +| `cachedir` | `"/path/to/bitcoin/test/cache"` | Sets the bitcoind datadir directory. | +| `chain` | `"regtest"` | Sets the chain-type for the underlying test bitcoind processes. | +| `configfile` | `"/path/to/bitcoin/test/config.ini"` | Sets the location of the test framework config file. | +| `coveragedir` | `None` | Records bitcoind RPC test coverage into this directory if set. | +| `loglevel` | `INFO` | Logs events at this level and higher. Can be set to `DEBUG`, `INFO`, `WARNING`, `ERROR` or `CRITICAL`. | +| `nocleanup` | `False` | Cleans up temporary test directory if set to `True` during `shutdown`. | +| `noshutdown` | `False` | Does not stop bitcoind instances after `shutdown` if set to `True`. | +| `num_nodes` | `1` | Sets the number of initialized bitcoind processes. | +| `perf` | False | Profiles running nodes with `perf` for the duration of the test if set to `True`. | +| `rpc_timeout` | `60` | Sets the RPC server timeout for the underlying bitcoind processes. | +| `setup_clean_chain` | `False` | Initializes an empty blockchain by default. A 199-block-long chain is initialized if set to `True`. | +| `randomseed` | Random Integer | `TestShell.options.randomseed` is a member of `TestShell` which can be accessed during a test to seed a random generator. User can override default with a constant value for reproducible test runs. | +| `supports_cli` | `False` | Whether the bitcoin-cli utility is compiled and available for the test. | +| `tmpdir` | `"/var/folders/.../"` | Sets directory for test logs. Will be deleted upon a successful test run unless `nocleanup` is set to `True` | +| `trace_rpc` | `False` | Logs all RPC calls if set to `True`. | +| `usecli` | `False` | Uses the bitcoin-cli interface for all bitcoind commands instead of directly calling the RPC server. Requires `supports_cli`. | diff --git a/test/functional/test_framework/address.py b/test/functional/test_framework/address.py index 194f2f061b..6a7e91216a 100644 --- a/test/functional/test_framework/address.py +++ b/test/functional/test_framework/address.py @@ -12,6 +12,9 @@ from .util import hex_str_to_bytes from . import segwit_addr ADDRESS_BCRT1_UNSPENDABLE = 'bcrt1qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq3xueyj' +ADDRESS_BCRT1_UNSPENDABLE_DESCRIPTOR = 'addr(bcrt1qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq3xueyj)#juyq9d97' +# Coins sent to this address can be spent with a witness stack of just OP_TRUE +ADDRESS_BCRT1_P2WSH_OP_TRUE = 'bcrt1qft5p2uhsdcdc3l2ua4ap5qqfg4pjaqlp250x7us7a8qqhrxrxfsqseac85' class AddressType(enum.Enum): diff --git a/test/functional/test_framework/descriptors.py b/test/functional/test_framework/descriptors.py index 29482ce01e..46b405749b 100644 --- a/test/functional/test_framework/descriptors.py +++ b/test/functional/test_framework/descriptors.py @@ -4,6 +4,8 @@ # file COPYING or http://www.opensource.org/licenses/mit-license.php. """Utility functions related to output descriptors""" +import re + INPUT_CHARSET = "0123456789()[],'/*abcdefgh@:$%{}IJKLMNOPQRSTUVWXYZ&+-.;<=>?!^_|~ijklmnopqrstuvwxyzABCDEFGH`#\"\\ " CHECKSUM_CHARSET = "qpzry9x8gf2tvdw0s3jn54khce6mua7l" GENERATOR = [0xf5dee51989, 0xa9fdca3312, 0x1bab10e32d, 0x3706b1677a, 0x644d626ffd] @@ -53,3 +55,10 @@ def descsum_check(s, require=True): return False symbols = descsum_expand(s[:-9]) + [CHECKSUM_CHARSET.find(x) for x in s[-8:]] return descsum_polymod(symbols) == 1 + +def drop_origins(s): + '''Drop the key origins from a descriptor''' + desc = re.sub(r'\[.+?\]', '', s) + if '#' in s: + desc = desc[:desc.index('#')] + return descsum_create(desc) diff --git a/test/functional/test_framework/messages.py b/test/functional/test_framework/messages.py index 25520a2151..285a3fbbf4 100755 --- a/test/functional/test_framework/messages.py +++ b/test/functional/test_framework/messages.py @@ -39,6 +39,7 @@ MAX_LOCATOR_SZ = 101 MAX_BLOCK_BASE_SIZE = 1000000 COIN = 100000000 # 1 btc in satoshis +MAX_MONEY = 21000000 * COIN BIP125_SEQUENCE_NUMBER = 0xfffffffd # Sequence number that is BIP 125 opt-in and BIP 68-opt-out @@ -1105,17 +1106,17 @@ class msg_tx: self.tx.deserialize(f) def serialize(self): - return self.tx.serialize_without_witness() + return self.tx.serialize_with_witness() def __repr__(self): return "msg_tx(tx=%s)" % (repr(self.tx)) -class msg_witness_tx(msg_tx): +class msg_no_witness_tx(msg_tx): __slots__ = () def serialize(self): - return self.tx.serialize_with_witness() + return self.tx.serialize_without_witness() class msg_block: diff --git a/test/functional/test_framework/mininode.py b/test/functional/test_framework/mininode.py index f95c158a68..a9e669fea9 100755 --- a/test/functional/test_framework/mininode.py +++ b/test/functional/test_framework/mininode.py @@ -478,7 +478,8 @@ class NetworkThread(threading.Thread): wait_until(lambda: not self.network_event_loop.is_running(), timeout=timeout) self.network_event_loop.close() self.join(timeout) - + # Safe to remove event loop. + NetworkThread.network_event_loop = None class P2PDataStore(P2PInterface): """A P2P data store class. diff --git a/test/functional/test_framework/script.py b/test/functional/test_framework/script.py index 384062b808..51aa9057f7 100644 --- a/test/functional/test_framework/script.py +++ b/test/functional/test_framework/script.py @@ -2,7 +2,7 @@ # Copyright (c) 2015-2019 The Bitcoin Core developers # Distributed under the MIT software license, see the accompanying # file COPYING or http://www.opensource.org/licenses/mit-license.php. -"""Functionality to build scripts, as well as SignatureHash(). +"""Functionality to build scripts, as well as signature hash functions. This file is modified from python-bitcoinlib. """ @@ -608,7 +608,7 @@ def FindAndDelete(script, sig): return CScript(r) -def SignatureHash(script, txTo, inIdx, hashtype): +def LegacySignatureHash(script, txTo, inIdx, hashtype): """Consensus-correct SignatureHash Returns (hash, err) to precisely match the consensus-critical behavior of @@ -662,7 +662,7 @@ def SignatureHash(script, txTo, inIdx, hashtype): # Performance optimization probably not necessary for python tests, however. # Note that this corresponds to sigversion == 1 in EvalScript, which is used # for version 0 witnesses. -def SegwitVersion1SignatureHash(script, txTo, inIdx, hashtype, amount): +def SegwitV0SignatureHash(script, txTo, inIdx, hashtype, amount): hashPrevouts = 0 hashSequence = 0 diff --git a/test/functional/test_framework/script_util.py b/test/functional/test_framework/script_util.py new file mode 100755 index 0000000000..5ef67226c4 --- /dev/null +++ b/test/functional/test_framework/script_util.py @@ -0,0 +1,25 @@ +#!/usr/bin/env python3 +# Copyright (c) 2019 The Bitcoin Core developers +# Distributed under the MIT software license, see the accompanying +# file COPYING or http://www.opensource.org/licenses/mit-license.php. +"""Useful Script constants and utils.""" +from test_framework.script import CScript + +# To prevent a "tx-size-small" policy rule error, a transaction has to have a +# non-witness size of at least 82 bytes (MIN_STANDARD_TX_NONWITNESS_SIZE in +# src/policy/policy.h). Considering a Tx with the smallest possible single +# input (blank, empty scriptSig), and with an output omitting the scriptPubKey, +# we get to a minimum size of 60 bytes: +# +# Tx Skeleton: 4 [Version] + 1 [InCount] + 1 [OutCount] + 4 [LockTime] = 10 bytes +# Blank Input: 32 [PrevTxHash] + 4 [Index] + 1 [scriptSigLen] + 4 [SeqNo] = 41 bytes +# Output: 8 [Amount] + 1 [scriptPubKeyLen] = 9 bytes +# +# Hence, the scriptPubKey of the single output has to have a size of at +# least 22 bytes, which corresponds to the size of a P2WPKH scriptPubKey. +# The following script constant consists of a single push of 21 bytes of 'a': +# <PUSH_21> <21-bytes of 'a'> +# resulting in a 22-byte size. It should be used whenever (small) fake +# scriptPubKeys are needed, to guarantee that the minimum transaction size is +# met. +DUMMY_P2WPKH_SCRIPT = CScript([b'a' * 21]) diff --git a/test/functional/test_framework/test_framework.py b/test/functional/test_framework/test_framework.py index 780aa5fe03..e36fb350c6 100755 --- a/test/functional/test_framework/test_framework.py +++ b/test/functional/test_framework/test_framework.py @@ -12,6 +12,7 @@ import os import pdb import random import shutil +import subprocess import sys import tempfile import time @@ -96,15 +97,45 @@ class BitcoinTestFramework(metaclass=BitcoinTestMetaClass): self.nodes = [] self.network_thread = None self.rpc_timeout = 60 # Wait for up to 60 seconds for the RPC server to respond - self.supports_cli = False + self.supports_cli = True self.bind_to_localhost_only = True self.set_test_params() - - assert hasattr(self, "num_nodes"), "Test must set self.num_nodes in set_test_params()" + self.parse_args() def main(self): """Main function. This should not be overridden by the subclass test scripts.""" + assert hasattr(self, "num_nodes"), "Test must set self.num_nodes in set_test_params()" + + try: + self.setup() + self.run_test() + except JSONRPCException: + self.log.exception("JSONRPC error") + self.success = TestStatus.FAILED + except SkipTest as e: + self.log.warning("Test Skipped: %s" % e.message) + self.success = TestStatus.SKIPPED + except AssertionError: + self.log.exception("Assertion failed") + self.success = TestStatus.FAILED + except KeyError: + self.log.exception("Key error") + self.success = TestStatus.FAILED + except subprocess.CalledProcessError as e: + self.log.exception("Called Process failed with '{}'".format(e.output)) + self.success = TestStatus.FAILED + except Exception: + self.log.exception("Unexpected exception caught during testing") + self.success = TestStatus.FAILED + except KeyboardInterrupt: + self.log.warning("Exiting after keyboard interrupt") + self.success = TestStatus.FAILED + finally: + exit_code = self.shutdown() + sys.exit(exit_code) + + def parse_args(self): parser = argparse.ArgumentParser(usage="%(prog)s [options]") parser.add_argument("--nocleanup", dest="nocleanup", default=False, action="store_true", help="Leave bitcoinds and test.* datadir on exit or error") @@ -130,11 +161,16 @@ class BitcoinTestFramework(metaclass=BitcoinTestMetaClass): help="use bitcoin-cli instead of RPC for all commands") parser.add_argument("--perf", dest="perf", default=False, action="store_true", help="profile running nodes with perf for the duration of the test") + parser.add_argument("--valgrind", dest="valgrind", default=False, action="store_true", + help="run nodes under the valgrind memory error detector: expect at least a ~10x slowdown, valgrind 3.14 or later required") parser.add_argument("--randomseed", type=int, help="set a random seed for deterministically reproducing a previous test run") self.add_options(parser) self.options = parser.parse_args() + def setup(self): + """Call this method to start up the test framework object with options set.""" + PortSeed.n = self.options.port_seed check_json_precision() @@ -181,33 +217,20 @@ class BitcoinTestFramework(metaclass=BitcoinTestMetaClass): self.network_thread = NetworkThread() self.network_thread.start() - success = TestStatus.FAILED + if self.options.usecli: + if not self.supports_cli: + raise SkipTest("--usecli specified but test does not support using CLI") + self.skip_if_no_cli() + self.skip_test_if_missing_module() + self.setup_chain() + self.setup_network() - try: - if self.options.usecli: - if not self.supports_cli: - raise SkipTest("--usecli specified but test does not support using CLI") - self.skip_if_no_cli() - self.skip_test_if_missing_module() - self.setup_chain() - self.setup_network() - self.run_test() - success = TestStatus.PASSED - except JSONRPCException: - self.log.exception("JSONRPC error") - except SkipTest as e: - self.log.warning("Test Skipped: %s" % e.message) - success = TestStatus.SKIPPED - except AssertionError: - self.log.exception("Assertion failed") - except KeyError: - self.log.exception("Key error") - except Exception: - self.log.exception("Unexpected exception caught during testing") - except KeyboardInterrupt: - self.log.warning("Exiting after keyboard interrupt") + self.success = TestStatus.PASSED - if success == TestStatus.FAILED and self.options.pdbonfailure: + def shutdown(self): + """Call this method to shut down the test framework object.""" + + if self.success == TestStatus.FAILED and self.options.pdbonfailure: print("Testcase failed. Attaching python debugger. Enter ? for help") pdb.set_trace() @@ -225,7 +248,7 @@ class BitcoinTestFramework(metaclass=BitcoinTestMetaClass): should_clean_up = ( not self.options.nocleanup and not self.options.noshutdown and - success != TestStatus.FAILED and + self.success != TestStatus.FAILED and not self.options.perf ) if should_clean_up: @@ -238,20 +261,33 @@ class BitcoinTestFramework(metaclass=BitcoinTestMetaClass): self.log.warning("Not cleaning up dir {}".format(self.options.tmpdir)) cleanup_tree_on_exit = False - if success == TestStatus.PASSED: + if self.success == TestStatus.PASSED: self.log.info("Tests successful") exit_code = TEST_EXIT_PASSED - elif success == TestStatus.SKIPPED: + elif self.success == TestStatus.SKIPPED: self.log.info("Test skipped") exit_code = TEST_EXIT_SKIPPED else: self.log.error("Test failed. Test logging available at %s/test_framework.log", self.options.tmpdir) self.log.error("Hint: Call {} '{}' to consolidate all logs".format(os.path.normpath(os.path.dirname(os.path.realpath(__file__)) + "/../combine_logs.py"), self.options.tmpdir)) exit_code = TEST_EXIT_FAILED - logging.shutdown() + # Logging.shutdown will not remove stream- and filehandlers, so we must + # do it explicitly. Handlers are removed so the next test run can apply + # different log handler settings. + # See: https://docs.python.org/3/library/logging.html#logging.shutdown + for h in list(self.log.handlers): + h.flush() + h.close() + self.log.removeHandler(h) + rpc_logger = logging.getLogger("BitcoinRPC") + for h in list(rpc_logger.handlers): + h.flush() + rpc_logger.removeHandler(h) if cleanup_tree_on_exit: shutil.rmtree(self.options.tmpdir) - sys.exit(exit_code) + + self.nodes.clear() + return exit_code # Methods to override in subclass test scripts. def set_test_params(self): @@ -333,7 +369,7 @@ class BitcoinTestFramework(metaclass=BitcoinTestMetaClass): # Public helper methods. These can be accessed by the subclass test scripts. - def add_nodes(self, num_nodes, extra_args=None, *, rpchost=None, binary=None): + def add_nodes(self, num_nodes, extra_args=None, *, rpchost=None, binary=None, binary_cli=None, versions=None): """Instantiate TestNode objects. Should only be called once after the nodes have been specified in @@ -344,11 +380,17 @@ class BitcoinTestFramework(metaclass=BitcoinTestMetaClass): extra_confs = [[]] * num_nodes if extra_args is None: extra_args = [[]] * num_nodes + if versions is None: + versions = [None] * num_nodes if binary is None: binary = [self.options.bitcoind] * num_nodes + if binary_cli is None: + binary_cli = [self.options.bitcoincli] * num_nodes assert_equal(len(extra_confs), num_nodes) assert_equal(len(extra_args), num_nodes) + assert_equal(len(versions), num_nodes) assert_equal(len(binary), num_nodes) + assert_equal(len(binary_cli), num_nodes) for i in range(num_nodes): self.nodes.append(TestNode( i, @@ -357,13 +399,15 @@ class BitcoinTestFramework(metaclass=BitcoinTestMetaClass): rpchost=rpchost, timewait=self.rpc_timeout, bitcoind=binary[i], - bitcoin_cli=self.options.bitcoincli, + bitcoin_cli=binary_cli[i], + version=versions[i], coverage_dir=self.options.coveragedir, cwd=self.options.tmpdir, extra_conf=extra_confs[i], extra_args=extra_args[i], use_cli=self.options.usecli, start_perf=self.options.perf, + use_valgrind=self.options.valgrind, )) def start_node(self, i, *args, **kwargs): @@ -566,6 +610,11 @@ class BitcoinTestFramework(metaclass=BitcoinTestMetaClass): if not self.is_wallet_compiled(): raise SkipTest("wallet has not been compiled.") + def skip_if_no_wallet_tool(self): + """Skip the running test if bitcoin-wallet has not been compiled.""" + if not self.is_wallet_tool_compiled(): + raise SkipTest("bitcoin-wallet has not been compiled") + def skip_if_no_cli(self): """Skip the running test if bitcoin-cli has not been compiled.""" if not self.is_cli_compiled(): @@ -579,6 +628,10 @@ class BitcoinTestFramework(metaclass=BitcoinTestMetaClass): """Checks whether the wallet module was compiled.""" return self.config["components"].getboolean("ENABLE_WALLET") + def is_wallet_tool_compiled(self): + """Checks whether bitcoin-wallet was compiled.""" + return self.config["components"].getboolean("ENABLE_WALLET_TOOL") + def is_zmq_compiled(self): """Checks whether the zmq module was compiled.""" return self.config["components"].getboolean("ENABLE_ZMQ") diff --git a/test/functional/test_framework/test_node.py b/test/functional/test_framework/test_node.py index d8bfdfd040..c7559ac7c8 100755 --- a/test/functional/test_framework/test_node.py +++ b/test/functional/test_framework/test_node.py @@ -30,6 +30,7 @@ from .util import ( rpc_url, wait_until, p2p_port, + EncodeDecimal, ) BITCOIND_PROC_WAIT_TIMEOUT = 60 @@ -59,7 +60,7 @@ class TestNode(): To make things easier for the test writer, any unrecognised messages will be dispatched to the RPC connection.""" - def __init__(self, i, datadir, *, chain, rpchost, timewait, bitcoind, bitcoin_cli, coverage_dir, cwd, extra_conf=None, extra_args=None, use_cli=False, start_perf=False): + def __init__(self, i, datadir, *, chain, rpchost, timewait, bitcoind, bitcoin_cli, coverage_dir, cwd, extra_conf=None, extra_args=None, use_cli=False, start_perf=False, use_valgrind=False, version=None): """ Kwargs: start_perf (bool): If True, begin profiling the node with `perf` as soon as @@ -83,6 +84,7 @@ class TestNode(): # For those callers that need more flexibility, they can just set the args property directly. # Note that common args are set in the config file (see initialize_datadir) self.extra_args = extra_args + self.version = version # Configuration for logging is set as command-line args rather than in the bitcoin.conf file. # This means that starting a bitcoind using the temp dir to debug a failed test won't # spam debug.log. @@ -90,12 +92,23 @@ class TestNode(): self.binary, "-datadir=" + self.datadir, "-logtimemicros", - "-logthreadnames", "-debug", "-debugexclude=libevent", "-debugexclude=leveldb", "-uacomment=testnode%d" % i, ] + if use_valgrind: + default_suppressions_file = os.path.join( + os.path.dirname(os.path.realpath(__file__)), + "..", "..", "..", "contrib", "valgrind.supp") + suppressions_file = os.getenv("VALGRIND_SUPPRESSIONS_FILE", + default_suppressions_file) + self.args = ["valgrind", "--suppressions={}".format(suppressions_file), + "--gen-suppressions=all", "--exit-on-first-error=yes", + "--error-exitcode=1", "--quiet"] + self.args + + if self.version is None or self.version >= 190000: + self.args.append("-logthreadnames") self.cli = TestNodeCLI(bitcoin_cli, self.datadir) self.use_cli = use_cli @@ -135,25 +148,6 @@ class TestNode(): assert len(self.PRIV_KEYS) == MAX_NODES return self.PRIV_KEYS[self.index] - def get_mem_rss_kilobytes(self): - """Get the memory usage (RSS) per `ps`. - - Returns None if `ps` is unavailable. - """ - assert self.running - - try: - return int(subprocess.check_output( - ["ps", "h", "-o", "rss", "{}".format(self.process.pid)], - stderr=subprocess.DEVNULL).split()[-1]) - - # Avoid failing on platforms where ps isn't installed. - # - # We could later use something like `psutils` to work across platforms. - except (FileNotFoundError, subprocess.SubprocessError): - self.log.exception("Unable to get memory usage") - return None - def _node_msg(self, msg: str) -> str: """Return a modified msg that identifies this node by its index as a debugging aid.""" return "[node %d] %s" % (self.index, msg) @@ -263,7 +257,11 @@ class TestNode(): return self.log.debug("Stopping node") try: - self.stop(wait=wait) + # Do not use wait argument when testing older nodes, e.g. in feature_backwards_compatibility.py + if self.version is None or self.version >= 180000: + self.stop(wait=wait) + else: + self.stop() except http.client.CannotSendRequest: self.log.exception("Unable to stop node.") @@ -307,7 +305,9 @@ class TestNode(): wait_until(self.is_node_stopped, timeout=timeout) @contextlib.contextmanager - def assert_debug_log(self, expected_msgs, timeout=2): + def assert_debug_log(self, expected_msgs, unexpected_msgs=None, timeout=2): + if unexpected_msgs is None: + unexpected_msgs = [] time_end = time.time() + timeout debug_log = os.path.join(self.datadir, self.chain, 'debug.log') with open(debug_log, encoding='utf-8') as dl: @@ -322,6 +322,9 @@ class TestNode(): dl.seek(prev_size) log = dl.read() print_log = " - " + "\n - ".join(log.splitlines()) + for unexpected_msg in unexpected_msgs: + if re.search(re.escape(unexpected_msg), log, flags=re.MULTILINE): + self._raise_assertion_error('Unexpected message "{}" partially matches log:\n\n{}\n\n'.format(unexpected_msg, print_log)) for expected_msg in expected_msgs: if re.search(re.escape(expected_msg), log, flags=re.MULTILINE) is None: found = False @@ -333,33 +336,6 @@ class TestNode(): self._raise_assertion_error('Expected messages "{}" does not partially match log:\n\n{}\n\n'.format(str(expected_msgs), print_log)) @contextlib.contextmanager - def assert_memory_usage_stable(self, *, increase_allowed=0.03): - """Context manager that allows the user to assert that a node's memory usage (RSS) - hasn't increased beyond some threshold percentage. - - Args: - increase_allowed (float): the fractional increase in memory allowed until failure; - e.g. `0.12` for up to 12% increase allowed. - """ - before_memory_usage = self.get_mem_rss_kilobytes() - - yield - - after_memory_usage = self.get_mem_rss_kilobytes() - - if not (before_memory_usage and after_memory_usage): - self.log.warning("Unable to detect memory usage (RSS) - skipping memory check.") - return - - perc_increase_memory_usage = (after_memory_usage / before_memory_usage) - 1 - - if perc_increase_memory_usage > increase_allowed: - self._raise_assertion_error( - "Memory usage increased over threshold of {:.3f}% from {} to {} ({:.3f}%)".format( - increase_allowed * 100, before_memory_usage, after_memory_usage, - perc_increase_memory_usage * 100)) - - @contextlib.contextmanager def profile_with_perf(self, profile_name): """ Context manager that allows easy profiling of node activity using `perf`. @@ -526,7 +502,7 @@ def arg_to_cli(arg): if isinstance(arg, bool): return str(arg).lower() elif isinstance(arg, dict) or isinstance(arg, list): - return json.dumps(arg) + return json.dumps(arg, default=EncodeDecimal) else: return str(arg) diff --git a/test/functional/test_framework/test_shell.py b/test/functional/test_framework/test_shell.py new file mode 100644 index 0000000000..26df128f1f --- /dev/null +++ b/test/functional/test_framework/test_shell.py @@ -0,0 +1,75 @@ +#!/usr/bin/env python3 +# Copyright (c) 2019 The Bitcoin Core developers +# Distributed under the MIT software license, see the accompanying +# file COPYING or http://www.opensource.org/licenses/mit-license.php. + +from test_framework.test_framework import BitcoinTestFramework + +class TestShell: + """Wrapper Class for BitcoinTestFramework. + + The TestShell class extends the BitcoinTestFramework + rpc & daemon process management functionality to external + python environments. + + It is a singleton class, which ensures that users only + start a single TestShell at a time.""" + + class __TestShell(BitcoinTestFramework): + def set_test_params(self): + pass + + def run_test(self): + pass + + def setup(self, **kwargs): + if self.running: + print("TestShell is already running!") + return + + # Num_nodes parameter must be set + # by BitcoinTestFramework child class. + self.num_nodes = 1 + + # User parameters override default values. + for key, value in kwargs.items(): + if hasattr(self, key): + setattr(self, key, value) + elif hasattr(self.options, key): + setattr(self.options, key, value) + else: + raise KeyError(key + " not a valid parameter key!") + + super().setup() + self.running = True + return self + + def shutdown(self): + if not self.running: + print("TestShell is not running!") + else: + super().shutdown() + self.running = False + + def reset(self): + if self.running: + print("Shutdown TestShell before resetting!") + else: + self.num_nodes = None + super().__init__() + + instance = None + + def __new__(cls): + # This implementation enforces singleton pattern, and will return the + # previously initialized instance if available + if not TestShell.instance: + TestShell.instance = TestShell.__TestShell() + TestShell.instance.running = False + return TestShell.instance + + def __getattr__(self, name): + return getattr(self.instance, name) + + def __setattr__(self, name, value): + return setattr(self.instance, name, value) diff --git a/test/functional/test_framework/util.py b/test/functional/test_framework/util.py index cde99a2219..5bb73aee7e 100644 --- a/test/functional/test_framework/util.py +++ b/test/functional/test_framework/util.py @@ -190,6 +190,11 @@ def check_json_precision(): if satoshis != 2000000000000003: raise RuntimeError("JSON encode/decode loses precision") +def EncodeDecimal(o): + if isinstance(o, Decimal): + return str(o) + raise TypeError(repr(o) + " is not JSON serializable") + def count_bytes(hex_string): return len(bytearray.fromhex(hex_string)) @@ -316,6 +321,7 @@ def initialize_datadir(dirname, n, chain): f.write("listenonion=0\n") f.write("printtoconsole=0\n") f.write("upnp=0\n") + f.write("shrinkdebugfile=0\n") os.makedirs(os.path.join(datadir, 'stderr'), exist_ok=True) os.makedirs(os.path.join(datadir, 'stdout'), exist_ok=True) return datadir diff --git a/test/functional/test_framework/wallet_util.py b/test/functional/test_framework/wallet_util.py index c0dfa4c3f0..eb537015fb 100755 --- a/test/functional/test_framework/wallet_util.py +++ b/test/functional/test_framework/wallet_util.py @@ -1,5 +1,5 @@ #!/usr/bin/env python3 -# Copyright (c) 2018 The Bitcoin Core developers +# Copyright (c) 2018-2020 The Bitcoin Core developers # Distributed under the MIT software license, see the accompanying # file COPYING or http://www.opensource.org/licenses/mit-license.php. """Useful util functions for testing the wallet""" diff --git a/test/functional/test_runner.py b/test/functional/test_runner.py index c40ccf30f3..9d87b5638c 100755 --- a/test/functional/test_runner.py +++ b/test/functional/test_runner.py @@ -136,6 +136,7 @@ BASE_SCRIPTS = [ 'interface_rpc.py', 'rpc_psbt.py', 'rpc_users.py', + 'rpc_whitelist.py', 'feature_proxy.py', 'rpc_signrawtransaction.py', 'wallet_groups.py', @@ -157,6 +158,7 @@ BASE_SCRIPTS = [ 'feature_assumevalid.py', 'example_test.py', 'wallet_txn_doublespend.py', + 'feature_backwards_compatibility.py', 'wallet_txn_clone.py --mineblock', 'feature_notifications.py', 'rpc_getblockfilter.py', @@ -173,6 +175,7 @@ BASE_SCRIPTS = [ 'wallet_balance.py', 'feature_nulldummy.py', 'mempool_accept.py', + 'mempool_expiry.py', 'wallet_import_rescan.py', 'wallet_import_with_label.py', 'rpc_bind.py --ipv4', @@ -181,6 +184,7 @@ BASE_SCRIPTS = [ 'mining_basic.py', 'wallet_bumpfee.py', 'wallet_bumpfee_totalfee_deprecation.py', + 'wallet_implicitsegwit.py', 'rpc_named_arguments.py', 'wallet_listsinceblock.py', 'p2p_leak.py', @@ -190,6 +194,7 @@ BASE_SCRIPTS = [ 'rpc_uptime.py', 'wallet_resendwallettransactions.py', 'wallet_fallbackfee.py', + 'rpc_dumptxoutset.py', 'feature_minchainwork.py', 'rpc_estimatefee.py', 'rpc_getblockstats.py', @@ -198,9 +203,11 @@ BASE_SCRIPTS = [ 'feature_uacomment.py', 'wallet_coinbase_category.py', 'feature_filelock.py', + 'feature_loadblock.py', 'p2p_dos_header_tree.py', 'p2p_unrequested_blocks.py', 'feature_includeconf.py', + 'feature_asmap.py', 'rpc_deriveaddresses.py', 'rpc_deriveaddresses.py --usecli', 'rpc_scantxoutset.py', @@ -209,6 +216,9 @@ BASE_SCRIPTS = [ 'p2p_permissions.py', 'feature_blocksdir.py', 'feature_config_args.py', + 'rpc_getaddressinfo_labels_purpose_deprecation.py', + 'rpc_getaddressinfo_label_deprecation.py', + 'rpc_getdescriptorinfo.py', 'rpc_help.py', 'feature_help.py', 'feature_shutdown.py', @@ -362,9 +372,10 @@ def main(): def run_tests(*, test_list, src_dir, build_dir, tmpdir, jobs=1, enable_coverage=False, args=None, combined_logs_len=0, failfast=False, runs_ci, use_term_control): args = args or [] - # Warn if bitcoind is already running (unix only) + # Warn if bitcoind is already running + # pidof might fail or return an empty string if bitcoind is not running try: - if subprocess.check_output(["pidof", "bitcoind"]) is not None: + if subprocess.check_output(["pidof", "bitcoind"]) not in [b'']: print("%sWARNING!%s There is already a bitcoind process running on this system. Tests may fail unexpectedly due to resource contention!" % (BOLD[1], BOLD[0])) except (OSError, subprocess.SubprocessError): pass diff --git a/test/functional/tool_wallet.py b/test/functional/tool_wallet.py index 355cd7af75..d2629ff1ed 100755 --- a/test/functional/tool_wallet.py +++ b/test/functional/tool_wallet.py @@ -23,10 +23,11 @@ class ToolWalletTest(BitcoinTestFramework): def skip_test_if_missing_module(self): self.skip_if_no_wallet() + self.skip_if_no_wallet_tool() def bitcoin_wallet_process(self, *args): binary = self.config["environment"]["BUILDDIR"] + '/src/bitcoin-wallet' + self.config["environment"]["EXEEXT"] - args = ['-datadir={}'.format(self.nodes[0].datadir), '-regtest'] + list(args) + args = ['-datadir={}'.format(self.nodes[0].datadir), '-chain=%s' % self.chain] + list(args) return subprocess.Popen([binary] + args, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, universal_newlines=True) def assert_raises_tool_error(self, error, *args): @@ -197,7 +198,7 @@ class ToolWalletTest(BitcoinTestFramework): self.log.debug('Wallet file shasum unchanged\n') def run_test(self): - self.wallet_path = os.path.join(self.nodes[0].datadir, 'regtest', 'wallets', 'wallet.dat') + self.wallet_path = os.path.join(self.nodes[0].datadir, self.chain, 'wallets', 'wallet.dat') self.test_invalid_tool_commands_and_args() # Warning: The following tests are order-dependent. self.test_tool_wallet_info() diff --git a/test/functional/wallet_abandonconflict.py b/test/functional/wallet_abandonconflict.py index e86679bc31..1122daaf83 100755 --- a/test/functional/wallet_abandonconflict.py +++ b/test/functional/wallet_abandonconflict.py @@ -18,6 +18,7 @@ from test_framework.util import ( assert_raises_rpc_error, connect_nodes, disconnect_nodes, + wait_until, ) @@ -97,6 +98,7 @@ class AbandonConflictTest(BitcoinTestFramework): # TODO: redo with eviction self.stop_node(0) self.start_node(0, extra_args=["-minrelaytxfee=0.0001"]) + wait_until(lambda: self.nodes[0].getmempoolinfo()['loaded']) # Verify txs no longer in either node's mempool assert_equal(len(self.nodes[0].getrawmempool()), 0) @@ -124,6 +126,8 @@ class AbandonConflictTest(BitcoinTestFramework): # Verify that even with a low min relay fee, the tx is not reaccepted from wallet on startup once abandoned self.stop_node(0) self.start_node(0, extra_args=["-minrelaytxfee=0.00001"]) + wait_until(lambda: self.nodes[0].getmempoolinfo()['loaded']) + assert_equal(len(self.nodes[0].getrawmempool()), 0) assert_equal(self.nodes[0].getbalance(), balance) @@ -144,6 +148,7 @@ class AbandonConflictTest(BitcoinTestFramework): # Remove using high relay fee again self.stop_node(0) self.start_node(0, extra_args=["-minrelaytxfee=0.0001"]) + wait_until(lambda: self.nodes[0].getmempoolinfo()['loaded']) assert_equal(len(self.nodes[0].getrawmempool()), 0) newbalance = self.nodes[0].getbalance() assert_equal(newbalance, balance - Decimal("24.9996")) diff --git a/test/functional/wallet_address_types.py b/test/functional/wallet_address_types.py index b72f5c6008..79b6db986b 100755 --- a/test/functional/wallet_address_types.py +++ b/test/functional/wallet_address_types.py @@ -80,6 +80,10 @@ class AddressTypeTest(BitcoinTestFramework): ["-changetype=p2sh-segwit"], [], ] + # whitelist all peers to speed up tx relay / mempool sync + for args in self.extra_args: + args.append("-whitelist=noban@127.0.0.1") + self.supports_cli = False def skip_test_if_missing_module(self): self.skip_if_no_wallet() diff --git a/test/functional/wallet_avoidreuse.py b/test/functional/wallet_avoidreuse.py index 3c8064ea2d..8e2dc03ac2 100755 --- a/test/functional/wallet_avoidreuse.py +++ b/test/functional/wallet_avoidreuse.py @@ -1,5 +1,5 @@ #!/usr/bin/env python3 -# Copyright (c) 2018 The Bitcoin Core developers +# Copyright (c) 2018-2020 The Bitcoin Core developers # Distributed under the MIT software license, see the accompanying # file COPYING or http://www.opensource.org/licenses/mit-license.php. """Test the avoid_reuse and setwalletflag features.""" @@ -68,6 +68,9 @@ class AvoidReuseTest(BitcoinTestFramework): def set_test_params(self): self.setup_clean_chain = False self.num_nodes = 2 + # This test isn't testing txn relay/timing, so set whitelist on the + # peers for instant txn relay. This speeds up the test run time 2-3x. + self.extra_args = [["-whitelist=noban@127.0.0.1"]] * self.num_nodes def skip_test_if_missing_module(self): self.skip_if_no_wallet() @@ -83,10 +86,18 @@ class AvoidReuseTest(BitcoinTestFramework): reset_balance(self.nodes[1], self.nodes[0].getnewaddress()) self.test_fund_send_fund_senddirty() reset_balance(self.nodes[1], self.nodes[0].getnewaddress()) - self.test_fund_send_fund_send() + self.test_fund_send_fund_send("legacy") + reset_balance(self.nodes[1], self.nodes[0].getnewaddress()) + self.test_fund_send_fund_send("p2sh-segwit") + reset_balance(self.nodes[1], self.nodes[0].getnewaddress()) + self.test_fund_send_fund_send("bech32") + reset_balance(self.nodes[1], self.nodes[0].getnewaddress()) + self.test_getbalances_used() def test_persistence(self): '''Test that wallet files persist the avoid_reuse flag.''' + self.log.info("Test wallet files persist avoid_reuse flag") + # Configure node 1 to use avoid_reuse self.nodes[1].setwalletflag('avoid_reuse') @@ -109,6 +120,8 @@ class AvoidReuseTest(BitcoinTestFramework): def test_immutable(self): '''Test immutable wallet flags''' + self.log.info("Test immutable wallet flags") + # Attempt to set the disable_private_keys flag; this should not work assert_raises_rpc_error(-8, "Wallet flag is immutable", self.nodes[1].setwalletflag, 'disable_private_keys') @@ -130,6 +143,7 @@ class AvoidReuseTest(BitcoinTestFramework): the avoid_reuse flag set to false. This means the 10 BTC send should succeed, where it fails in test_fund_send_fund_send. ''' + self.log.info("Test fund send fund send dirty") fundaddr = self.nodes[1].getnewaddress() retaddr = self.nodes[0].getnewaddress() @@ -174,7 +188,7 @@ class AvoidReuseTest(BitcoinTestFramework): assert_approx(self.nodes[1].getbalance(), 5, 0.001) assert_approx(self.nodes[1].getbalance(avoid_reuse=False), 5, 0.001) - def test_fund_send_fund_send(self): + def test_fund_send_fund_send(self, second_addr_type): ''' Test the simple case where [1] generates a new address A, then [0] sends 10 BTC to A. @@ -183,8 +197,9 @@ class AvoidReuseTest(BitcoinTestFramework): [1] tries to spend 10 BTC (fails; dirty). [1] tries to spend 4 BTC (succeeds; change address sufficient) ''' + self.log.info("Test fund send fund send") - fundaddr = self.nodes[1].getnewaddress() + fundaddr = self.nodes[1].getnewaddress(label="", address_type="legacy") retaddr = self.nodes[0].getnewaddress() self.nodes[0].sendtoaddress(fundaddr, 10) @@ -205,7 +220,19 @@ class AvoidReuseTest(BitcoinTestFramework): # getbalances should show no used, 5 btc trusted assert_balances(self.nodes[1], mine={"used": 0, "trusted": 5}) - self.nodes[0].sendtoaddress(fundaddr, 10) + # For the second send, we transmute it to a related single-key address + # to make sure it's also detected as re-use + fund_spk = self.nodes[0].getaddressinfo(fundaddr)["scriptPubKey"] + fund_decoded = self.nodes[0].decodescript(fund_spk) + if second_addr_type == "p2sh-segwit": + new_fundaddr = fund_decoded["segwit"]["p2sh-segwit"] + elif second_addr_type == "bech32": + new_fundaddr = fund_decoded["segwit"]["addresses"][0] + else: + new_fundaddr = fundaddr + assert_equal(second_addr_type, "legacy") + + self.nodes[0].sendtoaddress(new_fundaddr, 10) self.nodes[0].generate(1) self.sync_all() @@ -231,5 +258,35 @@ class AvoidReuseTest(BitcoinTestFramework): assert_approx(self.nodes[1].getbalance(), 1, 0.001) assert_approx(self.nodes[1].getbalance(avoid_reuse=False), 11, 0.001) + def test_getbalances_used(self): + ''' + getbalances and listunspent should pick up on reused addresses + immediately, even for address reusing outputs created before the first + transaction was spending from that address + ''' + self.log.info("Test getbalances used category") + + # node under test should be completely empty + assert_equal(self.nodes[1].getbalance(avoid_reuse=False), 0) + + new_addr = self.nodes[1].getnewaddress() + ret_addr = self.nodes[0].getnewaddress() + + # send multiple transactions, reusing one address + for _ in range(11): + self.nodes[0].sendtoaddress(new_addr, 1) + + self.nodes[0].generate(1) + self.sync_all() + + # send transaction that should not use all the available outputs + # per the current coin selection algorithm + self.nodes[1].sendtoaddress(ret_addr, 5) + + # getbalances and listunspent should show the remaining outputs + # in the reused address as used/reused + assert_unspent(self.nodes[1], total_count=2, total_sum=6, reused_count=1, reused_sum=1) + assert_balances(self.nodes[1], mine={"used": 1, "trusted": 5}) + if __name__ == '__main__': AvoidReuseTest().main() diff --git a/test/functional/wallet_backup.py b/test/functional/wallet_backup.py index 93178c5ab2..fb80a06433 100755 --- a/test/functional/wallet_backup.py +++ b/test/functional/wallet_backup.py @@ -48,7 +48,13 @@ class WalletBackupTest(BitcoinTestFramework): self.num_nodes = 4 self.setup_clean_chain = True # nodes 1, 2,3 are spenders, let's give them a keypool=100 - self.extra_args = [["-keypool=100"], ["-keypool=100"], ["-keypool=100"], []] + # whitelist all peers to speed up tx relay / mempool sync + self.extra_args = [ + ["-whitelist=noban@127.0.0.1", "-keypool=100"], + ["-whitelist=noban@127.0.0.1", "-keypool=100"], + ["-whitelist=noban@127.0.0.1", "-keypool=100"], + ["-whitelist=noban@127.0.0.1"], + ] self.rpc_timeout = 120 def skip_test_if_missing_module(self): @@ -101,9 +107,9 @@ class WalletBackupTest(BitcoinTestFramework): self.stop_node(2) def erase_three(self): - os.remove(os.path.join(self.nodes[0].datadir, 'regtest', 'wallets', 'wallet.dat')) - os.remove(os.path.join(self.nodes[1].datadir, 'regtest', 'wallets', 'wallet.dat')) - os.remove(os.path.join(self.nodes[2].datadir, 'regtest', 'wallets', 'wallet.dat')) + os.remove(os.path.join(self.nodes[0].datadir, self.chain, 'wallets', 'wallet.dat')) + os.remove(os.path.join(self.nodes[1].datadir, self.chain, 'wallets', 'wallet.dat')) + os.remove(os.path.join(self.nodes[2].datadir, self.chain, 'wallets', 'wallet.dat')) def run_test(self): self.log.info("Generating initial blockchain") @@ -161,13 +167,13 @@ class WalletBackupTest(BitcoinTestFramework): self.erase_three() # Start node2 with no chain - shutil.rmtree(os.path.join(self.nodes[2].datadir, 'regtest', 'blocks')) - shutil.rmtree(os.path.join(self.nodes[2].datadir, 'regtest', 'chainstate')) + shutil.rmtree(os.path.join(self.nodes[2].datadir, self.chain, 'blocks')) + shutil.rmtree(os.path.join(self.nodes[2].datadir, self.chain, 'chainstate')) # Restore wallets from backup - shutil.copyfile(os.path.join(self.nodes[0].datadir, 'wallet.bak'), os.path.join(self.nodes[0].datadir, 'regtest', 'wallets', 'wallet.dat')) - shutil.copyfile(os.path.join(self.nodes[1].datadir, 'wallet.bak'), os.path.join(self.nodes[1].datadir, 'regtest', 'wallets', 'wallet.dat')) - shutil.copyfile(os.path.join(self.nodes[2].datadir, 'wallet.bak'), os.path.join(self.nodes[2].datadir, 'regtest', 'wallets', 'wallet.dat')) + shutil.copyfile(os.path.join(self.nodes[0].datadir, 'wallet.bak'), os.path.join(self.nodes[0].datadir, self.chain, 'wallets', 'wallet.dat')) + shutil.copyfile(os.path.join(self.nodes[1].datadir, 'wallet.bak'), os.path.join(self.nodes[1].datadir, self.chain, 'wallets', 'wallet.dat')) + shutil.copyfile(os.path.join(self.nodes[2].datadir, 'wallet.bak'), os.path.join(self.nodes[2].datadir, self.chain, 'wallets', 'wallet.dat')) self.log.info("Re-starting nodes") self.start_three() @@ -182,8 +188,8 @@ class WalletBackupTest(BitcoinTestFramework): self.erase_three() #start node2 with no chain - shutil.rmtree(os.path.join(self.nodes[2].datadir, 'regtest', 'blocks')) - shutil.rmtree(os.path.join(self.nodes[2].datadir, 'regtest', 'chainstate')) + shutil.rmtree(os.path.join(self.nodes[2].datadir, self.chain, 'blocks')) + shutil.rmtree(os.path.join(self.nodes[2].datadir, self.chain, 'chainstate')) self.start_three() @@ -203,10 +209,10 @@ class WalletBackupTest(BitcoinTestFramework): # Backup to source wallet file must fail sourcePaths = [ - os.path.join(self.nodes[0].datadir, 'regtest', 'wallets', 'wallet.dat'), - os.path.join(self.nodes[0].datadir, 'regtest', '.', 'wallets', 'wallet.dat'), - os.path.join(self.nodes[0].datadir, 'regtest', 'wallets', ''), - os.path.join(self.nodes[0].datadir, 'regtest', 'wallets')] + os.path.join(self.nodes[0].datadir, self.chain, 'wallets', 'wallet.dat'), + os.path.join(self.nodes[0].datadir, self.chain, '.', 'wallets', 'wallet.dat'), + os.path.join(self.nodes[0].datadir, self.chain, 'wallets', ''), + os.path.join(self.nodes[0].datadir, self.chain, 'wallets')] for sourcePath in sourcePaths: assert_raises_rpc_error(-4, "backup failed", self.nodes[0].backupwallet, sourcePath) diff --git a/test/functional/wallet_balance.py b/test/functional/wallet_balance.py index c50dcd987a..a5f9a047ed 100755 --- a/test/functional/wallet_balance.py +++ b/test/functional/wallet_balance.py @@ -109,13 +109,51 @@ class WalletTest(BitcoinTestFramework): self.log.info("Test getbalance and getunconfirmedbalance with unconfirmed inputs") + # Before `test_balance()`, we have had two nodes with a balance of 50 + # each and then we: + # + # 1) Sent 40 from node A to node B with fee 0.01 + # 2) Sent 60 from node B to node A with fee 0.01 + # + # Then we check the balances: + # + # 1) As is + # 2) With transaction 2 from above with 2x the fee + # + # Prior to #16766, in this situation, the node would immediately report + # a balance of 30 on node B as unconfirmed and trusted. + # + # After #16766, we show that balance as unconfirmed. + # + # The balance is indeed "trusted" and "confirmed" insofar as removing + # the mempool transactions would return at least that much money. But + # the algorithm after #16766 marks it as unconfirmed because the 'taint' + # tracking of transaction trust for summing balances doesn't consider + # which inputs belong to a user. In this case, the change output in + # question could be "destroyed" by replace the 1st transaction above. + # + # The post #16766 behavior is correct; we shouldn't be treating those + # funds as confirmed. If you want to rely on that specific UTXO existing + # which has given you that balance, you cannot, as a third party + # spending the other input would destroy that unconfirmed. + # + # For example, if the test transactions were: + # + # 1) Sent 40 from node A to node B with fee 0.01 + # 2) Sent 10 from node B to node A with fee 0.01 + # + # Then our node would report a confirmed balance of 40 + 50 - 10 = 80 + # BTC, which is more than would be available if transaction 1 were + # replaced. + + def test_balances(*, fee_node_1=0): # getbalance without any arguments includes unconfirmed transactions, but not untrusted transactions assert_equal(self.nodes[0].getbalance(), Decimal('9.99')) # change from node 0's send - assert_equal(self.nodes[1].getbalance(), Decimal('30') - fee_node_1) # change from node 1's send + assert_equal(self.nodes[1].getbalance(), Decimal('0')) # node 1's send had an unsafe input # Same with minconf=0 assert_equal(self.nodes[0].getbalance(minconf=0), Decimal('9.99')) - assert_equal(self.nodes[1].getbalance(minconf=0), Decimal('30') - fee_node_1) + assert_equal(self.nodes[1].getbalance(minconf=0), Decimal('0')) # getbalance with a minconf incorrectly excludes coins that have been spent more recently than the minconf blocks ago # TODO: fix getbalance tracking of coin spentness depth assert_equal(self.nodes[0].getbalance(minconf=1), Decimal('0')) @@ -125,9 +163,9 @@ class WalletTest(BitcoinTestFramework): assert_equal(self.nodes[0].getbalances()['mine']['untrusted_pending'], Decimal('60')) assert_equal(self.nodes[0].getwalletinfo()["unconfirmed_balance"], Decimal('60')) - assert_equal(self.nodes[1].getunconfirmedbalance(), Decimal('0')) # Doesn't include output of node 0's send since it was spent - assert_equal(self.nodes[1].getbalances()['mine']['untrusted_pending'], Decimal('0')) - assert_equal(self.nodes[1].getwalletinfo()["unconfirmed_balance"], Decimal('0')) + assert_equal(self.nodes[1].getunconfirmedbalance(), Decimal('30') - fee_node_1) # Doesn't include output of node 0's send since it was spent + assert_equal(self.nodes[1].getbalances()['mine']['untrusted_pending'], Decimal('30') - fee_node_1) + assert_equal(self.nodes[1].getwalletinfo()["unconfirmed_balance"], Decimal('30') - fee_node_1) test_balances(fee_node_1=Decimal('0.01')) diff --git a/test/functional/wallet_basic.py b/test/functional/wallet_basic.py index 550037923e..15746d312c 100755 --- a/test/functional/wallet_basic.py +++ b/test/functional/wallet_basic.py @@ -15,6 +15,7 @@ from test_framework.util import ( connect_nodes, wait_until, ) +from test_framework.wallet_util import test_address class WalletTest(BitcoinTestFramework): @@ -24,6 +25,7 @@ class WalletTest(BitcoinTestFramework): "-acceptnonstdtxn=1", ]] * self.num_nodes self.setup_clean_chain = True + self.supports_cli = False def skip_test_if_missing_module(self): self.skip_if_no_wallet() @@ -390,7 +392,7 @@ class WalletTest(BitcoinTestFramework): for label in [u'рыба', u'𝅘𝅥𝅯']: addr = self.nodes[0].getnewaddress() self.nodes[0].setlabel(addr, label) - assert_equal(self.nodes[0].getaddressinfo(addr)['label'], label) + test_address(self.nodes[0], addr, labels=[label]) assert label in self.nodes[0].listlabels() self.nodes[0].rpc.ensure_ascii = True # restore to default diff --git a/test/functional/wallet_bumpfee.py b/test/functional/wallet_bumpfee.py index 0948d47653..336e246e33 100755 --- a/test/functional/wallet_bumpfee.py +++ b/test/functional/wallet_bumpfee.py @@ -38,7 +38,7 @@ class BumpFeeTest(BitcoinTestFramework): "-walletrbf={}".format(i), "-mintxfee=0.00002", "-deprecatedrpc=totalFee", - "-addresstype=p2sh-segwit", # TODO update constants in test and remove + "-addresstype=bech32", ] for i in range(self.num_nodes)] def skip_test_if_missing_module(self): @@ -71,27 +71,29 @@ class BumpFeeTest(BitcoinTestFramework): test_simple_bumpfee_succeeds(self, "default", rbf_node, peer_node, dest_address) test_simple_bumpfee_succeeds(self, "fee_rate", rbf_node, peer_node, dest_address) test_feerate_args(self, rbf_node, peer_node, dest_address) - test_segwit_bumpfee_succeeds(rbf_node, dest_address) - test_nonrbf_bumpfee_fails(peer_node, dest_address) - test_notmine_bumpfee_fails(rbf_node, peer_node, dest_address) - test_bumpfee_with_descendant_fails(rbf_node, rbf_node_address, dest_address) - test_small_output_fails(rbf_node, dest_address) - test_dust_to_fee(rbf_node, dest_address) - test_settxfee(rbf_node, dest_address) - test_rebumping(rbf_node, dest_address) - test_rebumping_not_replaceable(rbf_node, dest_address) - test_unconfirmed_not_spendable(rbf_node, rbf_node_address) - test_bumpfee_metadata(rbf_node, dest_address) - test_locked_wallet_fails(rbf_node, dest_address) - test_change_script_match(rbf_node, dest_address) + test_segwit_bumpfee_succeeds(self, rbf_node, dest_address) + test_nonrbf_bumpfee_fails(self, peer_node, dest_address) + test_notmine_bumpfee_fails(self, rbf_node, peer_node, dest_address) + test_bumpfee_with_descendant_fails(self, rbf_node, rbf_node_address, dest_address) + test_small_output_fails(self, rbf_node, dest_address) + test_dust_to_fee(self, rbf_node, dest_address) + test_settxfee(self, rbf_node, dest_address) + test_watchonly_psbt(self, peer_node, rbf_node, dest_address) + test_rebumping(self, rbf_node, dest_address) + test_rebumping_not_replaceable(self, rbf_node, dest_address) + test_unconfirmed_not_spendable(self, rbf_node, rbf_node_address) + test_bumpfee_metadata(self, rbf_node, dest_address) + test_locked_wallet_fails(self, rbf_node, dest_address) + test_change_script_match(self, rbf_node, dest_address) test_maxtxfee_fails(self, rbf_node, dest_address) # These tests wipe out a number of utxos that are expected in other tests - test_small_output_with_feerate_succeeds(rbf_node, dest_address) - test_no_more_inputs_fails(rbf_node, dest_address) + test_small_output_with_feerate_succeeds(self, rbf_node, dest_address) + test_no_more_inputs_fails(self, rbf_node, dest_address) self.log.info("Success") def test_simple_bumpfee_succeeds(self, mode, rbf_node, peer_node, dest_address): + self.log.info('Test simple bumpfee') rbfid = spend_one_input(rbf_node, dest_address) rbftx = rbf_node.gettransaction(rbfid) self.sync_mempools((rbf_node, peer_node)) @@ -101,7 +103,9 @@ def test_simple_bumpfee_succeeds(self, mode, rbf_node, peer_node, dest_address): else: bumped_tx = rbf_node.bumpfee(rbfid) assert_equal(bumped_tx["errors"], []) - assert bumped_tx["fee"] - abs(rbftx["fee"]) > 0 + assert bumped_tx["fee"] > -rbftx["fee"] + assert_equal(bumped_tx["origfee"], -rbftx["fee"]) + assert "psbt" not in bumped_tx # check that bumped_tx propagates, original tx was evicted and has a wallet conflict self.sync_mempools((rbf_node, peer_node)) assert bumped_tx["txid"] in rbf_node.getrawmempool() @@ -116,6 +120,7 @@ def test_simple_bumpfee_succeeds(self, mode, rbf_node, peer_node, dest_address): assert_equal(bumpedwtx["replaces_txid"], rbfid) def test_feerate_args(self, rbf_node, peer_node, dest_address): + self.log.info('Test feerate args') rbfid = spend_one_input(rbf_node, dest_address) self.sync_mempools((rbf_node, peer_node)) assert rbfid in rbf_node.getrawmempool() and rbfid in peer_node.getrawmempool() @@ -132,7 +137,8 @@ def test_feerate_args(self, rbf_node, peer_node, dest_address): assert_raises_rpc_error(-4, "is too high (cannot be higher than", rbf_node.bumpfee, rbfid, {"fee_rate":1}) -def test_segwit_bumpfee_succeeds(rbf_node, dest_address): +def test_segwit_bumpfee_succeeds(self, rbf_node, dest_address): + self.log.info('Test that segwit-sourcing bumpfee works') # Create a transaction with segwit output, then create an RBF transaction # which spends it, and make sure bumpfee can be called on it. @@ -162,14 +168,14 @@ def test_segwit_bumpfee_succeeds(rbf_node, dest_address): assert rbfid not in rbf_node.getrawmempool() -def test_nonrbf_bumpfee_fails(peer_node, dest_address): - # cannot replace a non RBF transaction (from node which did not enable RBF) +def test_nonrbf_bumpfee_fails(self, peer_node, dest_address): + self.log.info('Test that we cannot replace a non RBF transaction') not_rbfid = peer_node.sendtoaddress(dest_address, Decimal("0.00090000")) assert_raises_rpc_error(-4, "not BIP 125 replaceable", peer_node.bumpfee, not_rbfid) -def test_notmine_bumpfee_fails(rbf_node, peer_node, dest_address): - # cannot bump fee unless the tx has only inputs that we own. +def test_notmine_bumpfee_fails(self, rbf_node, peer_node, dest_address): + self.log.info('Test that it cannot bump fee if non-owned inputs are included') # here, the rbftx has a peer_node coin and then adds a rbf_node input # Note that this test depends upon the RPC code checking input ownership prior to change outputs # (since it can't use fundrawtransaction, it lacks a proper change output) @@ -189,8 +195,8 @@ def test_notmine_bumpfee_fails(rbf_node, peer_node, dest_address): rbf_node.bumpfee, rbfid) -def test_bumpfee_with_descendant_fails(rbf_node, rbf_node_address, dest_address): - # cannot bump fee if the transaction has a descendant +def test_bumpfee_with_descendant_fails(self, rbf_node, rbf_node_address, dest_address): + self.log.info('Test that fee cannot be bumped when it has descendant') # parent is send-to-self, so we don't have to check which output is change when creating the child tx parent_id = spend_one_input(rbf_node, rbf_node_address) tx = rbf_node.createrawtransaction([{"txid": parent_id, "vout": 0}], {dest_address: 0.00020000}) @@ -198,7 +204,8 @@ def test_bumpfee_with_descendant_fails(rbf_node, rbf_node_address, dest_address) rbf_node.sendrawtransaction(tx["hex"]) assert_raises_rpc_error(-8, "Transaction has descendants in the wallet", rbf_node.bumpfee, parent_id) -def test_small_output_fails(rbf_node, dest_address): +def test_small_output_fails(self, rbf_node, dest_address): + self.log.info('Test totalFee bump with small output fails') # cannot bump fee with a too-small output rbfid = spend_one_input(rbf_node, dest_address) rbf_node.bumpfee(rbfid, {"totalFee": 50000}) @@ -206,20 +213,21 @@ def test_small_output_fails(rbf_node, dest_address): rbfid = spend_one_input(rbf_node, dest_address) assert_raises_rpc_error(-4, "Change output is too small", rbf_node.bumpfee, rbfid, {"totalFee": 50001}) -def test_small_output_with_feerate_succeeds(rbf_node, dest_address): +def test_small_output_with_feerate_succeeds(self, rbf_node, dest_address): + self.log.info('Testing small output with feerate bump succeeds') # Make sure additional inputs exist rbf_node.generatetoaddress(101, rbf_node.getnewaddress()) rbfid = spend_one_input(rbf_node, dest_address) - original_input_list = rbf_node.getrawtransaction(rbfid, 1)["vin"] - assert_equal(len(original_input_list), 1) - original_txin = original_input_list[0] - # Keep bumping until we out-spend change output + input_list = rbf_node.getrawtransaction(rbfid, 1)["vin"] + assert_equal(len(input_list), 1) + original_txin = input_list[0] + self.log.info('Keep bumping until transaction fee out-spends non-destination value') tx_fee = 0 - while tx_fee < Decimal("0.0005"): - new_input_list = rbf_node.getrawtransaction(rbfid, 1)["vin"] - new_item = list(new_input_list)[0] - assert_equal(len(original_input_list), 1) + while True: + input_list = rbf_node.getrawtransaction(rbfid, 1)["vin"] + new_item = list(input_list)[0] + assert_equal(len(input_list), 1) assert_equal(original_txin["txid"], new_item["txid"]) assert_equal(original_txin["vout"], new_item["vout"]) rbfid_new_details = rbf_node.bumpfee(rbfid) @@ -228,7 +236,11 @@ def test_small_output_with_feerate_succeeds(rbf_node, dest_address): assert rbfid not in raw_pool assert rbfid_new in raw_pool rbfid = rbfid_new - tx_fee = rbfid_new_details["origfee"] + tx_fee = rbfid_new_details["fee"] + + # Total value from input not going to destination + if tx_fee > Decimal('0.00050000'): + break # input(s) have been added final_input_list = rbf_node.getrawtransaction(rbfid, 1)["vin"] @@ -241,22 +253,21 @@ def test_small_output_with_feerate_succeeds(rbf_node, dest_address): rbf_node.generatetoaddress(1, rbf_node.getnewaddress()) assert_equal(rbf_node.gettransaction(rbfid)["confirmations"], 1) -def test_dust_to_fee(rbf_node, dest_address): - # check that if output is reduced to dust, it will be converted to fee +def test_dust_to_fee(self, rbf_node, dest_address): + self.log.info('Test that bumped output that is dust is dropped to fee') # the bumped tx sets fee=49,900, but it converts to 50,000 rbfid = spend_one_input(rbf_node, dest_address) fulltx = rbf_node.getrawtransaction(rbfid, 1) - # (32-byte p2sh-pwpkh output size + 148 p2pkh spend estimate) * 10k(discard_rate) / 1000 = 1800 - # P2SH outputs are slightly "over-discarding" due to the IsDust calculation assuming it will - # be spent as a P2PKH. - bumped_tx = rbf_node.bumpfee(rbfid, {"totalFee": 50000 - 1800}) + # (31-vbyte p2wpkh output size + 67-vbyte p2wpkh spend estimate) * 10k(discard_rate) / 1000 = 980 + bumped_tx = rbf_node.bumpfee(rbfid, {"totalFee": 50000 - 980}) full_bumped_tx = rbf_node.getrawtransaction(bumped_tx["txid"], 1) assert_equal(bumped_tx["fee"], Decimal("0.00050000")) assert_equal(len(fulltx["vout"]), 2) assert_equal(len(full_bumped_tx["vout"]), 1) # change output is eliminated -def test_settxfee(rbf_node, dest_address): +def test_settxfee(self, rbf_node, dest_address): + self.log.info('Test settxfee') assert_raises_rpc_error(-8, "txfee cannot be less than min relay tx fee", rbf_node.settxfee, Decimal('0.000005')) assert_raises_rpc_error(-8, "txfee cannot be less than wallet min fee", rbf_node.settxfee, Decimal('0.000015')) # check that bumpfee reacts correctly to the use of settxfee (paytxfee) @@ -271,33 +282,117 @@ def test_settxfee(rbf_node, dest_address): rbf_node.settxfee(Decimal("0.00000000")) # unset paytxfee -def test_maxtxfee_fails(test, rbf_node, dest_address): - test.restart_node(1, ['-maxtxfee=0.00003'] + test.extra_args[1]) +def test_maxtxfee_fails(self, rbf_node, dest_address): + self.log.info('Test that bumpfee fails when it hits -matxfee') + # size of bumped transaction (p2wpkh, 1 input, 2 outputs): 141 vbytes + # expected bumping feerate of 20 sats/vbyte => 141*20 sats = 0.00002820 btc + self.restart_node(1, ['-maxtxfee=0.000025'] + self.extra_args[1]) rbf_node.walletpassphrase(WALLET_PASSPHRASE, WALLET_PASSPHRASE_TIMEOUT) rbfid = spend_one_input(rbf_node, dest_address) assert_raises_rpc_error(-4, "Unable to create transaction: Fee exceeds maximum configured by -maxtxfee", rbf_node.bumpfee, rbfid) - test.restart_node(1, test.extra_args[1]) + self.restart_node(1, self.extra_args[1]) rbf_node.walletpassphrase(WALLET_PASSPHRASE, WALLET_PASSPHRASE_TIMEOUT) - -def test_rebumping(rbf_node, dest_address): - # check that re-bumping the original tx fails, but bumping the bumper succeeds +def test_watchonly_psbt(self, peer_node, rbf_node, dest_address): + self.log.info('Test that PSBT is returned for bumpfee in watchonly wallets') + priv_rec_desc = "wpkh([00000001/84'/1'/0']tprv8ZgxMBicQKsPd7Uf69XL1XwhmjHopUGep8GuEiJDZmbQz6o58LninorQAfcKZWARbtRtfnLcJ5MQ2AtHcQJCCRUcMRvmDUjyEmNUWwx8UbK/0/*)#rweraev0" + pub_rec_desc = rbf_node.getdescriptorinfo(priv_rec_desc)["descriptor"] + priv_change_desc = "wpkh([00000001/84'/1'/0']tprv8ZgxMBicQKsPd7Uf69XL1XwhmjHopUGep8GuEiJDZmbQz6o58LninorQAfcKZWARbtRtfnLcJ5MQ2AtHcQJCCRUcMRvmDUjyEmNUWwx8UbK/1/*)#j6uzqvuh" + pub_change_desc = rbf_node.getdescriptorinfo(priv_change_desc)["descriptor"] + # Create a wallet with private keys that can sign PSBTs + rbf_node.createwallet(wallet_name="signer", disable_private_keys=False, blank=True) + signer = rbf_node.get_wallet_rpc("signer") + assert signer.getwalletinfo()['private_keys_enabled'] + result = signer.importmulti([{ + "desc": priv_rec_desc, + "timestamp": 0, + "range": [0,1], + "internal": False, + "keypool": False # Keys can only be imported to the keypool when private keys are disabled + }, + { + "desc": priv_change_desc, + "timestamp": 0, + "range": [0, 0], + "internal": True, + "keypool": False + }]) + assert_equal(result, [{'success': True}, {'success': True}]) + + # Create another wallet with just the public keys, which creates PSBTs + rbf_node.createwallet(wallet_name="watcher", disable_private_keys=True, blank=True) + watcher = rbf_node.get_wallet_rpc("watcher") + assert not watcher.getwalletinfo()['private_keys_enabled'] + + result = watcher.importmulti([{ + "desc": pub_rec_desc, + "timestamp": 0, + "range": [0,10], + "internal": False, + "keypool": True, + "watchonly": True + }, + { + "desc": pub_change_desc, + "timestamp": 0, + "range": [0, 10], + "internal": True, + "keypool": True, + "watchonly": True + }]) + assert_equal(result, [{'success': True}, {'success': True}]) + + funding_address1 = watcher.getnewaddress(address_type='bech32') + funding_address2 = watcher.getnewaddress(address_type='bech32') + peer_node.sendmany("", {funding_address1: 0.001, funding_address2: 0.001}) + peer_node.generate(1) + self.sync_all() + + # Create single-input PSBT for transaction to be bumped + psbt = watcher.walletcreatefundedpsbt([], {dest_address:0.0005}, 0, {"feeRate": 0.00001}, True)['psbt'] + psbt_signed = signer.walletprocesspsbt(psbt=psbt, sign=True, sighashtype="ALL", bip32derivs=True) + psbt_final = watcher.finalizepsbt(psbt_signed["psbt"]) + original_txid = watcher.sendrawtransaction(psbt_final["hex"]) + assert_equal(len(watcher.decodepsbt(psbt)["tx"]["vin"]), 1) + + # Bump fee, obnoxiously high to add additional watchonly input + bumped_psbt = watcher.bumpfee(original_txid, {"fee_rate":0.005}) + assert_greater_than(len(watcher.decodepsbt(bumped_psbt['psbt'])["tx"]["vin"]), 1) + assert "txid" not in bumped_psbt + assert_equal(bumped_psbt["origfee"], -watcher.gettransaction(original_txid)["fee"]) + assert not watcher.finalizepsbt(bumped_psbt["psbt"])["complete"] + + # Sign bumped transaction + bumped_psbt_signed = signer.walletprocesspsbt(psbt=bumped_psbt["psbt"], sign=True, sighashtype="ALL", bip32derivs=True) + bumped_psbt_final = watcher.finalizepsbt(bumped_psbt_signed["psbt"]) + assert bumped_psbt_final["complete"] + + # Broadcast bumped transaction + bumped_txid = watcher.sendrawtransaction(bumped_psbt_final["hex"]) + assert bumped_txid in rbf_node.getrawmempool() + assert original_txid not in rbf_node.getrawmempool() + + rbf_node.unloadwallet("watcher") + rbf_node.unloadwallet("signer") + +def test_rebumping(self, rbf_node, dest_address): + self.log.info('Test that re-bumping the original tx fails, but bumping successor works') rbfid = spend_one_input(rbf_node, dest_address) bumped = rbf_node.bumpfee(rbfid, {"totalFee": 2000}) assert_raises_rpc_error(-4, "already bumped", rbf_node.bumpfee, rbfid, {"totalFee": 3000}) rbf_node.bumpfee(bumped["txid"], {"totalFee": 3000}) -def test_rebumping_not_replaceable(rbf_node, dest_address): - # check that re-bumping a non-replaceable bump tx fails +def test_rebumping_not_replaceable(self, rbf_node, dest_address): + self.log.info('Test that re-bumping non-replaceable fails') rbfid = spend_one_input(rbf_node, dest_address) bumped = rbf_node.bumpfee(rbfid, {"totalFee": 10000, "replaceable": False}) assert_raises_rpc_error(-4, "Transaction is not BIP 125 replaceable", rbf_node.bumpfee, bumped["txid"], {"totalFee": 20000}) -def test_unconfirmed_not_spendable(rbf_node, rbf_node_address): - # check that unconfirmed outputs from bumped transactions are not spendable +def test_unconfirmed_not_spendable(self, rbf_node, rbf_node_address): + self.log.info('Test that unconfirmed outputs from bumped txns are not spendable') rbfid = spend_one_input(rbf_node, rbf_node_address) rbftx = rbf_node.gettransaction(rbfid)["hex"] assert rbfid in rbf_node.getrawmempool() @@ -335,7 +430,8 @@ def test_unconfirmed_not_spendable(rbf_node, rbf_node_address): if t["txid"] == rbfid and t["address"] == rbf_node_address and t["spendable"]), 1) -def test_bumpfee_metadata(rbf_node, dest_address): +def test_bumpfee_metadata(self, rbf_node, dest_address): + self.log.info('Test that bumped txn metadata persists to new txn record') assert(rbf_node.getbalance() < 49) rbf_node.generatetoaddress(101, rbf_node.getnewaddress()) rbfid = rbf_node.sendtoaddress(dest_address, 49, "comment value", "to value") @@ -345,15 +441,17 @@ def test_bumpfee_metadata(rbf_node, dest_address): assert_equal(bumped_wtx["to"], "to value") -def test_locked_wallet_fails(rbf_node, dest_address): +def test_locked_wallet_fails(self, rbf_node, dest_address): + self.log.info('Test that locked wallet cannot bump txn') rbfid = spend_one_input(rbf_node, dest_address) rbf_node.walletlock() assert_raises_rpc_error(-13, "Please enter the wallet passphrase with walletpassphrase first.", rbf_node.bumpfee, rbfid) rbf_node.walletpassphrase(WALLET_PASSPHRASE, WALLET_PASSPHRASE_TIMEOUT) -def test_change_script_match(rbf_node, dest_address): - """Test that the same change addresses is used for the replacement transaction when possible.""" +def test_change_script_match(self, rbf_node, dest_address): + self.log.info('Test that the same change addresses is used for the replacement transaction when possible.') + def get_change_address(tx): tx_details = rbf_node.getrawtransaction(tx, 1) txout_addresses = [txout['scriptPubKey']['addresses'][0] for txout in tx_details["vout"]] @@ -397,7 +495,8 @@ def submit_block_with_tx(node, tx): node.submitblock(block.serialize().hex()) return block -def test_no_more_inputs_fails(rbf_node, dest_address): +def test_no_more_inputs_fails(self, rbf_node, dest_address): + self.log.info('Test that bumpfee fails when there are no available confirmed outputs') # feerate rbf requires confirmed outputs when change output doesn't exist or is insufficient rbf_node.generatetoaddress(1, dest_address) # spend all funds, no change output diff --git a/test/functional/wallet_createwallet.py b/test/functional/wallet_createwallet.py index e302e499f4..dabd78f66c 100755 --- a/test/functional/wallet_createwallet.py +++ b/test/functional/wallet_createwallet.py @@ -15,7 +15,6 @@ class CreateWalletTest(BitcoinTestFramework): def set_test_params(self): self.setup_clean_chain = False self.num_nodes = 1 - self.supports_cli = True def skip_test_if_missing_module(self): self.skip_if_no_wallet() @@ -80,7 +79,7 @@ class CreateWalletTest(BitcoinTestFramework): assert_raises_rpc_error(-4, "Error: This wallet has no available keys", w4.getnewaddress) assert_raises_rpc_error(-4, "Error: This wallet has no available keys", w4.getrawchangeaddress) # Now set a seed and it should work. Wallet should also be encrypted - w4.walletpassphrase('pass', 2) + w4.walletpassphrase('pass', 60) w4.sethdseed() w4.getnewaddress() w4.getrawchangeaddress() diff --git a/test/functional/wallet_dump.py b/test/functional/wallet_dump.py index 53edf710b9..a39dfc7895 100755 --- a/test/functional/wallet_dump.py +++ b/test/functional/wallet_dump.py @@ -137,7 +137,7 @@ class WalletDumpTest(BitcoinTestFramework): # encrypt wallet, restart, unlock and dump self.nodes[0].encryptwallet('test') - self.nodes[0].walletpassphrase('test', 10) + self.nodes[0].walletpassphrase('test', 100) # Should be a no-op: self.nodes[0].keypoolrefill() self.nodes[0].dumpwallet(wallet_enc_dump) diff --git a/test/functional/wallet_encryption.py b/test/functional/wallet_encryption.py index fbcb4e75ba..bc7e3cca59 100755 --- a/test/functional/wallet_encryption.py +++ b/test/functional/wallet_encryption.py @@ -1,5 +1,5 @@ #!/usr/bin/env python3 -# Copyright (c) 2016-2018 The Bitcoin Core developers +# Copyright (c) 2016-2019 The Bitcoin Core developers # Distributed under the MIT software license, see the accompanying # file COPYING or http://www.opensource.org/licenses/mit-license.php. """Test Wallet encryption""" diff --git a/test/functional/wallet_groups.py b/test/functional/wallet_groups.py index d1178611bd..f2fa1d3e40 100755 --- a/test/functional/wallet_groups.py +++ b/test/functional/wallet_groups.py @@ -1,5 +1,5 @@ #!/usr/bin/env python3 -# Copyright (c) 2018 The Bitcoin Core developers +# Copyright (c) 2018-2019 The Bitcoin Core developers # Distributed under the MIT software license, see the accompanying # file COPYING or http://www.opensource.org/licenses/mit-license.php. """Test wallet group functionality.""" @@ -16,7 +16,7 @@ class WalletGroupTest(BitcoinTestFramework): self.setup_clean_chain = True self.num_nodes = 3 self.extra_args = [[], [], ['-avoidpartialspends']] - self.rpc_timeout = 120 + self.rpc_timeout = 240 def skip_test_if_missing_module(self): self.skip_if_no_wallet() diff --git a/test/functional/wallet_hd.py b/test/functional/wallet_hd.py index fa5d5a8878..7497475b67 100755 --- a/test/functional/wallet_hd.py +++ b/test/functional/wallet_hd.py @@ -20,6 +20,7 @@ class WalletHDTest(BitcoinTestFramework): self.setup_clean_chain = True self.num_nodes = 2 self.extra_args = [[], ['-keypool=0']] + self.supports_cli = False def skip_test_if_missing_module(self): self.skip_if_no_wallet() @@ -67,11 +68,11 @@ class WalletHDTest(BitcoinTestFramework): self.log.info("Restore backup ...") self.stop_node(1) - # we need to delete the complete regtest directory + # we need to delete the complete chain directory # otherwise node1 would auto-recover all funds in flag the keypool keys as used - shutil.rmtree(os.path.join(self.nodes[1].datadir, "regtest", "blocks")) - shutil.rmtree(os.path.join(self.nodes[1].datadir, "regtest", "chainstate")) - shutil.copyfile(os.path.join(self.nodes[1].datadir, "hd.bak"), os.path.join(self.nodes[1].datadir, "regtest", "wallets", "wallet.dat")) + shutil.rmtree(os.path.join(self.nodes[1].datadir, self.chain, "blocks")) + shutil.rmtree(os.path.join(self.nodes[1].datadir, self.chain, "chainstate")) + shutil.copyfile(os.path.join(self.nodes[1].datadir, "hd.bak"), os.path.join(self.nodes[1].datadir, self.chain, "wallets", "wallet.dat")) self.start_node(1) # Assert that derivation is deterministic @@ -92,9 +93,9 @@ class WalletHDTest(BitcoinTestFramework): # Try a RPC based rescan self.stop_node(1) - shutil.rmtree(os.path.join(self.nodes[1].datadir, "regtest", "blocks")) - shutil.rmtree(os.path.join(self.nodes[1].datadir, "regtest", "chainstate")) - shutil.copyfile(os.path.join(self.nodes[1].datadir, "hd.bak"), os.path.join(self.nodes[1].datadir, "regtest", "wallets", "wallet.dat")) + shutil.rmtree(os.path.join(self.nodes[1].datadir, self.chain, "blocks")) + shutil.rmtree(os.path.join(self.nodes[1].datadir, self.chain, "chainstate")) + shutil.copyfile(os.path.join(self.nodes[1].datadir, "hd.bak"), os.path.join(self.nodes[1].datadir, self.chain, "wallets", "wallet.dat")) self.start_node(1, extra_args=self.extra_args[1]) connect_nodes(self.nodes[0], 1) self.sync_all() diff --git a/test/functional/wallet_implicitsegwit.py b/test/functional/wallet_implicitsegwit.py new file mode 100755 index 0000000000..a8583e2879 --- /dev/null +++ b/test/functional/wallet_implicitsegwit.py @@ -0,0 +1,65 @@ +#!/usr/bin/env python3 +# Copyright (c) 2019 The Bitcoin Core developers +# Distributed under the MIT software license, see the accompanying +# file COPYING or http://www.opensource.org/licenses/mit-license.php. +"""Test the wallet implicit segwit feature.""" + +import test_framework.address as address +from test_framework.test_framework import BitcoinTestFramework + +# TODO: Might be nice to test p2pk here too +address_types = ('legacy', 'bech32', 'p2sh-segwit') + +def key_to_address(key, address_type): + if address_type == 'legacy': + return address.key_to_p2pkh(key) + elif address_type == 'p2sh-segwit': + return address.key_to_p2sh_p2wpkh(key) + elif address_type == 'bech32': + return address.key_to_p2wpkh(key) + +def send_a_to_b(receive_node, send_node): + keys = {} + for a in address_types: + a_address = receive_node.getnewaddress(address_type=a) + pubkey = receive_node.getaddressinfo(a_address)['pubkey'] + keys[a] = pubkey + for b in address_types: + b_address = key_to_address(pubkey, b) + send_node.sendtoaddress(address=b_address, amount=1) + return keys + +def check_implicit_transactions(implicit_keys, implicit_node): + # The implicit segwit node allows conversion all possible ways + txs = implicit_node.listtransactions(None, 99999) + for a in address_types: + pubkey = implicit_keys[a] + for b in address_types: + b_address = key_to_address(pubkey, b) + assert(('receive', b_address) in tuple((tx['category'], tx['address']) for tx in txs)) + +class ImplicitSegwitTest(BitcoinTestFramework): + def set_test_params(self): + self.num_nodes = 2 + self.supports_cli = False + + def skip_test_if_missing_module(self): + self.skip_if_no_wallet() + + def run_test(self): + self.log.info("Manipulating addresses and sending transactions to all variations") + implicit_keys = send_a_to_b(self.nodes[0], self.nodes[1]) + + self.sync_all() + + self.log.info("Checking that transactions show up correctly without a restart") + check_implicit_transactions(implicit_keys, self.nodes[0]) + + self.log.info("Checking that transactions still show up correctly after a restart") + self.restart_node(0) + self.restart_node(1) + + check_implicit_transactions(implicit_keys, self.nodes[0]) + +if __name__ == '__main__': + ImplicitSegwitTest().main() diff --git a/test/functional/wallet_import_rescan.py b/test/functional/wallet_import_rescan.py index 79062a4a29..b8b85b7a19 100755 --- a/test/functional/wallet_import_rescan.py +++ b/test/functional/wallet_import_rescan.py @@ -145,6 +145,7 @@ def get_rand_amount(): class ImportRescanTest(BitcoinTestFramework): def set_test_params(self): self.num_nodes = 2 + len(IMPORT_NODES) + self.supports_cli = False def skip_test_if_missing_module(self): self.skip_if_no_wallet() diff --git a/test/functional/wallet_import_with_label.py b/test/functional/wallet_import_with_label.py index 2a9051b1e8..6a9d2e8290 100755 --- a/test/functional/wallet_import_with_label.py +++ b/test/functional/wallet_import_with_label.py @@ -1,5 +1,5 @@ #!/usr/bin/env python3 -# Copyright (c) 2018 The Bitcoin Core developers +# Copyright (c) 2018-2020 The Bitcoin Core developers # Distributed under the MIT software license, see the accompanying # file COPYING or http://www.opensource.org/licenses/mit-license.php. """Test the behavior of RPC importprivkey on set and unset labels of @@ -36,7 +36,7 @@ class ImportWithLabel(BitcoinTestFramework): address, iswatchonly=True, ismine=False, - label=label) + labels=[label]) self.log.info( "Import the watch-only address's private key without a " @@ -44,10 +44,7 @@ class ImportWithLabel(BitcoinTestFramework): ) priv_key = self.nodes[0].dumpprivkey(address) self.nodes[1].importprivkey(priv_key) - - test_address(self.nodes[1], - address, - label=label) + test_address(self.nodes[1], address, labels=[label]) self.log.info( "Test importaddress without label and importprivkey with label." @@ -59,7 +56,7 @@ class ImportWithLabel(BitcoinTestFramework): address2, iswatchonly=True, ismine=False, - label="") + labels=[""]) self.log.info( "Import the watch-only address's private key with a " @@ -69,9 +66,7 @@ class ImportWithLabel(BitcoinTestFramework): label2 = "Test Label 2" self.nodes[1].importprivkey(priv_key2, label2) - test_address(self.nodes[1], - address2, - label=label2) + test_address(self.nodes[1], address2, labels=[label2]) self.log.info("Test importaddress with label and importprivkey with label.") self.log.info("Import a watch-only address with a label.") @@ -82,7 +77,7 @@ class ImportWithLabel(BitcoinTestFramework): address3, iswatchonly=True, ismine=False, - label=label3_addr) + labels=[label3_addr]) self.log.info( "Import the watch-only address's private key with a " @@ -92,9 +87,7 @@ class ImportWithLabel(BitcoinTestFramework): label3_priv = "Test Label 3 for importprivkey" self.nodes[1].importprivkey(priv_key3, label3_priv) - test_address(self.nodes[1], - address3, - label=label3_priv) + test_address(self.nodes[1], address3, labels=[label3_priv]) self.log.info( "Test importprivkey won't label new dests with the same " @@ -108,7 +101,7 @@ class ImportWithLabel(BitcoinTestFramework): address4, iswatchonly=True, ismine=False, - label=label4_addr, + labels=[label4_addr], embedded=None) self.log.info( @@ -121,12 +114,9 @@ class ImportWithLabel(BitcoinTestFramework): self.nodes[1].importprivkey(priv_key4) embedded_addr = self.nodes[1].getaddressinfo(address4)['embedded']['address'] - test_address(self.nodes[1], - embedded_addr, - label="") - test_address(self.nodes[1], - address4, - label=label4_addr) + test_address(self.nodes[1], embedded_addr, labels=[""]) + + test_address(self.nodes[1], address4, labels=[label4_addr]) self.stop_nodes() diff --git a/test/functional/wallet_importmulti.py b/test/functional/wallet_importmulti.py index 23748e5dd7..f152fcd1a4 100755 --- a/test/functional/wallet_importmulti.py +++ b/test/functional/wallet_importmulti.py @@ -121,7 +121,7 @@ class ImportMultiTest(BitcoinTestFramework): self.test_importmulti({"scriptPubKey": key.p2pkh_script, "timestamp": "now", "internal": True, - "label": "Example label"}, + "label": "Unsuccessful labelling for internal addresses"}, success=False, error_code=-8, error_message='Internal addresses should not have a label') @@ -550,7 +550,7 @@ class ImportMultiTest(BitcoinTestFramework): self.log.info("Should not import a p2sh-p2wpkh address from descriptor without checksum and private key") self.test_importmulti({"desc": "sh(wpkh(" + key.pubkey + "))", "timestamp": "now", - "label": "Descriptor import test", + "label": "Unsuccessful P2SH-P2WPKH descriptor import", "keys": [key.privkey]}, success=False, error_code=-5, @@ -558,17 +558,18 @@ class ImportMultiTest(BitcoinTestFramework): # Test importing of a P2SH-P2WPKH address via descriptor + private key key = get_key(self.nodes[0]) + p2sh_p2wpkh_label = "Successful P2SH-P2WPKH descriptor import" self.log.info("Should import a p2sh-p2wpkh address from descriptor and private key") self.test_importmulti({"desc": descsum_create("sh(wpkh(" + key.pubkey + "))"), "timestamp": "now", - "label": "Descriptor import test", + "label": p2sh_p2wpkh_label, "keys": [key.privkey]}, success=True) test_address(self.nodes[1], key.p2sh_p2wpkh_addr, solvable=True, ismine=True, - label="Descriptor import test") + labels=[p2sh_p2wpkh_label]) # Test ranged descriptor fails if range is not specified xpriv = "tprv8ZgxMBicQKsPeuVhWwi6wuMQGfPKi9Li5GtX35jVNknACgqe3CY4g5xgkfDDJcmtF7o1QnxWDRYw4H5P26PXq7sbcUkEqeR4fg3Kxp2tigg" @@ -628,17 +629,18 @@ class ImportMultiTest(BitcoinTestFramework): # Test importing of a P2PKH address via descriptor key = get_key(self.nodes[0]) + p2pkh_label = "P2PKH descriptor import" self.log.info("Should import a p2pkh address from descriptor") self.test_importmulti({"desc": descsum_create("pkh(" + key.pubkey + ")"), "timestamp": "now", - "label": "Descriptor import test"}, + "label": p2pkh_label}, True, warnings=["Some private keys are missing, outputs will be considered watchonly. If this is intentional, specify the watchonly flag."]) test_address(self.nodes[1], key.p2pkh_addr, solvable=True, ismine=False, - label="Descriptor import test") + labels=[p2pkh_label]) # Test import fails if both desc and scriptPubKey are provided key = get_key(self.nodes[0]) @@ -675,7 +677,7 @@ class ImportMultiTest(BitcoinTestFramework): # Import pubkeys with key origin info self.log.info("Addresses should have hd keypath and master key id after import with key origin") pub_addr = self.nodes[1].getnewaddress() - pub_addr = self.nodes[1].getnewaddress() + pub_addr = self.nodes[1].getnewaddress(address_type="bech32") info = self.nodes[1].getaddressinfo(pub_addr) pub = info['pubkey'] pub_keypath = info['hdkeypath'] @@ -693,7 +695,7 @@ class ImportMultiTest(BitcoinTestFramework): assert_equal(pub_import_info['hdkeypath'], pub_keypath) # Import privkeys with key origin info - priv_addr = self.nodes[1].getnewaddress() + priv_addr = self.nodes[1].getnewaddress(address_type="bech32") info = self.nodes[1].getaddressinfo(priv_addr) priv = self.nodes[1].dumpprivkey(priv_addr) priv_keypath = info['hdkeypath'] @@ -742,8 +744,8 @@ class ImportMultiTest(BitcoinTestFramework): self.nodes[1].createwallet(wallet_name="noprivkeys", disable_private_keys=True) wrpc = self.nodes[1].get_wallet_rpc("noprivkeys") - addr1 = self.nodes[0].getnewaddress() - addr2 = self.nodes[0].getnewaddress() + addr1 = self.nodes[0].getnewaddress(address_type="bech32") + addr2 = self.nodes[0].getnewaddress(address_type="bech32") pub1 = self.nodes[0].getaddressinfo(addr1)['pubkey'] pub2 = self.nodes[0].getaddressinfo(addr2)['pubkey'] result = wrpc.importmulti( @@ -761,15 +763,15 @@ class ImportMultiTest(BitcoinTestFramework): assert result[0]['success'] assert result[1]['success'] assert_equal(wrpc.getwalletinfo()["keypoolsize"], 2) - newaddr1 = wrpc.getnewaddress() + newaddr1 = wrpc.getnewaddress(address_type="bech32") assert_equal(addr1, newaddr1) - newaddr2 = wrpc.getnewaddress() + newaddr2 = wrpc.getnewaddress(address_type="bech32") assert_equal(addr2, newaddr2) # Import some public keys to the internal keypool of a no privkey wallet self.log.info("Adding pubkey to internal keypool of disableprivkey wallet") - addr1 = self.nodes[0].getnewaddress() - addr2 = self.nodes[0].getnewaddress() + addr1 = self.nodes[0].getnewaddress(address_type="bech32") + addr2 = self.nodes[0].getnewaddress(address_type="bech32") pub1 = self.nodes[0].getaddressinfo(addr1)['pubkey'] pub2 = self.nodes[0].getaddressinfo(addr2)['pubkey'] result = wrpc.importmulti( @@ -789,15 +791,15 @@ class ImportMultiTest(BitcoinTestFramework): assert result[0]['success'] assert result[1]['success'] assert_equal(wrpc.getwalletinfo()["keypoolsize_hd_internal"], 2) - newaddr1 = wrpc.getrawchangeaddress() + newaddr1 = wrpc.getrawchangeaddress(address_type="bech32") assert_equal(addr1, newaddr1) - newaddr2 = wrpc.getrawchangeaddress() + newaddr2 = wrpc.getrawchangeaddress(address_type="bech32") assert_equal(addr2, newaddr2) # Import a multisig and make sure the keys don't go into the keypool self.log.info('Imported scripts with pubkeys should not have their pubkeys go into the keypool') - addr1 = self.nodes[0].getnewaddress() - addr2 = self.nodes[0].getnewaddress() + addr1 = self.nodes[0].getnewaddress(address_type="bech32") + addr2 = self.nodes[0].getnewaddress(address_type="bech32") pub1 = self.nodes[0].getaddressinfo(addr1)['pubkey'] pub2 = self.nodes[0].getaddressinfo(addr2)['pubkey'] result = wrpc.importmulti( diff --git a/test/functional/wallet_keypool_topup.py b/test/functional/wallet_keypool_topup.py index 2e70a9e0a5..829633a050 100755 --- a/test/functional/wallet_keypool_topup.py +++ b/test/functional/wallet_keypool_topup.py @@ -30,7 +30,7 @@ class KeypoolRestoreTest(BitcoinTestFramework): self.skip_if_no_wallet() def run_test(self): - wallet_path = os.path.join(self.nodes[1].datadir, "regtest", "wallets", "wallet.dat") + wallet_path = os.path.join(self.nodes[1].datadir, self.chain, "wallets", "wallet.dat") wallet_backup_path = os.path.join(self.nodes[1].datadir, "wallet.bak") self.nodes[0].generate(101) diff --git a/test/functional/wallet_labels.py b/test/functional/wallet_labels.py index b71dae9f40..337d2e55d9 100755 --- a/test/functional/wallet_labels.py +++ b/test/functional/wallet_labels.py @@ -1,5 +1,5 @@ #!/usr/bin/env python3 -# Copyright (c) 2016-2018 The Bitcoin Core developers +# Copyright (c) 2016-2020 The Bitcoin Core developers # Distributed under the MIT software license, see the accompanying # file COPYING or http://www.opensource.org/licenses/mit-license.php. """Test label RPCs. @@ -13,6 +13,8 @@ from collections import defaultdict from test_framework.test_framework import BitcoinTestFramework from test_framework.util import assert_equal, assert_raises_rpc_error +from test_framework.wallet_util import test_address + class WalletLabelsTest(BitcoinTestFramework): def set_test_params(self): @@ -152,14 +154,9 @@ class Label: def verify(self, node): if self.receive_address is not None: assert self.receive_address in self.addresses - for address in self.addresses: - assert_equal( - node.getaddressinfo(address)['labels'][0], - {"name": self.name, - "purpose": self.purpose[address]}) - assert_equal(node.getaddressinfo(address)['label'], self.name) - + test_address(node, address, labels=[self.name]) + assert self.name in node.listlabels() assert_equal( node.getaddressesbylabel(self.name), {address: {"purpose": self.purpose[address]} for address in self.addresses}) diff --git a/test/functional/wallet_listreceivedby.py b/test/functional/wallet_listreceivedby.py index 5e94068930..b0590b149a 100755 --- a/test/functional/wallet_listreceivedby.py +++ b/test/functional/wallet_listreceivedby.py @@ -11,6 +11,7 @@ from test_framework.util import ( assert_equal, assert_raises_rpc_error, ) +from test_framework.wallet_util import test_address class ReceivedByTest(BitcoinTestFramework): @@ -19,6 +20,7 @@ class ReceivedByTest(BitcoinTestFramework): def skip_test_if_missing_module(self): self.skip_if_no_wallet() + self.skip_if_no_cli() def run_test(self): # Generate block to get out of IBD @@ -126,7 +128,7 @@ class ReceivedByTest(BitcoinTestFramework): # set pre-state label = '' address = self.nodes[1].getnewaddress() - assert_equal(self.nodes[1].getaddressinfo(address)['label'], label) + test_address(self.nodes[1], address, labels=[label]) received_by_label_json = [r for r in self.nodes[1].listreceivedbylabel() if r["label"] == label][0] balance_by_label = self.nodes[1].getreceivedbylabel(label) diff --git a/test/functional/wallet_listsinceblock.py b/test/functional/wallet_listsinceblock.py index 4aeb393255..229eda9806 100755 --- a/test/functional/wallet_listsinceblock.py +++ b/test/functional/wallet_listsinceblock.py @@ -2,9 +2,10 @@ # Copyright (c) 2017-2019 The Bitcoin Core developers # Distributed under the MIT software license, see the accompanying # file COPYING or http://www.opensource.org/licenses/mit-license.php. -"""Test the listsincelast RPC.""" +"""Test the listsinceblock RPC.""" from test_framework.test_framework import BitcoinTestFramework +from test_framework.messages import BIP125_SEQUENCE_NUMBER from test_framework.util import ( assert_array_result, assert_equal, @@ -12,6 +13,7 @@ from test_framework.util import ( connect_nodes, ) +from decimal import Decimal class ListSinceBlockTest(BitcoinTestFramework): def set_test_params(self): @@ -33,10 +35,13 @@ class ListSinceBlockTest(BitcoinTestFramework): self.test_reorg() self.test_double_spend() self.test_double_send() + self.double_spends_filtered() def test_no_blockhash(self): + self.log.info("Test no blockhash") txid = self.nodes[2].sendtoaddress(self.nodes[0].getnewaddress(), 1) blockhash, = self.nodes[2].generate(1) + blockheight = self.nodes[2].getblockheader(blockhash)['height'] self.sync_all() txs = self.nodes[0].listtransactions() @@ -44,6 +49,7 @@ class ListSinceBlockTest(BitcoinTestFramework): "category": "receive", "amount": 1, "blockhash": blockhash, + "blockheight": blockheight, "confirmations": 1, }) assert_equal( @@ -58,6 +64,7 @@ class ListSinceBlockTest(BitcoinTestFramework): "transactions": txs}) def test_invalid_blockhash(self): + self.log.info("Test invalid blockhash") assert_raises_rpc_error(-5, "Block not found", self.nodes[0].listsinceblock, "42759cde25462784395a337460bde75f58e73d3f08bd31fdc3507cbac856a2c4") assert_raises_rpc_error(-5, "Block not found", self.nodes[0].listsinceblock, @@ -95,6 +102,7 @@ class ListSinceBlockTest(BitcoinTestFramework): This test only checks that [tx0] is present. ''' + self.log.info("Test reorg") # Split network into two self.split_network() @@ -105,7 +113,7 @@ class ListSinceBlockTest(BitcoinTestFramework): # generate on both sides lastblockhash = self.nodes[1].generate(6)[5] self.nodes[2].generate(7) - self.log.info('lastblockhash=%s' % (lastblockhash)) + self.log.debug('lastblockhash={}'.format(lastblockhash)) self.sync_all(self.nodes[:2]) self.sync_all(self.nodes[2:]) @@ -150,6 +158,7 @@ class ListSinceBlockTest(BitcoinTestFramework): until the fork point, and to include all transactions that relate to the node wallet. ''' + self.log.info("Test double spend") self.sync_all() @@ -229,6 +238,7 @@ class ListSinceBlockTest(BitcoinTestFramework): 3. It is listed with a confirmation count of 2 (bb3, bb4), not 3 (aa1, aa2, aa3). ''' + self.log.info("Test double send") self.sync_all() @@ -273,7 +283,8 @@ class ListSinceBlockTest(BitcoinTestFramework): self.sync_all() # gettransaction should work for txid1 - self.nodes[0].gettransaction(txid1) + tx1 = self.nodes[0].gettransaction(txid1) + assert_equal(tx1['blockheight'], self.nodes[0].getblockheader(tx1['blockhash'])['height']) # listsinceblock(lastblockhash) should now include txid1 in transactions # as well as in removed @@ -291,5 +302,52 @@ class ListSinceBlockTest(BitcoinTestFramework): if tx['txid'] == txid1: assert_equal(tx['confirmations'], 2) + def double_spends_filtered(self): + ''' + `listsinceblock` was returning conflicted transactions even if they + occurred before the specified cutoff blockhash + ''' + self.log.info("Test spends filtered") + spending_node = self.nodes[2] + dest_address = spending_node.getnewaddress() + + tx_input = dict( + sequence=BIP125_SEQUENCE_NUMBER, **next(u for u in spending_node.listunspent())) + rawtx = spending_node.createrawtransaction( + [tx_input], {dest_address: tx_input["amount"] - Decimal("0.00051000"), + spending_node.getrawchangeaddress(): Decimal("0.00050000")}) + signedtx = spending_node.signrawtransactionwithwallet(rawtx) + orig_tx_id = spending_node.sendrawtransaction(signedtx["hex"]) + original_tx = spending_node.gettransaction(orig_tx_id) + + double_tx = spending_node.bumpfee(orig_tx_id) + + # check that both transactions exist + block_hash = spending_node.listsinceblock( + spending_node.getblockhash(spending_node.getblockcount())) + original_found = False + double_found = False + for tx in block_hash['transactions']: + if tx['txid'] == original_tx['txid']: + original_found = True + if tx['txid'] == double_tx['txid']: + double_found = True + assert_equal(original_found, True) + assert_equal(double_found, True) + + lastblockhash = spending_node.generate(1)[0] + + # check that neither transaction exists + block_hash = spending_node.listsinceblock(lastblockhash) + original_found = False + double_found = False + for tx in block_hash['transactions']: + if tx['txid'] == original_tx['txid']: + original_found = True + if tx['txid'] == double_tx['txid']: + double_found = True + assert_equal(original_found, False) + assert_equal(double_found, False) + if __name__ == '__main__': ListSinceBlockTest().main() diff --git a/test/functional/wallet_listtransactions.py b/test/functional/wallet_listtransactions.py index 997d6e702c..8c44a070b8 100755 --- a/test/functional/wallet_listtransactions.py +++ b/test/functional/wallet_listtransactions.py @@ -40,14 +40,15 @@ class ListTransactionsTest(BitcoinTestFramework): {"txid": txid}, {"category": "receive", "amount": Decimal("0.1"), "confirmations": 0}) # mine a block, confirmations should change: - self.nodes[0].generate(1) + blockhash = self.nodes[0].generate(1)[0] + blockheight = self.nodes[0].getblockheader(blockhash)['height'] self.sync_all() assert_array_result(self.nodes[0].listtransactions(), {"txid": txid}, - {"category": "send", "amount": Decimal("-0.1"), "confirmations": 1}) + {"category": "send", "amount": Decimal("-0.1"), "confirmations": 1, "blockhash": blockhash, "blockheight": blockheight}) assert_array_result(self.nodes[1].listtransactions(), {"txid": txid}, - {"category": "receive", "amount": Decimal("0.1"), "confirmations": 1}) + {"category": "receive", "amount": Decimal("0.1"), "confirmations": 1, "blockhash": blockhash, "blockheight": blockheight}) # send-to-self: txid = self.nodes[0].sendtoaddress(self.nodes[0].getnewaddress(), 0.2) diff --git a/test/functional/wallet_multiwallet.py b/test/functional/wallet_multiwallet.py index 68bc45f986..f2fa41b647 100755 --- a/test/functional/wallet_multiwallet.py +++ b/test/functional/wallet_multiwallet.py @@ -24,7 +24,6 @@ class MultiWalletTest(BitcoinTestFramework): def set_test_params(self): self.setup_clean_chain = True self.num_nodes = 2 - self.supports_cli = True def skip_test_if_missing_module(self): self.skip_if_no_wallet() @@ -39,7 +38,7 @@ class MultiWalletTest(BitcoinTestFramework): def run_test(self): node = self.nodes[0] - data_dir = lambda *p: os.path.join(node.datadir, 'regtest', *p) + data_dir = lambda *p: os.path.join(node.datadir, self.chain, *p) wallet_dir = lambda *p: data_dir('wallets', *p) wallet = lambda name: node.get_wallet_rpc(name) @@ -187,7 +186,7 @@ class MultiWalletTest(BitcoinTestFramework): assert_equal(w4.getbalance(), 3) batch = w1.batch([w1.getblockchaininfo.get_request(), w1.getwalletinfo.get_request()]) - assert_equal(batch[0]["result"]["chain"], "regtest") + assert_equal(batch[0]["result"]["chain"], self.chain) assert_equal(batch[1]["result"]["walletname"], "w1") self.log.info('Check for per-wallet settxfee call') @@ -339,14 +338,10 @@ class MultiWalletTest(BitcoinTestFramework): self.log.info("Fail -upgradewallet that results in downgrade") assert_raises_rpc_error( -4, - "Wallet loading failed.", + 'Wallet loading failed: Error loading {}: Wallet requires newer version of {}'.format( + wallet_dir('high_minversion', 'wallet.dat'), self.config['environment']['PACKAGE_NAME']), lambda: self.nodes[0].loadwallet(filename='high_minversion'), ) - self.stop_node( - i=0, - expected_stderr='Error: Error loading {}: Wallet requires newer version of Bitcoin Core'.format( - wallet_dir('high_minversion', 'wallet.dat')), - ) if __name__ == '__main__': diff --git a/test/functional/wallet_reorgsrestore.py b/test/functional/wallet_reorgsrestore.py index f48018e9fb..497a5dd95e 100755 --- a/test/functional/wallet_reorgsrestore.py +++ b/test/functional/wallet_reorgsrestore.py @@ -90,7 +90,7 @@ class ReorgsRestoreTest(BitcoinTestFramework): # Node0 wallet file is loaded on longest sync'ed node1 self.stop_node(1) self.nodes[0].backupwallet(os.path.join(self.nodes[0].datadir, 'wallet.bak')) - shutil.copyfile(os.path.join(self.nodes[0].datadir, 'wallet.bak'), os.path.join(self.nodes[1].datadir, 'regtest', 'wallet.dat')) + shutil.copyfile(os.path.join(self.nodes[0].datadir, 'wallet.bak'), os.path.join(self.nodes[1].datadir, self.chain, 'wallet.dat')) self.start_node(1) tx_after_reorg = self.nodes[1].gettransaction(txid) # Check that normal confirmed tx is confirmed again but with different blockhash diff --git a/test/functional/wallet_txn_clone.py b/test/functional/wallet_txn_clone.py index 60d7205887..99559090ee 100755 --- a/test/functional/wallet_txn_clone.py +++ b/test/functional/wallet_txn_clone.py @@ -16,6 +16,7 @@ from test_framework.messages import CTransaction, COIN class TxnMallTest(BitcoinTestFramework): def set_test_params(self): self.num_nodes = 4 + self.supports_cli = False def skip_test_if_missing_module(self): self.skip_if_no_wallet() diff --git a/test/functional/wallet_txn_doublespend.py b/test/functional/wallet_txn_doublespend.py index 40eeb4048c..1891cd9190 100755 --- a/test/functional/wallet_txn_doublespend.py +++ b/test/functional/wallet_txn_doublespend.py @@ -16,6 +16,7 @@ from test_framework.util import ( class TxnMallTest(BitcoinTestFramework): def set_test_params(self): self.num_nodes = 4 + self.supports_cli = False def skip_test_if_missing_module(self): self.skip_if_no_wallet() diff --git a/test/functional/wallet_watchonly.py b/test/functional/wallet_watchonly.py index be8d7714fb..b0c41b2738 100755 --- a/test/functional/wallet_watchonly.py +++ b/test/functional/wallet_watchonly.py @@ -16,7 +16,6 @@ class CreateWalletWatchonlyTest(BitcoinTestFramework): def set_test_params(self): self.setup_clean_chain = False self.num_nodes = 1 - self.supports_cli = True def skip_test_if_missing_module(self): self.skip_if_no_wallet() diff --git a/test/fuzz/test_runner.py b/test/fuzz/test_runner.py index f19cf924d2..520a2b5a95 100755 --- a/test/fuzz/test_runner.py +++ b/test/fuzz/test_runner.py @@ -28,6 +28,16 @@ def main(): help='If true, export coverage information to files in the seed corpus', ) parser.add_argument( + '--valgrind', + action='store_true', + help='If true, run fuzzing binaries under the valgrind memory error detector', + ) + parser.add_argument( + '-x', + '--exclude', + help="A comma-separated list of targets to exclude", + ) + parser.add_argument( 'seed_dir', help='The seed corpus to run on (must contain subfolders for each fuzz target).', ) @@ -61,7 +71,7 @@ def main(): logging.error("No fuzz targets found") sys.exit(1) - logging.info("Fuzz targets found: {}".format(test_list_all)) + logging.debug("{} fuzz target(s) found: {}".format(len(test_list_all), " ".join(sorted(test_list_all)))) args.target = args.target or test_list_all # By default run all test_list_error = list(set(args.target).difference(set(test_list_all))) @@ -70,7 +80,29 @@ def main(): test_list_selection = list(set(test_list_all).intersection(set(args.target))) if not test_list_selection: logging.error("No fuzz targets selected") - logging.info("Fuzz targets selected: {}".format(test_list_selection)) + if args.exclude: + for excluded_target in args.exclude.split(","): + if excluded_target not in test_list_selection: + logging.error("Target \"{}\" not found in current target list.".format(excluded_target)) + continue + test_list_selection.remove(excluded_target) + test_list_selection.sort() + + logging.info("{} of {} detected fuzz target(s) selected: {}".format(len(test_list_selection), len(test_list_all), " ".join(test_list_selection))) + + test_list_seedless = [] + for t in test_list_selection: + corpus_path = os.path.join(args.seed_dir, t) + if not os.path.exists(corpus_path) or len(os.listdir(corpus_path)) == 0: + test_list_seedless.append(t) + test_list_seedless.sort() + if test_list_seedless: + logging.info( + "Fuzzing harnesses lacking a seed corpus: {}".format( + " ".join(test_list_seedless) + ) + ) + logging.info("Please consider adding a fuzz seed corpus at https://github.com/bitcoin-core/qa-assets") try: help_output = subprocess.run( @@ -78,7 +110,7 @@ def main(): os.path.join(config["environment"]["BUILDDIR"], 'src', 'test', 'fuzz', test_list_selection[0]), '-help=1', ], - timeout=1, + timeout=20, check=True, stderr=subprocess.PIPE, universal_newlines=True, @@ -95,21 +127,34 @@ def main(): test_list=test_list_selection, build_dir=config["environment"]["BUILDDIR"], export_coverage=args.export_coverage, + use_valgrind=args.valgrind, ) -def run_once(*, corpus, test_list, build_dir, export_coverage): +def run_once(*, corpus, test_list, build_dir, export_coverage, use_valgrind): for t in test_list: + corpus_path = os.path.join(corpus, t) + os.makedirs(corpus_path, exist_ok=True) args = [ os.path.join(build_dir, 'src', 'test', 'fuzz', t), '-runs=1', - os.path.join(corpus, t), + corpus_path, ] + if use_valgrind: + args = ['valgrind', '--quiet', '--error-exitcode=1'] + args logging.debug('Run {} with args {}'.format(t, args)) result = subprocess.run(args, stderr=subprocess.PIPE, universal_newlines=True) output = result.stderr logging.debug('Output: {}'.format(output)) - result.check_returncode() + try: + result.check_returncode() + except subprocess.CalledProcessError as e: + if e.stdout: + logging.info(e.stdout) + if e.stderr: + logging.info(e.stderr) + logging.info("Target \"{}\" failed with exit code {}: {}".format(t, e.returncode, " ".join(args))) + sys.exit(1) if not export_coverage: continue for l in output.splitlines(): diff --git a/test/lint/README.md b/test/lint/README.md index 15974a3598..6b95cc3540 100644 --- a/test/lint/README.md +++ b/test/lint/README.md @@ -7,6 +7,8 @@ Check for missing documentation of command line options. commit-script-check.sh ====================== Verification of [scripted diffs](/doc/developer-notes.md#scripted-diffs). +Scripted diffs are only assumed to run on the latest LTS release of Ubuntu. Running them on other operating systems +might require installing GNU tools, such as GNU sed. git-subtree-check.sh ==================== @@ -19,6 +21,7 @@ maintained: * for `src/leveldb`: https://github.com/bitcoin-core/leveldb.git (branch bitcoin-fork) * for `src/univalue`: https://github.com/bitcoin-core/univalue.git (branch master) * for `src/crypto/ctaes`: https://github.com/bitcoin-core/ctaes.git (branch master) +* for `src/crc32c`: https://github.com/google/crc32c.git (branch master) Usage: `git-subtree-check.sh DIR (COMMIT)` diff --git a/test/lint/check-rpc-mappings.py b/test/lint/check-rpc-mappings.py index a33ab17f3f..0a4cc875d0 100755 --- a/test/lint/check-rpc-mappings.py +++ b/test/lint/check-rpc-mappings.py @@ -1,5 +1,5 @@ #!/usr/bin/env python3 -# Copyright (c) 2017-2018 The Bitcoin Core developers +# Copyright (c) 2017-2019 The Bitcoin Core developers # Distributed under the MIT software license, see the accompanying # file COPYING or http://www.opensource.org/licenses/mit-license.php. """Check RPC argument consistency.""" diff --git a/test/lint/commit-script-check.sh b/test/lint/commit-script-check.sh index 5603456e62..ff3f784437 100755 --- a/test/lint/commit-script-check.sh +++ b/test/lint/commit-script-check.sh @@ -1,5 +1,5 @@ #!/bin/sh -# Copyright (c) 2017 The Bitcoin Core developers +# Copyright (c) 2017-2019 The Bitcoin Core developers # Distributed under the MIT software license, see the accompanying # file COPYING or http://www.opensource.org/licenses/mit-license.php. diff --git a/test/lint/extended-lint-cppcheck.sh b/test/lint/extended-lint-cppcheck.sh index 47df25ba6b..ae18d74ebf 100755 --- a/test/lint/extended-lint-cppcheck.sh +++ b/test/lint/extended-lint-cppcheck.sh @@ -65,8 +65,8 @@ function join_array { ENABLED_CHECKS_REGEXP=$(join_array "|" "${ENABLED_CHECKS[@]}") IGNORED_WARNINGS_REGEXP=$(join_array "|" "${IGNORED_WARNINGS[@]}") -WARNINGS=$(git ls-files -- "*.cpp" "*.h" ":(exclude)src/leveldb/" ":(exclude)src/secp256k1/" ":(exclude)src/univalue/" | \ - xargs cppcheck --enable=all -j "$(getconf _NPROCESSORS_ONLN)" --language=c++ --std=c++11 --template=gcc -D__cplusplus -DCLIENT_VERSION_BUILD -DCLIENT_VERSION_IS_RELEASE -DCLIENT_VERSION_MAJOR -DCLIENT_VERSION_MINOR -DCLIENT_VERSION_REVISION -DCOPYRIGHT_YEAR -DDEBUG -DHAVE_WORKING_BOOST_SLEEP_FOR -I src/ -q 2>&1 | sort -u | \ +WARNINGS=$(git ls-files -- "*.cpp" "*.h" ":(exclude)src/leveldb/" ":(exclude)src/crc32c/" ":(exclude)src/secp256k1/" ":(exclude)src/univalue/" | \ + xargs cppcheck --enable=all -j "$(getconf _NPROCESSORS_ONLN)" --language=c++ --std=c++11 --template=gcc -D__cplusplus -DCLIENT_VERSION_BUILD -DCLIENT_VERSION_IS_RELEASE -DCLIENT_VERSION_MAJOR -DCLIENT_VERSION_MINOR -DCLIENT_VERSION_REVISION -DCOPYRIGHT_YEAR -DDEBUG -I src/ -q 2>&1 | sort -u | \ grep -E "${ENABLED_CHECKS_REGEXP}" | \ grep -vE "${IGNORED_WARNINGS_REGEXP}") if [[ ${WARNINGS} != "" ]]; then diff --git a/test/lint/git-subtree-check.sh b/test/lint/git-subtree-check.sh index 85e8b841b6..caa7affc63 100755 --- a/test/lint/git-subtree-check.sh +++ b/test/lint/git-subtree-check.sh @@ -1,10 +1,11 @@ #!/bin/sh -# Copyright (c) 2015 The Bitcoin Core developers +# Copyright (c) 2015-2019 The Bitcoin Core developers # Distributed under the MIT software license, see the accompanying # file COPYING or http://www.opensource.org/licenses/mit-license.php. export LC_ALL=C -DIR="$1" +# Strip trailing / from directory path (in case it was added by autocomplete) +DIR="${1%/}" COMMIT="$2" if [ -z "$COMMIT" ]; then COMMIT=HEAD diff --git a/test/lint/lint-assertions.sh b/test/lint/lint-assertions.sh index 5bbcae79eb..1aacc09bcc 100755 --- a/test/lint/lint-assertions.sh +++ b/test/lint/lint-assertions.sh @@ -1,6 +1,6 @@ #!/usr/bin/env bash # -# Copyright (c) 2018 The Bitcoin Core developers +# Copyright (c) 2018-2019 The Bitcoin Core developers # Distributed under the MIT software license, see the accompanying # file COPYING or http://www.opensource.org/licenses/mit-license.php. # @@ -20,4 +20,15 @@ if [[ ${OUTPUT} != "" ]]; then EXIT_CODE=1 fi +# Macro CHECK_NONFATAL(condition) should be used instead of assert for RPC code, where it +# is undesirable to crash the whole program. See: src/util/check.h +# src/rpc/server.cpp is excluded from this check since it's mostly meta-code. +OUTPUT=$(git grep -nE 'assert *\(.*\);' -- "src/rpc/" "src/wallet/rpc*" ":(exclude)src/rpc/server.cpp") +if [[ ${OUTPUT} != "" ]]; then + echo "CHECK_NONFATAL(condition) should be used instead of assert for RPC code." + echo + echo "${OUTPUT}" + EXIT_CODE=1 +fi + exit ${EXIT_CODE} diff --git a/test/lint/lint-circular-dependencies.sh b/test/lint/lint-circular-dependencies.sh index 8607fc4371..bbd94dd6c7 100755 --- a/test/lint/lint-circular-dependencies.sh +++ b/test/lint/lint-circular-dependencies.sh @@ -1,6 +1,6 @@ #!/usr/bin/env bash # -# Copyright (c) 2018 The Bitcoin Core developers +# Copyright (c) 2018-2019 The Bitcoin Core developers # Distributed under the MIT software license, see the accompanying # file COPYING or http://www.opensource.org/licenses/mit-license.php. # @@ -13,24 +13,14 @@ EXPECTED_CIRCULAR_DEPENDENCIES=( "index/txindex -> validation -> index/txindex" "policy/fees -> txmempool -> policy/fees" "qt/addresstablemodel -> qt/walletmodel -> qt/addresstablemodel" - "qt/bantablemodel -> qt/clientmodel -> qt/bantablemodel" - "qt/bitcoingui -> qt/utilitydialog -> qt/bitcoingui" "qt/bitcoingui -> qt/walletframe -> qt/bitcoingui" - "qt/bitcoingui -> qt/walletview -> qt/bitcoingui" - "qt/clientmodel -> qt/peertablemodel -> qt/clientmodel" - "qt/paymentserver -> qt/walletmodel -> qt/paymentserver" "qt/recentrequeststablemodel -> qt/walletmodel -> qt/recentrequeststablemodel" "qt/sendcoinsdialog -> qt/walletmodel -> qt/sendcoinsdialog" "qt/transactiontablemodel -> qt/walletmodel -> qt/transactiontablemodel" - "qt/walletmodel -> qt/walletmodeltransaction -> qt/walletmodel" "txmempool -> validation -> txmempool" - "wallet/coincontrol -> wallet/wallet -> wallet/coincontrol" "wallet/fees -> wallet/wallet -> wallet/fees" "wallet/wallet -> wallet/walletdb -> wallet/wallet" "policy/fees -> txmempool -> validation -> policy/fees" - "qt/guiutil -> qt/walletmodel -> qt/optionsmodel -> qt/guiutil" - "txmempool -> validation -> validationinterface -> txmempool" - "wallet/ismine -> wallet/wallet -> wallet/ismine" ) EXIT_CODE=0 diff --git a/test/lint/lint-filenames.sh b/test/lint/lint-filenames.sh index 5391e43d91..3f7491cd2b 100755 --- a/test/lint/lint-filenames.sh +++ b/test/lint/lint-filenames.sh @@ -1,6 +1,6 @@ #!/usr/bin/env bash # -# Copyright (c) 2018 The Bitcoin Core developers +# Copyright (c) 2018-2019 The Bitcoin Core developers # Distributed under the MIT software license, see the accompanying # file COPYING or http://www.opensource.org/licenses/mit-license.php. # @@ -12,7 +12,7 @@ export LC_ALL=C EXIT_CODE=0 OUTPUT=$(git ls-files --full-name -- "*.[cC][pP][pP]" "*.[hH]" "*.[pP][yY]" "*.[sS][hH]" | \ grep -vE '^[a-z0-9_./-]+$' | \ - grep -vE '^src/(secp256k1|univalue)/') + grep -vE '^src/(secp256k1/|univalue/|test/fuzz/FuzzedDataProvider.h)') if [[ ${OUTPUT} != "" ]]; then echo "Use only lowercase alphanumerics (a-z0-9), underscores (_), hyphens (-) and dots (.)" diff --git a/test/lint/lint-format-strings.py b/test/lint/lint-format-strings.py index 9f34d0f4dd..2870432bff 100755 --- a/test/lint/lint-format-strings.py +++ b/test/lint/lint-format-strings.py @@ -17,9 +17,13 @@ FALSE_POSITIVES = [ ("src/index/base.cpp", "FatalError(const char* fmt, const Args&... args)"), ("src/netbase.cpp", "LogConnectFailure(bool manual_connection, const char* fmt, const Args&... args)"), ("src/util/system.cpp", "strprintf(_(COPYRIGHT_HOLDERS).translated, COPYRIGHT_HOLDERS_SUBSTITUTION)"), + ("src/validationinterface.cpp", "LogPrint(BCLog::VALIDATION, fmt \"\\n\", __VA_ARGS__)"), ("src/wallet/wallet.h", "WalletLogPrintf(std::string fmt, Params... parameters)"), ("src/wallet/wallet.h", "LogPrintf((\"%s \" + fmt).c_str(), GetDisplayName(), parameters...)"), + ("src/wallet/scriptpubkeyman.h", "WalletLogPrintf(std::string fmt, Params... parameters)"), + ("src/wallet/scriptpubkeyman.h", "LogPrintf((\"%s \" + fmt).c_str(), m_storage.GetDisplayName(), parameters...)"), ("src/logging.h", "LogPrintf(const char* fmt, const Args&... args)"), + ("src/wallet/scriptpubkeyman.h", "WalletLogPrintf(const std::string& fmt, const Params&... parameters)"), ] diff --git a/test/lint/lint-format-strings.sh b/test/lint/lint-format-strings.sh index cb630c78ad..184c3682c8 100755 --- a/test/lint/lint-format-strings.sh +++ b/test/lint/lint-format-strings.sh @@ -1,6 +1,6 @@ #!/usr/bin/env bash # -# Copyright (c) 2018 The Bitcoin Core developers +# Copyright (c) 2018-2019 The Bitcoin Core developers # Distributed under the MIT software license, see the accompanying # file COPYING or http://www.opensource.org/licenses/mit-license.php. # @@ -34,7 +34,7 @@ if ! python3 -m doctest test/lint/lint-format-strings.py; then fi for S in "${FUNCTION_NAMES_AND_NUMBER_OF_LEADING_ARGUMENTS[@]}"; do IFS="," read -r FUNCTION_NAME SKIP_ARGUMENTS <<< "${S}" - for MATCHING_FILE in $(git grep --full-name -l "${FUNCTION_NAME}" -- "*.c" "*.cpp" "*.h" | sort | grep -vE "^src/(leveldb|secp256k1|tinyformat|univalue)"); do + for MATCHING_FILE in $(git grep --full-name -l "${FUNCTION_NAME}" -- "*.c" "*.cpp" "*.h" | sort | grep -vE "^src/(leveldb|secp256k1|tinyformat|univalue|test/fuzz/strprintf.cpp)"); do MATCHING_FILES+=("${MATCHING_FILE}") done if ! test/lint/lint-format-strings.py --skip-arguments "${SKIP_ARGUMENTS}" "${FUNCTION_NAME}" "${MATCHING_FILES[@]}"; then diff --git a/test/lint/lint-include-guards.sh b/test/lint/lint-include-guards.sh index 464969794b..3a0494c190 100755 --- a/test/lint/lint-include-guards.sh +++ b/test/lint/lint-include-guards.sh @@ -1,6 +1,6 @@ #!/usr/bin/env bash # -# Copyright (c) 2018 The Bitcoin Core developers +# Copyright (c) 2018-2019 The Bitcoin Core developers # Distributed under the MIT software license, see the accompanying # file COPYING or http://www.opensource.org/licenses/mit-license.php. # @@ -10,7 +10,7 @@ export LC_ALL=C HEADER_ID_PREFIX="BITCOIN_" HEADER_ID_SUFFIX="_H" -REGEXP_EXCLUDE_FILES_WITH_PREFIX="src/(crypto/ctaes/|leveldb/|secp256k1/|tinyformat.h|univalue/)" +REGEXP_EXCLUDE_FILES_WITH_PREFIX="src/(crypto/ctaes/|leveldb/|crc32c/|secp256k1/|test/fuzz/FuzzedDataProvider.h|tinyformat.h|univalue/)" EXIT_CODE=0 for HEADER_FILE in $(git ls-files -- "*.h" | grep -vE "^${REGEXP_EXCLUDE_FILES_WITH_PREFIX}") diff --git a/test/lint/lint-includes.sh b/test/lint/lint-includes.sh index d27e45a23f..1cece6a525 100755 --- a/test/lint/lint-includes.sh +++ b/test/lint/lint-includes.sh @@ -1,6 +1,6 @@ #!/usr/bin/env bash # -# Copyright (c) 2018 The Bitcoin Core developers +# Copyright (c) 2018-2019 The Bitcoin Core developers # Distributed under the MIT software license, see the accompanying # file COPYING or http://www.opensource.org/licenses/mit-license.php. # @@ -9,7 +9,7 @@ # Check includes: Check for duplicate includes. Enforce bracket syntax includes. export LC_ALL=C -IGNORE_REGEXP="/(leveldb|secp256k1|univalue)/" +IGNORE_REGEXP="/(leveldb|secp256k1|univalue|crc32c)/" # cd to root folder of git repo for git ls-files to work properly cd "$(dirname $0)/../.." || exit 1 @@ -53,7 +53,6 @@ EXPECTED_BOOST_INCLUDES=( boost/algorithm/string/classification.hpp boost/algorithm/string/replace.hpp boost/algorithm/string/split.hpp - boost/chrono/chrono.hpp boost/date_time/posix_time/posix_time.hpp boost/filesystem.hpp boost/filesystem/fstream.hpp diff --git a/test/lint/lint-locale-dependence.sh b/test/lint/lint-locale-dependence.sh index 9a1aa766f7..0cb38b6fdb 100755 --- a/test/lint/lint-locale-dependence.sh +++ b/test/lint/lint-locale-dependence.sh @@ -1,16 +1,36 @@ #!/usr/bin/env bash +# Copyright (c) 2018-2019 The Bitcoin Core developers +# Distributed under the MIT software license, see the accompanying +# file COPYING or http://www.opensource.org/licenses/mit-license.php. export LC_ALL=C KNOWN_VIOLATIONS=( "src/bitcoin-tx.cpp.*stoul" + "src/bitcoin-tx.cpp.*std::to_string" "src/bitcoin-tx.cpp.*trim_right" "src/dbwrapper.cpp.*stoul" "src/dbwrapper.cpp:.*vsnprintf" "src/httprpc.cpp.*trim" "src/init.cpp:.*atoi" + "src/qt/optionsmodel.cpp.*std::to_string" "src/qt/rpcconsole.cpp:.*atoi" "src/rest.cpp:.*strtol" + "src/rpc/net.cpp.*std::to_string" + "src/rpc/rawtransaction.cpp.*std::to_string" + "src/rpc/util.cpp.*std::to_string" + "src/test/addrman_tests.cpp.*std::to_string" + "src/test/blockchain_tests.cpp.*std::to_string" "src/test/dbwrapper_tests.cpp:.*snprintf" + "src/test/denialofservice_tests.cpp.*std::to_string" + "src/test/fuzz/locale.cpp" + "src/test/fuzz/parse_numbers.cpp:.*atoi" + "src/test/key_tests.cpp.*std::to_string" + "src/test/net_tests.cpp.*std::to_string" + "src/test/settings_tests.cpp.*std::to_string" + "src/test/timedata_tests.cpp.*std::to_string" + "src/test/util/setup_common.cpp.*std::to_string" + "src/test/util_tests.cpp.*std::to_string" + "src/test/util_threadnames_tests.cpp.*std::to_string" "src/torcontrol.cpp:.*atoi" "src/torcontrol.cpp:.*strtol" "src/util/strencodings.cpp:.*atoi" @@ -18,6 +38,7 @@ KNOWN_VIOLATIONS=( "src/util/strencodings.cpp:.*strtoul" "src/util/strencodings.h:.*atoi" "src/util/system.cpp:.*atoi" + "src/wallet/scriptpubkeyman.cpp.*std::to_string" ) REGEXP_IGNORE_EXTERNAL_DEPENDENCIES="^src/(crypto/ctaes/|leveldb/|secp256k1/|tinyformat.h|univalue/)" @@ -92,6 +113,7 @@ LOCALE_DEPENDENT_FUNCTIONS=( snprintf sprintf sscanf + std::to_string stod stof stoi diff --git a/test/lint/lint-logs.sh b/test/lint/lint-logs.sh index 632ed7c812..2fbb4a38e7 100755 --- a/test/lint/lint-logs.sh +++ b/test/lint/lint-logs.sh @@ -1,6 +1,6 @@ #!/usr/bin/env bash # -# Copyright (c) 2018 The Bitcoin Core developers +# Copyright (c) 2018-2019 The Bitcoin Core developers # Distributed under the MIT software license, see the accompanying # file COPYING or http://www.opensource.org/licenses/mit-license.php. # @@ -15,6 +15,7 @@ export LC_ALL=C UNTERMINATED_LOGS=$(git grep --extended-regexp "LogPrintf?\(" -- "*.cpp" | \ grep -v '\\n"' | \ + grep -v '\.\.\.' | \ grep -v "/\* Continued \*/" | \ grep -v "LogPrint()" | \ grep -v "LogPrintf()") diff --git a/test/lint/lint-python-utf8-encoding.sh b/test/lint/lint-python-utf8-encoding.sh index d03c20205d..773855bed1 100755 --- a/test/lint/lint-python-utf8-encoding.sh +++ b/test/lint/lint-python-utf8-encoding.sh @@ -9,7 +9,7 @@ export LC_ALL=C EXIT_CODE=0 -OUTPUT=$(git grep " open(" -- "*.py" | grep -vE "encoding=.(ascii|utf8|utf-8)." | grep -vE "open\([^,]*, ['\"][^'\"]*b[^'\"]*['\"]") +OUTPUT=$(git grep " open(" -- "*.py" ":(exclude)src/crc32c/" | grep -vE "encoding=.(ascii|utf8|utf-8)." | grep -vE "open\([^,]*, ['\"][^'\"]*b[^'\"]*['\"]") if [[ ${OUTPUT} != "" ]]; then echo "Python's open(...) seems to be used to open text files without explicitly" echo "specifying encoding=\"utf8\":" @@ -17,7 +17,7 @@ if [[ ${OUTPUT} != "" ]]; then echo "${OUTPUT}" EXIT_CODE=1 fi -OUTPUT=$(git grep "check_output(" -- "*.py" | grep "universal_newlines=True" | grep -vE "encoding=.(ascii|utf8|utf-8).") +OUTPUT=$(git grep "check_output(" -- "*.py" ":(exclude)src/crc32c/"| grep "universal_newlines=True" | grep -vE "encoding=.(ascii|utf8|utf-8).") if [[ ${OUTPUT} != "" ]]; then echo "Python's check_output(...) seems to be used to get program outputs without explicitly" echo "specifying encoding=\"utf8\":" diff --git a/test/lint/lint-python.sh b/test/lint/lint-python.sh index 3c82ec19e3..86ac5a930f 100755 --- a/test/lint/lint-python.sh +++ b/test/lint/lint-python.sh @@ -1,6 +1,6 @@ #!/usr/bin/env bash # -# Copyright (c) 2017 The Bitcoin Core developers +# Copyright (c) 2017-2019 The Bitcoin Core developers # Distributed under the MIT software license, see the accompanying # file COPYING or http://www.opensource.org/licenses/mit-license.php. # @@ -82,10 +82,10 @@ enabled=( ) if ! command -v flake8 > /dev/null; then - echo "Skipping Python linting since flake8 is not installed. Install by running \"pip3 install flake8\"" + echo "Skipping Python linting since flake8 is not installed." exit 0 elif PYTHONWARNINGS="ignore" flake8 --version | grep -q "Python 2"; then - echo "Skipping Python linting since flake8 is running under Python 2. Install the Python 3 version of flake8 by running \"pip3 install flake8\"" + echo "Skipping Python linting since flake8 is running under Python 2. Install the Python 3 version of flake8." exit 0 fi diff --git a/test/lint/lint-shebang.sh b/test/lint/lint-shebang.sh index fda22592d3..a666fdfecf 100755 --- a/test/lint/lint-shebang.sh +++ b/test/lint/lint-shebang.sh @@ -1,4 +1,8 @@ #!/usr/bin/env bash +# Copyright (c) 2018 The Bitcoin Core developers +# Distributed under the MIT software license, see the accompanying +# file COPYING or http://www.opensource.org/licenses/mit-license.php. + # Assert expected shebang lines export LC_ALL=C diff --git a/test/lint/lint-shell.sh b/test/lint/lint-shell.sh index 69fc3cf368..f59b2c9945 100755 --- a/test/lint/lint-shell.sh +++ b/test/lint/lint-shell.sh @@ -1,6 +1,6 @@ #!/usr/bin/env bash # -# Copyright (c) 2018 The Bitcoin Core developers +# Copyright (c) 2018-2019 The Bitcoin Core developers # Distributed under the MIT software license, see the accompanying # file COPYING or http://www.opensource.org/licenses/mit-license.php. # @@ -16,16 +16,45 @@ if [ "$TRAVIS" = "true" ]; then unset LC_ALL fi -if ! command -v shellcheck > /dev/null; then - echo "Skipping shell linting since shellcheck is not installed." - exit 0 -fi - # Disabled warnings: disabled=( SC2046 # Quote this to prevent word splitting. SC2086 # Double quote to prevent globbing and word splitting. SC2162 # read without -r will mangle backslashes. ) -shellcheck -e "$(IFS=","; echo "${disabled[*]}")" \ - $(git ls-files -- "*.sh" | grep -vE 'src/(secp256k1|univalue)/') +disabled_gitian=( + SC2094 # Make sure not to read and write the same file in the same pipeline. + SC2129 # Consider using { cmd1; cmd2; } >> file instead of individual redirects. + SC2230 # which is non-standard. Use builtin 'command -v' instead. +) + +EXIT_CODE=0 + +if ! command -v shellcheck > /dev/null; then + echo "Skipping shell linting since shellcheck is not installed." + exit $EXIT_CODE +fi + +EXCLUDE="--exclude=$(IFS=','; echo "${disabled[*]}")" +if ! shellcheck "$EXCLUDE" $(git ls-files -- '*.sh' | grep -vE 'src/(leveldb|secp256k1|univalue)/'); then + EXIT_CODE=1 +fi + +if ! command -v yq > /dev/null; then + echo "Skipping Gitian descriptor scripts checking since yq is not installed." + exit $EXIT_CODE +fi + +EXCLUDE_GITIAN=${EXCLUDE}",$(IFS=','; echo "${disabled_gitian[*]}")" +for descriptor in $(git ls-files -- 'contrib/gitian-descriptors/*.yml') +do + echo + echo "$descriptor" + # Use #!/bin/bash as gitian-builder/bin/gbuild does to complete a script. + SCRIPT=$'#!/bin/bash\n'$(yq -r .script "$descriptor") + if ! echo "$SCRIPT" | shellcheck "$EXCLUDE_GITIAN" -; then + EXIT_CODE=1 + fi +done + +exit $EXIT_CODE diff --git a/test/lint/lint-spelling.ignore-words.txt b/test/lint/lint-spelling.ignore-words.txt index b08837c1d4..a7a97eb41f 100644 --- a/test/lint/lint-spelling.ignore-words.txt +++ b/test/lint/lint-spelling.ignore-words.txt @@ -1,9 +1,7 @@ -cas hights mor mut objext -unselect useable wit unparseable @@ -13,3 +11,6 @@ errorstring keyserver homogenous setban +hist +ser +unselect diff --git a/test/lint/lint-spelling.sh b/test/lint/lint-spelling.sh index e70b73e1cc..cb84727ba5 100755 --- a/test/lint/lint-spelling.sh +++ b/test/lint/lint-spelling.sh @@ -1,6 +1,6 @@ #!/usr/bin/env bash # -# Copyright (c) 2018 The Bitcoin Core developers +# Copyright (c) 2018-2019 The Bitcoin Core developers # Distributed under the MIT software license, see the accompanying # file COPYING or http://www.opensource.org/licenses/mit-license.php. # @@ -15,6 +15,6 @@ if ! command -v codespell > /dev/null; then fi IGNORE_WORDS_FILE=test/lint/lint-spelling.ignore-words.txt -if ! codespell --check-filenames --disable-colors --quiet-level=7 --ignore-words=${IGNORE_WORDS_FILE} $(git ls-files -- ":(exclude)build-aux/m4/" ":(exclude)contrib/seeds/*.txt" ":(exclude)depends/" ":(exclude)doc/release-notes/" ":(exclude)src/leveldb/" ":(exclude)src/qt/locale/" ":(exclude)src/secp256k1/" ":(exclude)src/univalue/"); then +if ! codespell --check-filenames --disable-colors --quiet-level=7 --ignore-words=${IGNORE_WORDS_FILE} $(git ls-files -- ":(exclude)build-aux/m4/" ":(exclude)contrib/seeds/*.txt" ":(exclude)depends/" ":(exclude)doc/release-notes/" ":(exclude)src/leveldb/" ":(exclude)src/crc32c/" ":(exclude)src/qt/locale/" ":(exclude)src/qt/*.qrc" ":(exclude)src/secp256k1/" ":(exclude)src/univalue/"); then echo "^ Warning: codespell identified likely spelling errors. Any false positives? Add them to the list of ignored words in ${IGNORE_WORDS_FILE}" fi diff --git a/test/lint/lint-submodule.sh b/test/lint/lint-submodule.sh new file mode 100755 index 0000000000..d9aa021df7 --- /dev/null +++ b/test/lint/lint-submodule.sh @@ -0,0 +1,20 @@ +#!/usr/bin/env bash +# +# Copyright (c) 2020 The Bitcoin Core developers +# Distributed under the MIT software license, see the accompanying +# file COPYING or http://www.opensource.org/licenses/mit-license.php. +# +# This script checks for git modules +export LC_ALL=C +EXIT_CODE=0 + +CMD=$(git submodule status --recursive) +if test -n "$CMD"; +then + echo These submodules were found, delete them: + echo "$CMD" + EXIT_CODE=1 +fi + +exit $EXIT_CODE + diff --git a/test/lint/lint-whitespace.sh b/test/lint/lint-whitespace.sh index 861faf8516..d8bdb0a8d7 100755 --- a/test/lint/lint-whitespace.sh +++ b/test/lint/lint-whitespace.sh @@ -31,14 +31,14 @@ if [ -z "${TRAVIS_COMMIT_RANGE}" ]; then fi showdiff() { - if ! git diff -U0 "${TRAVIS_COMMIT_RANGE}" -- "." ":(exclude)depends/patches/" ":(exclude)src/leveldb/" ":(exclude)src/secp256k1/" ":(exclude)src/univalue/" ":(exclude)doc/release-notes/" ":(exclude)src/qt/locale/"; then + if ! git diff -U0 "${TRAVIS_COMMIT_RANGE}" -- "." ":(exclude)depends/patches/" ":(exclude)src/leveldb/" ":(exclude)src/crc32c/" ":(exclude)src/secp256k1/" ":(exclude)src/univalue/" ":(exclude)doc/release-notes/" ":(exclude)src/qt/locale/"; then echo "Failed to get a diff" exit 1 fi } showcodediff() { - if ! git diff -U0 "${TRAVIS_COMMIT_RANGE}" -- *.cpp *.h *.md *.py *.sh ":(exclude)src/leveldb/" ":(exclude)src/secp256k1/" ":(exclude)src/univalue/" ":(exclude)doc/release-notes/" ":(exclude)src/qt/locale/"; then + if ! git diff -U0 "${TRAVIS_COMMIT_RANGE}" -- *.cpp *.h *.md *.py *.sh ":(exclude)src/leveldb/" ":(exclude)src/crc32c/" ":(exclude)src/secp256k1/" ":(exclude)src/univalue/" ":(exclude)doc/release-notes/" ":(exclude)src/qt/locale/"; then echo "Failed to get a diff" exit 1 fi diff --git a/test/sanitizer_suppressions/lsan b/test/sanitizer_suppressions/lsan index 90a92a5115..d2cb618d4e 100644 --- a/test/sanitizer_suppressions/lsan +++ b/test/sanitizer_suppressions/lsan @@ -1,5 +1,4 @@ # Suppress warnings triggered in dependencies -leak:libcrypto leak:libqminimal leak:libQt5Core leak:libQt5Gui diff --git a/test/sanitizer_suppressions/tsan b/test/sanitizer_suppressions/tsan index 70eea34363..b9c5c038d0 100644 --- a/test/sanitizer_suppressions/tsan +++ b/test/sanitizer_suppressions/tsan @@ -7,6 +7,14 @@ deadlock:WalletBatch # Intentional deadlock in tests deadlock:TestPotentialDeadLockDetected +# Race due to unprotected calls to thread-unsafe BOOST_TEST_MESSAGE from different threads: +# * G_TEST_LOG_FUN in the index thread +# * boost test case invoker (entering a test case) in the main thread +# TODO: get rid of BOOST_ macros, see also https://github.com/bitcoin/bitcoin/issues/8670 +race:blockfilter_index_initial_sync_invoker +race:txindex_initial_sync_invoker +race:validation_block_tests::TestSubscriber + # Wildcard for all gui tests, should be replaced with non-wildcard suppressions race:src/qt/test/* deadlock:src/qt/test/* diff --git a/test/sanitizer_suppressions/ubsan b/test/sanitizer_suppressions/ubsan index 643272de52..b3d9b9e6ec 100644 --- a/test/sanitizer_suppressions/ubsan +++ b/test/sanitizer_suppressions/ubsan @@ -1,9 +1,17 @@ -alignment:move.h -alignment:prevector.h -bool:wallet/wallet.cpp +# -fsanitize=undefined suppressions +# ================================= float-divide-by-zero:policy/fees.cpp float-divide-by-zero:validation.cpp float-divide-by-zero:wallet/wallet.cpp + +# -fsanitize=integer suppressions +# =============================== +# Unsigned integer overflow occurs when the result of an unsigned integer +# computation cannot be represented in its type. Unlike signed integer overflow, +# this is not undefined behavior, but it is often unintentional. The list below +# contains files in which we expect unsigned integer overflows to occur. The +# list is used to suppress -fsanitize=integer warnings when running our CI UBSan +# job. unsigned-integer-overflow:arith_uint256.h unsigned-integer-overflow:basic_string.h unsigned-integer-overflow:bench/bench.h @@ -32,4 +40,46 @@ unsigned-integer-overflow:stl_bvector.h unsigned-integer-overflow:txmempool.cpp unsigned-integer-overflow:util/strencodings.cpp unsigned-integer-overflow:validation.cpp -vptr:fs.cpp + +implicit-integer-sign-change:*/include/c++/*/bits/*.h +implicit-integer-sign-change:*/new_allocator.h +implicit-integer-sign-change:/usr/include/boost/date_time/format_date_parser.hpp +implicit-integer-sign-change:arith_uint256.cpp +implicit-integer-sign-change:bech32.cpp +implicit-integer-sign-change:bloom.cpp +implicit-integer-sign-change:chain.* +implicit-integer-sign-change:coins.h +implicit-integer-sign-change:compat/stdin.cpp +implicit-integer-sign-change:compressor.h +implicit-integer-sign-change:crypto/* +implicit-integer-sign-change:key.cpp +implicit-integer-sign-change:noui.cpp +implicit-integer-sign-change:prevector.h +implicit-integer-sign-change:protocol.cpp +implicit-integer-sign-change:script/bitcoinconsensus.cpp +implicit-integer-sign-change:script/interpreter.cpp +implicit-integer-sign-change:serialize.h +implicit-integer-sign-change:test/arith_uint256_tests.cpp +implicit-integer-sign-change:test/coins_tests.cpp +implicit-integer-sign-change:test/pow_tests.cpp +implicit-integer-sign-change:test/prevector_tests.cpp +implicit-integer-sign-change:test/sighash_tests.cpp +implicit-integer-sign-change:test/streams_tests.cpp +implicit-integer-sign-change:test/transaction_tests.cpp +implicit-integer-sign-change:txmempool.cpp +implicit-integer-sign-change:util/strencodings.* +implicit-integer-sign-change:validation.cpp +implicit-integer-sign-change:zmq/zmqpublishnotifier.cpp +implicit-signed-integer-truncation,implicit-integer-sign-change:chain.h +implicit-signed-integer-truncation,implicit-integer-sign-change:test/skiplist_tests.cpp +implicit-signed-integer-truncation:chain.h +implicit-signed-integer-truncation:crypto/* +implicit-signed-integer-truncation:cuckoocache.h +implicit-signed-integer-truncation:leveldb/* +implicit-signed-integer-truncation:streams.h +implicit-signed-integer-truncation:test/arith_uint256_tests.cpp +implicit-signed-integer-truncation:test/skiplist_tests.cpp +implicit-signed-integer-truncation:torcontrol.cpp +implicit-unsigned-integer-truncation:crypto/* +implicit-unsigned-integer-truncation:leveldb/* +implicit-integer-sign-change:crc32c/* |