diff options
Diffstat (limited to 'test/functional')
28 files changed, 1291 insertions, 563 deletions
diff --git a/test/functional/README.md b/test/functional/README.md index e6c4849702..96fe0becce 100644 --- a/test/functional/README.md +++ b/test/functional/README.md @@ -1,108 +1,154 @@ -Regression tests -================ +# Functional tests -### [test_framework/authproxy.py](test_framework/authproxy.py) -Taken from the [python-bitcoinrpc repository](https://github.com/jgarzik/python-bitcoinrpc). +### Writing Functional Tests -### [test_framework/test_framework.py](test_framework/test_framework.py) -Base class for new regression tests. +#### Example test -### [test_framework/util.py](test_framework/util.py) -Generally useful functions. +The [example_test.py](example_test.py) is a heavily commented example of a test case that uses both +the RPC and P2P interfaces. If you are writing your first test, copy that file +and modify to fit your needs. -### [test_framework/mininode.py](test_framework/mininode.py) -Basic code to support p2p connectivity to a bitcoind. +#### Coverage -### [test_framework/comptool.py](test_framework/comptool.py) -Framework for comparison-tool style, p2p tests. +Running `test_runner.py` with the `--coverage` argument tracks which RPCs are +called by the tests and prints a report of uncovered RPCs in the summary. This +can be used (along with the `--extended` argument) to find out which RPCs we +don't have test cases for. -### [test_framework/script.py](test_framework/script.py) -Utilities for manipulating transaction scripts (originally from python-bitcoinlib) +#### Style guidelines -### [test_framework/blockstore.py](test_framework/blockstore.py) -Implements disk-backed block and tx storage. +- Where possible, try to adhere to [PEP-8 guidelines]([https://www.python.org/dev/peps/pep-0008/) +- Use a python linter like flake8 before submitting PRs to catch common style + nits (eg trailing whitespace, unused imports, etc) +- Avoid wildcard imports where possible +- Use a module-level docstring to describe what the test is testing, and how it + is testing it. +- When subclassing the BitcoinTestFramwork, place overrides for the + `__init__()`, and `setup_xxxx()` methods at the top of the subclass, then + locally-defined helper methods, then the `run_test()` method. -### [test_framework/key.py](test_framework/key.py) -Wrapper around OpenSSL EC_Key (originally from python-bitcoinlib) +#### General test-writing advice -### [test_framework/bignum.py](test_framework/bignum.py) -Helpers for script.py +- Set `self.num_nodes` to the minimum number of nodes necessary for the test. + Having additional unrequired nodes adds to the execution time of the test as + well as memory/CPU/disk requirements (which is important when running tests in + parallel or on Travis). +- Avoid stop-starting the nodes multiple times during the test if possible. A + stop-start takes several seconds, so doing it several times blows up the + runtime of the test. +- Set the `self.setup_clean_chain` variable in `__init__()` to control whether + or not to use the cached data directories. The cached data directories + contain a 200-block pre-mined blockchain and wallets for four nodes. Each node + has 25 mature blocks (25x50=1250 BTC) in its wallet. +- When calling RPCs with lots of arguments, consider using named keyword + arguments instead of positional arguments to make the intent of the call + clear to readers. -### [test_framework/blocktools.py](test_framework/blocktools.py) -Helper functions for creating blocks and transactions. +#### RPC and P2P definitions -P2P test design notes ---------------------- +Test writers may find it helpful to refer to the definitions for the RPC and +P2P messages. These can be found in the following source files: -## Mininode +- `/src/rpc/*` for RPCs +- `/src/wallet/rpc*` for wallet RPCs +- `ProcessMessage()` in `/src/net_processing.cpp` for parsing P2P messages -* ```mininode.py``` contains all the definitions for objects that pass -over the network (```CBlock```, ```CTransaction```, etc, along with the network-level -wrappers for them, ```msg_block```, ```msg_tx```, etc). +#### Using the P2P interface -* P2P tests have two threads. One thread handles all network communication +- `mininode.py` contains all the definitions for objects that pass +over the network (`CBlock`, `CTransaction`, etc, along with the network-level +wrappers for them, `msg_block`, `msg_tx`, etc). + +- P2P tests have two threads. One thread handles all network communication with the bitcoind(s) being tested (using python's asyncore package); the other implements the test logic. -* ```NodeConn``` is the class used to connect to a bitcoind. If you implement -a callback class that derives from ```NodeConnCB``` and pass that to the -```NodeConn``` object, your code will receive the appropriate callbacks when +- `NodeConn` is the class used to connect to a bitcoind. If you implement +a callback class that derives from `NodeConnCB` and pass that to the +`NodeConn` object, your code will receive the appropriate callbacks when events of interest arrive. -* You can pass the same handler to multiple ```NodeConn```'s if you like, or pass -different ones to each -- whatever makes the most sense for your test. - -* Call ```NetworkThread.start()``` after all ```NodeConn``` objects are created to +- Call `NetworkThread.start()` after all `NodeConn` objects are created to start the networking thread. (Continue with the test logic in your existing thread.) -* RPC calls are available in p2p tests. +- Can be used to write tests where specific P2P protocol behavior is tested. +Examples tests are `p2p-accept-block.py`, `p2p-compactblocks.py`. -* Can be used to write free-form tests, where specific p2p-protocol behavior -is tested. Examples: ```p2p-accept-block.py```, ```p2p-compactblocks.py```. +#### Comptool -## Comptool +- Comptool is a Testing framework for writing tests that compare the block/tx acceptance +behavior of a bitcoind against 1 or more other bitcoind instances. It should not be used +to write static tests with known outcomes, since that type of test is easier to write and +maintain using the standard BitcoinTestFramework. -* Testing framework for writing tests that compare the block/tx acceptance -behavior of a bitcoind against 1 or more other bitcoind instances, or against -known outcomes, or both. - -* Set the ```num_nodes``` variable (defined in ```ComparisonTestFramework```) to start up -1 or more nodes. If using 1 node, then ```--testbinary``` can be used as a command line +- Set the `num_nodes` variable (defined in `ComparisonTestFramework`) to start up +1 or more nodes. If using 1 node, then `--testbinary` can be used as a command line option to change the bitcoind binary used by the test. If using 2 or more nodes, -then ```--refbinary``` can be optionally used to change the bitcoind that will be used +then `--refbinary` can be optionally used to change the bitcoind that will be used on nodes 2 and up. -* Implement a (generator) function called ```get_tests()``` which yields ```TestInstance```s. -Each ```TestInstance``` consists of: - - a list of ```[object, outcome, hash]``` entries - * ```object``` is a ```CBlock```, ```CTransaction```, or - ```CBlockHeader```. ```CBlock```'s and ```CTransaction```'s are tested for - acceptance. ```CBlockHeader```s can be used so that the test runner can deliver +- Implement a (generator) function called `get_tests()` which yields `TestInstance`s. +Each `TestInstance` consists of: + - a list of `[object, outcome, hash]` entries + * `object` is a `CBlock`, `CTransaction`, or + `CBlockHeader`. `CBlock`'s and `CTransaction`'s are tested for + acceptance. `CBlockHeader`s can be used so that the test runner can deliver complete headers-chains when requested from the bitcoind, to allow writing tests where blocks can be delivered out of order but still processed by headers-first bitcoind's. - * ```outcome``` is ```True```, ```False```, or ```None```. If ```True``` - or ```False```, the tip is compared with the expected tip -- either the + * `outcome` is `True`, `False`, or `None`. If `True` + or `False`, the tip is compared with the expected tip -- either the block passed in, or the hash specified as the optional 3rd entry. If - ```None``` is specified, then the test will compare all the bitcoind's + `None` is specified, then the test will compare all the bitcoind's being tested to see if they all agree on what the best tip is. - * ```hash``` is the block hash of the tip to compare against. Optional to + * `hash` is the block hash of the tip to compare against. Optional to specify; if left out then the hash of the block passed in will be used as the expected tip. This allows for specifying an expected tip while testing the handling of either invalid blocks or blocks delivered out of order, which complete a longer chain. - - ```sync_every_block```: ```True/False```. If ```False```, then all blocks + - `sync_every_block`: `True/False`. If `False`, then all blocks are inv'ed together, and the test runner waits until the node receives the last one, and tests only the last block for tip acceptance using the - outcome and specified tip. If ```True```, then each block is tested in + outcome and specified tip. If `True`, then each block is tested in sequence and synced (this is slower when processing many blocks). - - ```sync_every_transaction```: ```True/False```. Analogous to - ```sync_every_block```, except if the outcome on the last tx is "None", + - `sync_every_transaction`: `True/False`. Analogous to + `sync_every_block`, except if the outcome on the last tx is "None", then the contents of the entire mempool are compared across all bitcoind - connections. If ```True``` or ```False```, then only the last tx's + connections. If `True` or `False`, then only the last tx's acceptance is tested against the given outcome. -* For examples of tests written in this framework, see - ```invalidblockrequest.py``` and ```p2p-fullblocktest.py```. +- For examples of tests written in this framework, see + `invalidblockrequest.py` and `p2p-fullblocktest.py`. + +### test-framework modules + +#### [test_framework/authproxy.py](test_framework/authproxy.py) +Taken from the [python-bitcoinrpc repository](https://github.com/jgarzik/python-bitcoinrpc). + +#### [test_framework/test_framework.py](test_framework/test_framework.py) +Base class for functional tests. +#### [test_framework/util.py](test_framework/util.py) +Generally useful functions. + +#### [test_framework/mininode.py](test_framework/mininode.py) +Basic code to support P2P connectivity to a bitcoind. + +#### [test_framework/comptool.py](test_framework/comptool.py) +Framework for comparison-tool style, P2P tests. + +#### [test_framework/script.py](test_framework/script.py) +Utilities for manipulating transaction scripts (originally from python-bitcoinlib) + +#### [test_framework/blockstore.py](test_framework/blockstore.py) +Implements disk-backed block and tx storage. + +#### [test_framework/key.py](test_framework/key.py) +Wrapper around OpenSSL EC_Key (originally from python-bitcoinlib) + +#### [test_framework/bignum.py](test_framework/bignum.py) +Helpers for script.py + +#### [test_framework/blocktools.py](test_framework/blocktools.py) +Helper functions for creating blocks and transactions. diff --git a/test/functional/bip9-softforks.py b/test/functional/bip9-softforks.py index b90b0ca628..f00232c9ff 100755 --- a/test/functional/bip9-softforks.py +++ b/test/functional/bip9-softforks.py @@ -15,6 +15,10 @@ mine a further 143 blocks (LOCKED_IN) test that enforcement has not triggered (which triggers ACTIVE) test that enforcement has triggered """ +from io import BytesIO +import shutil +import time +import itertools from test_framework.test_framework import ComparisonTestFramework from test_framework.util import * @@ -22,9 +26,6 @@ from test_framework.mininode import CTransaction, NetworkThread from test_framework.blocktools import create_coinbase, create_block from test_framework.comptool import TestInstance, TestManager from test_framework.script import CScript, OP_1NEGATE, OP_CHECKSEQUENCEVERIFY, OP_DROP -from io import BytesIO -import time -import itertools class BIP9SoftForksTest(ComparisonTestFramework): diff --git a/test/functional/blockchain.py b/test/functional/blockchain.py index 6aef6d4489..a7034e6bcd 100755 --- a/test/functional/blockchain.py +++ b/test/functional/blockchain.py @@ -18,10 +18,13 @@ Tests correspond to code in rpc/blockchain.cpp. """ from decimal import Decimal +import http.client +import subprocess -from test_framework.test_framework import BitcoinTestFramework +from test_framework.test_framework import (BitcoinTestFramework, BITCOIND_PROC_WAIT_TIMEOUT) from test_framework.util import ( assert_equal, + assert_raises, assert_raises_jsonrpc, assert_is_hex_string, assert_is_hash_string, @@ -34,6 +37,7 @@ class BlockchainTest(BitcoinTestFramework): super().__init__() self.setup_clean_chain = False self.num_nodes = 1 + self.extra_args = [['-stopatheight=207']] def run_test(self): self._test_getchaintxstats() @@ -41,7 +45,8 @@ class BlockchainTest(BitcoinTestFramework): self._test_getblockheader() self._test_getdifficulty() self._test_getnetworkhashps() - self.nodes[0].verifychain(4, 0) + self._test_stopatheight() + assert self.nodes[0].verifychain(4, 0) def _test_getchaintxstats(self): chaintxstats = self.nodes[0].getchaintxstats(1) @@ -129,5 +134,21 @@ class BlockchainTest(BitcoinTestFramework): # This should be 2 hashes every 10 minutes or 1/300 assert abs(hashes_per_second * 300 - 1) < 0.0001 + def _test_stopatheight(self): + assert_equal(self.nodes[0].getblockcount(), 200) + self.nodes[0].generate(6) + assert_equal(self.nodes[0].getblockcount(), 206) + self.log.debug('Node should not stop at this height') + assert_raises(subprocess.TimeoutExpired, lambda: self.bitcoind_processes[0].wait(timeout=3)) + try: + self.nodes[0].generate(1) + except (ConnectionError, http.client.BadStatusLine): + pass # The node already shut down before response + self.log.debug('Node should stop at this height...') + self.bitcoind_processes[0].wait(timeout=BITCOIND_PROC_WAIT_TIMEOUT) + self.nodes[0] = self.start_node(0, self.options.tmpdir) + assert_equal(self.nodes[0].getblockcount(), 207) + + if __name__ == '__main__': BlockchainTest().main() diff --git a/test/functional/bumpfee.py b/test/functional/bumpfee.py index d42bab6cbf..9237f09240 100755 --- a/test/functional/bumpfee.py +++ b/test/functional/bumpfee.py @@ -10,7 +10,7 @@ its preconditions are met, and returns appropriate errors in other cases. This module consists of around a dozen individual test cases implemented in the top-level functions named as test_<test_case_description>. The test functions can be disabled or reordered if needed for debugging. If new test cases are -added in the the future, they should try to follow the same convention and not +added in the future, they should try to follow the same convention and not make assumptions about execution order. """ @@ -42,7 +42,7 @@ class BumpFeeTest(BitcoinTestFramework): # Encrypt wallet for test_locked_wallet_fails test self.nodes[1].encryptwallet(WALLET_PASSPHRASE) - bitcoind_processes[1].wait() + self.bitcoind_processes[1].wait() self.nodes[1] = self.start_node(1, self.options.tmpdir, extra_args[1]) self.nodes[1].walletpassphrase(WALLET_PASSPHRASE, WALLET_PASSPHRASE_TIMEOUT) diff --git a/test/functional/dbcrash.py b/test/functional/dbcrash.py new file mode 100755 index 0000000000..8339305f5e --- /dev/null +++ b/test/functional/dbcrash.py @@ -0,0 +1,281 @@ +#!/usr/bin/env python3 +# Copyright (c) 2017 The Bitcoin Core developers +# Distributed under the MIT software license, see the accompanying +# file COPYING or http://www.opensource.org/licenses/mit-license.php. +"""Test recovery from a crash during chainstate writing. + +- 4 nodes + * node0, node1, and node2 will have different dbcrash ratios, and different + dbcache sizes + * node3 will be a regular node, with no crashing. + * The nodes will not connect to each other. + +- use default test framework starting chain. initialize starting_tip_height to + tip height. + +- Main loop: + * generate lots of transactions on node3, enough to fill up a block. + * uniformly randomly pick a tip height from starting_tip_height to + tip_height; with probability 1/(height_difference+4), invalidate this block. + * mine enough blocks to overtake tip_height at start of loop. + * for each node in [node0,node1,node2]: + - for each mined block: + * submit block to node + * if node crashed on/after submitting: + - restart until recovery succeeds + - check that utxo matches node3 using gettxoutsetinfo""" + +import errno +import http.client +import random +import sys +import time + +from test_framework.mininode import * +from test_framework.script import * +from test_framework.test_framework import BitcoinTestFramework +from test_framework.util import * + +HTTP_DISCONNECT_ERRORS = [http.client.CannotSendRequest] +try: + HTTP_DISCONNECT_ERRORS.append(http.client.RemoteDisconnected) +except AttributeError: + pass + +class ChainstateWriteCrashTest(BitcoinTestFramework): + def __init__(self): + super().__init__() + self.num_nodes = 4 + self.setup_clean_chain = False + + # Set -maxmempool=0 to turn off mempool memory sharing with dbcache + # Set -rpcservertimeout=900 to reduce socket disconnects in this + # long-running test + self.base_args = ["-limitdescendantsize=0", "-maxmempool=0", "-rpcservertimeout=900", "-dbbatchsize=200000"] + + # Set different crash ratios and cache sizes. Note that not all of + # -dbcache goes to pcoinsTip. + self.node0_args = ["-dbcrashratio=8", "-dbcache=4"] + self.base_args + self.node1_args = ["-dbcrashratio=16", "-dbcache=8"] + self.base_args + self.node2_args = ["-dbcrashratio=24", "-dbcache=16"] + self.base_args + + # Node3 is a normal node with default args, except will mine full blocks + self.node3_args = ["-blockmaxweight=4000000"] + self.extra_args = [self.node0_args, self.node1_args, self.node2_args, self.node3_args] + + def setup_network(self): + self.setup_nodes() + # Leave them unconnected, we'll use submitblock directly in this test + + def restart_node(self, node_index, expected_tip): + """Start up a given node id, wait for the tip to reach the given block hash, and calculate the utxo hash. + + Exceptions on startup should indicate node crash (due to -dbcrashratio), in which case we try again. Give up + after 60 seconds. Returns the utxo hash of the given node.""" + + time_start = time.time() + while time.time() - time_start < 60: + try: + # Any of these RPC calls could throw due to node crash + self.nodes[node_index] = self.start_node(node_index, self.options.tmpdir, self.extra_args[node_index]) + self.nodes[node_index].waitforblock(expected_tip) + utxo_hash = self.nodes[node_index].gettxoutsetinfo()['hash_serialized_2'] + return utxo_hash + except: + # An exception here should mean the node is about to crash. + # If bitcoind exits, then try again. wait_for_node_exit() + # should raise an exception if bitcoind doesn't exit. + self.wait_for_node_exit(node_index, timeout=10) + self.crashed_on_restart += 1 + time.sleep(1) + + # If we got here, bitcoind isn't coming back up on restart. Could be a + # bug in bitcoind, or we've gotten unlucky with our dbcrash ratio -- + # perhaps we generated a test case that blew up our cache? + # TODO: If this happens a lot, we should try to restart without -dbcrashratio + # and make sure that recovery happens. + raise AssertionError("Unable to successfully restart node %d in allotted time", node_index) + + def submit_block_catch_error(self, node_index, block): + """Try submitting a block to the given node. + + Catch any exceptions that indicate the node has crashed. + Returns true if the block was submitted successfully; false otherwise.""" + + try: + self.nodes[node_index].submitblock(block) + return True + except http.client.BadStatusLine as e: + # Prior to 3.5 BadStatusLine('') was raised for a remote disconnect error. + if sys.version_info[0] == 3 and sys.version_info[1] < 5 and e.line == "''": + self.log.debug("node %d submitblock raised exception: %s", node_index, e) + return False + else: + raise + except tuple(HTTP_DISCONNECT_ERRORS) as e: + self.log.debug("node %d submitblock raised exception: %s", node_index, e) + return False + except OSError as e: + self.log.debug("node %d submitblock raised OSError exception: errno=%s", node_index, e.errno) + if e.errno in [errno.EPIPE, errno.ECONNREFUSED, errno.ECONNRESET]: + # The node has likely crashed + return False + else: + # Unexpected exception, raise + raise + + def sync_node3blocks(self, block_hashes): + """Use submitblock to sync node3's chain with the other nodes + + If submitblock fails, restart the node and get the new utxo hash. + If any nodes crash while updating, we'll compare utxo hashes to + ensure recovery was successful.""" + + node3_utxo_hash = self.nodes[3].gettxoutsetinfo()['hash_serialized_2'] + + # Retrieve all the blocks from node3 + blocks = [] + for block_hash in block_hashes: + blocks.append([block_hash, self.nodes[3].getblock(block_hash, 0)]) + + # Deliver each block to each other node + for i in range(3): + nodei_utxo_hash = None + self.log.debug("Syncing blocks to node %d", i) + for (block_hash, block) in blocks: + # Get the block from node3, and submit to node_i + self.log.debug("submitting block %s", block_hash) + if not self.submit_block_catch_error(i, block): + # TODO: more carefully check that the crash is due to -dbcrashratio + # (change the exit code perhaps, and check that here?) + self.wait_for_node_exit(i, timeout=30) + self.log.debug("Restarting node %d after block hash %s", i, block_hash) + nodei_utxo_hash = self.restart_node(i, block_hash) + assert nodei_utxo_hash is not None + self.restart_counts[i] += 1 + else: + # Clear it out after successful submitblock calls -- the cached + # utxo hash will no longer be correct + nodei_utxo_hash = None + + # Check that the utxo hash matches node3's utxo set + # NOTE: we only check the utxo set if we had to restart the node + # after the last block submitted: + # - checking the utxo hash causes a cache flush, which we don't + # want to do every time; so + # - we only update the utxo cache after a node restart, since flushing + # the cache is a no-op at that point + if nodei_utxo_hash is not None: + self.log.debug("Checking txoutsetinfo matches for node %d", i) + assert_equal(nodei_utxo_hash, node3_utxo_hash) + + def verify_utxo_hash(self): + """Verify that the utxo hash of each node matches node3. + + Restart any nodes that crash while querying.""" + node3_utxo_hash = self.nodes[3].gettxoutsetinfo()['hash_serialized_2'] + self.log.info("Verifying utxo hash matches for all nodes") + + for i in range(3): + try: + nodei_utxo_hash = self.nodes[i].gettxoutsetinfo()['hash_serialized_2'] + except OSError: + # probably a crash on db flushing + nodei_utxo_hash = self.restart_node(i, self.nodes[3].getbestblockhash()) + assert_equal(nodei_utxo_hash, node3_utxo_hash) + + def generate_small_transactions(self, node, count, utxo_list): + FEE = 1000 # TODO: replace this with node relay fee based calculation + num_transactions = 0 + random.shuffle(utxo_list) + while len(utxo_list) >= 2 and num_transactions < count: + tx = CTransaction() + input_amount = 0 + for i in range(2): + utxo = utxo_list.pop() + tx.vin.append(CTxIn(COutPoint(int(utxo['txid'], 16), utxo['vout']))) + input_amount += int(utxo['amount'] * COIN) + output_amount = (input_amount - FEE) // 3 + + if output_amount <= 0: + # Sanity check -- if we chose inputs that are too small, skip + continue + + for i in range(3): + tx.vout.append(CTxOut(output_amount, hex_str_to_bytes(utxo['scriptPubKey']))) + + # Sign and send the transaction to get into the mempool + tx_signed_hex = node.signrawtransaction(ToHex(tx))['hex'] + node.sendrawtransaction(tx_signed_hex) + num_transactions += 1 + + def run_test(self): + # Track test coverage statistics + self.restart_counts = [0, 0, 0] # Track the restarts for nodes 0-2 + self.crashed_on_restart = 0 # Track count of crashes during recovery + + # Start by creating a lot of utxos on node3 + initial_height = self.nodes[3].getblockcount() + utxo_list = create_confirmed_utxos(self.nodes[3].getnetworkinfo()['relayfee'], self.nodes[3], 5000) + self.log.info("Prepped %d utxo entries", len(utxo_list)) + + # Sync these blocks with the other nodes + block_hashes_to_sync = [] + for height in range(initial_height + 1, self.nodes[3].getblockcount() + 1): + block_hashes_to_sync.append(self.nodes[3].getblockhash(height)) + + self.log.debug("Syncing %d blocks with other nodes", len(block_hashes_to_sync)) + # Syncing the blocks could cause nodes to crash, so the test begins here. + self.sync_node3blocks(block_hashes_to_sync) + + starting_tip_height = self.nodes[3].getblockcount() + + # Main test loop: + # each time through the loop, generate a bunch of transactions, + # and then either mine a single new block on the tip, or some-sized reorg. + for i in range(40): + self.log.info("Iteration %d, generating 2500 transactions %s", i, self.restart_counts) + # Generate a bunch of small-ish transactions + self.generate_small_transactions(self.nodes[3], 2500, utxo_list) + # Pick a random block between current tip, and starting tip + current_height = self.nodes[3].getblockcount() + random_height = random.randint(starting_tip_height, current_height) + self.log.debug("At height %d, considering height %d", current_height, random_height) + if random_height > starting_tip_height: + # Randomly reorg from this point with some probability (1/4 for + # tip, 1/5 for tip-1, ...) + if random.random() < 1.0 / (current_height + 4 - random_height): + self.log.debug("Invalidating block at height %d", random_height) + self.nodes[3].invalidateblock(self.nodes[3].getblockhash(random_height)) + + # Now generate new blocks until we pass the old tip height + self.log.debug("Mining longer tip") + block_hashes = [] + while current_height + 1 > self.nodes[3].getblockcount(): + block_hashes.extend(self.nodes[3].generate(min(10, current_height + 1 - self.nodes[3].getblockcount()))) + self.log.debug("Syncing %d new blocks...", len(block_hashes)) + self.sync_node3blocks(block_hashes) + utxo_list = self.nodes[3].listunspent() + self.log.debug("Node3 utxo count: %d", len(utxo_list)) + + # Check that the utxo hashes agree with node3 + # Useful side effect: each utxo cache gets flushed here, so that we + # won't get crashes on shutdown at the end of the test. + self.verify_utxo_hash() + + # Check the test coverage + self.log.info("Restarted nodes: %s; crashes on restart: %d", self.restart_counts, self.crashed_on_restart) + + # If no nodes were restarted, we didn't test anything. + assert self.restart_counts != [0, 0, 0] + + # Make sure we tested the case of crash-during-recovery. + assert self.crashed_on_restart > 0 + + # Warn if any of the nodes escaped restart. + for i in range(3): + if self.restart_counts[i] == 0: + self.log.warn("Node %d never crashed during utxo flush!", i) + +if __name__ == "__main__": + ChainstateWriteCrashTest().main() diff --git a/test/functional/example_test.py b/test/functional/example_test.py new file mode 100755 index 0000000000..1ba5f756cd --- /dev/null +++ b/test/functional/example_test.py @@ -0,0 +1,219 @@ +#!/usr/bin/env python3 +# Copyright (c) 2017 The Bitcoin Core developers +# Distributed under the MIT software license, see the accompanying +# file COPYING or http://www.opensource.org/licenses/mit-license.php. +"""An example functional test + +The module-level docstring should include a high-level description of +what the test is doing. It's the first thing people see when they open +the file and should give the reader information about *what* the test +is testing and *how* it's being tested +""" +# Imports should be in PEP8 ordering (std library first, then third party +# libraries then local imports). +from collections import defaultdict + +# Avoid wildcard * imports if possible +from test_framework.blocktools import (create_block, create_coinbase) +from test_framework.mininode import ( + CInv, + NetworkThread, + NodeConn, + NodeConnCB, + mininode_lock, + msg_block, + msg_getdata, + wait_until, +) +from test_framework.test_framework import BitcoinTestFramework +from test_framework.util import ( + assert_equal, + connect_nodes, + p2p_port, +) + +# NodeConnCB is a class containing callbacks to be executed when a P2P +# message is received from the node-under-test. Subclass NodeConnCB and +# override the on_*() methods if you need custom behaviour. +class BaseNode(NodeConnCB): + def __init__(self): + """Initialize the NodeConnCB + + Used to inialize custom properties for the Node that aren't + included by default in the base class. Be aware that the NodeConnCB + base class already stores a counter for each P2P message type and the + last received message of each type, which should be sufficient for the + needs of most tests. + + Call super().__init__() first for standard initialization and then + initialize custom properties.""" + super().__init__() + # Stores a dictionary of all blocks received + self.block_receive_map = defaultdict(int) + + def on_block(self, conn, message): + """Override the standard on_block callback + + Store the hash of a received block in the dictionary.""" + message.block.calc_sha256() + self.block_receive_map[message.block.sha256] += 1 + +def custom_function(): + """Do some custom behaviour + + If this function is more generally useful for other tests, consider + moving it to a module in test_framework.""" + # self.log.info("running custom_function") # Oops! Can't run self.log outside the BitcoinTestFramework + pass + +class ExampleTest(BitcoinTestFramework): + # Each functional test is a subclass of the BitcoinTestFramework class. + + # Override the __init__(), add_options(), setup_chain(), setup_network() + # and setup_nodes() methods to customize the test setup as required. + + def __init__(self): + """Initialize the test + + Call super().__init__() first, and then override any test parameters + for your individual test.""" + super().__init__() + self.setup_clean_chain = True + self.num_nodes = 3 + # Use self.extra_args to change command-line arguments for the nodes + self.extra_args = [[], ["-logips"], []] + + # self.log.info("I've finished __init__") # Oops! Can't run self.log before run_test() + + # Use add_options() to add specific command-line options for your test. + # In practice this is not used very much, since the tests are mostly written + # to be run in automated environments without command-line options. + # def add_options() + # pass + + # Use setup_chain() to customize the node data directories. In practice + # this is not used very much since the default behaviour is almost always + # fine + # def setup_chain(): + # pass + + def setup_network(self): + """Setup the test network topology + + Often you won't need to override this, since the standard network topology + (linear: node0 <-> node1 <-> node2 <-> ...) is fine for most tests. + + If you do override this method, remember to start the nodes, assign + them to self.nodes, connect them and then sync.""" + + self.setup_nodes() + + # In this test, we're not connecting node2 to node0 or node1. Calls to + # sync_all() should not include node2, since we're not expecting it to + # sync. + connect_nodes(self.nodes[0], 1) + self.sync_all([self.nodes[0:1]]) + + # Use setup_nodes() to customize the node start behaviour (for example if + # you don't want to start all nodes at the start of the test). + # def setup_nodes(): + # pass + + def custom_method(self): + """Do some custom behaviour for this test + + Define it in a method here because you're going to use it repeatedly. + If you think it's useful in general, consider moving it to the base + BitcoinTestFramework class so other tests can use it.""" + + self.log.info("Running custom_method") + + def run_test(self): + """Main test logic""" + + # Create a P2P connection to one of the nodes + node0 = BaseNode() + connections = [] + connections.append(NodeConn('127.0.0.1', p2p_port(0), self.nodes[0], node0)) + node0.add_connection(connections[0]) + + # Start up network handling in another thread. This needs to be called + # after the P2P connections have been created. + NetworkThread().start() + # wait_for_verack ensures that the P2P connection is fully up. + node0.wait_for_verack() + + # Generating a block on one of the nodes will get us out of IBD + blocks = [int(self.nodes[0].generate(nblocks=1)[0], 16)] + self.sync_all([self.nodes[0:1]]) + + # Notice above how we called an RPC by calling a method with the same + # name on the node object. Notice also how we used a keyword argument + # to specify a named RPC argument. Neither of those are defined on the + # node object. Instead there's some __getattr__() magic going on under + # the covers to dispatch unrecognised attribute calls to the RPC + # interface. + + # Logs are nice. Do plenty of them. They can be used in place of comments for + # breaking the test into sub-sections. + self.log.info("Starting test!") + + self.log.info("Calling a custom function") + custom_function() + + self.log.info("Calling a custom method") + self.custom_method() + + self.log.info("Create some blocks") + self.tip = int(self.nodes[0].getbestblockhash(), 16) + self.block_time = self.nodes[0].getblock(self.nodes[0].getbestblockhash())['time'] + 1 + + height = 1 + + for i in range(10): + # Use the mininode and blocktools functionality to manually build a block + # Calling the generate() rpc is easier, but this allows us to exactly + # control the blocks and transactions. + block = create_block(self.tip, create_coinbase(height), self.block_time) + block.solve() + block_message = msg_block(block) + # Send message is used to send a P2P message to the node over our NodeConn connection + node0.send_message(block_message) + self.tip = block.sha256 + blocks.append(self.tip) + self.block_time += 1 + height += 1 + + self.log.info("Wait for node1 to reach current tip (height 11) using RPC") + self.nodes[1].waitforblockheight(11) + + self.log.info("Connect node2 and node1") + connect_nodes(self.nodes[1], 2) + + self.log.info("Add P2P connection to node2") + node2 = BaseNode() + connections.append(NodeConn('127.0.0.1', p2p_port(2), self.nodes[2], node2)) + node2.add_connection(connections[1]) + node2.wait_for_verack() + + self.log.info("Wait for node2 reach current tip. Test that it has propogated all the blocks to us") + + for block in blocks: + getdata_request = msg_getdata() + getdata_request.inv.append(CInv(2, block)) + node2.send_message(getdata_request) + + # wait_until() will loop until a predicate condition is met. Use it to test properties of the + # NodeConnCB objects. + assert wait_until(lambda: sorted(blocks) == sorted(list(node2.block_receive_map.keys())), timeout=5) + + self.log.info("Check that each block was received only once") + # The network thread uses a global lock on data access to the NodeConn objects when sending and receiving + # messages. The test thread should acquire the global lock before accessing any NodeConn data to avoid locking + # and synchronization issues. Note wait_until() acquires this global lock when testing the predicate. + with mininode_lock: + for block in node2.block_receive_map.values(): + assert_equal(block, 1) + +if __name__ == '__main__': + ExampleTest().main() diff --git a/test/functional/fundrawtransaction.py b/test/functional/fundrawtransaction.py index 0a3166b89b..0baab6d01c 100755 --- a/test/functional/fundrawtransaction.py +++ b/test/functional/fundrawtransaction.py @@ -4,7 +4,7 @@ # file COPYING or http://www.opensource.org/licenses/mit-license.php. """Test the fundrawtransaction RPC.""" -from test_framework.test_framework import BitcoinTestFramework +from test_framework.test_framework import BitcoinTestFramework, BITCOIND_PROC_WAIT_TIMEOUT from test_framework.util import * @@ -452,7 +452,7 @@ class RawTransactionsTest(BitcoinTestFramework): self.stop_node(2) self.stop_node(3) self.nodes[1].encryptwallet("test") - bitcoind_processes[1].wait(timeout=BITCOIND_PROC_WAIT_TIMEOUT) + self.bitcoind_processes[1].wait(timeout=BITCOIND_PROC_WAIT_TIMEOUT) self.nodes = self.start_nodes(self.num_nodes, self.options.tmpdir) # This test is not meant to test fee estimation and we'd like diff --git a/test/functional/keypool.py b/test/functional/keypool.py index f23a427d1f..e8be559918 100755 --- a/test/functional/keypool.py +++ b/test/functional/keypool.py @@ -18,7 +18,7 @@ class KeyPoolTest(BitcoinTestFramework): # Encrypt wallet and wait to terminate nodes[0].encryptwallet('test') - bitcoind_processes[0].wait() + self.bitcoind_processes[0].wait() # Restart node 0 nodes[0] = self.start_node(0, self.options.tmpdir) # Keep creating keys diff --git a/test/functional/listtransactions.py b/test/functional/listtransactions.py index f69f1c5724..f75a8e29cc 100755 --- a/test/functional/listtransactions.py +++ b/test/functional/listtransactions.py @@ -23,7 +23,7 @@ class ListTransactionsTest(BitcoinTestFramework): def setup_nodes(self): #This test requires mocktime - enable_mocktime() + self.enable_mocktime() self.nodes = self.start_nodes(self.num_nodes, self.options.tmpdir) def run_test(self): diff --git a/test/functional/merkle_blocks.py b/test/functional/merkle_blocks.py index 06af72ef10..bcc65c8408 100755 --- a/test/functional/merkle_blocks.py +++ b/test/functional/merkle_blocks.py @@ -39,7 +39,8 @@ class MerkleBlockTest(BitcoinTestFramework): txid1 = self.nodes[0].sendrawtransaction(self.nodes[0].signrawtransaction(tx1)["hex"]) tx2 = self.nodes[0].createrawtransaction([node0utxos.pop()], {self.nodes[1].getnewaddress(): 49.99}) txid2 = self.nodes[0].sendrawtransaction(self.nodes[0].signrawtransaction(tx2)["hex"]) - assert_raises(JSONRPCException, self.nodes[0].gettxoutproof, [txid1]) + # This will raise an exception because the transaction is not yet in a block + assert_raises_jsonrpc(-5, "Transaction not yet in block", self.nodes[0].gettxoutproof, [txid1]) self.nodes[0].generate(1) blockhash = self.nodes[0].getblockhash(chain_height + 1) @@ -56,7 +57,7 @@ class MerkleBlockTest(BitcoinTestFramework): txin_spent = self.nodes[1].listunspent(1).pop() tx3 = self.nodes[1].createrawtransaction([txin_spent], {self.nodes[0].getnewaddress(): 49.98}) - self.nodes[0].sendrawtransaction(self.nodes[1].signrawtransaction(tx3)["hex"]) + txid3 = self.nodes[0].sendrawtransaction(self.nodes[1].signrawtransaction(tx3)["hex"]) self.nodes[0].generate(1) self.sync_all() @@ -64,17 +65,21 @@ class MerkleBlockTest(BitcoinTestFramework): txid_unspent = txid1 if txin_spent["txid"] != txid1 else txid2 # We can't find the block from a fully-spent tx - assert_raises(JSONRPCException, self.nodes[2].gettxoutproof, [txid_spent]) - # ...but we can if we specify the block + assert_raises_jsonrpc(-5, "Transaction not yet in block", self.nodes[2].gettxoutproof, [txid_spent]) + # We can get the proof if we specify the block assert_equal(self.nodes[2].verifytxoutproof(self.nodes[2].gettxoutproof([txid_spent], blockhash)), [txid_spent]) - # ...or if the first tx is not fully-spent + # We can't get the proof if we specify a non-existent block + assert_raises_jsonrpc(-5, "Block not found", self.nodes[2].gettxoutproof, [txid_spent], "00000000000000000000000000000000") + # We can get the proof if the transaction is unspent assert_equal(self.nodes[2].verifytxoutproof(self.nodes[2].gettxoutproof([txid_unspent])), [txid_unspent]) - try: - assert_equal(self.nodes[2].verifytxoutproof(self.nodes[2].gettxoutproof([txid1, txid2])), txlist) - except JSONRPCException: - assert_equal(self.nodes[2].verifytxoutproof(self.nodes[2].gettxoutproof([txid2, txid1])), txlist) - # ...or if we have a -txindex + # We can get the proof if we provide a list of transactions and one of them is unspent. The ordering of the list should not matter. + assert_equal(sorted(self.nodes[2].verifytxoutproof(self.nodes[2].gettxoutproof([txid1, txid2]))), sorted(txlist)) + assert_equal(sorted(self.nodes[2].verifytxoutproof(self.nodes[2].gettxoutproof([txid2, txid1]))), sorted(txlist)) + # We can always get a proof if we have a -txindex assert_equal(self.nodes[2].verifytxoutproof(self.nodes[3].gettxoutproof([txid_spent])), [txid_spent]) + # We can't get a proof if we specify transactions from different blocks + assert_raises_jsonrpc(-5, "Not all transactions found in specified or retrieved block", self.nodes[2].gettxoutproof, [txid1, txid3]) + if __name__ == '__main__': MerkleBlockTest().main() diff --git a/test/functional/multi_rpc.py b/test/functional/multi_rpc.py index 6ff91a960b..a30e15ace9 100755 --- a/test/functional/multi_rpc.py +++ b/test/functional/multi_rpc.py @@ -16,16 +16,21 @@ class HTTPBasicsTest (BitcoinTestFramework): def __init__(self): super().__init__() self.setup_clean_chain = False - self.num_nodes = 1 + self.num_nodes = 2 def setup_chain(self): super().setup_chain() #Append rpcauth to bitcoin.conf before initialization rpcauth = "rpcauth=rt:93648e835a54c573682c2eb19f882535$7681e9c5b74bdd85e78166031d2058e1069b3ed7ed967c93fc63abba06f31144" rpcauth2 = "rpcauth=rt2:f8607b1a88861fac29dfccf9b52ff9f$ff36a0c23c8c62b4846112e50fa888416e94c17bfd4c42f88fd8f55ec6a3137e" + rpcuser = "rpcuser=rpcuser💻" + rpcpassword = "rpcpassword=rpcpassword🔑" with open(os.path.join(self.options.tmpdir+"/node0", "bitcoin.conf"), 'a', encoding='utf8') as f: f.write(rpcauth+"\n") f.write(rpcauth2+"\n") + with open(os.path.join(self.options.tmpdir+"/node1", "bitcoin.conf"), 'a', encoding='utf8') as f: + f.write(rpcuser+"\n") + f.write(rpcpassword+"\n") def run_test(self): @@ -50,7 +55,7 @@ class HTTPBasicsTest (BitcoinTestFramework): conn.connect() conn.request('POST', '/', '{"method": "getbestblockhash"}', headers) resp = conn.getresponse() - assert_equal(resp.status==401, False) + assert_equal(resp.status, 200) conn.close() #Use new authpair to confirm both work @@ -60,7 +65,7 @@ class HTTPBasicsTest (BitcoinTestFramework): conn.connect() conn.request('POST', '/', '{"method": "getbestblockhash"}', headers) resp = conn.getresponse() - assert_equal(resp.status==401, False) + assert_equal(resp.status, 200) conn.close() #Wrong login name with rt's password @@ -71,7 +76,7 @@ class HTTPBasicsTest (BitcoinTestFramework): conn.connect() conn.request('POST', '/', '{"method": "getbestblockhash"}', headers) resp = conn.getresponse() - assert_equal(resp.status==401, True) + assert_equal(resp.status, 401) conn.close() #Wrong password for rt @@ -82,7 +87,7 @@ class HTTPBasicsTest (BitcoinTestFramework): conn.connect() conn.request('POST', '/', '{"method": "getbestblockhash"}', headers) resp = conn.getresponse() - assert_equal(resp.status==401, True) + assert_equal(resp.status, 401) conn.close() #Correct for rt2 @@ -93,7 +98,7 @@ class HTTPBasicsTest (BitcoinTestFramework): conn.connect() conn.request('POST', '/', '{"method": "getbestblockhash"}', headers) resp = conn.getresponse() - assert_equal(resp.status==401, False) + assert_equal(resp.status, 200) conn.close() #Wrong password for rt2 @@ -104,7 +109,46 @@ class HTTPBasicsTest (BitcoinTestFramework): conn.connect() conn.request('POST', '/', '{"method": "getbestblockhash"}', headers) resp = conn.getresponse() - assert_equal(resp.status==401, True) + assert_equal(resp.status, 401) + conn.close() + + ############################################################### + # Check correctness of the rpcuser/rpcpassword config options # + ############################################################### + url = urllib.parse.urlparse(self.nodes[1].url) + + # rpcuser and rpcpassword authpair + rpcuserauthpair = "rpcuser💻:rpcpassword🔑" + + headers = {"Authorization": "Basic " + str_to_b64str(rpcuserauthpair)} + + conn = http.client.HTTPConnection(url.hostname, url.port) + conn.connect() + conn.request('POST', '/', '{"method": "getbestblockhash"}', headers) + resp = conn.getresponse() + assert_equal(resp.status, 200) + conn.close() + + #Wrong login name with rpcuser's password + rpcuserauthpair = "rpcuserwrong:rpcpassword" + headers = {"Authorization": "Basic " + str_to_b64str(rpcuserauthpair)} + + conn = http.client.HTTPConnection(url.hostname, url.port) + conn.connect() + conn.request('POST', '/', '{"method": "getbestblockhash"}', headers) + resp = conn.getresponse() + assert_equal(resp.status, 401) + conn.close() + + #Wrong password for rpcuser + rpcuserauthpair = "rpcuser:rpcpasswordwrong" + headers = {"Authorization": "Basic " + str_to_b64str(rpcuserauthpair)} + + conn = http.client.HTTPConnection(url.hostname, url.port) + conn.connect() + conn.request('POST', '/', '{"method": "getbestblockhash"}', headers) + resp = conn.getresponse() + assert_equal(resp.status, 401) conn.close() diff --git a/test/functional/p2p-segwit.py b/test/functional/p2p-segwit.py index dbc61d21fc..63dfbb8ae6 100755 --- a/test/functional/p2p-segwit.py +++ b/test/functional/p2p-segwit.py @@ -1486,7 +1486,7 @@ class SegWitTest(BitcoinTestFramework): # nodes would have stored, this requires special handling. # To enable this test, pass --oldbinary=<path-to-pre-segwit-bitcoind> to # the test. - def test_upgrade_after_activation(self, node, node_id): + def test_upgrade_after_activation(self, node_id): self.log.info("Testing software upgrade after softfork activation") assert(node_id != 0) # node0 is assumed to be a segwit-active bitcoind @@ -1502,14 +1502,14 @@ class SegWitTest(BitcoinTestFramework): sync_blocks(self.nodes) # Make sure that this peer thinks segwit has activated. - assert(get_bip9_status(node, 'segwit')['status'] == "active") + assert(get_bip9_status(self.nodes[node_id], 'segwit')['status'] == "active") # Make sure this peers blocks match those of node0. - height = node.getblockcount() + height = self.nodes[node_id].getblockcount() while height >= 0: - block_hash = node.getblockhash(height) + block_hash = self.nodes[node_id].getblockhash(height) assert_equal(block_hash, self.nodes[0].getblockhash(height)) - assert_equal(self.nodes[0].getblock(block_hash), node.getblock(block_hash)) + assert_equal(self.nodes[0].getblock(block_hash), self.nodes[node_id].getblock(block_hash)) height -= 1 @@ -1944,7 +1944,7 @@ class SegWitTest(BitcoinTestFramework): self.test_signature_version_1() self.test_non_standard_witness() sync_blocks(self.nodes) - self.test_upgrade_after_activation(self.nodes[2], 2) + self.test_upgrade_after_activation(node_id=2) self.test_witness_sigops() diff --git a/test/functional/pruning.py b/test/functional/pruning.py index 4c3501ad77..0af91e0658 100755 --- a/test/functional/pruning.py +++ b/test/functional/pruning.py @@ -315,17 +315,17 @@ class PruneTest(BitcoinTestFramework): # check that the pruning node's wallet is still in good shape self.log.info("Stop and start pruning node to trigger wallet rescan") self.stop_node(2) - self.start_node(2, self.options.tmpdir, ["-prune=550"]) + self.nodes[2] = self.start_node(2, self.options.tmpdir, ["-prune=550"]) self.log.info("Success") - # check that wallet loads loads successfully when restarting a pruned node after IBD. + # check that wallet loads successfully when restarting a pruned node after IBD. # this was reported to fail in #7494. self.log.info("Syncing node 5 to test wallet") connect_nodes(self.nodes[0], 5) nds = [self.nodes[0], self.nodes[5]] sync_blocks(nds, wait=5, timeout=300) self.stop_node(5) #stop and start to trigger rescan - self.start_node(5, self.options.tmpdir, ["-prune=550"]) + self.nodes[5] = self.start_node(5, self.options.tmpdir, ["-prune=550"]) self.log.info("Success") def run_test(self): diff --git a/test/functional/receivedby.py b/test/functional/receivedby.py index 2cad6269ac..19d99c9c9e 100755 --- a/test/functional/receivedby.py +++ b/test/functional/receivedby.py @@ -31,7 +31,7 @@ class ReceivedByTest(BitcoinTestFramework): def setup_nodes(self): #This test requires mocktime - enable_mocktime() + self.enable_mocktime() self.nodes = self.start_nodes(self.num_nodes, self.options.tmpdir) def run_test(self): diff --git a/test/functional/rest.py b/test/functional/rest.py index fbcceba0fa..a69dbb5013 100755 --- a/test/functional/rest.py +++ b/test/functional/rest.py @@ -82,9 +82,9 @@ class RESTTest (BitcoinTestFramework): n = vout['n'] - ###################################### - # GETUTXOS: query a unspent outpoint # - ###################################### + ####################################### + # GETUTXOS: query an unspent outpoint # + ####################################### json_request = '/checkmempool/'+txid+'-'+str(n) json_string = http_get_call(url.hostname, url.port, '/rest/getutxos'+json_request+self.FORMAT_SEPARATOR+'json') json_obj = json.loads(json_string) @@ -97,9 +97,9 @@ class RESTTest (BitcoinTestFramework): assert_equal(json_obj['utxos'][0]['value'], 0.1) - ################################################ - # GETUTXOS: now query a already spent outpoint # - ################################################ + ################################################# + # GETUTXOS: now query an already spent outpoint # + ################################################# json_request = '/checkmempool/'+vintx+'-0' json_string = http_get_call(url.hostname, url.port, '/rest/getutxos'+json_request+self.FORMAT_SEPARATOR+'json') json_obj = json.loads(json_string) @@ -161,24 +161,24 @@ class RESTTest (BitcoinTestFramework): json_request = '/'+txid+'-'+str(n) json_string = http_get_call(url.hostname, url.port, '/rest/getutxos'+json_request+self.FORMAT_SEPARATOR+'json') json_obj = json.loads(json_string) - assert_equal(len(json_obj['utxos']), 0) #there should be a outpoint because it has just added to the mempool + assert_equal(len(json_obj['utxos']), 0) #there should be an outpoint because it has just added to the mempool json_request = '/checkmempool/'+txid+'-'+str(n) json_string = http_get_call(url.hostname, url.port, '/rest/getutxos'+json_request+self.FORMAT_SEPARATOR+'json') json_obj = json.loads(json_string) - assert_equal(len(json_obj['utxos']), 1) #there should be a outpoint because it has just added to the mempool + assert_equal(len(json_obj['utxos']), 1) #there should be an outpoint because it has just added to the mempool #do some invalid requests json_request = '{"checkmempool' response = http_post_call(url.hostname, url.port, '/rest/getutxos'+self.FORMAT_SEPARATOR+'json', json_request, True) - assert_equal(response.status, 400) #must be a 400 because we send a invalid json request + assert_equal(response.status, 400) #must be a 400 because we send an invalid json request json_request = '{"checkmempool' response = http_post_call(url.hostname, url.port, '/rest/getutxos'+self.FORMAT_SEPARATOR+'bin', json_request, True) - assert_equal(response.status, 400) #must be a 400 because we send a invalid bin request + assert_equal(response.status, 400) #must be a 400 because we send an invalid bin request response = http_post_call(url.hostname, url.port, '/rest/getutxos/checkmempool'+self.FORMAT_SEPARATOR+'bin', '', True) - assert_equal(response.status, 400) #must be a 400 because we send a invalid bin request + assert_equal(response.status, 400) #must be a 400 because we send an invalid bin request #test limits json_request = '/checkmempool/' diff --git a/test/functional/rpcbind_test.py b/test/functional/rpcbind_test.py index 5336cf2ec8..951685aa76 100755 --- a/test/functional/rpcbind_test.py +++ b/test/functional/rpcbind_test.py @@ -37,7 +37,7 @@ class RPCBindTest(BitcoinTestFramework): base_args += ['-rpcallowip=' + x for x in allow_ips] binds = ['-rpcbind='+addr for addr in addresses] self.nodes = self.start_nodes(self.num_nodes, self.options.tmpdir, [base_args + binds], connect_to) - pid = bitcoind_processes[0].pid + pid = self.bitcoind_processes[0].pid assert_equal(set(get_bind_addrs(pid)), set(expected)) self.stop_nodes() @@ -49,7 +49,7 @@ class RPCBindTest(BitcoinTestFramework): base_args = ['-disablewallet', '-nolisten'] + ['-rpcallowip='+x for x in allow_ips] self.nodes = self.start_nodes(self.num_nodes, self.options.tmpdir, [base_args]) # connect to node through non-loopback interface - node = get_rpc_proxy(rpc_url(0, "%s:%d" % (rpchost, rpcport)), 0) + node = get_rpc_proxy(rpc_url(get_datadir_path(self.options.tmpdir, 0), 0, "%s:%d" % (rpchost, rpcport)), 0) node.getnetworkinfo() self.stop_nodes() diff --git a/test/functional/smartfees.py b/test/functional/smartfees.py index 482c77863f..bc42a319df 100755 --- a/test/functional/smartfees.py +++ b/test/functional/smartfees.py @@ -10,7 +10,7 @@ from test_framework.script import CScript, OP_1, OP_DROP, OP_2, OP_HASH160, OP_E from test_framework.mininode import CTransaction, CTxIn, CTxOut, COutPoint, ToHex, COIN # Construct 2 trivial P2SH's and the ScriptSigs that spend them -# So we can create many many transactions without needing to spend +# So we can create many transactions without needing to spend # time signing. redeem_script_1 = CScript([OP_1, OP_DROP]) redeem_script_2 = CScript([OP_2, OP_DROP]) diff --git a/test/functional/test_framework/mininode.py b/test/functional/test_framework/mininode.py index fb3ed1473a..688347a68f 100755 --- a/test/functional/test_framework/mininode.py +++ b/test/functional/test_framework/mininode.py @@ -61,7 +61,7 @@ mininode_socket_map = dict() # One lock for synchronizing all data access between the networking thread (see # NetworkThread below) and the thread running the test logic. For simplicity, -# NodeConn acquires this lock whenever delivering a message to to a NodeConnCB, +# NodeConn acquires this lock whenever delivering a message to a NodeConnCB, # and whenever adding anything to the send buffer (in send_message()). This # lock should be acquired in the thread running the test logic to synchronize # access to any data shared with the NodeConnCB or NodeConn. diff --git a/test/functional/test_framework/test_framework.py b/test/functional/test_framework/test_framework.py index 67abf35687..8d698a7327 100755 --- a/test/functional/test_framework/test_framework.py +++ b/test/functional/test_framework/test_framework.py @@ -5,7 +5,9 @@ """Base class for RPC testing.""" from collections import deque +import errno from enum import Enum +import http.client import logging import optparse import os @@ -14,33 +16,27 @@ import subprocess import sys import tempfile import time +import traceback +from .authproxy import JSONRPCException +from . import coverage from .util import ( - PortSeed, MAX_NODES, - bitcoind_processes, + PortSeed, + assert_equal, check_json_precision, connect_nodes_bi, - disable_mocktime, disconnect_nodes, - enable_coverage, - enable_mocktime, - get_mocktime, get_rpc_proxy, initialize_datadir, + get_datadir_path, log_filename, p2p_port, rpc_url, set_node_times, - _start_node, - _start_nodes, - _stop_node, - _stop_nodes, sync_blocks, sync_mempools, - wait_for_bitcoind_start, ) -from .authproxy import JSONRPCException class TestStatus(Enum): PASSED = 1 @@ -51,6 +47,8 @@ TEST_EXIT_PASSED = 0 TEST_EXIT_FAILED = 1 TEST_EXIT_SKIPPED = 77 +BITCOIND_PROC_WAIT_TIMEOUT = 60 + class BitcoinTestFramework(object): """Base class for a bitcoin test script. @@ -70,13 +68,15 @@ class BitcoinTestFramework(object): def __init__(self): self.num_nodes = 4 self.setup_clean_chain = False - self.nodes = None + self.nodes = [] + self.bitcoind_processes = {} + self.mocktime = 0 def add_options(self, parser): pass def setup_chain(self): - self.log.info("Initializing test directory "+self.options.tmpdir) + self.log.info("Initializing test directory " + self.options.tmpdir) if self.setup_clean_chain: self._initialize_chain_clean(self.options.tmpdir, self.num_nodes) else: @@ -96,7 +96,7 @@ class BitcoinTestFramework(object): extra_args = None if hasattr(self, "extra_args"): extra_args = self.extra_args - self.nodes = _start_nodes(self.num_nodes, self.options.tmpdir, extra_args) + self.nodes = self.start_nodes(self.num_nodes, self.options.tmpdir, extra_args) def run_test(self): raise NotImplementedError @@ -110,9 +110,9 @@ class BitcoinTestFramework(object): help="Leave bitcoinds and test.* datadir on exit or error") parser.add_option("--noshutdown", dest="noshutdown", default=False, action="store_true", help="Don't stop bitcoinds after the test execution") - parser.add_option("--srcdir", dest="srcdir", default=os.path.normpath(os.path.dirname(os.path.realpath(__file__))+"/../../../src"), + parser.add_option("--srcdir", dest="srcdir", default=os.path.normpath(os.path.dirname(os.path.realpath(__file__)) + "/../../../src"), help="Source directory containing bitcoind/bitcoin-cli (default: %default)") - parser.add_option("--cachedir", dest="cachedir", default=os.path.normpath(os.path.dirname(os.path.realpath(__file__))+"/../../cache"), + parser.add_option("--cachedir", dest="cachedir", default=os.path.normpath(os.path.dirname(os.path.realpath(__file__)) + "/../../cache"), help="Directory for caching pregenerated datadirs") parser.add_option("--tmpdir", dest="tmpdir", help="Root directory for datadirs") parser.add_option("-l", "--loglevel", dest="loglevel", default="INFO", @@ -128,12 +128,9 @@ class BitcoinTestFramework(object): self.add_options(parser) (self.options, self.args) = parser.parse_args() - if self.options.coveragedir: - enable_coverage(self.options.coveragedir) - PortSeed.n = self.options.port_seed - os.environ['PATH'] = self.options.srcdir+":"+self.options.srcdir+"/qt:"+os.environ['PATH'] + os.environ['PATH'] = self.options.srcdir + ":" + self.options.srcdir + "/qt:" + os.environ['PATH'] check_json_precision() @@ -187,7 +184,7 @@ class BitcoinTestFramework(object): for fn in filenames: try: with open(fn, 'r') as f: - print("From" , fn, ":") + print("From", fn, ":") print("".join(deque(f, MAX_LINES_TO_PRINT))) except OSError: print("Opening file %s failed." % fn) @@ -207,16 +204,88 @@ class BitcoinTestFramework(object): # Public helper methods. These can be accessed by the subclass test scripts. def start_node(self, i, dirname, extra_args=None, rpchost=None, timewait=None, binary=None, stderr=None): - return _start_node(i, dirname, extra_args, rpchost, timewait, binary, stderr) + """Start a bitcoind and return RPC connection to it""" + + datadir = os.path.join(dirname, "node" + str(i)) + if binary is None: + binary = os.getenv("BITCOIND", "bitcoind") + args = [binary, "-datadir=" + datadir, "-server", "-keypool=1", "-discover=0", "-rest", "-logtimemicros", "-debug", "-debugexclude=libevent", "-debugexclude=leveldb", "-mocktime=" + str(self.mocktime), "-uacomment=testnode%d" % i] + if extra_args is not None: + args.extend(extra_args) + self.bitcoind_processes[i] = subprocess.Popen(args, stderr=stderr) + self.log.debug("initialize_chain: bitcoind started, waiting for RPC to come up") + self._wait_for_bitcoind_start(self.bitcoind_processes[i], datadir, i, rpchost) + self.log.debug("initialize_chain: RPC successfully started") + proxy = get_rpc_proxy(rpc_url(datadir, i, rpchost), i, timeout=timewait) + + if self.options.coveragedir: + coverage.write_all_rpc_commands(self.options.coveragedir, proxy) + + return proxy def start_nodes(self, num_nodes, dirname, extra_args=None, rpchost=None, timewait=None, binary=None): - return _start_nodes(num_nodes, dirname, extra_args, rpchost, timewait, binary) + """Start multiple bitcoinds, return RPC connections to them""" + + if extra_args is None: + extra_args = [None] * num_nodes + if binary is None: + binary = [None] * num_nodes + assert_equal(len(extra_args), num_nodes) + assert_equal(len(binary), num_nodes) + rpcs = [] + try: + for i in range(num_nodes): + rpcs.append(self.start_node(i, dirname, extra_args[i], rpchost, timewait=timewait, binary=binary[i])) + except: + # If one node failed to start, stop the others + # TODO: abusing self.nodes in this way is a little hacky. + # Eventually we should do a better job of tracking nodes + self.nodes.extend(rpcs) + self.stop_nodes() + self.nodes = [] + raise + return rpcs + + def stop_node(self, i): + """Stop a bitcoind test node""" - def stop_node(self, num_node): - _stop_node(self.nodes[num_node], num_node) + self.log.debug("Stopping node %d" % i) + try: + self.nodes[i].stop() + except http.client.CannotSendRequest as e: + self.log.exception("Unable to stop node") + return_code = self.bitcoind_processes[i].wait(timeout=BITCOIND_PROC_WAIT_TIMEOUT) + del self.bitcoind_processes[i] + assert_equal(return_code, 0) def stop_nodes(self): - _stop_nodes(self.nodes) + """Stop multiple bitcoind test nodes""" + + for i in range(len(self.nodes)): + self.stop_node(i) + assert not self.bitcoind_processes.values() # All connections must be gone now + + def assert_start_raises_init_error(self, i, dirname, extra_args=None, expected_msg=None): + with tempfile.SpooledTemporaryFile(max_size=2**16) as log_stderr: + try: + self.start_node(i, dirname, extra_args, stderr=log_stderr) + self.stop_node(i) + except Exception as e: + assert 'bitcoind exited' in str(e) # node must have shutdown + if expected_msg is not None: + log_stderr.seek(0) + stderr = log_stderr.read().decode('utf-8') + if expected_msg not in stderr: + raise AssertionError("Expected error \"" + expected_msg + "\" not found in:\n" + stderr) + else: + if expected_msg is None: + assert_msg = "bitcoind should have exited with an error" + else: + assert_msg = "bitcoind should have exited with expected error " + expected_msg + raise AssertionError(assert_msg) + + def wait_for_node_exit(self, i, timeout): + self.bitcoind_processes[i].wait(timeout) def split_network(self): """ @@ -241,6 +310,21 @@ class BitcoinTestFramework(object): sync_blocks(group) sync_mempools(group) + def enable_mocktime(self): + """Enable mocktime for the script. + + mocktime may be needed for scripts that use the cached version of the + blockchain. If the cached version of the blockchain is used without + mocktime then the mempools will not sync due to IBD. + + For backwared compatibility of the python scripts with previous + versions of the cache, this helper function sets mocktime to Jan 1, + 2014 + (201 * 10 * 60)""" + self.mocktime = 1388534400 + (201 * 10 * 60) + + def disable_mocktime(self): + self.mocktime = 0 + # Private helper methods. These should not be accessed by the subclass test scripts. def _start_logging(self): @@ -256,7 +340,7 @@ class BitcoinTestFramework(object): ll = int(self.options.loglevel) if self.options.loglevel.isdigit() else self.options.loglevel.upper() ch.setLevel(ll) # Format logs the same as bitcoind's debug.log with microprecision (so log files can be concatenated and sorted) - formatter = logging.Formatter(fmt = '%(asctime)s.%(msecs)03d000 %(name)s (%(levelname)s): %(message)s', datefmt='%Y-%m-%d %H:%M:%S') + formatter = logging.Formatter(fmt='%(asctime)s.%(msecs)03d000 %(name)s (%(levelname)s): %(message)s', datefmt='%Y-%m-%d %H:%M:%S') formatter.converter = time.gmtime fh.setFormatter(formatter) ch.setFormatter(formatter) @@ -298,15 +382,15 @@ class BitcoinTestFramework(object): args = [os.getenv("BITCOIND", "bitcoind"), "-server", "-keypool=1", "-datadir=" + datadir, "-discover=0"] if i > 0: args.append("-connect=127.0.0.1:" + str(p2p_port(0))) - bitcoind_processes[i] = subprocess.Popen(args) + self.bitcoind_processes[i] = subprocess.Popen(args) self.log.debug("initialize_chain: bitcoind started, waiting for RPC to come up") - wait_for_bitcoind_start(bitcoind_processes[i], rpc_url(i), i) + self._wait_for_bitcoind_start(self.bitcoind_processes[i], datadir, i) self.log.debug("initialize_chain: RPC successfully started") self.nodes = [] for i in range(MAX_NODES): try: - self.nodes.append(get_rpc_proxy(rpc_url(i), i)) + self.nodes.append(get_rpc_proxy(rpc_url(get_datadir_path(cachedir, i), i), i)) except: self.log.exception("Error connecting to node %d" % i) sys.exit(1) @@ -318,8 +402,8 @@ class BitcoinTestFramework(object): # # blocks are created with timestamps 10 minutes apart # starting from 2010 minutes in the past - enable_mocktime() - block_time = get_mocktime() - (201 * 10 * 60) + self.enable_mocktime() + block_time = self.mocktime - (201 * 10 * 60) for i in range(2): for peer in range(4): for j in range(25): @@ -332,7 +416,7 @@ class BitcoinTestFramework(object): # Shut them down, and clean up cache directories: self.stop_nodes() self.nodes = [] - disable_mocktime() + self.disable_mocktime() for i in range(MAX_NODES): os.remove(log_filename(cachedir, i, "debug.log")) os.remove(log_filename(cachedir, i, "db.log")) @@ -353,18 +437,37 @@ class BitcoinTestFramework(object): for i in range(num_nodes): initialize_datadir(test_dir, i) -# Test framework for doing p2p comparison testing, which sets up some bitcoind -# binaries: -# 1 binary: test binary -# 2 binaries: 1 test binary, 1 ref binary -# n>2 binaries: 1 test binary, n-1 ref binaries - -class SkipTest(Exception): - """This exception is raised to skip a test""" - def __init__(self, message): - self.message = message + def _wait_for_bitcoind_start(self, process, datadir, i, rpchost=None): + """Wait for bitcoind to start. + + This means that RPC is accessible and fully initialized. + Raise an exception if bitcoind exits during initialization.""" + while True: + if process.poll() is not None: + raise Exception('bitcoind exited with status %i during initialization' % process.returncode) + try: + # Check if .cookie file to be created + rpc = get_rpc_proxy(rpc_url(datadir, i, rpchost), i, coveragedir=self.options.coveragedir) + rpc.getblockcount() + break # break out of loop on success + except IOError as e: + if e.errno != errno.ECONNREFUSED: # Port not yet open? + raise # unknown IO error + except JSONRPCException as e: # Initialization phase + if e.error['code'] != -28: # RPC in warmup? + raise # unknown JSON RPC exception + except ValueError as e: # cookie file not found and no rpcuser or rpcassword. bitcoind still starting + if "No RPC credentials" not in str(e): + raise + time.sleep(0.25) class ComparisonTestFramework(BitcoinTestFramework): + """Test framework for doing p2p comparison testing + + Sets up some bitcoind binaries: + - 1 binary: test binary + - 2 binaries: 1 test binary, 1 ref binary + - n>2 binaries: 1 test binary, n-1 ref binaries""" def __init__(self): super().__init__() @@ -380,8 +483,15 @@ class ComparisonTestFramework(BitcoinTestFramework): help="bitcoind binary to use for reference nodes (if any)") def setup_network(self): + extra_args = [['-whitelist=127.0.0.1']]*self.num_nodes + if hasattr(self, "extra_args"): + extra_args = self.extra_args self.nodes = self.start_nodes( - self.num_nodes, self.options.tmpdir, - extra_args=[['-whitelist=127.0.0.1']] * self.num_nodes, + self.num_nodes, self.options.tmpdir, extra_args, binary=[self.options.testbinary] + - [self.options.refbinary]*(self.num_nodes-1)) + [self.options.refbinary] * (self.num_nodes - 1)) + +class SkipTest(Exception): + """This exception is raised to skip a test""" + def __init__(self, message): + self.message = message diff --git a/test/functional/test_framework/util.py b/test/functional/test_framework/util.py index 2b0f32c2b6..8a2d8de50e 100644 --- a/test/functional/test_framework/util.py +++ b/test/functional/test_framework/util.py @@ -4,30 +4,162 @@ # file COPYING or http://www.opensource.org/licenses/mit-license.php. """Helpful routines for regression testing.""" -import os -import sys - -from binascii import hexlify, unhexlify from base64 import b64encode +from binascii import hexlify, unhexlify from decimal import Decimal, ROUND_DOWN import json -import http.client +import logging +import os import random -import shutil -import subprocess -import tempfile -import time import re -import errno -import logging +import time from . import coverage from .authproxy import AuthServiceProxy, JSONRPCException -COVERAGE_DIR = None - logger = logging.getLogger("TestFramework.utils") +# Assert functions +################## + +def assert_fee_amount(fee, tx_size, fee_per_kB): + """Assert the fee was in range""" + target_fee = tx_size * fee_per_kB / 1000 + if fee < target_fee: + raise AssertionError("Fee of %s BTC too low! (Should be %s BTC)" % (str(fee), str(target_fee))) + # allow the wallet's estimation to be at most 2 bytes off + if fee > (tx_size + 2) * fee_per_kB / 1000: + raise AssertionError("Fee of %s BTC too high! (Should be %s BTC)" % (str(fee), str(target_fee))) + +def assert_equal(thing1, thing2, *args): + if thing1 != thing2 or any(thing1 != arg for arg in args): + raise AssertionError("not(%s)" % " == ".join(str(arg) for arg in (thing1, thing2) + args)) + +def assert_greater_than(thing1, thing2): + if thing1 <= thing2: + raise AssertionError("%s <= %s" % (str(thing1), str(thing2))) + +def assert_greater_than_or_equal(thing1, thing2): + if thing1 < thing2: + raise AssertionError("%s < %s" % (str(thing1), str(thing2))) + +def assert_raises(exc, fun, *args, **kwds): + assert_raises_message(exc, None, fun, *args, **kwds) + +def assert_raises_message(exc, message, fun, *args, **kwds): + try: + fun(*args, **kwds) + except exc as e: + if message is not None and message not in e.error['message']: + raise AssertionError("Expected substring not found:" + e.error['message']) + except Exception as e: + raise AssertionError("Unexpected exception raised: " + type(e).__name__) + else: + raise AssertionError("No exception raised") + +def assert_raises_jsonrpc(code, message, fun, *args, **kwds): + """Run an RPC and verify that a specific JSONRPC exception code and message is raised. + + Calls function `fun` with arguments `args` and `kwds`. Catches a JSONRPCException + and verifies that the error code and message are as expected. Throws AssertionError if + no JSONRPCException was returned or if the error code/message are not as expected. + + Args: + code (int), optional: the error code returned by the RPC call (defined + in src/rpc/protocol.h). Set to None if checking the error code is not required. + message (string), optional: [a substring of] the error string returned by the + RPC call. Set to None if checking the error string is not required + fun (function): the function to call. This should be the name of an RPC. + args*: positional arguments for the function. + kwds**: named arguments for the function. + """ + try: + fun(*args, **kwds) + except JSONRPCException as e: + # JSONRPCException was thrown as expected. Check the code and message values are correct. + if (code is not None) and (code != e.error["code"]): + raise AssertionError("Unexpected JSONRPC error code %i" % e.error["code"]) + if (message is not None) and (message not in e.error['message']): + raise AssertionError("Expected substring not found:" + e.error['message']) + except Exception as e: + raise AssertionError("Unexpected exception raised: " + type(e).__name__) + else: + raise AssertionError("No exception raised") + +def assert_is_hex_string(string): + try: + int(string, 16) + except Exception as e: + raise AssertionError( + "Couldn't interpret %r as hexadecimal; raised: %s" % (string, e)) + +def assert_is_hash_string(string, length=64): + if not isinstance(string, str): + raise AssertionError("Expected a string, got type %r" % type(string)) + elif length and len(string) != length: + raise AssertionError( + "String of length %d expected; got %d" % (length, len(string))) + elif not re.match('[abcdef0-9]+$', string): + raise AssertionError( + "String %r contains invalid characters for a hash." % string) + +def assert_array_result(object_array, to_match, expected, should_not_find=False): + """ + Pass in array of JSON objects, a dictionary with key/value pairs + to match against, and another dictionary with expected key/value + pairs. + If the should_not_find flag is true, to_match should not be found + in object_array + """ + if should_not_find: + assert_equal(expected, {}) + num_matched = 0 + for item in object_array: + all_match = True + for key, value in to_match.items(): + if item[key] != value: + all_match = False + if not all_match: + continue + elif should_not_find: + num_matched = num_matched + 1 + for key, value in expected.items(): + if item[key] != value: + raise AssertionError("%s : expected %s=%s" % (str(item), str(key), str(value))) + num_matched = num_matched + 1 + if num_matched == 0 and not should_not_find: + raise AssertionError("No objects matched %s" % (str(to_match))) + if num_matched > 0 and should_not_find: + raise AssertionError("Objects were found %s" % (str(to_match))) + +# Utility functions +################### + +def check_json_precision(): + """Make sure json library being used does not lose precision converting BTC values""" + n = Decimal("20000000.00000003") + satoshis = int(json.loads(json.dumps(float(n))) * 1.0e8) + if satoshis != 2000000000000003: + raise RuntimeError("JSON encode/decode loses precision") + +def count_bytes(hex_string): + return len(bytearray.fromhex(hex_string)) + +def bytes_to_hex_str(byte_str): + return hexlify(byte_str).decode('ascii') + +def hex_str_to_bytes(hex_str): + return unhexlify(hex_str.encode('ascii')) + +def str_to_b64str(string): + return b64encode(string.encode('utf-8')).decode('ascii') + +def satoshi_round(amount): + return Decimal(amount).quantize(Decimal('0.00000001'), rounding=ROUND_DOWN) + +# RPC/P2P connection constants and functions +############################################ + # The maximum number of nodes a single test can spawn MAX_NODES = 8 # Don't assign rpc or p2p ports lower than this @@ -35,41 +167,11 @@ PORT_MIN = 11000 # The number of ports to "reserve" for p2p and rpc, each PORT_RANGE = 5000 -BITCOIND_PROC_WAIT_TIMEOUT = 60 - - class PortSeed: # Must be initialized with a unique integer for each process n = None -#Set Mocktime default to OFF. -#MOCKTIME is only needed for scripts that use the -#cached version of the blockchain. If the cached -#version of the blockchain is used without MOCKTIME -#then the mempools will not sync due to IBD. -MOCKTIME = 0 - -def enable_mocktime(): - #For backwared compatibility of the python scripts - #with previous versions of the cache, set MOCKTIME - #to Jan 1, 2014 + (201 * 10 * 60) - global MOCKTIME - MOCKTIME = 1388534400 + (201 * 10 * 60) - -def disable_mocktime(): - global MOCKTIME - MOCKTIME = 0 - -def get_mocktime(): - return MOCKTIME - -def enable_coverage(dirname): - """Maintain a log of which RPC calls are made during testing.""" - global COVERAGE_DIR - COVERAGE_DIR = dirname - - -def get_rpc_proxy(url, node_number, timeout=None): +def get_rpc_proxy(url, node_number, timeout=None, coveragedir=None): """ Args: url (str): URL of the RPC server to call @@ -90,11 +192,10 @@ def get_rpc_proxy(url, node_number, timeout=None): proxy.url = url # store URL on proxy for info coverage_logfile = coverage.get_filename( - COVERAGE_DIR, node_number) if COVERAGE_DIR else None + coveragedir, node_number) if coveragedir else None return coverage.AuthServiceProxyWrapper(proxy, coverage_logfile) - def p2p_port(n): assert(n <= MAX_NODES) return PORT_MIN + n + (MAX_NODES * PortSeed.n) % (PORT_RANGE - 1 - MAX_NODES) @@ -102,24 +203,90 @@ def p2p_port(n): def rpc_port(n): return PORT_MIN + PORT_RANGE + n + (MAX_NODES * PortSeed.n) % (PORT_RANGE - 1 - MAX_NODES) -def check_json_precision(): - """Make sure json library being used does not lose precision converting BTC values""" - n = Decimal("20000000.00000003") - satoshis = int(json.loads(json.dumps(float(n)))*1.0e8) - if satoshis != 2000000000000003: - raise RuntimeError("JSON encode/decode loses precision") +def rpc_url(datadir, i, rpchost=None): + rpc_u, rpc_p = get_auth_cookie(datadir, i) + host = '127.0.0.1' + port = rpc_port(i) + if rpchost: + parts = rpchost.split(':') + if len(parts) == 2: + host, port = parts + else: + host = rpchost + return "http://%s:%s@%s:%d" % (rpc_u, rpc_p, host, int(port)) -def count_bytes(hex_string): - return len(bytearray.fromhex(hex_string)) +# Node functions +################ -def bytes_to_hex_str(byte_str): - return hexlify(byte_str).decode('ascii') +def initialize_datadir(dirname, n): + datadir = os.path.join(dirname, "node" + str(n)) + if not os.path.isdir(datadir): + os.makedirs(datadir) + with open(os.path.join(datadir, "bitcoin.conf"), 'w', encoding='utf8') as f: + f.write("regtest=1\n") + f.write("port=" + str(p2p_port(n)) + "\n") + f.write("rpcport=" + str(rpc_port(n)) + "\n") + f.write("listenonion=0\n") + return datadir -def hex_str_to_bytes(hex_str): - return unhexlify(hex_str.encode('ascii')) +def get_datadir_path(dirname, n): + return os.path.join(dirname, "node" + str(n)) + +def get_auth_cookie(datadir, n): + user = None + password = None + if os.path.isfile(os.path.join(datadir, "bitcoin.conf")): + with open(os.path.join(datadir, "bitcoin.conf"), 'r') as f: + for line in f: + if line.startswith("rpcuser="): + assert user is None # Ensure that there is only one rpcuser line + user = line.split("=")[1].strip("\n") + if line.startswith("rpcpassword="): + assert password is None # Ensure that there is only one rpcpassword line + password = line.split("=")[1].strip("\n") + if os.path.isfile(os.path.join(datadir, "regtest", ".cookie")): + with open(os.path.join(datadir, "regtest", ".cookie"), 'r') as f: + userpass = f.read() + split_userpass = userpass.split(':') + user = split_userpass[0] + password = split_userpass[1] + if user is None or password is None: + raise ValueError("No RPC credentials") + return user, password -def str_to_b64str(string): - return b64encode(string.encode('utf-8')).decode('ascii') +def log_filename(dirname, n_node, logname): + return os.path.join(dirname, "node" + str(n_node), "regtest", logname) + +def get_bip9_status(node, key): + info = node.getblockchaininfo() + return info['bip9_softforks'][key] + +def set_node_times(nodes, t): + for node in nodes: + node.setmocktime(t) + +def disconnect_nodes(from_connection, node_num): + for peer_id in [peer['id'] for peer in from_connection.getpeerinfo() if "testnode%d" % node_num in peer['subver']]: + from_connection.disconnectnode(nodeid=peer_id) + + for _ in range(50): + if [peer['id'] for peer in from_connection.getpeerinfo() if "testnode%d" % node_num in peer['subver']] == []: + break + time.sleep(0.1) + else: + raise AssertionError("timed out waiting for disconnect") + +def connect_nodes(from_connection, node_num): + ip_port = "127.0.0.1:" + str(p2p_port(node_num)) + from_connection.addnode(ip_port, "onetry") + # poll until version handshake complete to avoid race conditions + # with transaction relaying + while any(peer['version'] == 0 for peer in from_connection.getpeerinfo()): + time.sleep(0.1) + +def connect_nodes_bi(nodes, a, b): + connect_nodes(nodes[a], b) + connect_nodes(nodes[b], a) def sync_blocks(rpc_connections, *, wait=1, timeout=60): """ @@ -152,7 +319,7 @@ def sync_chain(rpc_connections, *, wait=1, timeout=60): """ while timeout > 0: best_hash = [x.getbestblockhash() for x in rpc_connections] - if best_hash == [best_hash[0]]*len(best_hash): + if best_hash == [best_hash[0]] * len(best_hash): return time.sleep(wait) timeout -= wait @@ -168,176 +335,15 @@ def sync_mempools(rpc_connections, *, wait=1, timeout=60): num_match = 1 for i in range(1, len(rpc_connections)): if set(rpc_connections[i].getrawmempool()) == pool: - num_match = num_match+1 + num_match = num_match + 1 if num_match == len(rpc_connections): return time.sleep(wait) timeout -= wait raise AssertionError("Mempool sync failed") -bitcoind_processes = {} - -def initialize_datadir(dirname, n): - datadir = os.path.join(dirname, "node"+str(n)) - if not os.path.isdir(datadir): - os.makedirs(datadir) - rpc_u, rpc_p = rpc_auth_pair(n) - with open(os.path.join(datadir, "bitcoin.conf"), 'w', encoding='utf8') as f: - f.write("regtest=1\n") - f.write("rpcuser=" + rpc_u + "\n") - f.write("rpcpassword=" + rpc_p + "\n") - f.write("port="+str(p2p_port(n))+"\n") - f.write("rpcport="+str(rpc_port(n))+"\n") - f.write("listenonion=0\n") - return datadir - -def rpc_auth_pair(n): - return 'rpcuser💻' + str(n), 'rpcpass🔑' + str(n) - -def rpc_url(i, rpchost=None): - rpc_u, rpc_p = rpc_auth_pair(i) - host = '127.0.0.1' - port = rpc_port(i) - if rpchost: - parts = rpchost.split(':') - if len(parts) == 2: - host, port = parts - else: - host = rpchost - return "http://%s:%s@%s:%d" % (rpc_u, rpc_p, host, int(port)) - -def wait_for_bitcoind_start(process, url, i): - ''' - Wait for bitcoind to start. This means that RPC is accessible and fully initialized. - Raise an exception if bitcoind exits during initialization. - ''' - while True: - if process.poll() is not None: - raise Exception('bitcoind exited with status %i during initialization' % process.returncode) - try: - rpc = get_rpc_proxy(url, i) - blocks = rpc.getblockcount() - break # break out of loop on success - except IOError as e: - if e.errno != errno.ECONNREFUSED: # Port not yet open? - raise # unknown IO error - except JSONRPCException as e: # Initialization phase - if e.error['code'] != -28: # RPC in warmup? - raise # unknown JSON RPC exception - time.sleep(0.25) - - -def _start_node(i, dirname, extra_args=None, rpchost=None, timewait=None, binary=None, stderr=None): - """Start a bitcoind and return RPC connection to it - - This function should only be called from within test_framework, not by individual test scripts.""" - - datadir = os.path.join(dirname, "node"+str(i)) - if binary is None: - binary = os.getenv("BITCOIND", "bitcoind") - args = [binary, "-datadir=" + datadir, "-server", "-keypool=1", "-discover=0", "-rest", "-logtimemicros", "-debug", "-debugexclude=libevent", "-debugexclude=leveldb", "-mocktime=" + str(get_mocktime()), "-uacomment=testnode%d" % i] - if extra_args is not None: args.extend(extra_args) - bitcoind_processes[i] = subprocess.Popen(args, stderr=stderr) - logger.debug("initialize_chain: bitcoind started, waiting for RPC to come up") - url = rpc_url(i, rpchost) - wait_for_bitcoind_start(bitcoind_processes[i], url, i) - logger.debug("initialize_chain: RPC successfully started") - proxy = get_rpc_proxy(url, i, timeout=timewait) - - if COVERAGE_DIR: - coverage.write_all_rpc_commands(COVERAGE_DIR, proxy) - - return proxy - -def assert_start_raises_init_error(i, dirname, extra_args=None, expected_msg=None): - with tempfile.SpooledTemporaryFile(max_size=2**16) as log_stderr: - try: - node = _start_node(i, dirname, extra_args, stderr=log_stderr) - _stop_node(node, i) - except Exception as e: - assert 'bitcoind exited' in str(e) #node must have shutdown - if expected_msg is not None: - log_stderr.seek(0) - stderr = log_stderr.read().decode('utf-8') - if expected_msg not in stderr: - raise AssertionError("Expected error \"" + expected_msg + "\" not found in:\n" + stderr) - else: - if expected_msg is None: - assert_msg = "bitcoind should have exited with an error" - else: - assert_msg = "bitcoind should have exited with expected error " + expected_msg - raise AssertionError(assert_msg) - -def _start_nodes(num_nodes, dirname, extra_args=None, rpchost=None, timewait=None, binary=None): - """Start multiple bitcoinds, return RPC connections to them - - This function should only be called from within test_framework, not by individual test scripts.""" - - if extra_args is None: extra_args = [ None for _ in range(num_nodes) ] - if binary is None: binary = [ None for _ in range(num_nodes) ] - assert_equal(len(extra_args), num_nodes) - assert_equal(len(binary), num_nodes) - rpcs = [] - try: - for i in range(num_nodes): - rpcs.append(_start_node(i, dirname, extra_args[i], rpchost, timewait=timewait, binary=binary[i])) - except: # If one node failed to start, stop the others - _stop_nodes(rpcs) - raise - return rpcs - -def log_filename(dirname, n_node, logname): - return os.path.join(dirname, "node"+str(n_node), "regtest", logname) - -def _stop_node(node, i): - """Stop a bitcoind test node - - This function should only be called from within test_framework, not by individual test scripts.""" - - logger.debug("Stopping node %d" % i) - try: - node.stop() - except http.client.CannotSendRequest as e: - logger.exception("Unable to stop node") - return_code = bitcoind_processes[i].wait(timeout=BITCOIND_PROC_WAIT_TIMEOUT) - assert_equal(return_code, 0) - del bitcoind_processes[i] - -def _stop_nodes(nodes): - """Stop multiple bitcoind test nodes - - This function should only be called from within test_framework, not by individual test scripts.""" - - for i, node in enumerate(nodes): - _stop_node(node, i) - assert not bitcoind_processes.values() # All connections must be gone now - -def set_node_times(nodes, t): - for node in nodes: - node.setmocktime(t) - -def disconnect_nodes(from_connection, node_num): - for peer_id in [peer['id'] for peer in from_connection.getpeerinfo() if "testnode%d" % node_num in peer['subver']]: - from_connection.disconnectnode(nodeid=peer_id) - - for _ in range(50): - if [peer['id'] for peer in from_connection.getpeerinfo() if "testnode%d" % node_num in peer['subver']] == []: - break - time.sleep(0.1) - else: - raise AssertionError("timed out waiting for disconnect") - -def connect_nodes(from_connection, node_num): - ip_port = "127.0.0.1:"+str(p2p_port(node_num)) - from_connection.addnode(ip_port, "onetry") - # poll until version handshake complete to avoid race conditions - # with transaction relaying - while any(peer['version'] == 0 for peer in from_connection.getpeerinfo()): - time.sleep(0.1) - -def connect_nodes_bi(nodes, a, b): - connect_nodes(nodes[a], b) - connect_nodes(nodes[b], a) +# Transaction/Block functions +############################# def find_output(node, txid, amount): """ @@ -348,14 +354,13 @@ def find_output(node, txid, amount): for i in range(len(txdata["vout"])): if txdata["vout"][i]["value"] == amount: return i - raise RuntimeError("find_output txid %s : %s not found"%(txid,str(amount))) - + raise RuntimeError("find_output txid %s : %s not found" % (txid, str(amount))) def gather_inputs(from_node, amount_needed, confirmations_required=1): """ Return a random set of unspent txouts that are enough to pay amount_needed """ - assert(confirmations_required >=0) + assert(confirmations_required >= 0) utxo = from_node.listunspent(confirmations_required) random.shuffle(utxo) inputs = [] @@ -363,9 +368,9 @@ def gather_inputs(from_node, amount_needed, confirmations_required=1): while total_in < amount_needed and len(utxo) > 0: t = utxo.pop() total_in += t["amount"] - inputs.append({ "txid" : t["txid"], "vout" : t["vout"], "address" : t["address"] } ) + inputs.append({"txid": t["txid"], "vout": t["vout"], "address": t["address"]}) if total_in < amount_needed: - raise RuntimeError("Insufficient funds: need %d, have %d"%(amount_needed, total_in)) + raise RuntimeError("Insufficient funds: need %d, have %d" % (amount_needed, total_in)) return (total_in, inputs) def make_change(from_node, amount_in, amount_out, fee): @@ -373,13 +378,13 @@ def make_change(from_node, amount_in, amount_out, fee): Create change output(s), return them """ outputs = {} - amount = amount_out+fee + amount = amount_out + fee change = amount_in - amount - if change > amount*2: + if change > amount * 2: # Create an extra change output to break up big inputs change_address = from_node.getnewaddress() # Split change in two, being careful of rounding: - outputs[change_address] = Decimal(change/2).quantize(Decimal('0.00000001'), rounding=ROUND_DOWN) + outputs[change_address] = Decimal(change / 2).quantize(Decimal('0.00000001'), rounding=ROUND_DOWN) change = amount_in - amount - outputs[change_address] if change > 0: outputs[from_node.getnewaddress()] = change @@ -392,9 +397,9 @@ def random_transaction(nodes, amount, min_fee, fee_increment, fee_variants): """ from_node = random.choice(nodes) to_node = random.choice(nodes) - fee = min_fee + fee_increment*random.randint(0,fee_variants) + fee = min_fee + fee_increment * random.randint(0, fee_variants) - (total_in, inputs) = gather_inputs(from_node, amount+fee) + (total_in, inputs) = gather_inputs(from_node, amount + fee) outputs = make_change(from_node, total_in, amount, fee) outputs[to_node.getnewaddress()] = float(amount) @@ -404,123 +409,13 @@ def random_transaction(nodes, amount, min_fee, fee_increment, fee_variants): return (txid, signresult["hex"], fee) -def assert_fee_amount(fee, tx_size, fee_per_kB): - """Assert the fee was in range""" - target_fee = tx_size * fee_per_kB / 1000 - if fee < target_fee: - raise AssertionError("Fee of %s BTC too low! (Should be %s BTC)"%(str(fee), str(target_fee))) - # allow the wallet's estimation to be at most 2 bytes off - if fee > (tx_size + 2) * fee_per_kB / 1000: - raise AssertionError("Fee of %s BTC too high! (Should be %s BTC)"%(str(fee), str(target_fee))) - -def assert_equal(thing1, thing2, *args): - if thing1 != thing2 or any(thing1 != arg for arg in args): - raise AssertionError("not(%s)" % " == ".join(str(arg) for arg in (thing1, thing2) + args)) - -def assert_greater_than(thing1, thing2): - if thing1 <= thing2: - raise AssertionError("%s <= %s"%(str(thing1),str(thing2))) - -def assert_greater_than_or_equal(thing1, thing2): - if thing1 < thing2: - raise AssertionError("%s < %s"%(str(thing1),str(thing2))) - -def assert_raises(exc, fun, *args, **kwds): - assert_raises_message(exc, None, fun, *args, **kwds) - -def assert_raises_message(exc, message, fun, *args, **kwds): - try: - fun(*args, **kwds) - except exc as e: - if message is not None and message not in e.error['message']: - raise AssertionError("Expected substring not found:"+e.error['message']) - except Exception as e: - raise AssertionError("Unexpected exception raised: "+type(e).__name__) - else: - raise AssertionError("No exception raised") - -def assert_raises_jsonrpc(code, message, fun, *args, **kwds): - """Run an RPC and verify that a specific JSONRPC exception code and message is raised. - - Calls function `fun` with arguments `args` and `kwds`. Catches a JSONRPCException - and verifies that the error code and message are as expected. Throws AssertionError if - no JSONRPCException was returned or if the error code/message are not as expected. - - Args: - code (int), optional: the error code returned by the RPC call (defined - in src/rpc/protocol.h). Set to None if checking the error code is not required. - message (string), optional: [a substring of] the error string returned by the - RPC call. Set to None if checking the error string is not required - fun (function): the function to call. This should be the name of an RPC. - args*: positional arguments for the function. - kwds**: named arguments for the function. - """ - try: - fun(*args, **kwds) - except JSONRPCException as e: - # JSONRPCException was thrown as expected. Check the code and message values are correct. - if (code is not None) and (code != e.error["code"]): - raise AssertionError("Unexpected JSONRPC error code %i" % e.error["code"]) - if (message is not None) and (message not in e.error['message']): - raise AssertionError("Expected substring not found:"+e.error['message']) - except Exception as e: - raise AssertionError("Unexpected exception raised: "+type(e).__name__) - else: - raise AssertionError("No exception raised") - -def assert_is_hex_string(string): - try: - int(string, 16) - except Exception as e: - raise AssertionError( - "Couldn't interpret %r as hexadecimal; raised: %s" % (string, e)) - -def assert_is_hash_string(string, length=64): - if not isinstance(string, str): - raise AssertionError("Expected a string, got type %r" % type(string)) - elif length and len(string) != length: - raise AssertionError( - "String of length %d expected; got %d" % (length, len(string))) - elif not re.match('[abcdef0-9]+$', string): - raise AssertionError( - "String %r contains invalid characters for a hash." % string) - -def assert_array_result(object_array, to_match, expected, should_not_find = False): - """ - Pass in array of JSON objects, a dictionary with key/value pairs - to match against, and another dictionary with expected key/value - pairs. - If the should_not_find flag is true, to_match should not be found - in object_array - """ - if should_not_find == True: - assert_equal(expected, { }) - num_matched = 0 - for item in object_array: - all_match = True - for key,value in to_match.items(): - if item[key] != value: - all_match = False - if not all_match: - continue - elif should_not_find == True: - num_matched = num_matched+1 - for key,value in expected.items(): - if item[key] != value: - raise AssertionError("%s : expected %s=%s"%(str(item), str(key), str(value))) - num_matched = num_matched+1 - if num_matched == 0 and should_not_find != True: - raise AssertionError("No objects matched %s"%(str(to_match))) - if num_matched > 0 and should_not_find == True: - raise AssertionError("Objects were found %s"%(str(to_match))) - -def satoshi_round(amount): - return Decimal(amount).quantize(Decimal('0.00000001'), rounding=ROUND_DOWN) - # Helper to create at least "count" utxos # Pass in a fee that is sufficient for relay and mining new transactions. def create_confirmed_utxos(fee, node, count): - node.generate(int(0.5*count)+101) + to_generate = int(0.5 * count) + 101 + while to_generate > 0: + node.generate(min(25, to_generate)) + to_generate -= 25 utxos = node.listunspent() iterations = count - len(utxos) addr1 = node.getnewaddress() @@ -530,14 +425,14 @@ def create_confirmed_utxos(fee, node, count): for i in range(iterations): t = utxos.pop() inputs = [] - inputs.append({ "txid" : t["txid"], "vout" : t["vout"]}) + inputs.append({"txid": t["txid"], "vout": t["vout"]}) outputs = {} send_value = t['amount'] - fee - outputs[addr1] = satoshi_round(send_value/2) - outputs[addr2] = satoshi_round(send_value/2) + outputs[addr1] = satoshi_round(send_value / 2) + outputs[addr2] = satoshi_round(send_value / 2) raw_tx = node.createrawtransaction(inputs, outputs) signed_tx = node.signrawtransaction(raw_tx)["hex"] - txid = node.sendrawtransaction(signed_tx) + node.sendrawtransaction(signed_tx) while (node.getmempoolinfo()['size'] > 0): node.generate(1) @@ -552,8 +447,8 @@ def gen_return_txouts(): # Some pre-processing to create a bunch of OP_RETURN txouts to insert into transactions we create # So we have big transactions (and therefore can't fit very many into each block) # create one script_pubkey - script_pubkey = "6a4d0200" #OP_RETURN OP_PUSH2 512 bytes - for i in range (512): + script_pubkey = "6a4d0200" # OP_RETURN OP_PUSH2 512 bytes + for i in range(512): script_pubkey = script_pubkey + "01" # concatenate 128 txouts of above script_pubkey which we'll insert before the txout for change txouts = "81" @@ -567,8 +462,8 @@ def gen_return_txouts(): return txouts def create_tx(node, coinbase, to_address, amount): - inputs = [{ "txid" : coinbase, "vout" : 0}] - outputs = { to_address : amount } + inputs = [{"txid": coinbase, "vout": 0}] + outputs = {to_address: amount} rawtx = node.createrawtransaction(inputs, outputs) signresult = node.signrawtransaction(rawtx) assert_equal(signresult["complete"], True) @@ -581,7 +476,7 @@ def create_lots_of_big_transactions(node, txouts, utxos, num, fee): txids = [] for _ in range(num): t = utxos.pop() - inputs=[{ "txid" : t["txid"], "vout" : t["vout"]}] + inputs = [{"txid": t["txid"], "vout": t["vout"]}] outputs = {} change = t['amount'] - fee outputs[addr] = satoshi_round(change) @@ -606,7 +501,3 @@ def mine_large_block(node, utxos=None): fee = 100 * node.getnetworkinfo()["relayfee"] create_lots_of_big_transactions(node, txouts, utxos, num, fee=fee) node.generate(1) - -def get_bip9_status(node, key): - info = node.getblockchaininfo() - return info['bip9_softforks'][key] diff --git a/test/functional/test_runner.py b/test/functional/test_runner.py index 4702f2d773..54f625514b 100755 --- a/test/functional/test_runner.py +++ b/test/functional/test_runner.py @@ -20,6 +20,7 @@ import datetime import os import time import shutil +import signal import sys import subprocess import tempfile @@ -78,7 +79,7 @@ BASE_SCRIPTS= [ 'rawtransactions.py', 'reindex.py', # vv Tests less than 30s vv - "zmq_test.py", + 'zmq_test.py', 'mempool_resurrect_test.py', 'txn_doublespend.py --mineblock', 'txn_clone.py', @@ -111,6 +112,8 @@ BASE_SCRIPTS= [ 'rpcnamedargs.py', 'listsinceblock.py', 'p2p-leaktests.py', + 'wallet-encryption.py', + 'uptime.py', ] EXTENDED_SCRIPTS = [ @@ -122,6 +125,7 @@ EXTENDED_SCRIPTS = [ # vv Tests less than 5m vv 'maxuploadtarget.py', 'mempool_packages.py', + 'dbcrash.py', # vv Tests less than 2m vv 'bip68-sequence.py', 'getblocktemplate_longpoll.py', @@ -136,6 +140,7 @@ EXTENDED_SCRIPTS = [ 'bip65-cltv-p2p.py', 'bipdersig-p2p.py', 'bipdersig.py', + 'example_test.py', 'getblocktemplate_proposals.py', 'txn_doublespend.py', 'txn_clone.py --mineblock', @@ -389,6 +394,10 @@ class TestHandler: time.sleep(.5) for j in self.jobs: (name, time0, proc, log_out, log_err) = j + if os.getenv('TRAVIS') == 'true' and int(time.time() - time0) > 20 * 60: + # In travis, timeout individual tests after 20 minutes (to stop tests hanging and not + # providing useful output. + proc.send_signal(signal.SIGINT) if proc.poll() is not None: log_out.seek(0), log_err.seek(0) [stdout, stderr] = [l.read().decode('utf-8') for l in (log_out, log_err)] diff --git a/test/functional/uptime.py b/test/functional/uptime.py new file mode 100755 index 0000000000..b20d6f5cb6 --- /dev/null +++ b/test/functional/uptime.py @@ -0,0 +1,32 @@ +#!/usr/bin/env python3 +# Copyright (c) 2017 The Bitcoin Core developers +# Distributed under the MIT software license, see the accompanying +# file COPYING or http://www.opensource.org/licenses/mit-license.php. +"""Test the RPC call related to the uptime command. + +Test corresponds to code in rpc/server.cpp. +""" + +import time + +from test_framework.test_framework import BitcoinTestFramework + + +class UptimeTest(BitcoinTestFramework): + def __init__(self): + super().__init__() + + self.num_nodes = 1 + self.setup_clean_chain = True + + def run_test(self): + self._test_uptime() + + def _test_uptime(self): + wait_time = 10 + self.nodes[0].setmocktime(int(time.time() + wait_time)) + assert(self.nodes[0].uptime() >= wait_time) + + +if __name__ == '__main__': + UptimeTest().main() diff --git a/test/functional/wallet-dump.py b/test/functional/wallet-dump.py index 9cb32d4650..569cc46e6c 100755 --- a/test/functional/wallet-dump.py +++ b/test/functional/wallet-dump.py @@ -7,7 +7,7 @@ import os from test_framework.test_framework import BitcoinTestFramework -from test_framework.util import (assert_equal, bitcoind_processes) +from test_framework.util import assert_equal def read_dump(file_name, addrs, hd_master_addr_old): @@ -95,7 +95,7 @@ class WalletDumpTest(BitcoinTestFramework): #encrypt wallet, restart, unlock and dump self.nodes[0].encryptwallet('test') - bitcoind_processes[0].wait() + self.bitcoind_processes[0].wait() self.nodes[0] = self.start_node(0, self.options.tmpdir, self.extra_args[0]) self.nodes[0].walletpassphrase('test', 10) # Should be a no-op: diff --git a/test/functional/wallet-encryption.py b/test/functional/wallet-encryption.py new file mode 100755 index 0000000000..ba72918fe1 --- /dev/null +++ b/test/functional/wallet-encryption.py @@ -0,0 +1,64 @@ +#!/usr/bin/env python3 +# Copyright (c) 2016 The Bitcoin Core developers +# Distributed under the MIT software license, see the accompanying +# file COPYING or http://www.opensource.org/licenses/mit-license.php. +"""Test Wallet encryption""" + +import time + +from test_framework.test_framework import BitcoinTestFramework, BITCOIND_PROC_WAIT_TIMEOUT +from test_framework.util import ( + assert_equal, + assert_raises_jsonrpc, +) + +class WalletEncryptionTest(BitcoinTestFramework): + + def __init__(self): + super().__init__() + self.setup_clean_chain = True + self.num_nodes = 1 + + def run_test(self): + passphrase = "WalletPassphrase" + passphrase2 = "SecondWalletPassphrase" + + # Make sure the wallet isn't encrypted first + address = self.nodes[0].getnewaddress() + privkey = self.nodes[0].dumpprivkey(address) + assert_equal(privkey[:1], "c") + assert_equal(len(privkey), 52) + + # Encrypt the wallet + self.nodes[0].encryptwallet(passphrase) + self.bitcoind_processes[0].wait(timeout=BITCOIND_PROC_WAIT_TIMEOUT) + self.nodes[0] = self.start_node(0, self.options.tmpdir) + + # Test that the wallet is encrypted + assert_raises_jsonrpc(-13, "Please enter the wallet passphrase with walletpassphrase first", self.nodes[0].dumpprivkey, address) + + # Check that walletpassphrase works + self.nodes[0].walletpassphrase(passphrase, 2) + assert_equal(privkey, self.nodes[0].dumpprivkey(address)) + + # Check that the timeout is right + time.sleep(2) + assert_raises_jsonrpc(-13, "Please enter the wallet passphrase with walletpassphrase first", self.nodes[0].dumpprivkey, address) + + # Test wrong passphrase + assert_raises_jsonrpc(-14, "wallet passphrase entered was incorrect", self.nodes[0].walletpassphrase, passphrase + "wrong", 10) + + # Test walletlock + self.nodes[0].walletpassphrase(passphrase, 84600) + assert_equal(privkey, self.nodes[0].dumpprivkey(address)) + self.nodes[0].walletlock() + assert_raises_jsonrpc(-13, "Please enter the wallet passphrase with walletpassphrase first", self.nodes[0].dumpprivkey, address) + + # Test passphrase changes + self.nodes[0].walletpassphrasechange(passphrase, passphrase2) + assert_raises_jsonrpc(-14, "wallet passphrase entered was incorrect", self.nodes[0].walletpassphrase, passphrase, 10) + self.nodes[0].walletpassphrase(passphrase2, 10) + assert_equal(privkey, self.nodes[0].dumpprivkey(address)) + +if __name__ == '__main__': + WalletEncryptionTest().main() diff --git a/test/functional/wallet-hd.py b/test/functional/wallet-hd.py index e7ec72a248..dfd3dc83c5 100755 --- a/test/functional/wallet-hd.py +++ b/test/functional/wallet-hd.py @@ -8,7 +8,6 @@ from test_framework.test_framework import BitcoinTestFramework from test_framework.util import ( assert_equal, connect_nodes_bi, - assert_start_raises_init_error ) import os import shutil @@ -27,7 +26,7 @@ class WalletHDTest(BitcoinTestFramework): # Make sure can't switch off usehd after wallet creation self.stop_node(1) - assert_start_raises_init_error(1, self.options.tmpdir, ['-usehd=0'], 'already existing HD wallet') + self.assert_start_raises_init_error(1, self.options.tmpdir, ['-usehd=0'], 'already existing HD wallet') self.nodes[1] = self.start_node(1, self.options.tmpdir, self.extra_args[1]) connect_nodes_bi(self.nodes, 0, 1) diff --git a/test/functional/walletbackup.py b/test/functional/walletbackup.py index a4507182a2..ff51cba4b3 100755 --- a/test/functional/walletbackup.py +++ b/test/functional/walletbackup.py @@ -30,10 +30,11 @@ confirm 1/2/3/4 balances are same as before. Shutdown again, restore using importwallet, and confirm again balances are correct. """ +from random import randint +import shutil from test_framework.test_framework import BitcoinTestFramework from test_framework.util import * -from random import randint class WalletBackupTest(BitcoinTestFramework): diff --git a/test/functional/zapwallettxes.py b/test/functional/zapwallettxes.py index a8600e82f6..e4d40520ef 100755 --- a/test/functional/zapwallettxes.py +++ b/test/functional/zapwallettxes.py @@ -70,7 +70,7 @@ class ZapWalletTXesTest (BitcoinTestFramework): self.nodes[0] = self.start_node(0,self.options.tmpdir, ["-zapwallettxes=1"]) assert_raises(JSONRPCException, self.nodes[0].gettransaction, [txid3]) - #there must be a expection because the unconfirmed wallettx0 must be gone by now + #there must be an exception because the unconfirmed wallettx0 must be gone by now tx0 = self.nodes[0].gettransaction(txid0) assert_equal(tx0['txid'], txid0) #tx0 (confirmed) must still be available because it was confirmed diff --git a/test/functional/zmq_test.py b/test/functional/zmq_test.py index ce39cfefdc..26c946d215 100755 --- a/test/functional/zmq_test.py +++ b/test/functional/zmq_test.py @@ -8,15 +8,15 @@ import os import struct from test_framework.test_framework import BitcoinTestFramework, SkipTest -from test_framework.util import * +from test_framework.util import (assert_equal, + bytes_to_hex_str, + ) class ZMQTest (BitcoinTestFramework): def __init__(self): super().__init__() - self.num_nodes = 4 - - port = 28332 + self.num_nodes = 2 def setup_nodes(self): # Try to import python3-zmq. Skip this test if the import fails. @@ -28,7 +28,7 @@ class ZMQTest (BitcoinTestFramework): # Check that bitcoin has been built with ZMQ enabled config = configparser.ConfigParser() if not self.options.configfile: - self.options.configfile = os.path.dirname(__file__) + "/config.ini" + self.options.configfile = os.path.dirname(__file__) + "/../config.ini" config.read_file(open(self.options.configfile)) if not config["components"].getboolean("ENABLE_ZMQ"): @@ -36,59 +36,66 @@ class ZMQTest (BitcoinTestFramework): self.zmqContext = zmq.Context() self.zmqSubSocket = self.zmqContext.socket(zmq.SUB) + self.zmqSubSocket.set(zmq.RCVTIMEO, 60000) self.zmqSubSocket.setsockopt(zmq.SUBSCRIBE, b"hashblock") self.zmqSubSocket.setsockopt(zmq.SUBSCRIBE, b"hashtx") - self.zmqSubSocket.connect("tcp://127.0.0.1:%i" % self.port) - self.nodes = self.start_nodes(self.num_nodes, self.options.tmpdir, extra_args=[ - ['-zmqpubhashtx=tcp://127.0.0.1:'+str(self.port), '-zmqpubhashblock=tcp://127.0.0.1:'+str(self.port)], - [], - [], - [] - ]) + ip_address = "tcp://127.0.0.1:28332" + self.zmqSubSocket.connect(ip_address) + extra_args = [['-zmqpubhashtx=%s' % ip_address, '-zmqpubhashblock=%s' % ip_address], []] + self.nodes = self.start_nodes(self.num_nodes, self.options.tmpdir, extra_args) def run_test(self): - self.sync_all() + try: + self._zmq_test() + finally: + # Destroy the zmq context + self.log.debug("Destroying zmq context") + self.zmqContext.destroy(linger=None) + def _zmq_test(self): genhashes = self.nodes[0].generate(1) self.sync_all() - self.log.info("listen...") + self.log.info("Wait for tx") msg = self.zmqSubSocket.recv_multipart() topic = msg[0] assert_equal(topic, b"hashtx") body = msg[1] msgSequence = struct.unpack('<I', msg[-1])[-1] - assert_equal(msgSequence, 0) #must be sequence 0 on hashtx + assert_equal(msgSequence, 0) # must be sequence 0 on hashtx + self.log.info("Wait for block") msg = self.zmqSubSocket.recv_multipart() topic = msg[0] body = msg[1] msgSequence = struct.unpack('<I', msg[-1])[-1] - assert_equal(msgSequence, 0) #must be sequence 0 on hashblock + assert_equal(msgSequence, 0) # must be sequence 0 on hashblock blkhash = bytes_to_hex_str(body) - assert_equal(genhashes[0], blkhash) #blockhash from generate must be equal to the hash received over zmq + assert_equal(genhashes[0], blkhash) # blockhash from generate must be equal to the hash received over zmq + self.log.info("Generate 10 blocks (and 10 coinbase txes)") n = 10 genhashes = self.nodes[1].generate(n) self.sync_all() zmqHashes = [] blockcount = 0 - for x in range(0,n*2): + for x in range(n * 2): msg = self.zmqSubSocket.recv_multipart() topic = msg[0] body = msg[1] if topic == b"hashblock": zmqHashes.append(bytes_to_hex_str(body)) msgSequence = struct.unpack('<I', msg[-1])[-1] - assert_equal(msgSequence, blockcount+1) + assert_equal(msgSequence, blockcount + 1) blockcount += 1 - for x in range(0,n): - assert_equal(genhashes[x], zmqHashes[x]) #blockhash from generate must be equal to the hash received over zmq + for x in range(n): + assert_equal(genhashes[x], zmqHashes[x]) # blockhash from generate must be equal to the hash received over zmq - #test tx from a second node + self.log.info("Wait for tx from second node") + # test tx from a second node hashRPC = self.nodes[1].sendtoaddress(self.nodes[0].getnewaddress(), 1.0) self.sync_all() @@ -96,14 +103,12 @@ class ZMQTest (BitcoinTestFramework): msg = self.zmqSubSocket.recv_multipart() topic = msg[0] body = msg[1] - hashZMQ = "" - if topic == b"hashtx": - hashZMQ = bytes_to_hex_str(body) - msgSequence = struct.unpack('<I', msg[-1])[-1] - assert_equal(msgSequence, blockcount+1) - - assert_equal(hashRPC, hashZMQ) #blockhash from generate must be equal to the hash received over zmq + assert_equal(topic, b"hashtx") + hashZMQ = bytes_to_hex_str(body) + msgSequence = struct.unpack('<I', msg[-1])[-1] + assert_equal(msgSequence, blockcount + 1) + assert_equal(hashRPC, hashZMQ) # txid from sendtoaddress must be equal to the hash received over zmq if __name__ == '__main__': - ZMQTest ().main () + ZMQTest().main() |