aboutsummaryrefslogtreecommitdiff
path: root/test/functional/test_framework
AgeCommit message (Collapse)Author
2017-09-07[test] Replace check_output with low level versionJoão Barbosa
2017-09-07[test] Add assert_raises_process_error to assert process errorsJoão Barbosa
2017-09-07[test] Add support for custom arguments to TestNodeCLIJoão Barbosa
2017-09-07[test] Improve assert_raises_jsonrpc docstringJoão Barbosa
2017-09-06[qa] TestNode: Add wait_until_stopped helper methodMarcoFalke
2017-09-01Merge #11121: TestNode tidyupsMarcoFalke
7148b74dc [tests] Functional tests must explicitly set num_nodes (John Newbery) 5448a1471 [tests] don't override __init__() in individual tests (John Newbery) 6cf094a02 [tests] Avoid passing around member variables in test_framework (John Newbery) 36b626867 [tests] TestNode: separate add_node from start_node (John Newbery) be2a2ab6a [tests] fix - use rpc_timeout as rpc timeout (John Newbery) Pull request description: Some additional tidyups after the introduction of TestNode: - commit 1 makes TestNode use the correct rpc timeout. This should have been included in #11077 - commit 2 separates `add_node()` from `start_node()` as originally discussed here: https://github.com/bitcoin/bitcoin/pull/10556#discussion_r121161453 with @kallewoof . The test writer no longer needs to assign to `self.nodes` when starting/stopping nodes. - commit 3 adds a `set_test_params()` method, so individual tests don't need to override `__init__()` and call `super().__init__()` Tree-SHA512: 0adb030623b96675b5c29e2890ce99ccd837ed05f721d0c91b35378c5ac01b6658174aac12f1f77402e1d38b61f39b3c43b4df85c96952565dde1cda05b0db84
2017-09-01[tests] Functional tests must explicitly set num_nodesJohn Newbery
2017-09-01[tests] don't override __init__() in individual testsJohn Newbery
Almost all test scripts currently need to override the __init__() method. When they do that they need to call into super().__init__() as the base class does some generic initialization. This commit makes the base class __init__() call into set_test_params() method. Individual test cases can override set_test_params() to setup their test parameters.
2017-09-01[tests] Avoid passing around member variables in test_frameworkJohn Newbery
2017-09-01[tests] TestNode: separate add_node from start_nodeJohn Newbery
Separates the act of creating a TestNode object from starting the node. The test_framework now keeps track of its list of TestNodes, and test writers can call start_node() and stop_node() without having to update the self.nodes list.
2017-09-01[tests] fix - use rpc_timeout as rpc timeoutJohn Newbery
2017-08-28Use the variable name _ for unused return valuespracticalswift
2017-08-28Remove unused variables and/or function callspracticalswift
2017-08-24Merge #10798: [tests] [utils] test bitcoin-cliMarcoFalke
c6ec4358a [tests] Add bitcoin_cli.py test script (John Newbery) b23549f6e [tests] add TestNodeCLI class for calling bitcoin-cli for a node (John Newbery) Pull request description: We don't test bitcoin-cli at all. That means that we can miss inconsistencies between the bitcoin-cli client and the RPC interface, such as #10698 and #10747. It also means that the various bitcoin-cli options and features are untested and regressions could be silently introduced. Let's fix that. This PR adds bitcoin-cli testing in the python functional test_framework: 1. Add a bitcoin_cli.py test script that tests bitcoin-cli. At the moment it only tests that the result of `getinfo` is the same if you run it as an RPC or through bitcoin-cli, but can easily be extended to test additional bitcoin-cli features **EDIT: `--usecli` option is moved to a separate PR. This PR now only covers the bitcoin_cli.py test.** 2. ~Add a `--usecli` option to the test framework. This changes the test to use bitcoin-cli for all RPC calls instead of using direct HTTP requests. This is somewhat experimental. It works for most tests, but there are some cases where it can't work transparently because:~ - ~the testcase is asserting on a specific error code, and bitcoin-cli returns a different error code from the direct RPC~ - ~we're sending a very large RPC request (eg `submitblock`) and it can't be serialized into a shell bitcoin-cli call.~ ~I think that even though `--usecli` doesn't work on all tests, it's still a useful experimental feature. Future potential enhancements:~ - ~enhance the framework to automatically skip tests that are known to fail with bitcoin-cli if the `--usecli` option is used.~ - ~run a subset of tests in Travis with `-usecli`~ This builds on and requires the `TestNode` PR #10711 . As an aside, this is a good demonstration of how tidy it is to add additional features/interfaces now that test node logic/state is encapsulated in a TestNode class. Addresses #10791 Tree-SHA512: a1e6be12e8e007f6f67b3d3bbcd142d835787300831eb38e6027a1ad25ca9d79c4bc99a41b19e31ee95205cba1b3b2d21a688b5909316aad70bfc2b4eb6d8a52
2017-08-23Merge #11068: qa: Move wait_until to utilMarcoFalke
08ce33f8e qa: Move wait_until to util (MarcoFalke) Pull request description: This moves `wait_until` to `util.py` to make it generally available to python tests. Also, `wait_until` now takes an optional lock that is acquired while testing the predicate. Previously the lock was always acquired, even when it was not necessary, cf. `disconnect_ban.py`. Tree-SHA512: 18e452a017a6566fa8ad09bde058e1b841e167039dc63299e70cfa7a6dcbc779581e60ca3e8eb2f1b610767d5208b9376c203eb11015b250fd0542b5eb4215a8
2017-08-23[tests] fix timeout issues from TestNodeJohn Newbery
Fixes a couple of bugs from the introduction of TestNode: - test scripts were no longer able to specify a custom timeout for starting a node. Therefore tests with nodes that take a long time to start up (eg pruning.py) would fail. - the test for whether a node has failed on start up was broken by changing 'assert x is None' to 'assert not x'. Since subprocess.poll() can return None (indicating the node is still running) or 0 (indicating the node exited with return code 0), this was a regression.
2017-08-20test: Increase initial RPC timeout to 60 secondsWladimir J. van der Laan
When running the tests locally with a parallelism of 4 on an otherwise busy system, RPC can take quite a wait to come up. Change the timeout to 60 seconds just to be safe.
2017-08-17qa: Move wait_until to utilMarcoFalke
2017-08-15[tests] add TestNodeCLI class for calling bitcoin-cli for a nodeJohn Newbery
2017-08-15Merge #10711: [tests] Introduce TestNodeMarcoFalke
789733891 [tests] Introduce TestNode (John Newbery) Pull request description: Continues #10082 TestNode is a class responsible for all state related to a bitcoind node under test. It stores local state, is responsible for tracking the bitcoind process and delegates unrecognised messages to the RPC connection. This commit changes start_nodes and stop_nodes to start and stop the bitcoind nodes in parallel, making test setup and teardown much faster. On my vm, this changeset reduces total test_runner runtime for the base set of tests (including building the cache) from 263s to 195s (a 25% speedup). Note that the time reported by test_runner does not include time spent building the cache: *with TestNode*: ``` → date +"%T" ; ./test_runner.py -q ; date +"%T" 12:48:04 .................................................................................................................................................................................................................................................................................................................................. TEST | STATUS | DURATION abandonconflict.py | ✓ Passed | 12 s bip68-112-113-p2p.py | ✓ Passed | 19 s blockchain.py | ✓ Passed | 8 s bumpfee.py | ✓ Passed | 13 s decodescript.py | ✓ Passed | 3 s disablewallet.py | ✓ Passed | 3 s disconnect_ban.py | ✓ Passed | 6 s fundrawtransaction.py | ✓ Passed | 37 s getchaintips.py | ✓ Passed | 4 s httpbasics.py | ✓ Passed | 3 s import-rescan.py | ✓ Passed | 4 s importmulti.py | ✓ Passed | 6 s importprunedfunds.py | ✓ Passed | 3 s invalidblockrequest.py | ✓ Passed | 4 s invalidtxrequest.py | ✓ Passed | 4 s keypool.py | ✓ Passed | 7 s listsinceblock.py | ✓ Passed | 4 s listtransactions.py | ✓ Passed | 33 s mempool_limit.py | ✓ Passed | 4 s mempool_persist.py | ✓ Passed | 15 s mempool_reorg.py | ✓ Passed | 4 s mempool_resurrect_test.py | ✓ Passed | 3 s mempool_spendcoinbase.py | ✓ Passed | 3 s merkle_blocks.py | ✓ Passed | 3 s multi_rpc.py | ✓ Passed | 4 s net.py | ✓ Passed | 3 s nulldummy.py | ✓ Passed | 3 s p2p-compactblocks.py | ✓ Passed | 28 s p2p-fullblocktest.py | ✓ Passed | 126 s p2p-leaktests.py | ✓ Passed | 8 s p2p-mempool.py | ✓ Passed | 3 s p2p-segwit.py | ✓ Passed | 59 s p2p-versionbits-warning.py | ✓ Passed | 8 s preciousblock.py | ✓ Passed | 3 s prioritise_transaction.py | ✓ Passed | 5 s proxy_test.py | ✓ Passed | 3 s rawtransactions.py | ✓ Passed | 9 s receivedby.py | ✓ Passed | 19 s reindex.py | ✓ Passed | 12 s rest.py | ✓ Passed | 9 s rpcnamedargs.py | ✓ Passed | 3 s segwit.py | ✓ Passed | 7 s sendheaders.py | ✓ Passed | 24 s signmessages.py | ✓ Passed | 3 s signrawtransactions.py | ✓ Passed | 3 s txn_clone.py | ✓ Passed | 4 s txn_doublespend.py --mineblock | ✓ Passed | 4 s uptime.py | ✓ Passed | 3 s wallet-accounts.py | ✓ Passed | 3 s wallet-dump.py | ✓ Passed | 7 s wallet-encryption.py | ✓ Passed | 8 s wallet-hd.py | ✓ Passed | 15 s wallet.py | ✓ Passed | 31 s walletbackup.py | ✓ Passed | 104 s zapwallettxes.py | ✓ Passed | 9 s zmq_test.py | ○ Skipped | 0 s ALL | ✓ Passed | 735 s (accumulated) Runtime: 189 s 12:51:19 ``` *master*: ``` → date +"%T" ; ./test_runner.py -q ; date +"%T" 12:40:13 .......................................................................................................................................................................................................................................................................................................................................................................................................................................... TEST | STATUS | DURATION abandonconflict.py | ✓ Passed | 15 s bip68-112-113-p2p.py | ✓ Passed | 19 s blockchain.py | ✓ Passed | 8 s bumpfee.py | ✓ Passed | 20 s decodescript.py | ✓ Passed | 3 s disablewallet.py | ✓ Passed | 3 s disconnect_ban.py | ✓ Passed | 8 s fundrawtransaction.py | ✓ Passed | 36 s getchaintips.py | ✓ Passed | 11 s httpbasics.py | ✓ Passed | 7 s import-rescan.py | ✓ Passed | 16 s importmulti.py | ✓ Passed | 10 s importprunedfunds.py | ✓ Passed | 5 s invalidblockrequest.py | ✓ Passed | 4 s invalidtxrequest.py | ✓ Passed | 3 s keypool.py | ✓ Passed | 7 s listsinceblock.py | ✓ Passed | 11 s listtransactions.py | ✓ Passed | 37 s mempool_limit.py | ✓ Passed | 4 s mempool_persist.py | ✓ Passed | 23 s mempool_reorg.py | ✓ Passed | 7 s mempool_resurrect_test.py | ✓ Passed | 3 s mempool_spendcoinbase.py | ✓ Passed | 3 s merkle_blocks.py | ✓ Passed | 10 s multi_rpc.py | ✓ Passed | 6 s net.py | ✓ Passed | 6 s nulldummy.py | ✓ Passed | 3 s p2p-compactblocks.py | ✓ Passed | 30 s p2p-fullblocktest.py | ✓ Passed | 126 s p2p-leaktests.py | ✓ Passed | 8 s p2p-mempool.py | ✓ Passed | 3 s p2p-segwit.py | ✓ Passed | 62 s p2p-versionbits-warning.py | ✓ Passed | 8 s preciousblock.py | ✓ Passed | 8 s prioritise_transaction.py | ✓ Passed | 7 s proxy_test.py | ✓ Passed | 10 s rawtransactions.py | ✓ Passed | 15 s receivedby.py | ✓ Passed | 28 s reindex.py | ✓ Passed | 12 s rest.py | ✓ Passed | 12 s rpcnamedargs.py | ✓ Passed | 3 s segwit.py | ✓ Passed | 12 s sendheaders.py | ✓ Passed | 26 s signmessages.py | ✓ Passed | 3 s signrawtransactions.py | ✓ Passed | 3 s txn_clone.py | ✓ Passed | 10 s txn_doublespend.py --mineblock | ✓ Passed | 10 s uptime.py | ✓ Passed | 3 s wallet-accounts.py | ✓ Passed | 3 s wallet-dump.py | ✓ Passed | 6 s wallet-encryption.py | ✓ Passed | 8 s wallet-hd.py | ✓ Passed | 18 s wallet.py | ✓ Passed | 69 s walletbackup.py | ✓ Passed | 130 s zapwallettxes.py | ✓ Passed | 15 s zmq_test.py | ○ Skipped | 0 s ALL | ✓ Passed | 936 s (accumulated) Runtime: 242 s 12:44:36 ``` Tree-SHA512: 6dfc4c11fd0caf7de6954c93679cf22c3df0acc6f432e616d1151062a61f456faa8ae2fe670b427868af55bb564802df84c8fd76e90b4b338750dbc23f46ad88
2017-08-15[tests] Introduce TestNodeJohn Newbery
TestNode is a class responsible for all state related to a bitcoind node under test. It stores local state, is responsible for tracking the bitcoind process and delegates unrecognised messages to the RPC connection. This commit changes start_nodes and stop_nodes to start and stop the bitcoind nodes in parallel, making test setup and teardown much faster.
2017-08-11Merge #11023: [tests] Add option to attach a python debugger if functional ↵MarcoFalke
test fails cc5d38f4b Add option to attach a python debugger if test fails (John Newbery) Pull request description: Adds a simple option to the test_framework to attach pdb if the test fails. Helpful for catching and debugging intermittent failures: Run the test in a loop with this option. The first failure will cause execution to pause and nodes will be left running for interactive debugging. @sdaftuar Tree-SHA512: 01cfae15fa3f04ed6ec6a99fef60a6c6a59723429309e81eacd6767caf12f5758f59b337804291ecab33a38a2958f36e2b513d201bee72a2eeb207a67046f952
2017-08-10Add option to attach a python debugger if test failsJohn Newbery
2017-08-07[tests] Test disconnecting unsupported service bits logic.John Newbery
In v0.15, we disconnect nodes that send us version messages with unsupported service bits (1 << 5 and 1 << 7). This commit adds a test that bitcoind will disconnect those nodes before August 1st 2018, and won't disconnect those nodes after August 1st 2018.
2017-07-17[tests] [wallet] Add wallet endpoint support to authproxyJohn Newbery
2017-07-06Fix multi_rpc test for hosts that dont default to utf8Matt Corallo
Otherwise the utf8 written to bitcoin.conf throws an exception when read from get_auth_cookie
2017-07-02[tests] nits in dbcrash.pyJohn Newbery
2017-06-29[tests] fix nits.John Newbery
Thanks to Marco Falke.
2017-06-29[tests] reorganize utils.py module (code move only)John Newbery
This commit re-organizes the utils.py module into logical sections.
2017-06-29[tests] move mocktime property and functions to BitcoinTestFrameworkJohn Newbery
2017-06-29[tests] Move stop_node and start_node methods to BitcoinTestFrameworkJohn Newbery
This commit moves functions start_node, start_nodes, stop_node and stop_nodes functions into the BitcoinTestFramework class. It also moves the bitcoind_processes dict and coverage variables into BitcoinTestFramework.
2017-06-29[tests] fix flake8 warnings in test_framework.py and util.pyJohn Newbery
2017-06-29[tests] remove unused imports from utils.pyJohn Newbery
2017-06-28Merge #10148: Use non-atomic flushing with block replayWladimir J. van der Laan
176c021 [qa] Test non-atomic chainstate writes (Suhas Daftuar) d6af06d Dont create pcoinsTip until after ReplayBlocks. (Matt Corallo) eaca1b7 Random db flush crash simulator (Pieter Wuille) 0580ee0 Adapt memory usage estimation for flushing (Pieter Wuille) 013a56a Non-atomic flushing using the blockchain as replay journal (Pieter Wuille) b3a279c [MOVEONLY] Move LastCommonAncestor to chain (Pieter Wuille) Tree-SHA512: 47ccc62303f9075c44d2a914be75bd6969ff881a857a2ff1227f05ec7def6f4c71c46680c5a28cb150c814999526797dc05cf2701fde1369c06169f46eccddee
2017-06-28Merge #10690: [qa] Bugfix: allow overriding extra_args in ↵MarcoFalke
ComparisonTestFramework 4ed3653 [qa] Bugfix: allow overriding extra_args in ComparisonTestFramework (Suhas Daftuar) Tree-SHA512: d8f724b3324aad73a7b15cf87ff394e8d615bf3cd5a394d5715347d833f2ae9ac745a944202986866255eca5fc105ea06ab3abe12e168b67de34482f751c68e2
2017-06-28[qa] Bugfix: allow overriding extra_args in ComparisonTestFrameworkSuhas Daftuar
2017-06-26[qa] Test non-atomic chainstate writesSuhas Daftuar
Adds new functional test, dbcrash.py, which uses -dbcrashratio to exercise the logic for recovering from a crash during chainstate flush. dbcrash.py is added to the extended tests, as it may take ~10 minutes to run Use _Exit() instead of exit() for crash simulation This eliminates stderr output such as: terminate called without an active exception or Assertion failed: (!pthread_mutex_destroy(&m)), function ~recursive_mutex, file /usr/local/include/boost/thread/pthread/recursive_mutex.hpp, line 104. Eliminating the stderr output on crash simulation allows testing with test_runner.py, which reports a test as failed if stderr is produced.
2017-06-22Merge #10633: doc: Fix various typosWladimir J. van der Laan
0a5a6b9 Fixed multiple typos (Dimitris Tsapakidis) Tree-SHA512: 57748710bcbc03945b160db5e95bd686a2c64605f25d5e11d8ed9d0e1be3b3bf287a63588dc6eb33d0cef4ff17c765fda7c226d667a357acc539c8fcf2b9bb7e
2017-06-22Merge #10636: [qa] util: Check return code after closing bitcoind procMarcoFalke
999923e [qa] util: Check return code after closing bitcoind proc (MarcoFalke) Tree-SHA512: 1aa911e5ce25ef59b006b7f8f25e5eef3c6c31f1b4362564e158ed92dd7804625fccc08be8b98c8c32779b530a8ec988819947692698214e34f5cb1d411c9eda
2017-06-22Fixed multiple typosDimitris Tsapakidis
A few "a->an" and "an->a". "Shows, if the supplied default SOCKS5 proxy" -> "Shows if the supplied default SOCKS5 proxy". Change made on 3 occurrences. "without fully understanding the ramification of a command" -> "without fully understanding the ramifications of a command". Removed duplicate words such as "the the".
2017-06-20[qa] util: Check return code after closing bitcoind procMarcoFalke
This has no effect on the outcome of test cases, but prints shorter and less confusing tracebacks on fails. I.e. does not print an obvious "ConnectionRefusedError" when shutting down the nodes due to an invalid return code.
2017-06-18Check for rpcuser/rpcpassword first then for cookieAndrew Chow
Better to check that rpcuser and rpcpassword exist then to check for the cookie in the test framework. Name an argument for consistency in p2p-segwit.py
2017-06-08Add test for rpcuser/rpcpasswordAndrew Chow
2017-06-08Replace cookie auth in testsAndrew Chow
Since rpcuser and rpcpassword are now deprecated, replace them with cookie auth. Fix test failures with cookie auth
2017-06-02Merge #10359: [tests] functional tests should call BitcoinTestFramework ↵MarcoFalke
start/stop node methods 53f6775 fixup: fix nits (John Newbery) a433d8a [tests] Update start/stop node functions to be private module functions (John Newbery) d8c218f [tests] Functional tests call self.start_node(s) and self.stop_node(s) (John Newbery) Tree-SHA512: 9cc01584a5e57686b7e7cb1c4c5186ad8cc7eb650d6d4f27b06bdb5e249a10966705814bdfb22d9ff2d5d3326911e489bf3d22257d751a299c0b24b7f40bffb5
2017-05-31[tests] Update start/stop node functions to be private module functionsJohn Newbery
This commit marks the start/stop functions in util.py as private module functions. A future PR will remove these entirely and move the functionality directly into the BitcoinTestFramework class, but setting them as private in this PR will prevent anyone from accidentally calling them before that future PR is merged.
2017-05-22[tests] skipped tests should clean up after themselvesJohn Newbery
2017-05-18[tests] improve tmpdir structureJohn Newbery
2017-05-09[tests] Make wait_until timeout 60 seconds by defaultJohn Newbery
2017-05-08Merge #10352: test: Add elapsed time to RPC tracingMarcoFalke
20187e4 test: Add elapsed time to RPC tracing (Wladimir J. van der Laan) Tree-SHA512: f271acedd14020cf911711577f6dd940850fa84d2577618af06a2247c940fcc5b339a86c1c7a179899c556d217a6c967c785fb311bba43a9b6073cbe470b6737
generated by cgit v1.2.3 (git 2.26.2) at 2024-08-24 09:13:30 +0000