aboutsummaryrefslogtreecommitdiff
path: root/test/README.md
diff options
context:
space:
mode:
Diffstat (limited to 'test/README.md')
-rw-r--r--test/README.md174
1 files changed, 131 insertions, 43 deletions
diff --git a/test/README.md b/test/README.md
index 4dd512638d..868eb667ae 100644
--- a/test/README.md
+++ b/test/README.md
@@ -15,84 +15,172 @@ The util tests are run as part of `make check` target. The functional
tests are run by the travis continuous build process whenever a pull
request is opened. Both sets of tests can also be run locally.
-Functional Test dependencies
-============================
+# Running tests locally
+
+Build for your system first. Be sure to enable wallet, utils and daemon when you configure. Tests will not run otherwise.
+
+### Functional tests
+
+#### Dependencies
+
The ZMQ functional test requires a python ZMQ library. To install it:
- on Unix, run `sudo apt-get install python3-zmq`
- on mac OS, run `pip3 install pyzmq`
-Running tests locally
-=====================
+#### Running the tests
-Build for your system first. Be sure to enable wallet, utils and daemon when you configure. Tests will not run otherwise.
+Individual tests can be run by directly calling the test script, eg:
-Functional tests
-----------------
+```
+test/functional/replace-by-fee.py
+```
-You can run any single test by calling
+or can be run through the test_runner harness, eg:
- test/functional/test_runner.py <testname>
+```
+test/functional/test_runner.py replace-by-fee.py
+```
-Or you can run any combination (incl. duplicates) of tests by calling
+You can run any combination (incl. duplicates) of tests by calling:
- test/functional/test_runner.py <testname1> <testname2> <testname3> ...
+```
+test/functional/test_runner.py <testname1> <testname2> <testname3> ...
+```
-Run the regression test suite with
+Run the regression test suite with:
- test/functional/test_runner.py
+```
+test/functional/test_runner.py
+```
Run all possible tests with
- test/functional/test_runner.py --extended
+```
+test/functional/test_runner.py --extended
+```
-By default, tests will be run in parallel. To specify how many jobs to run,
-append `--jobs=n` (default n=4).
+By default, up to 4 tests will be run in parallel by test_runner. To specify
+how many jobs to run, append `--jobs=n`
-If you want to create a basic coverage report for the RPC test suite, append `--coverage`.
+The individual tests and the test_runner harness have many command-line
+options. Run `test_runner.py -h` to see them all.
-Possible options, which apply to each individual test run:
+#### Troubleshooting and debugging test failures
-```
- -h, --help show this help message and exit
- --nocleanup Leave bitcoinds and test.* datadir on exit or error
- --noshutdown Don't stop bitcoinds after the test execution
- --srcdir=SRCDIR Source directory containing bitcoind/bitcoin-cli
- (default: ../../src)
- --tmpdir=TMPDIR Root directory for datadirs
- --tracerpc Print out all RPC calls as they are made
- --coveragedir=COVERAGEDIR
- Write tested RPC commands into this directory
+##### Resource contention
+
+The P2P and RPC ports used by the bitcoind nodes-under-test are chosen to make
+conflicts with other processes unlikely. However, if there is another bitcoind
+process running on the system (perhaps from a previous test which hasn't successfully
+killed all its bitcoind nodes), then there may be a port conflict which will
+cause the test to fail. It is recommended that you run the tests on a system
+where no other bitcoind processes are running.
+
+On linux, the test_framework will warn if there is another
+bitcoind process running when the tests are started.
+
+If there are zombie bitcoind processes after test failure, you can kill them
+by running the following commands. **Note that these commands will kill all
+bitcoind processes running on the system, so should not be used if any non-test
+bitcoind processes are being run.**
+
+```bash
+killall bitcoind
```
-If you set the environment variable `PYTHON_DEBUG=1` you will get some debug
-output (example: `PYTHON_DEBUG=1 test/functional/test_runner.py wallet`).
+or
+
+```bash
+pkill -9 bitcoind
+```
-A 200-block -regtest blockchain and wallets for four nodes
-is created the first time a regression test is run and
-is stored in the cache/ directory. Each node has 25 mature
-blocks (25*50=1250 BTC) in its wallet.
-After the first run, the cache/ blockchain and wallets are
-copied into a temporary directory and used as the initial
-test state.
+##### Data directory cache
-If you get into a bad state, you should be able
-to recover with:
+A pre-mined blockchain with 200 blocks is generated the first time a
+functional test is run and is stored in test/cache. This speeds up
+test startup times since new blockchains don't need to be generated for
+each test. However, the cache may get into a bad state, in which case
+tests will fail. If this happens, remove the cache directory (and make
+sure bitcoind processes are stopped as above):
```bash
rm -rf cache
killall bitcoind
```
-Util tests
-----------
+##### Test logging
+
+The tests contain logging at different levels (debug, info, warning, etc). By
+default:
+
+- when run through the test_runner harness, *all* logs are written to
+ `test_framework.log` and no logs are output to the console.
+- when run directly, *all* logs are written to `test_framework.log` and INFO
+ level and above are output to the console.
+- when run on Travis, no logs are output to the console. However, if a test
+ fails, the `test_framework.log` and bitcoind `debug.log`s will all be dumped
+ to the console to help troubleshooting.
+
+To change the level of logs output to the console, use the `-l` command line
+argument.
+
+`test_framework.log` and bitcoind `debug.log`s can be combined into a single
+aggregate log by running the `combine_logs.py` script. The output can be plain
+text, colorized text or html. For example:
+
+```
+combine_logs.py -c <test data directory> | less -r
+```
+
+will pipe the colorized logs from the test into less.
+
+Use `--tracerpc` to trace out all the RPC calls and responses to the console. For
+some tests (eg any that use `submitblock` to submit a full block over RPC),
+this can result in a lot of screen output.
+
+By default, the test data directory will be deleted after a successful run.
+Use `--nocleanup` to leave the test data directory intact. The test data
+directory is never deleted after a failed test.
+
+##### Attaching a debugger
+
+A python debugger can be attached to tests at any point. Just add the line:
+
+```py
+import pdb; pdb.set_trace()
+```
+
+anywhere in the test. You will then be able to inspect variables, as well as
+call methods that interact with the bitcoind nodes-under-test.
+
+If further introspection of the bitcoind instances themselves becomes
+necessary, this can be accomplished by first setting a pdb breakpoint
+at an appropriate location, running the test to that point, then using
+`gdb` to attach to the process and debug.
+
+For instance, to attach to `self.node[1]` during a run:
+
+```bash
+2017-06-27 14:13:56.686000 TestFramework (INFO): Initializing test directory /tmp/user/1000/testo9vsdjo3
+```
+
+use the directory path to get the pid from the pid file:
+
+```bash
+cat /tmp/user/1000/testo9vsdjo3/node1/regtest/bitcoind.pid
+gdb /home/example/bitcoind <pid>
+```
+
+Note: gdb attach step may require `sudo`
+
+### Util tests
Util tests can be run locally by running `test/util/bitcoin-util-test.py`.
Use the `-v` option for verbose output.
-Writing functional tests
-========================
+# Writing functional tests
You are encouraged to write functional tests for new or existing features.
Further information about the functional test framework and individual