Merge bitcoin/bitcoin#23289: test: add stress tests for initialization

d9803f7a0a33688f7429cf10384244f4770851ca test: add stress tests for initialization (James O'Beirne)
23f85616a8d9c9a1b054e492eca4d199028f34dc test: add node.chain_path and node.debug_log_path (James O'Beirne)

Pull request description:

  In the course of coming up with a test plan for #23280, I thought it would be neat to include a Python snippet showing how I tested the initialization process. I quickly realized I was reinventing the functional test framework... so here's a new test.

  This change bangs init around like the Fonz hitting a jukebox. It adds some interesting (read: lazy and random) coverage for the initialization process by
  - interrupting init with SIGTERM after certain log statements,
  - interrupting init at random points, and
  - starting init with some essential data missing (block files, block indices, etc.) to test init error paths.

  As far as I can tell, some of these code paths are uncovered otherwise (namely the startup errors).

  ---

  Incidentally, I think I may have uncovered some kind of a bug or race condition with indexing initialization based on an intermittent failure in this testcase. This test sometimes fails after shutting down immediately after `loadblk` thread start:
  ```
  2021-10-15T21:14:51.295000Z TestFramework (INFO): Starting node and will exit after line 'loadblk thread start'
    36   │ 2021-10-15T21:14:51.296000Z TestFramework.node0 (DEBUG): bitcoind started, waiting for RPC to come up
    37   │ 2021-10-15T21:14:51.493000Z TestFramework (INFO): terminating node after 110 log lines seen
    38   │ 2021-10-15T21:14:51.625000Z TestFramework (INFO): Starting node and will exit after line 'txindex thread start'
    39   │ 2021-10-15T21:14:51.625000Z TestFramework.node0 (DEBUG): bitcoind started, waiting for RPC to come up
    ------> [[ FAILURE HERE ]] 2021-10-15T21:15:21.626000Z TestFramework (WARNING): missed line {bail_line}; bailing now after {num_lines} lines
  ```
  and then fails to start up afterwards. Combined logs showing `Error: txindex best block of the index goes beyond pruned data`, when the node under test is not pruned:

  ```
    node0 2021-10-15T21:16:51.848439Z [shutoff] [validationinterface.cpp:244] [ChainStateFlushed] Enqueuing ChainStateFlushed: block hash=1014bc4ff4917602ae53d10e9dfe230af4b7d52a6cdaa8a47798b9c288180907
     node0 2021-10-15T21:16:51.848954Z [shutoff] [init.cpp:302] [Shutdown] Shutdown: done
     test  2021-10-15T21:16:51.882000Z TestFramework (ERROR): Unexpected exception caught during testing
       Traceback (most recent call last):
         File "/home/james/src/bitcoin/test/functional/test_framework/test_framework.py", line 132, in main
   self.run_test()
         File "/home/james/src/bitcoin/./test/functional/stress_init.py", line 87, in run_test
   check_clean_start()
         File "/home/james/src/bitcoin/./test/functional/stress_init.py", line 60, in check_clean_start
   node.wait_for_rpc_connection()
         File "/home/james/src/bitcoin/test/functional/test_framework/test_node.py", line 224, in wait_for_rpc_connection
   raise FailedToStartError(self._node_msg(
       test_framework.test_node.FailedToStartError: [node 0] bitcoind exited with status 1 during initialization
     test  2021-10-15T21:16:51.882000Z TestFramework (DEBUG): Closing down network thread
     test  2021-10-15T21:16:51.933000Z TestFramework (INFO): Stopping nodes
     test  2021-10-15T21:16:51.933000Z TestFramework.node0 (DEBUG): Stopping node

     node0 stderr Error: txindex best block of the index goes beyond pruned data. Please disable the index or reindex (which will download the whole blockchain again)
     node0 stderr Error: txindex best block of the index goes beyond pruned data. Please disable the index or reindex (which will download the whole blockchain again)
     node0 stderr Error: txindex best block of the index goes beyond pruned data. Please disable the index or reindex (which will download the whole blockchain again)
     node0 stderr Error: txindex best block of the index goes beyond pruned data. Please disable the index or reindex (which will download the whole blockchain again)
     node0 stderr Error: txindex best block of the index goes beyond pruned data. Please disable the index or reindex (which will download the whole blockchain again)
  ```

ACKs for top commit:
  laanwj:
    Code review ACK d9803f7a0a33688f7429cf10384244f4770851ca

Tree-SHA512: 4d80dc399daf199a6222e81e47d12d830dc7af07355eddbb7f52479a676a645b8d3d45093ff54a9295f01a163b2f4fe0e038e83fc269969e03d4cfda69eaf111

1 files changed, 11 insertions, 3 deletions

diff --git a/test/functional/test_framework/test_node.py b/test/functional/test_framework/test_node.py
index 269f2442a9..b3279666b2 100755
--- a/test/functional/test_framework/test_node.py
+++ b/test/functional/test_framework/test_node.py
@@ -20,6 +20,7 @@ import urllib.parse
 import collections
 import shlex
 import sys
+from pathlib import Path
 
 from .authproxy import JSONRPCException
 from .descriptors import descsum_create
@@ -380,13 +381,20 @@ class TestNode():
     def wait_until_stopped(self, timeout=BITCOIND_PROC_WAIT_TIMEOUT):
         wait_until_helper(self.is_node_stopped, timeout=timeout, timeout_factor=self.timeout_factor)
 
+    @property
+    def chain_path(self) -> Path:
+        return Path(self.datadir) / self.chain
+
+    @property
+    def debug_log_path(self) -> Path:
+        return self.chain_path / 'debug.log'
+
     @contextlib.contextmanager
     def assert_debug_log(self, expected_msgs, unexpected_msgs=None, timeout=2):
         if unexpected_msgs is None:
             unexpected_msgs = []
         time_end = time.time() + timeout * self.timeout_factor
-        debug_log = os.path.join(self.datadir, self.chain, 'debug.log')
-        with open(debug_log, encoding='utf-8') as dl:
+        with open(self.debug_log_path, encoding='utf-8') as dl:
             dl.seek(0, 2)
             prev_size = dl.tell()
 
@@ -394,7 +402,7 @@ class TestNode():
 
         while True:
             found = True
-            with open(debug_log, encoding='utf-8') as dl:
+            with open(self.debug_log_path, encoding='utf-8') as dl:
                 dl.seek(prev_size)
                 log = dl.read()
             print_log = " - " + "\n - ".join(log.splitlines())