aboutsummaryrefslogtreecommitdiff
path: root/test/functional/interface_zmq.py
AgeCommit message (Collapse)Author
2024-04-16test: cover unix sockets in zmq interfaceMatthew Zipkin
2024-03-12Merge bitcoin/bitcoin#26415: rpc,rest,zmq: faster getblock, NotifyBlock and ↵Ava Chow
rest_block by reading raw block e710cefd5701cd33d1e55034b3e37cea78582733 rest: read raw block in rest_block and deserialize for json (Andrew Toth) 95ce0783a6dab325038a64d6c529c9e7816e3072 rpc: read raw block in getblock and deserialize for verbosity > 0 (Andrew Toth) 0865ab8712429761bc69f09d93760f8c6081c99c test: check more details on zmq raw block response (Andrew Toth) 38265cc14e7d646bf27882329d374d42167eb49f zmq: read raw block with ReadRawBlockFromDisk (Andrew Toth) da338aada7943c392013c36c542af621fbc6edd1 blockstorage: check nPos in ReadRawBlockFromDisk before seeking back (Andrew Toth) Pull request description: For the `getblock` endpoint with `verbosity=0`, the `rest_block` REST endpoint for `bin` and `hex`, and zmq `NotifyBlock` we don't have to deserialize the block since we're just sending the raw data. This PR uses `ReadRawBlockFromDisk` instead of `ReadBlockFromDisk` to serve these requests, and only deserializes for `verbosity > 0` and `json` REST requests. See benchmarks in https://github.com/bitcoin/bitcoin/pull/26684. Benchmarked using ApacheBench. Requesting block 750,000 in binary 10k times on a single core (set `-rest=1` in config): `ab -n 10000 -c 1 "http://127.0.0.1:8332/rest/block/0000000000000000000592a974b1b9f087cb77628bb4a097d5c2c11b3476a58e.bin"` On master, mean time 15ms. On this branch, mean time 1ms. For RPC ``` echo '{"jsonrpc": "1.0", "id": "curltest", "method": "getblock", "params": ["0000000000000000000592a974b1b9f087cb77628bb4a097d5c2c11b3476a58e", 0]}' > /tmp/data.json ab -p /tmp/data.json -n 1000 -c 1 -A user:password "http://127.0.0.1:8332/" ``` On master, mean time 32ms On this branch, mean time 13ms ACKs for top commit: achow101: re-ACK e710cefd5701cd33d1e55034b3e37cea78582733 Tree-SHA512: 4cea13c7b563b2139d041b1fdcfdb793c8cc688654ae08db07e7ee6b875c5e582b8185db3ae603abbfb06d2164724f29205774620b48c493726b991999af289e
2024-03-12test: check more details on zmq raw block responseAndrew Toth
2024-02-28test: add option to speed up tx relay/mempool syncbrunoerg
when `self.noban_tx_relay=True`, the following flag `-whitelist=noban,in,out@127.0.0.1`is added to `extra_args` to speed up tx relay/mempool sync.
2023-04-01test: refactor: replace unnecessary `BytesIO` usesSebastian Falbesoner
Rather than needing to create intermediate stream variables, we can use helper functions like `tx_from_hex` instead or access the result directly, leading both to increased readability and less code.
2023-01-16test: add `rescan_utxos` in MiniWallet's initializationkouloumos
this simplifies usage when MiniWallet is used with a pre-mined chain.
2023-01-03tests: Use unique port for ZMQ testsAndrew Chow
The ZMQ interface tests should use unique ports as we do for the p2p and rpc ports so that multiple instances of the test can be run at the same time.
2022-12-24scripted-diff: Bump copyright headersHennadii Stepanov
-BEGIN VERIFY SCRIPT- ./contrib/devtools/copyright_header.py update ./ -END VERIFY SCRIPT- Commits of previous years: - 2021: f47dda2c58b5d8d623e0e7ff4e74bc352dfa83d7 - 2020: fa0074e2d82928016a43ca408717154a1c70a4db - 2019: aaaaad6ac95b402fe18d019d67897ced6b316ee0
2022-03-24[move only] remove `is_wallet_compiled` checksjosibake
2022-03-24test: use MiniWallet in `interfaces_zmq`josibake
make interfaces_zmg run deterministically. this test is for the zmg notifications, so it doesn't need the wallet compiled to run
2021-11-17test: refactor: take use of `create_block` txlist parameterSebastian Falbesoner
Passing a list of transactions `txlist` to `create_block` appends them to the block, hence we don't need to do that manually anymore. The merkle root calculation can also be removed, since this is done in the end of the helper.
2021-11-10scripted-diff: Bump copyright headersMarcoFalke
The previous diff touched most files in ./test/, so bump the headers to avoid having to touch them again for a bump later. -BEGIN VERIFY SCRIPT- ./contrib/devtools/copyright_header.py update ./test/ -END VERIFY SCRIPT-
2021-11-10scripted-diff: Remove redundant sync_all and sync_blocksMarcoFalke
The sync calls are redundant after a call to generate, because generate already syncs itself. -BEGIN VERIFY SCRIPT- perl -0777 -pi -e 's/(generate[^\n]*\)[^\n]*)(\n|\s)+self.sync_(all|blocks)\([^\)]*\)\n/\1\n/g' $(git grep -l generate ./test) -END VERIFY SCRIPT-
2021-10-29test: Implicitly sync after generate*, unless opted outMarcoFalke
2021-10-06Use self.generate not node.generate throughout testsSamuel Dobson
2021-09-09Merge bitcoin/bitcoin#22079: zmq: Add support to listen on IPv6 addressesW. J. van der Laan
e6998838e5548991274ad2bf1697d862905b8837 doc: Add IPv6 address to zmq example (nthumann) 8abe5703a9bb76bc92204a6f69775790e96208fa test: Add IPv6 test to zmq (nthumann) ded449b726e47f35798ef1c4b1e59123a0dc2b61 zmq: Enable IPv6 on listening socket (nthumann) Pull request description: This PR adds support for listening on IPv6 addresses with bitcoinds ZMQ interface, just like the RPC server. Currently, it is not possible to specify an IPv6 address, as the `ZMQ_IPV6` [socket option](http://api.zeromq.org/master:zmq-setsockopt#toc27) is not set and therefore the ZMQ initialization fails, if one does so. The absence of this option has also been noted [here](https://github.com/bitcoin/bitcoin/issues/15198#issuecomment-617378512). With this PR one can e.g. set `-zmqpubhashblock=tcp://[::1]:28333` to listen on the IPv6 loopback address. ACKs for top commit: laanwj: Code review ACK e6998838e5548991274ad2bf1697d862905b8837 theStack: Tested ACK e6998838e5548991274ad2bf1697d862905b8837 🌱 Tree-SHA512: 43c3043d8d5c79794d475926259c1be975b694db4fcc1f7750a9a28e242f0fa1b531735a63ea5777498003aa5834f6243f39742d0f3941f2f37593d0c7890700
2021-09-02scripted-diff: Use generate* from TestFrameworkMarcoFalke
-BEGIN VERIFY SCRIPT- sed --regexp-extended -i \ 's/((self\.)?(nodes\[[^]]+\]|[a-z_]*(wallet|node)[0-9a-z_]*))\.(generate(|toaddress|block|todescriptor)(\(|, ))/self.\5\1, /g' \ $(git grep -l generate ./test | grep -v 'test_framework/' | grep -v 'feature_rbf') -END VERIFY SCRIPT-
2021-08-19Use generate* from TestFrameworkMarcoFalke
The changes in feature_rbf can be reviewed with --color-moved=dimmed-zebra --color-moved-ws=ignore-all-space
2021-08-18test: use f-strings in interface_*.py testsfanquake
2021-06-21test: remove `ToHex` helper, use .serialize().hex() insteadSebastian Falbesoner
2021-06-21test: introduce `tx_from_hex` helper for tx deserializationSebastian Falbesoner
`FromHex` is mostly used for transactions, so we introduce a shortcut `tx_from_hex` for `FromHex(CTransaction, hex_str)`.
2021-06-20test: Add IPv6 test to zmqnthumann
2021-03-01zmq test: fix sync-up by matching notification to generated blockSebastian Falbesoner
It turned out that the "sync up" procedure of repeatedly generating a block and waiting for a notification once with timeout is too naive in its current form, as the following scenario could happen: - generate block A - receive notification, timeout happens -> repeat procedure - generate block B - node publishes block A notification - receive notification, we receive the one caused by block A -> sync-up procedure is completed - node publishes block B - the actual test starts - on the first notification reception, one caused by block B is received, rather than the one actually caused by test code, leading to failure This change ensures that after each test block generation, we wait for the notification that is actually caused by that block and ignore others from possibly earlier blocks. Co-authored-by: Jon Atack <jon@atack.com>
2021-02-09zmq test: speedup test by whitelisting peers (immediate tx relay)Sebastian Falbesoner
Speeds up the zmq test roughly by a factor of 2x (~20 sec. instead of ~40 sec.) and also avoids timeouts on the synchronization methods (sync_mempool() / sync_blocks()) that happened with a slight chance. This is due to the fact that there is no upper bound on the trickle relay time, so even the default of 60s is sometimes too low. Fixed by enabling immediate tx relay on node1.
2021-02-09zmq test: fix flakiness by using more robust sync methodSebastian Falbesoner
After connecting the subscriber sockets to the node, there is no guarantee that the node's zmq publisher interfaces are ready yet, which means that potentially the first expected notification messages could get lost and the test fails. Currently this is handled by just waiting for a short period of time (200ms), which works most of the time but is still problematic, as in some rare cases the setup time takes much longer, even in the range of multiple seconds. The solution in this commit approaches the problem by using a more robust method of syncing up, originally proposed by instagibbs: 1. Generate a block on the node 2. Try to receive a notification on all subscribers 3. If all subscribers get a message within the timeout (1 second), we are done, otherwise repeat starting from step 1
2021-02-09zmq test: accept arbitrary sequence start number in ZMQSubscriberSebastian Falbesoner
The ZMQSubscriber reception methods currently assert that the first received publisher message has a sequence number of zero. In order to fix the current test flakiness via "syncing up" to nodes in the setup phase, we have to cope with the situation that messages get lost and the first actual received message has a sequence number larger than zero.
2021-02-09zmq test: dedup message reception handling in ZMQSubscriberSebastian Falbesoner
2021-01-27Setup wallets for interface_zmq.pyAndrew Chow
2021-01-17zmq test: deduplicate test setup code (node restart, topics subscription)Sebastian Falbesoner
2020-12-31scripted-diff: Bump copyright headersMarcoFalke
-BEGIN VERIFY SCRIPT- ./contrib/devtools/copyright_header.py update ./ -END VERIFY SCRIPT-
2020-10-20scripted-diff: test: Replace uses of (dis)?connect_nodes globalPrayank
-BEGIN VERIFY SCRIPT- # max-depth=0 excludes test/functional/test_framework/... FILES=$(git grep -l --max-depth 0 "connect_nodes" test/functional) # Replace (dis)?connect_nodes(self.nodes[a], b) with self.(dis)?connect_nodes(a, b) sed -i 's/\b\(dis\)\?connect_nodes(self\.nodes\[\(.*\)\]/self.\1connect_nodes(\2/g' $FILES # Remove imports in the middle of a line sed -i 's/\(dis\)\?connect_nodes, //g' $FILES sed -i 's/, \(dis\)\?connect_nodes//g' $FILES # Remove imports on a line by themselves sed -i '/^\s*\(dis\)\?connect_nodes,\?$/d' $FILES sed -i '/^from test_framework\.util import connect_nodes$/d' $FILES -END VERIFY SCRIPT- Co-authored-by: Elliott Jin <elliott.jin@gmail.com>
2020-10-01test: Add zmq test to support multiple interfacesnthumann
2020-09-22Add functional tests for zmq sequence topic and mempool sequence logicGregory Sanders
2020-09-22zmq test: Actually make reorg occurGregory Sanders
2020-07-14Have zmq reorg test cover mempool txnsGregory Sanders
2020-07-13Add zmq test for transaction pub during reorgGregory Sanders
2020-07-13Add test case for mempool->block zmq notificationGregory Sanders
2020-07-13Make ordering of zmq consumption irrelevant to functional testGregory Sanders
2019-11-11zmq: Fix due to invalid argument and multiple notifiersJoão Barbosa
2019-08-28qa: Relax so that the subscriber is ready before publishing zmq messagesJoão Barbosa
2019-08-19qa: Test ZMQ notification after chain reorgJoão Barbosa
2019-08-19qa: Refactor ZMQ testJoão Barbosa
2019-08-13Moved and renamed hash256 from util.py to zmq_interface.pyElichai Turkel
2019-03-02scripted-diff: Update copyright in ./testMarcoFalke
-BEGIN VERIFY SCRIPT- ./contrib/devtools/copyright_header.py update ./test/ -END VERIFY SCRIPT-
2019-03-02scripted-diff: test: Use py3.5 bytes::hex() methodMarcoFalke
-BEGIN VERIFY SCRIPT- sed -i -e "s/def bytes_to_hex_str/def b_2_x/g" $(git grep -l bytes_to_hex_str) export RE_B_0="[^()]*" # match no bracket export RE_B_1="${RE_B_0}\(${RE_B_0}\)${RE_B_0}" # match exactly one () export RE_B_2="${RE_B_0}\(${RE_B_1}\)${RE_B_0}" # match wrapped (()) export RE_M="(b2x|bytes_to_hex_str)\(((${RE_B_0}|${RE_B_1}|${RE_B_2})*)\)" sed -i --regexp-extended -e "s/${RE_M}/\2.hex()/g" $(git grep -l -E '(b2x|bytes_to_hex_str)') sed -i --regexp-extended -e "/ +bytes_to_hex_str( as b2x)?,/d" $(git grep -l bytes_to_hex_str) sed -i --regexp-extended -e "s/ +bytes_to_hex_str( as b2x)?,//g" $(git grep -l bytes_to_hex_str) sed -i --regexp-extended -e "s/, bytes_to_hex_str( as b2x)?//g" $(git grep -l bytes_to_hex_str) export RE_M="(binascii\.)?hexlify\(((${RE_B_0}|${RE_B_1}|${RE_B_2})*)\).decode\(${RE_B_0}\)" sed -i --regexp-extended -e "s/${RE_M}/\2.hex()/g" $(git grep -l hexlify -- ':(exclude)share') sed -i --regexp-extended -e "/from binascii import hexlify$/d" $(git grep -l hexlify -- ':(exclude)share') sed -i --regexp-extended -e "s/(from binascii import) .*hexlify/\1 unhexlify/g" $(git grep -l hexlify -- ':(exclude)share') sed -i -e 's/ignore-names "/ignore-names "b_2_x,/g' ./test/lint/lint-python-dead-code.sh -END VERIFY SCRIPT-
2018-11-05Merge #14060: ZMQ: add options to configure outbound message high water ↵Wladimir J. van der Laan
mark, aka SNDHWM a4edb168b635b6f5c36324e44961cd42cf9bbbaa ZMQ: add options to configure outbound message high water mark, aka SNDHWM (mruddy) Pull request description: ZMQ: add options to configure outbound message high water mark, aka SNDHWM This is my attempt at https://github.com/bitcoin/bitcoin/pull/13315 Tree-SHA512: a4cc3bcf179776899261a97c8c4f31f35d1d8950fd71a09a79c5c064879b38e600b26824c89c4091d941502ed5b0255390882f7d44baf9e6dc49d685a86e8edb
2018-11-01[tests] Move deterministic address import to setup_nodesJohn Newbery
This requires a small changes to a few tests, but means that deterministic addresses will always be imported (unless setup_nodes behaviour is explicitly overridden).
2018-10-19ZMQ: add options to configure outbound message high water mark, aka SNDHWMmruddy
2018-10-08Merge #14324: qa: Run more tests with wallet disabledMarcoFalke
faa4043c66 qa: Run more tests with wallet disabled (MarcoFalke) Pull request description: Instead of skipping the whole test, only skip the wallet specific section of a test if the wallet is not compiled in. This is mostly an indentation change, so can be reviewed with `--ignore-all-space`. Tree-SHA512: 5941a8b6b00dca5cf9438c5f6f010ba812115188a69e427d7ade4c1ab8cfe7a57c73daf52c66235dbb24b1cd9ab7c7a17c49bc23d931e041b605d79116a71f66
2018-10-07[tests] Remove rpc_zmq.pyJohn Newbery
rpc_zmq.py is racy and fails intermittently. Remove that test file and move the getzmqnotifications RPC test into interface_zmq.py
generated by cgit v1.2.3 (git 2.26.2) at 2024-11-06 16:26:49 +0000