aboutsummaryrefslogtreecommitdiff
path: root/contrib/linearize
AgeCommit message (Collapse)Author
2020-04-16scripted-diff: Bump copyright headersMarcoFalke
-BEGIN VERIFY SCRIPT- ./contrib/devtools/copyright_header.py update ./ -END VERIFY SCRIPT-
2020-02-05Merge #17336: scripts: search for first block file for linearize-data with ↵fanquake
some block files pruned 317fb96de9c6257972f1213b4ef2c3fe87dde99f Add search for first blk file with pruned node (Rjected) Pull request description: <!-- *** Please remove the following help text before submitting: *** Pull requests without a rationale and clear improvement may be closed immediately. --> <!-- Please provide clear motivation for your patch and explain how it improves Bitcoin Core user experience or Bitcoin Core developer experience significantly: * Any test improvements or new tests that improve coverage are always welcome. * All other changes should have accompanying unit tests (see `src/test/`) or functional tests (see `test/`). Contributors should note which tests cover modified code. If no tests exist for a region of modified code, new tests should accompany the change. * Bug fixes are most welcome when they come with steps to reproduce or an explanation of the potential issue as well as reasoning for the way the bug was fixed. * Features are welcome, but might be rejected due to design or scope issues. If a feature is based on a lot of dependencies, contributors should first consider building the system outside of Bitcoin Core, if possible. * Refactoring changes are only accepted if they are required for a feature or bug fix or otherwise improve developer experience significantly. For example, most "code style" refactoring changes require a thorough explanation why they are useful, what downsides they have and why they *significantly* improve developer experience or avoid serious programming bugs. Note that code style is often a subjective matter. Unless they are explicitly mentioned to be preferred in the [developer notes](/doc/developer-notes.md), stylistic code changes are usually rejected. --> When bitcoind is running in pruned mode, producing a hashlist with `./linearize-hashes.py linearize.cfg > hashlist.txt` and then executing `linearize-data.py linearize.cfg` will produce: ``` Read 313001 hashes Input file /home/dan/.bitcoin/blocks/blk00000.dat Premature end of block data ``` This happens because `linearize-data` starts by attempting to process `blk00000.dat` regardless of whether or not `blk00000.dat` actually exists - this may not be the case if working with a pruned node. This PR adds a function which finds the first block file that does exist, and calls that function when the `BlockDataCopier` is initialized. This is a refactor of #16431. <!-- Bitcoin Core has a thorough review process and even the most trivial change needs to pass a lot of eyes and requires non-zero or even substantial time effort to review. There is a huge lack of active reviewers on the project, so patches often sit for a long time. --> ACKs for top commit: darosior: ACK 317fb96de9c6257972f1213b4ef2c3fe87dde99f laanwj: Code review ACK 317fb96de9c6257972f1213b4ef2c3fe87dde99f theStack: Code review ACK https://github.com/bitcoin/bitcoin/pull/17336/commits/317fb96de9c6257972f1213b4ef2c3fe87dde99f Tree-SHA512: fc8014282df6cfe7b267e64db8ce7d82b86b758c302fbfea4a3c39b62d93512f5c2e31a0de4e9c5ec18fc0268c917f011257d37b45afaef6033eec90e4aa585f
2020-01-05Merge #17393: doc: Added regtest config for linearize scriptfanquake
582e66b6e75d58033987a7b0474226cfdd724ce0 doc: Added regtest config for linearize script (Gr0kchain) Pull request description: Updated the example-linearize.cfg file to include support for the regtest chain network config which is used by the ./linearize-data.py Problem: Without the regtest magic, genesis hash and path config, the `linearize-data.py` script cannot generate a bootstrap.dat file. Example: ./linearize-data.py ./linearize.cfg Read 102 hashes Genesis block not found in hashlist Solution: Added netmagic, genesis and input example parameters to file. Resolution 1. Starting bitcoind in regtest mode 2. bitcoin-cli generatetoaddress 101 $(bitcoin-cli getnewaddress) 3. ./linearize-hashes.py ./linearize.cfg > ./hashlist.txt 4. ./linearize-data.py ./linearize.cfg ``` $ ./linearize-data.py ./linearize.cfg Read 102 hashes Input file /Users/gr0kchain/.bitcoin/regtest/blocks/blk00000.dat Output file /Users/gr0kchain/Downloads/bootstrap.dat Done (102 blocks written) ``` ACKs for top commit: fanquake: ACK 582e66b6e75d58033987a7b0474226cfdd724ce0 Tree-SHA512: 699e92e740e68e2e5190ba37538efbbe3e4d4e725ebd6af704a0cf5517683b691754f7ea097bf840845d2b53b793c63258d406e9bd37922db810cf58bed053c3
2019-12-30scripted-diff: Bump copyright of files changed in 2019MarcoFalke
-BEGIN VERIFY SCRIPT- ./contrib/devtools/copyright_header.py update ./ -END VERIFY SCRIPT-
2019-11-07doc: Added regtest config for linearize scriptGr0kchain
Updated the example-linearize.cfg file to include support for the regtest chain network config which is used by the ./linearize-data.py Problem: Without the regtest magic, genesis hash and path config, the `linearize-data.py` script cannot generate a bootstrap.dat file. Example of error: ./linearize-data.py ./linearize.cfg Read 102 hashes Genesis block not found in hashlist Solution: Added netmagic, genesis and input example parameters to file. Resolution 1. Starting bitcoind in regtest mode 2. bitcoin-cli generatetoaddress 101 $(bitcoin-cli getnewaddress) 3. ./linearize-hashes.py ./linearize.cfg > ./hashlist.txt 4. ./linearize-data.py ./linearize.cfg Example after fix: $ ./linearize-data.py ./linearize.cfg Read 102 hashes Input file /Users/gr0kchain/.bitcoin/regtest/blocks/blk00000.dat Output file /Users/gr0kchain/Downloads/bootstrap.dat Done (102 blocks written)
2019-10-27Add search for first blk file with pruned nodeRjected
2019-10-08Merge #16802: scripts: In linearize, search for next position of magic bytes ↵Wladimir J. van der Laan
rather than fail 3284e6c09a84e9557ec72723ad636053d3ef7122 scripts: search for next position of magic bytes rather than fail (Tim Akinbo) Pull request description: When using the `linearize-data.py` contrib script to export block data, there are edge cases where the script fails with an `Invalid magic: 00000000` error. This error occurs due to the presence of padding bytes that occasionally appears between consecutive blocks in the block data file. There's an ongoing conversation about this in #14986. sipa also admitted that it is a bug in #5028. Fortunately, this is not an issue in bitcoin core as it handles this type of situation gracefully and so no fix in bitcoin core is required. This PR is an improvement on how the script handles these "invalid magic bytes". Rather than failing, this patch allows the script to search for the next occurrence of the magic bytes and then starts reading the block from there. ACKs for top commit: laanwj: ACK 3284e6c09a84e9557ec72723ad636053d3ef7122 Tree-SHA512: 18067ae0b4b62e822dfc558a86439ad6acaf939b98479e38e8e4248536574643b26eb48e96ec7139375c88b42cbe7705a64deb13a3c239e16025a6aad3d69bfa
2019-09-30scripts: search for next position of magic bytes rather than failTim Akinbo
document seek method for next position of magic bytes
2019-09-03test/contrib: Fix invalid escapes in regex stringsBen Woosley
Flagged by flake8 v3.6.0, as W605, plus a few others identified incidentally, e.g. 59ffecf66cf4d08c4b431e457b083878d66a3fd6. Note that r"\n" matches to "\n" under re.match/search.
2019-03-02build: Require python 3.5MarcoFalke
2019-03-02scripted-diff: test: Use py3.5 bytes::hex() methodMarcoFalke
-BEGIN VERIFY SCRIPT- sed -i -e "s/def bytes_to_hex_str/def b_2_x/g" $(git grep -l bytes_to_hex_str) export RE_B_0="[^()]*" # match no bracket export RE_B_1="${RE_B_0}\(${RE_B_0}\)${RE_B_0}" # match exactly one () export RE_B_2="${RE_B_0}\(${RE_B_1}\)${RE_B_0}" # match wrapped (()) export RE_M="(b2x|bytes_to_hex_str)\(((${RE_B_0}|${RE_B_1}|${RE_B_2})*)\)" sed -i --regexp-extended -e "s/${RE_M}/\2.hex()/g" $(git grep -l -E '(b2x|bytes_to_hex_str)') sed -i --regexp-extended -e "/ +bytes_to_hex_str( as b2x)?,/d" $(git grep -l bytes_to_hex_str) sed -i --regexp-extended -e "s/ +bytes_to_hex_str( as b2x)?,//g" $(git grep -l bytes_to_hex_str) sed -i --regexp-extended -e "s/, bytes_to_hex_str( as b2x)?//g" $(git grep -l bytes_to_hex_str) export RE_M="(binascii\.)?hexlify\(((${RE_B_0}|${RE_B_1}|${RE_B_2})*)\).decode\(${RE_B_0}\)" sed -i --regexp-extended -e "s/${RE_M}/\2.hex()/g" $(git grep -l hexlify -- ':(exclude)share') sed -i --regexp-extended -e "/from binascii import hexlify$/d" $(git grep -l hexlify -- ':(exclude)share') sed -i --regexp-extended -e "s/(from binascii import) .*hexlify/\1 unhexlify/g" $(git grep -l hexlify -- ':(exclude)share') sed -i -e 's/ignore-names "/ignore-names "b_2_x,/g' ./test/lint/lint-python-dead-code.sh -END VERIFY SCRIPT-
2019-02-05Update linearize-hashes.pyOverlordQ
Fix class case issue.
2018-12-13Remove Python 2 import workaroundspracticalswift
2018-07-27Update copyright headers to 2018DrahtBot
2018-06-12Explicitly specify encoding when opening text files in Python codepracticalswift
2018-05-11Enable W191 and W291 flake8 checks.John Bampton
Remove trailing whitespace from Python files. Convert tabs to spaces.
2018-04-16Minor Python cleanups to make flake8 pass with the new rules enabledpracticalswift
2018-01-03Increment MIT Licence copyright header year on files modified in 2017Akira Takizawa
2017-09-06Merge #10825: net: set regtest JSON-RPC port to 18443 to avoid conflict with ↵Wladimir J. van der Laan
testnet 18332 ce3baa193 changed regtest RPCport to 18443 to avoid conflict with testnet 18332 (Ferdinando M. Ametrano) Pull request description: using the same JSON-RPC default port for both testnet and regtest prevents running both at the same time on the same machine. Since RPCport=P2Pport-1 for both mainnet and testnet, and regtest P2Pport being 18444, 18443 is proposed for regtest RPCport Documentation has been updated (or created where missing); manpages doc/man/bitcoin*.1 could include information for regtest too Tree-SHA512: d42185f7ef54dc918ece19b543c8681d08bb9c5a971394e21f2d9a1091734b091b08df69fab622c207b46f402cf9323ded5b7a33fbd0af722388930169124e7f
2017-08-28Avoid reference to undefined name: stderr does not exist, sys.stderr doespracticalswift
2017-08-28Use sys.exit(...) instead of exit(...): exit(...) should not be used in programspracticalswift
2017-08-08Fix typos.practicalswift
2017-08-04changed regtest RPCport to 18443 to avoid conflict with testnet 18332Ferdinando M. Ametrano
2017-03-29Datadir option in linearize scriptsAndrew Chow
Adds a datadir configuration option to the linearize scripts to allow the script to use the RPC cookie instead of requiring the user to set a rpcuser and rpcpassword for the rpc server.
2017-02-02Merge #9580: Fix various minor linearization script issuesWladimir J. van der Laan
b9d95bd Fix various minor linearization script issues (Douglas Roark)
2017-01-27[trivial] Fix typos in commentspracticalswift
2017-01-25Fix various minor linearization script issuesDouglas Roark
- The last-timestamp-encountered variable wasn’t being used properly. Rewrite code to properly allow for new blockchain files to be written when split by month. - Properly set a blockchain file’s access and modify times. - Add a “debug output” option to quiet certain output that might not always be desirable. - Update the README.
2017-01-13Remove unused Python importspracticalswift
2017-01-05Make linearize scripts Python 3-compatible.Doug
2017-01-05Allow linearization scripts to support hash byte reversalDoug
Currently, the linearization scripts require input hashes to be in one endian form. Add support for byte reversal.
2016-12-31Increment MIT Licence copyright header year on files modified in 2016isle2983
Edited via: $ contrib/devtools/copyright_header.py update .
2016-06-26Use portable #! in python scripts (/usr/bin/env)Matthew King
2015-07-10Update Linearize tool to support Windows pathsPaul Georgiou
2015-05-20contrib/linearize: Support linearization of testnet blocksJeff Garzik
2014-12-19Added "Core" to copyright headerssandakersmann
Github-Pull: #5494 Rebased-From: 15de949bb9277e442302bdd8dee299a8d6deee60
2014-12-16Remove references to X11 licenceMichael Ford
2014-12-08Remove genesis block hash from errorMicha
2014-12-07Make several small string fixes in linearize-data.pyMicha
- Add a space after the fixed string prepended to file names when input or output file changes - Clarify the error message when the genesis block is not found in the hash list (...why do we have this at all?)
2014-10-08Merge pull request #5051Wladimir J. van der Laan
aedc74d contrib: make linearize-data.py cope with out-of-order blocks (Wladimir J. van der Laan)
2014-10-06contrib: make linearize-data.py cope with out-of-order blocksWladimir J. van der Laan
Make it possible to read blocks in any order. This will be required after headers-first (#4468), so should be merged before that. - Read block header. For expected blocks, continue, else skip. - For in-order blocks: copy block contents directly. Write prior out-of-order blocks if this connects a consecutive span. - For out-of-order blocks, store extents of block data for later retrieval. Cache out-of-order blocks in memory up to 100MB (configurable).
2014-10-06contrib: use batched JSON-RPC in linarize-hashesWladimir J. van der Laan
Batch up to 10000 requests for a ~30x speedup.
2014-09-11Catch error when done reading filesSuhas Daftuar
2014-09-04contrib/linearize: Add feature to set file's timestamp based on block header ↵Jeff Garzik
time.
2014-08-24contrib/linearize: split block files based on year-month, not just yearJeff Garzik
2014-08-24contrib/linearize: Guarantee that output is generated in-orderJeff Garzik
This was typically ensured implicitly by virtue of normal bitcoind operation. Adding an explicit check provides a stronger guarantee, and it is cheap to add.
2014-08-23contrib/linearize: split output files based on new-timestamp-year or ↵Jeff Garzik
max-file-size
2014-08-23Update linearize scripts.Jeff Garzik
Break into two steps: * Generate hash list * Build data file(s) from local bitcoind blocks/ directory. This supports building one large bootstrap.dat, or multiple smaller blocks/blkNNNNN.dat files.
2014-02-10linearize.py: Harmonize rpcpass to rpcpasswordSubo1978
linearize.py: harmonize rpcpass to rpcpassword according settings in bitcoin.conf
2014-01-18Update linearize.py defaults to block 279,000Micha
2014-01-12Tweak linearize.py to give more flexibilityMicha
Add the ability to start at a non-zero height, and allow for appending to an existing file.
generated by cgit v1.2.3 (git 2.26.2) at 2024-11-01 22:34:45 +0000