aboutsummaryrefslogtreecommitdiff
path: root/contrib/linearize/linearize-data.py
AgeCommit message (Collapse)Author
2022-04-27test, contrib, refactor: use `with` when opening a filebrunoerg
2022-02-17contrib: refactor: replace `hex_switchEndian` in linearize scriptsSebastian Falbesoner
Switching the endianness of a hex string `str` can simply be achieved by `bytes.fromhex(str)[::-1].hex()`, i.e. we can use that and remove those helper methods.
2022-02-17contrib: refactor: simplify block header string routine in linearize-data.pySebastian Falbesoner
The string representation of a block header hash is simply the hexlified byte-reversed double SHA256 hash of its serialization.
2021-12-30scripted-diff: Bump copyright headersHennadii Stepanov
-BEGIN VERIFY SCRIPT- ./contrib/devtools/copyright_header.py update ./ -END VERIFY SCRIPT- Commits of previous years: * 2020: fa0074e2d82928016a43ca408717154a1c70a4db * 2019: aaaaad6ac95b402fe18d019d67897ced6b316ee0
2021-08-16refactor: replace remaining binascii method calls Zero-1729
2020-04-16scripted-diff: Bump copyright headersMarcoFalke
-BEGIN VERIFY SCRIPT- ./contrib/devtools/copyright_header.py update ./ -END VERIFY SCRIPT-
2020-02-05Merge #17336: scripts: search for first block file for linearize-data with ↵fanquake
some block files pruned 317fb96de9c6257972f1213b4ef2c3fe87dde99f Add search for first blk file with pruned node (Rjected) Pull request description: <!-- *** Please remove the following help text before submitting: *** Pull requests without a rationale and clear improvement may be closed immediately. --> <!-- Please provide clear motivation for your patch and explain how it improves Bitcoin Core user experience or Bitcoin Core developer experience significantly: * Any test improvements or new tests that improve coverage are always welcome. * All other changes should have accompanying unit tests (see `src/test/`) or functional tests (see `test/`). Contributors should note which tests cover modified code. If no tests exist for a region of modified code, new tests should accompany the change. * Bug fixes are most welcome when they come with steps to reproduce or an explanation of the potential issue as well as reasoning for the way the bug was fixed. * Features are welcome, but might be rejected due to design or scope issues. If a feature is based on a lot of dependencies, contributors should first consider building the system outside of Bitcoin Core, if possible. * Refactoring changes are only accepted if they are required for a feature or bug fix or otherwise improve developer experience significantly. For example, most "code style" refactoring changes require a thorough explanation why they are useful, what downsides they have and why they *significantly* improve developer experience or avoid serious programming bugs. Note that code style is often a subjective matter. Unless they are explicitly mentioned to be preferred in the [developer notes](/doc/developer-notes.md), stylistic code changes are usually rejected. --> When bitcoind is running in pruned mode, producing a hashlist with `./linearize-hashes.py linearize.cfg > hashlist.txt` and then executing `linearize-data.py linearize.cfg` will produce: ``` Read 313001 hashes Input file /home/dan/.bitcoin/blocks/blk00000.dat Premature end of block data ``` This happens because `linearize-data` starts by attempting to process `blk00000.dat` regardless of whether or not `blk00000.dat` actually exists - this may not be the case if working with a pruned node. This PR adds a function which finds the first block file that does exist, and calls that function when the `BlockDataCopier` is initialized. This is a refactor of #16431. <!-- Bitcoin Core has a thorough review process and even the most trivial change needs to pass a lot of eyes and requires non-zero or even substantial time effort to review. There is a huge lack of active reviewers on the project, so patches often sit for a long time. --> ACKs for top commit: darosior: ACK 317fb96de9c6257972f1213b4ef2c3fe87dde99f laanwj: Code review ACK 317fb96de9c6257972f1213b4ef2c3fe87dde99f theStack: Code review ACK https://github.com/bitcoin/bitcoin/pull/17336/commits/317fb96de9c6257972f1213b4ef2c3fe87dde99f Tree-SHA512: fc8014282df6cfe7b267e64db8ce7d82b86b758c302fbfea4a3c39b62d93512f5c2e31a0de4e9c5ec18fc0268c917f011257d37b45afaef6033eec90e4aa585f
2019-12-30scripted-diff: Bump copyright of files changed in 2019MarcoFalke
-BEGIN VERIFY SCRIPT- ./contrib/devtools/copyright_header.py update ./ -END VERIFY SCRIPT-
2019-10-27Add search for first blk file with pruned nodeRjected
2019-10-08Merge #16802: scripts: In linearize, search for next position of magic bytes ↵Wladimir J. van der Laan
rather than fail 3284e6c09a84e9557ec72723ad636053d3ef7122 scripts: search for next position of magic bytes rather than fail (Tim Akinbo) Pull request description: When using the `linearize-data.py` contrib script to export block data, there are edge cases where the script fails with an `Invalid magic: 00000000` error. This error occurs due to the presence of padding bytes that occasionally appears between consecutive blocks in the block data file. There's an ongoing conversation about this in #14986. sipa also admitted that it is a bug in #5028. Fortunately, this is not an issue in bitcoin core as it handles this type of situation gracefully and so no fix in bitcoin core is required. This PR is an improvement on how the script handles these "invalid magic bytes". Rather than failing, this patch allows the script to search for the next occurrence of the magic bytes and then starts reading the block from there. ACKs for top commit: laanwj: ACK 3284e6c09a84e9557ec72723ad636053d3ef7122 Tree-SHA512: 18067ae0b4b62e822dfc558a86439ad6acaf939b98479e38e8e4248536574643b26eb48e96ec7139375c88b42cbe7705a64deb13a3c239e16025a6aad3d69bfa
2019-09-30scripts: search for next position of magic bytes rather than failTim Akinbo
document seek method for next position of magic bytes
2019-09-03test/contrib: Fix invalid escapes in regex stringsBen Woosley
Flagged by flake8 v3.6.0, as W605, plus a few others identified incidentally, e.g. 59ffecf66cf4d08c4b431e457b083878d66a3fd6. Note that r"\n" matches to "\n" under re.match/search.
2019-03-02scripted-diff: test: Use py3.5 bytes::hex() methodMarcoFalke
-BEGIN VERIFY SCRIPT- sed -i -e "s/def bytes_to_hex_str/def b_2_x/g" $(git grep -l bytes_to_hex_str) export RE_B_0="[^()]*" # match no bracket export RE_B_1="${RE_B_0}\(${RE_B_0}\)${RE_B_0}" # match exactly one () export RE_B_2="${RE_B_0}\(${RE_B_1}\)${RE_B_0}" # match wrapped (()) export RE_M="(b2x|bytes_to_hex_str)\(((${RE_B_0}|${RE_B_1}|${RE_B_2})*)\)" sed -i --regexp-extended -e "s/${RE_M}/\2.hex()/g" $(git grep -l -E '(b2x|bytes_to_hex_str)') sed -i --regexp-extended -e "/ +bytes_to_hex_str( as b2x)?,/d" $(git grep -l bytes_to_hex_str) sed -i --regexp-extended -e "s/ +bytes_to_hex_str( as b2x)?,//g" $(git grep -l bytes_to_hex_str) sed -i --regexp-extended -e "s/, bytes_to_hex_str( as b2x)?//g" $(git grep -l bytes_to_hex_str) export RE_M="(binascii\.)?hexlify\(((${RE_B_0}|${RE_B_1}|${RE_B_2})*)\).decode\(${RE_B_0}\)" sed -i --regexp-extended -e "s/${RE_M}/\2.hex()/g" $(git grep -l hexlify -- ':(exclude)share') sed -i --regexp-extended -e "/from binascii import hexlify$/d" $(git grep -l hexlify -- ':(exclude)share') sed -i --regexp-extended -e "s/(from binascii import) .*hexlify/\1 unhexlify/g" $(git grep -l hexlify -- ':(exclude)share') sed -i -e 's/ignore-names "/ignore-names "b_2_x,/g' ./test/lint/lint-python-dead-code.sh -END VERIFY SCRIPT-
2018-12-13Remove Python 2 import workaroundspracticalswift
2018-07-27Update copyright headers to 2018DrahtBot
2018-06-12Explicitly specify encoding when opening text files in Python codepracticalswift
2018-05-11Enable W191 and W291 flake8 checks.John Bampton
Remove trailing whitespace from Python files. Convert tabs to spaces.
2018-04-16Minor Python cleanups to make flake8 pass with the new rules enabledpracticalswift
2018-01-03Increment MIT Licence copyright header year on files modified in 2017Akira Takizawa
2017-01-25Fix various minor linearization script issuesDouglas Roark
- The last-timestamp-encountered variable wasn’t being used properly. Rewrite code to properly allow for new blockchain files to be written when split by month. - Properly set a blockchain file’s access and modify times. - Add a “debug output” option to quiet certain output that might not always be desirable. - Update the README.
2017-01-13Remove unused Python importspracticalswift
2017-01-05Make linearize scripts Python 3-compatible.Doug
2017-01-05Allow linearization scripts to support hash byte reversalDoug
Currently, the linearization scripts require input hashes to be in one endian form. Add support for byte reversal.
2016-12-31Increment MIT Licence copyright header year on files modified in 2016isle2983
Edited via: $ contrib/devtools/copyright_header.py update .
2016-06-26Use portable #! in python scripts (/usr/bin/env)Matthew King
2015-07-10Update Linearize tool to support Windows pathsPaul Georgiou
2015-05-20contrib/linearize: Support linearization of testnet blocksJeff Garzik
2014-12-19Added "Core" to copyright headerssandakersmann
Github-Pull: #5494 Rebased-From: 15de949bb9277e442302bdd8dee299a8d6deee60
2014-12-16Remove references to X11 licenceMichael Ford
2014-12-08Remove genesis block hash from errorMicha
2014-12-07Make several small string fixes in linearize-data.pyMicha
- Add a space after the fixed string prepended to file names when input or output file changes - Clarify the error message when the genesis block is not found in the hash list (...why do we have this at all?)
2014-10-06contrib: make linearize-data.py cope with out-of-order blocksWladimir J. van der Laan
Make it possible to read blocks in any order. This will be required after headers-first (#4468), so should be merged before that. - Read block header. For expected blocks, continue, else skip. - For in-order blocks: copy block contents directly. Write prior out-of-order blocks if this connects a consecutive span. - For out-of-order blocks, store extents of block data for later retrieval. Cache out-of-order blocks in memory up to 100MB (configurable).
2014-09-11Catch error when done reading filesSuhas Daftuar
2014-09-04contrib/linearize: Add feature to set file's timestamp based on block header ↵Jeff Garzik
time.
2014-08-24contrib/linearize: split block files based on year-month, not just yearJeff Garzik
2014-08-24contrib/linearize: Guarantee that output is generated in-orderJeff Garzik
This was typically ensured implicitly by virtue of normal bitcoind operation. Adding an explicit check provides a stronger guarantee, and it is cheap to add.
2014-08-23contrib/linearize: split output files based on new-timestamp-year or ↵Jeff Garzik
max-file-size
2014-08-23Update linearize scripts.Jeff Garzik
Break into two steps: * Generate hash list * Build data file(s) from local bitcoind blocks/ directory. This supports building one large bootstrap.dat, or multiple smaller blocks/blkNNNNN.dat files.
generated by cgit v1.2.3 (git 2.26.2) at 2024-11-02 13:59:35 +0000