| Age | Commit message (Collapse) | Author |
|---|
|
rather than fail
3284e6c09a84e9557ec72723ad636053d3ef7122 scripts: search for next position of magic bytes rather than fail (Tim Akinbo)
Pull request description:
When using the `linearize-data.py` contrib script to export block data, there are edge cases where the script fails with an `Invalid magic: 00000000` error. This error occurs due to the presence of padding bytes that occasionally appears between consecutive blocks in the block data file.
There's an ongoing conversation about this in #14986. sipa also admitted that it is a bug in #5028. Fortunately, this is not an issue in bitcoin core as it handles this type of situation gracefully and so no fix in bitcoin core is required.
This PR is an improvement on how the script handles these "invalid magic bytes". Rather than failing, this patch allows the script to search for the next occurrence of the magic bytes and then starts reading the block from there.
ACKs for top commit:
laanwj:
ACK 3284e6c09a84e9557ec72723ad636053d3ef7122
Tree-SHA512: 18067ae0b4b62e822dfc558a86439ad6acaf939b98479e38e8e4248536574643b26eb48e96ec7139375c88b42cbe7705a64deb13a3c239e16025a6aad3d69bfa
|
|
document seek method for next position of magic bytes
|
|
Flagged by flake8 v3.6.0, as W605, plus a few others identified
incidentally, e.g. 59ffecf66cf4d08c4b431e457b083878d66a3fd6.
Note that r"\n" matches to "\n" under re.match/search.
|
|
-BEGIN VERIFY SCRIPT-
sed -i -e "s/def bytes_to_hex_str/def b_2_x/g" $(git grep -l bytes_to_hex_str)
export RE_B_0="[^()]*" # match no bracket
export RE_B_1="${RE_B_0}\(${RE_B_0}\)${RE_B_0}" # match exactly one ()
export RE_B_2="${RE_B_0}\(${RE_B_1}\)${RE_B_0}" # match wrapped (())
export RE_M="(b2x|bytes_to_hex_str)\(((${RE_B_0}|${RE_B_1}|${RE_B_2})*)\)"
sed -i --regexp-extended -e "s/${RE_M}/\2.hex()/g" $(git grep -l -E '(b2x|bytes_to_hex_str)')
sed -i --regexp-extended -e "/ +bytes_to_hex_str( as b2x)?,/d" $(git grep -l bytes_to_hex_str)
sed -i --regexp-extended -e "s/ +bytes_to_hex_str( as b2x)?,//g" $(git grep -l bytes_to_hex_str)
sed -i --regexp-extended -e "s/, bytes_to_hex_str( as b2x)?//g" $(git grep -l bytes_to_hex_str)
export RE_M="(binascii\.)?hexlify\(((${RE_B_0}|${RE_B_1}|${RE_B_2})*)\).decode\(${RE_B_0}\)"
sed -i --regexp-extended -e "s/${RE_M}/\2.hex()/g" $(git grep -l hexlify -- ':(exclude)share')
sed -i --regexp-extended -e "/from binascii import hexlify$/d" $(git grep -l hexlify -- ':(exclude)share')
sed -i --regexp-extended -e "s/(from binascii import) .*hexlify/\1 unhexlify/g" $(git grep -l hexlify -- ':(exclude)share')
sed -i -e 's/ignore-names "/ignore-names "b_2_x,/g' ./test/lint/lint-python-dead-code.sh
-END VERIFY SCRIPT-
|
|
|
|
|
|
|
|
Remove trailing whitespace from Python files.
Convert tabs to spaces.
|
|
|
|
|
|
- The last-timestamp-encountered variable wasn’t being used properly. Rewrite code to properly allow for new blockchain files to be written when split by month.
- Properly set a blockchain file’s access and modify times.
- Add a “debug output” option to quiet certain output that might not always be desirable.
- Update the README.
|
|
|
|
|
|
Currently, the linearization scripts require input hashes to be in one endian form. Add support for byte reversal.
|
|
Edited via:
$ contrib/devtools/copyright_header.py update .
|
|
|
|
|
|
|
|
Github-Pull: #5494
Rebased-From: 15de949bb9277e442302bdd8dee299a8d6deee60
|
|
|
|
|
|
- Add a space after the fixed string prepended to file names when input or
output file changes
- Clarify the error message when the genesis block is not found in the
hash list (...why do we have this at all?)
|
|
Make it possible to read blocks in any order. This will be required
after headers-first (#4468), so should be merged before that.
- Read block header. For expected blocks, continue, else skip.
- For in-order blocks: copy block contents directly. Write prior
out-of-order blocks if this connects a consecutive span.
- For out-of-order blocks, store extents of block data for later
retrieval. Cache out-of-order blocks in memory up to 100MB
(configurable).
|
|
|
|
time.
|
|
|
|
This was typically ensured implicitly by virtue of normal bitcoind
operation. Adding an explicit check provides a stronger guarantee, and
it is cheap to add.
|
|
max-file-size
|
|
Break into two steps:
* Generate hash list
* Build data file(s) from local bitcoind blocks/ directory.
This supports building one large bootstrap.dat, or multiple
smaller blocks/blkNNNNN.dat files.
|
