| Age | Commit message (Collapse) | Author |
|---|
|
rather than fail
3284e6c09a84e9557ec72723ad636053d3ef7122 scripts: search for next position of magic bytes rather than fail (Tim Akinbo)
Pull request description:
When using the `linearize-data.py` contrib script to export block data, there are edge cases where the script fails with an `Invalid magic: 00000000` error. This error occurs due to the presence of padding bytes that occasionally appears between consecutive blocks in the block data file.
There's an ongoing conversation about this in #14986. sipa also admitted that it is a bug in #5028. Fortunately, this is not an issue in bitcoin core as it handles this type of situation gracefully and so no fix in bitcoin core is required.
This PR is an improvement on how the script handles these "invalid magic bytes". Rather than failing, this patch allows the script to search for the next occurrence of the magic bytes and then starts reading the block from there.
ACKs for top commit:
laanwj:
ACK 3284e6c09a84e9557ec72723ad636053d3ef7122
Tree-SHA512: 18067ae0b4b62e822dfc558a86439ad6acaf939b98479e38e8e4248536574643b26eb48e96ec7139375c88b42cbe7705a64deb13a3c239e16025a6aad3d69bfa
|
|
document seek method for next position of magic bytes
|
|
Flagged by flake8 v3.6.0, as W605, plus a few others identified
incidentally, e.g. 59ffecf66cf4d08c4b431e457b083878d66a3fd6.
Note that r"\n" matches to "\n" under re.match/search.
|
|
|
|
-BEGIN VERIFY SCRIPT-
sed -i -e "s/def bytes_to_hex_str/def b_2_x/g" $(git grep -l bytes_to_hex_str)
export RE_B_0="[^()]*" # match no bracket
export RE_B_1="${RE_B_0}\(${RE_B_0}\)${RE_B_0}" # match exactly one ()
export RE_B_2="${RE_B_0}\(${RE_B_1}\)${RE_B_0}" # match wrapped (())
export RE_M="(b2x|bytes_to_hex_str)\(((${RE_B_0}|${RE_B_1}|${RE_B_2})*)\)"
sed -i --regexp-extended -e "s/${RE_M}/\2.hex()/g" $(git grep -l -E '(b2x|bytes_to_hex_str)')
sed -i --regexp-extended -e "/ +bytes_to_hex_str( as b2x)?,/d" $(git grep -l bytes_to_hex_str)
sed -i --regexp-extended -e "s/ +bytes_to_hex_str( as b2x)?,//g" $(git grep -l bytes_to_hex_str)
sed -i --regexp-extended -e "s/, bytes_to_hex_str( as b2x)?//g" $(git grep -l bytes_to_hex_str)
export RE_M="(binascii\.)?hexlify\(((${RE_B_0}|${RE_B_1}|${RE_B_2})*)\).decode\(${RE_B_0}\)"
sed -i --regexp-extended -e "s/${RE_M}/\2.hex()/g" $(git grep -l hexlify -- ':(exclude)share')
sed -i --regexp-extended -e "/from binascii import hexlify$/d" $(git grep -l hexlify -- ':(exclude)share')
sed -i --regexp-extended -e "s/(from binascii import) .*hexlify/\1 unhexlify/g" $(git grep -l hexlify -- ':(exclude)share')
sed -i -e 's/ignore-names "/ignore-names "b_2_x,/g' ./test/lint/lint-python-dead-code.sh
-END VERIFY SCRIPT-
|
|
Fix class case issue.
|
|
|
|
|
|
|
|
Remove trailing whitespace from Python files.
Convert tabs to spaces.
|
|
|
|
|
|
testnet 18332
ce3baa193 changed regtest RPCport to 18443 to avoid conflict with testnet 18332 (Ferdinando M. Ametrano)
Pull request description:
using the same JSON-RPC default port for both testnet and regtest prevents running both at the same time on the same machine. Since RPCport=P2Pport-1 for both mainnet and testnet, and regtest P2Pport being 18444, 18443 is proposed for regtest RPCport
Documentation has been updated (or created where missing); manpages doc/man/bitcoin*.1 could include information for regtest too
Tree-SHA512: d42185f7ef54dc918ece19b543c8681d08bb9c5a971394e21f2d9a1091734b091b08df69fab622c207b46f402cf9323ded5b7a33fbd0af722388930169124e7f
|
|
|
|
|
|
|
|
|
|
Adds a datadir configuration option to the linearize scripts to allow the script to use the RPC cookie instead of requiring the user to set a rpcuser and rpcpassword for the rpc server.
|
|
b9d95bd Fix various minor linearization script issues (Douglas Roark)
|
|
|
|
- The last-timestamp-encountered variable wasn’t being used properly. Rewrite code to properly allow for new blockchain files to be written when split by month.
- Properly set a blockchain file’s access and modify times.
- Add a “debug output” option to quiet certain output that might not always be desirable.
- Update the README.
|
|
|
|
|
|
Currently, the linearization scripts require input hashes to be in one endian form. Add support for byte reversal.
|
|
Edited via:
$ contrib/devtools/copyright_header.py update .
|
|
|
|
|
|
|
|
Github-Pull: #5494
Rebased-From: 15de949bb9277e442302bdd8dee299a8d6deee60
|
|
|
|
|
|
- Add a space after the fixed string prepended to file names when input or
output file changes
- Clarify the error message when the genesis block is not found in the
hash list (...why do we have this at all?)
|
|
aedc74d contrib: make linearize-data.py cope with out-of-order blocks (Wladimir J. van der Laan)
|
|
Make it possible to read blocks in any order. This will be required
after headers-first (#4468), so should be merged before that.
- Read block header. For expected blocks, continue, else skip.
- For in-order blocks: copy block contents directly. Write prior
out-of-order blocks if this connects a consecutive span.
- For out-of-order blocks, store extents of block data for later
retrieval. Cache out-of-order blocks in memory up to 100MB
(configurable).
|
|
Batch up to 10000 requests for a ~30x speedup.
|
|
|
|
time.
|
|
|
|
This was typically ensured implicitly by virtue of normal bitcoind
operation. Adding an explicit check provides a stronger guarantee, and
it is cheap to add.
|
|
max-file-size
|
|
Break into two steps:
* Generate hash list
* Build data file(s) from local bitcoind blocks/ directory.
This supports building one large bootstrap.dat, or multiple
smaller blocks/blkNNNNN.dat files.
|
|
linearize.py: harmonize rpcpass to rpcpassword according settings in
bitcoin.conf
|
|
|
|
Add the ability to start at a non-zero height, and allow for appending to
an existing file.
|
|
File and Link Fix.
|
