Age | Commit message (Collapse) | Author |
|
Switching the endianness of a hex string `str` can simply be
achieved by `bytes.fromhex(str)[::-1].hex()`, i.e. we can use
that and remove those helper methods.
|
|
The string representation of a block header hash is simply the
hexlified byte-reversed double SHA256 hash of its serialization.
|
|
-BEGIN VERIFY SCRIPT-
./contrib/devtools/copyright_header.py update ./
-END VERIFY SCRIPT-
Commits of previous years:
* 2020: fa0074e2d82928016a43ca408717154a1c70a4db
* 2019: aaaaad6ac95b402fe18d019d67897ced6b316ee0
|
|
|
|
|
|
-BEGIN VERIFY SCRIPT-
./contrib/devtools/copyright_header.py update ./
-END VERIFY SCRIPT-
|
|
some block files pruned
317fb96de9c6257972f1213b4ef2c3fe87dde99f Add search for first blk file with pruned node (Rjected)
Pull request description:
<!--
*** Please remove the following help text before submitting: ***
Pull requests without a rationale and clear improvement may be closed
immediately.
-->
<!--
Please provide clear motivation for your patch and explain how it improves
Bitcoin Core user experience or Bitcoin Core developer experience
significantly:
* Any test improvements or new tests that improve coverage are always welcome.
* All other changes should have accompanying unit tests (see `src/test/`) or
functional tests (see `test/`). Contributors should note which tests cover
modified code. If no tests exist for a region of modified code, new tests
should accompany the change.
* Bug fixes are most welcome when they come with steps to reproduce or an
explanation of the potential issue as well as reasoning for the way the bug
was fixed.
* Features are welcome, but might be rejected due to design or scope issues.
If a feature is based on a lot of dependencies, contributors should first
consider building the system outside of Bitcoin Core, if possible.
* Refactoring changes are only accepted if they are required for a feature or
bug fix or otherwise improve developer experience significantly. For example,
most "code style" refactoring changes require a thorough explanation why they
are useful, what downsides they have and why they *significantly* improve
developer experience or avoid serious programming bugs. Note that code style
is often a subjective matter. Unless they are explicitly mentioned to be
preferred in the [developer notes](/doc/developer-notes.md), stylistic code
changes are usually rejected.
-->
When bitcoind is running in pruned mode, producing a hashlist with `./linearize-hashes.py linearize.cfg > hashlist.txt` and then executing `linearize-data.py linearize.cfg` will produce:
```
Read 313001 hashes
Input file /home/dan/.bitcoin/blocks/blk00000.dat
Premature end of block data
```
This happens because `linearize-data` starts by attempting to process `blk00000.dat` regardless of whether or not `blk00000.dat` actually exists - this may not be the case if working with a pruned node.
This PR adds a function which finds the first block file that does exist, and calls that function when the `BlockDataCopier` is initialized.
This is a refactor of #16431.
<!--
Bitcoin Core has a thorough review process and even the most trivial change
needs to pass a lot of eyes and requires non-zero or even substantial time
effort to review. There is a huge lack of active reviewers on the project, so
patches often sit for a long time.
-->
ACKs for top commit:
darosior:
ACK 317fb96de9c6257972f1213b4ef2c3fe87dde99f
laanwj:
Code review ACK 317fb96de9c6257972f1213b4ef2c3fe87dde99f
theStack:
Code review ACK https://github.com/bitcoin/bitcoin/pull/17336/commits/317fb96de9c6257972f1213b4ef2c3fe87dde99f
Tree-SHA512: fc8014282df6cfe7b267e64db8ce7d82b86b758c302fbfea4a3c39b62d93512f5c2e31a0de4e9c5ec18fc0268c917f011257d37b45afaef6033eec90e4aa585f
|
|
582e66b6e75d58033987a7b0474226cfdd724ce0 doc: Added regtest config for linearize script (Gr0kchain)
Pull request description:
Updated the example-linearize.cfg file to include support for the regtest chain network config which is used by the ./linearize-data.py
Problem:
Without the regtest magic, genesis hash and path config, the `linearize-data.py` script cannot generate a bootstrap.dat file.
Example:
./linearize-data.py ./linearize.cfg
Read 102 hashes
Genesis block not found in hashlist
Solution:
Added netmagic, genesis and input example parameters to file.
Resolution
1. Starting bitcoind in regtest mode
2. bitcoin-cli generatetoaddress 101 $(bitcoin-cli getnewaddress)
3. ./linearize-hashes.py ./linearize.cfg > ./hashlist.txt
4. ./linearize-data.py ./linearize.cfg
```
$ ./linearize-data.py ./linearize.cfg
Read 102 hashes
Input file /Users/gr0kchain/.bitcoin/regtest/blocks/blk00000.dat
Output file /Users/gr0kchain/Downloads/bootstrap.dat
Done (102 blocks written)
```
ACKs for top commit:
fanquake:
ACK 582e66b6e75d58033987a7b0474226cfdd724ce0
Tree-SHA512: 699e92e740e68e2e5190ba37538efbbe3e4d4e725ebd6af704a0cf5517683b691754f7ea097bf840845d2b53b793c63258d406e9bd37922db810cf58bed053c3
|
|
-BEGIN VERIFY SCRIPT-
./contrib/devtools/copyright_header.py update ./
-END VERIFY SCRIPT-
|
|
Updated the example-linearize.cfg file to include support for the regtest chain network config which is used by the ./linearize-data.py
Problem:
Without the regtest magic, genesis hash and path config, the `linearize-data.py` script cannot generate a bootstrap.dat file.
Example of error:
./linearize-data.py ./linearize.cfg
Read 102 hashes
Genesis block not found in hashlist
Solution:
Added netmagic, genesis and input example parameters to file.
Resolution
1. Starting bitcoind in regtest mode
2. bitcoin-cli generatetoaddress 101 $(bitcoin-cli getnewaddress)
3. ./linearize-hashes.py ./linearize.cfg > ./hashlist.txt
4. ./linearize-data.py ./linearize.cfg
Example after fix:
$ ./linearize-data.py ./linearize.cfg
Read 102 hashes
Input file /Users/gr0kchain/.bitcoin/regtest/blocks/blk00000.dat
Output file /Users/gr0kchain/Downloads/bootstrap.dat
Done (102 blocks written)
|
|
|
|
rather than fail
3284e6c09a84e9557ec72723ad636053d3ef7122 scripts: search for next position of magic bytes rather than fail (Tim Akinbo)
Pull request description:
When using the `linearize-data.py` contrib script to export block data, there are edge cases where the script fails with an `Invalid magic: 00000000` error. This error occurs due to the presence of padding bytes that occasionally appears between consecutive blocks in the block data file.
There's an ongoing conversation about this in #14986. sipa also admitted that it is a bug in #5028. Fortunately, this is not an issue in bitcoin core as it handles this type of situation gracefully and so no fix in bitcoin core is required.
This PR is an improvement on how the script handles these "invalid magic bytes". Rather than failing, this patch allows the script to search for the next occurrence of the magic bytes and then starts reading the block from there.
ACKs for top commit:
laanwj:
ACK 3284e6c09a84e9557ec72723ad636053d3ef7122
Tree-SHA512: 18067ae0b4b62e822dfc558a86439ad6acaf939b98479e38e8e4248536574643b26eb48e96ec7139375c88b42cbe7705a64deb13a3c239e16025a6aad3d69bfa
|
|
document seek method for next position of magic bytes
|
|
Flagged by flake8 v3.6.0, as W605, plus a few others identified
incidentally, e.g. 59ffecf66cf4d08c4b431e457b083878d66a3fd6.
Note that r"\n" matches to "\n" under re.match/search.
|
|
|
|
-BEGIN VERIFY SCRIPT-
sed -i -e "s/def bytes_to_hex_str/def b_2_x/g" $(git grep -l bytes_to_hex_str)
export RE_B_0="[^()]*" # match no bracket
export RE_B_1="${RE_B_0}\(${RE_B_0}\)${RE_B_0}" # match exactly one ()
export RE_B_2="${RE_B_0}\(${RE_B_1}\)${RE_B_0}" # match wrapped (())
export RE_M="(b2x|bytes_to_hex_str)\(((${RE_B_0}|${RE_B_1}|${RE_B_2})*)\)"
sed -i --regexp-extended -e "s/${RE_M}/\2.hex()/g" $(git grep -l -E '(b2x|bytes_to_hex_str)')
sed -i --regexp-extended -e "/ +bytes_to_hex_str( as b2x)?,/d" $(git grep -l bytes_to_hex_str)
sed -i --regexp-extended -e "s/ +bytes_to_hex_str( as b2x)?,//g" $(git grep -l bytes_to_hex_str)
sed -i --regexp-extended -e "s/, bytes_to_hex_str( as b2x)?//g" $(git grep -l bytes_to_hex_str)
export RE_M="(binascii\.)?hexlify\(((${RE_B_0}|${RE_B_1}|${RE_B_2})*)\).decode\(${RE_B_0}\)"
sed -i --regexp-extended -e "s/${RE_M}/\2.hex()/g" $(git grep -l hexlify -- ':(exclude)share')
sed -i --regexp-extended -e "/from binascii import hexlify$/d" $(git grep -l hexlify -- ':(exclude)share')
sed -i --regexp-extended -e "s/(from binascii import) .*hexlify/\1 unhexlify/g" $(git grep -l hexlify -- ':(exclude)share')
sed -i -e 's/ignore-names "/ignore-names "b_2_x,/g' ./test/lint/lint-python-dead-code.sh
-END VERIFY SCRIPT-
|
|
Fix class case issue.
|
|
|
|
|
|
|
|
Remove trailing whitespace from Python files.
Convert tabs to spaces.
|
|
|
|
|
|
testnet 18332
ce3baa193 changed regtest RPCport to 18443 to avoid conflict with testnet 18332 (Ferdinando M. Ametrano)
Pull request description:
using the same JSON-RPC default port for both testnet and regtest prevents running both at the same time on the same machine. Since RPCport=P2Pport-1 for both mainnet and testnet, and regtest P2Pport being 18444, 18443 is proposed for regtest RPCport
Documentation has been updated (or created where missing); manpages doc/man/bitcoin*.1 could include information for regtest too
Tree-SHA512: d42185f7ef54dc918ece19b543c8681d08bb9c5a971394e21f2d9a1091734b091b08df69fab622c207b46f402cf9323ded5b7a33fbd0af722388930169124e7f
|
|
|
|
|
|
|
|
|
|
Adds a datadir configuration option to the linearize scripts to allow the script to use the RPC cookie instead of requiring the user to set a rpcuser and rpcpassword for the rpc server.
|
|
b9d95bd Fix various minor linearization script issues (Douglas Roark)
|
|
|
|
- The last-timestamp-encountered variable wasn’t being used properly. Rewrite code to properly allow for new blockchain files to be written when split by month.
- Properly set a blockchain file’s access and modify times.
- Add a “debug output” option to quiet certain output that might not always be desirable.
- Update the README.
|
|
|
|
|
|
Currently, the linearization scripts require input hashes to be in one endian form. Add support for byte reversal.
|
|
Edited via:
$ contrib/devtools/copyright_header.py update .
|
|
|
|
|
|
|
|
Github-Pull: #5494
Rebased-From: 15de949bb9277e442302bdd8dee299a8d6deee60
|
|
|
|
|
|
- Add a space after the fixed string prepended to file names when input or
output file changes
- Clarify the error message when the genesis block is not found in the
hash list (...why do we have this at all?)
|
|
aedc74d contrib: make linearize-data.py cope with out-of-order blocks (Wladimir J. van der Laan)
|
|
Make it possible to read blocks in any order. This will be required
after headers-first (#4468), so should be merged before that.
- Read block header. For expected blocks, continue, else skip.
- For in-order blocks: copy block contents directly. Write prior
out-of-order blocks if this connects a consecutive span.
- For out-of-order blocks, store extents of block data for later
retrieval. Cache out-of-order blocks in memory up to 100MB
(configurable).
|
|
Batch up to 10000 requests for a ~30x speedup.
|
|
|
|
time.
|
|
|
|
This was typically ensured implicitly by virtue of normal bitcoind
operation. Adding an explicit check provides a stronger guarantee, and
it is cheap to add.
|