aboutsummaryrefslogtreecommitdiff
path: root/src/util
AgeCommit message (Collapse)Author
2021-03-11Merge #21007: bitcoind: Add -daemonwait option to wait for initializationWladimir J. van der Laan
e017a913d0d78ef0766cf73586fe7a38488e1a26 bitcoind: Add -daemonwait option to wait for initialization (Wladimir J. van der Laan) c3e6fdee6d39d3f52dec421b48a0ac8bad5006f7 shutdown: Use RAII TokenPipe in shutdown (Wladimir J. van der Laan) 612f746a8ffa265b6877bedbbe21fcbb392f1516 util: Add RAII TokenPipe (Wladimir J. van der Laan) Pull request description: This adds a `-daemonwait` flag that does the same as `-daemon` except that it, from a user perspective, backgrounds the process only after initialization is complete. This is similar to the behaviour of some other software such as c-lightning. This can be useful when the process launching bitcoind wants to guarantee that either the RPC server is running, or that initialization failed, before continuing. The exit code indicates the initialization result. The use of the libc function `daemon()` is replaced by a custom implementation which is inspired by the [glibc implementation](https://github.com/lattera/glibc/blob/master/misc/daemon.c#L44), but which also creates a pipe from the child to the parent process for communication. An additional advantage of having our own `daemon()` implementation is that no MACOS-specific pragmas are needed anymore to silence a deprecation warning. TODO: - [x] Factor out `token_read` and `token_write` to an utility, and use them in `shutdown.cpp` as well—this is exactly the same kind of communication mechanism. - [x] RAII-ify pipe endpoints. - [x] Improve granularity of the `configure.ac` checks. This currently still checks for the function `daemon()` which makes no sense as it's not used. It should check for individual functions such as `fork()` and `setsid()` etc—the former being required, the second optional. - [-] ~~Signal propagation during initialization: if say, pressing Ctrl-C during `-daemonwait` it would be good to pass this SIGINT on to the child process instead of detaching the parent process and letting the child run free.~~ This is not necessary, see https://github.com/bitcoin/bitcoin/pull/21007#issuecomment-769007341. Future: - Consider if it makes sense to use this in the RPC tests (there would be no more need for "is RPC ready" polling loops). I think this is out of scope for this PR. ACKs for top commit: jonatack: Tested ACK e017a913d0d78ef0766cf73586fe7a38488e1a26 checked change since previous review is move-only Tree-SHA512: 53369b8ca2247e4cf3af8cb2cfd5b3399e8e0e3296423d64be987004758162a7ddc1287b01a92d7692328edcb2da4cf05d279b1b4ef61a665b71440ab6a6dbe2
2021-03-11scripted-diff: remove MakeUnique<T>()fanquake
-BEGIN VERIFY SCRIPT- git rm src/util/memory.h sed -i -e 's/MakeUnique/std::make_unique/g' $(git grep -l MakeUnique src) sed -i -e '/#include <util\/memory.h>/d' $(git grep -l '#include <util/memory.h>' src) sed -i -e '/util\/memory.h \\/d' src/Makefile.am -END VERIFY SCRIPT-
2021-03-04util: Add RAII TokenPipeWladimir J. van der Laan
2021-03-04Merge #21015: Make all of net_processing (and some of net) use std::chrono typesfanquake
0eaea66e8bfdfb23ff86c0f0924c2d75f5aca75f Make tx relay data structure use std::chrono types (Pieter Wuille) 55e82881a1503bff146970856c1474a6ea659c94 Make all Poisson delays use std::chrono types (Pieter Wuille) c733ac4d8a597b1001555b142f488ed9fbecc405 Convert block/header sync timeouts to std::chrono types (Pieter Wuille) 4d98b401fbd821700f7a792b0a4cb52c9b71bc9f Change all ping times to std::chrono types (Pieter Wuille) Pull request description: (Picking up #20044. Rebased against master.) This changes various uses of integers to represent timestamps and durations to `std::chrono` duration types with type-safe conversions, getting rid of various `.count()`, constructors, and conversion factors. ACKs for top commit: jnewbery: utACK 0eaea66e8bfdfb23ff86c0f0924c2d75f5aca75f vasild: ACK 0eaea66e8bfdfb23ff86c0f0924c2d75f5aca75f MarcoFalke: re-ACK 0eaea66e8bfdfb23ff86c0f0924c2d75f5aca75f, only changes: minor rename, using C++11 member initializer, using 2min chrono literal, rebase 🤚 ajtowns: utACK 0eaea66e8bfdfb23ff86c0f0924c2d75f5aca75f Tree-SHA512: 2dbd8d53bf82e98f9b4611e61dc14c448e8957d1a02575b837fadfd59f80e98614d0ccf890fc351f960ade76a6fb8051b282e252e81675a8ee753dba8b1d7f57
2021-03-04util: Make Assume() usable as unary expressionMarcoFalke
2021-03-03Make all Poisson delays use std::chrono typesPieter Wuille
2021-03-03Change all ping times to std::chrono typesPieter Wuille
2021-03-02util: Avoid invalid integer negation in FormatMoney: make FormatMoney(const ↵practicalswift
CAmount& n) well-defined also when n is std::numeric_limits<CAmount>::min()
2021-03-01net: extend Sock with a method to check whether connectedVasil Dimov
This will be convenient in the I2P SAM implementation.
2021-03-01net: extend Sock with methods for robust send & read until terminatorVasil Dimov
Introduce two high level, convenience methods in the `Sock` class: * `SendComplete()`: keep trying to send the specified data until either successfully sent all of it, timeout or interrupted. * `RecvUntilTerminator()`: read until a terminator is encountered (never after it), timeout or interrupted. These will be convenient in the I2P SAM implementation. `SendComplete()` can also be used in the SOCKS5 implementation instead of calling `send()` directly.
2021-03-01net: extend Sock::Wait() to report a timeoutVasil Dimov
Previously `Sock::Wait()` would not have signaled to the caller whether a timeout or one of the requested events occurred since that was not needed by any of the callers. Such functionality will be needed in the I2P implementation, thus extend the `Sock::Wait()` method.
2021-03-01net: move the constant maxWait out of InterruptibleRecv()Vasil Dimov
Move `maxWait` out of `InterruptibleRecv()` and rename it to `MAX_WAIT_FOR_IO` so that it can be reused by other code.
2021-03-01util: fix WriteBinaryFile() claiming success even if error occurredVasil Dimov
`fclose()` is flushing any buffered data to disk, so if it fails then that could mean that the data was not completely written to disk. Thus, check if `fclose()` succeeds and only then claim success from `WriteBinaryFile()`.
2021-03-01util: fix ReadBinaryFile() returning partial contentsVasil Dimov
If an error occurs and `fread()` returns `0` (nothing was read) then the code before this patch would have returned "success" with a partially read contents of the file.
2021-03-01util: extract {Read,Write}BinaryFile() to its own filesVasil Dimov
Extract `ReadBinaryFile()` and `WriteBinaryFile()` from `torcontrol.cpp` to its own `readwritefile.{h,cpp}` files, so that it can be reused from other modules.
2021-02-24Merge #18017: txmempool: split epoch logic into classWladimir J. van der Laan
fd6580e405699ccb051fd2a34525e48d3253673d [refactor] txmempool: split epoch logic into class (Anthony Towns) Pull request description: Splits the epoch logic introduced in #17925 into a separate class. Uses clang's thread safety annotations and encapsulates the data more strongly to reduce chances of bugs from API misuse. ACKs for top commit: jonatack: ACK fd6580e405699ccb051fd2a34525e48d3253673d using clang thread safety annotations looks like a very good idea, and the encapsulation this change adds should improve robustness (and possible unit test-ability) of the code. Verified that changing some of the locking duly provoked build-time warnings with Clang 9 on Debian and that small changes in the new `Epoch` class were covered by failing functional test assertions in `mempool_updatefromblock.py`, `mempool_resurrect.py`, and `mempool_reorg.py` hebasto: re-ACK fd6580e405699ccb051fd2a34525e48d3253673d, since my [previous](https://github.com/bitcoin/bitcoin/pull/18017#pullrequestreview-569619362) review: Tree-SHA512: 7004623faa02b56639aa05ab7a078320a6d8d54ec62d8022876221e33f350f47df51ddff056c0de5be798f8eb39b5c03c2d3f035698555d70abc218e950f2f8c
2021-02-23rpc: send: support external signerSjors Provoost
2021-02-21configure: add --enable-external-signerSjors Provoost
This option replaces --with-boost-process This prepares external signer support to be disabled by default. It adds a configure option to enable this feature and to check if Boost::Process is present. This also exposes ENABLE_EXTERNAL_SIGNER to the test suite via test/config.ini
2021-02-18Merge #19809: log: Prefix log messages with function name and source code ↵Wladimir J. van der Laan
location if -logsourcelocations is set b4511e2e2ed1a6077ae6826a9ee6b7a311293d08 log: Prefix log messages with function name if -logsourcelocations is set (practicalswift) Pull request description: Prefix log messages with function name if `-logfunctionnames` is set. Yes, exactly like `-logthreadnames` but for function names instead of thread names :) This is a small developer ergonomics improvement: I've found this to be a cheap/simple way to correlate log output and originating function. For me it beats the ordinary cycle of 1.) try to figure out a regexp matching the static part of the dynamic log message, 2.) `git grep -E 'Using .* MiB out of .* requested for signature cache'`, 3.) `mcedit filename.cpp` (`openemacs filename.cpp` works too!) and 4.) search for log message and scroll up to find the function name :) Without any logging parameters: ``` $ src/bitcoind -regtest 2020-08-25T03:29:04Z Using RdRand as an additional entropy source 2020-08-25T03:29:04Z Using 16 MiB out of 32/2 requested for signature cache, able to store 524288 elements 2020-08-25T03:29:04Z Using 16 MiB out of 32/2 requested for script execution cache, able to store 524288 elements 2020-08-25T03:29:04Z Loaded best chain: hashBestChain=0fff88f13cb7b2c71f2a335e3a4fc328bf5beb436012afca590b1a11466e22ff height=0 date=2011-02-02T23:16:42Z progress=1.000000 2020-08-25T03:29:04Z block tree size = 1 2020-08-25T03:29:04Z nBestHeight = 0 2020-08-25T03:29:04Z Imported mempool transactions from disk: 0 succeeded, 0 failed, 0 expired, 0 already there, 0 waiting for initial broadcast 2020-08-25T03:29:04Z 0 addresses found from DNS seeds ``` With `-logthreadnames` and `-logfunctionnames`: ``` $ src/bitcoind -regtest -logthreadnames -logfunctionnames 2020-08-25T03:29:04Z [init] [ReportHardwareRand] Using RdRand as an additional entropy source 2020-08-25T03:29:04Z [init] [InitSignatureCache] Using 16 MiB out of 32/2 requested for signature cache, able to store 524288 elements 2020-08-25T03:29:04Z [init] [InitScriptExecutionCache] Using 16 MiB out of 32/2 requested for script execution cache, able to store 524288 elements 2020-08-25T03:29:04Z [init] [LoadChainTip] Loaded best chain: hashBestChain=0fff88f13cb7b2c71f2a335e3a4fc328bf5beb436012afca590b1a11466e22ff height=0 date=2011-02-02T23:16:42Z progress=1.000000 2020-08-25T03:29:04Z [init] [AppInitMain] block tree size = 1 2020-08-25T03:29:04Z [init] [AppInitMain] nBestHeight = 0 2020-08-25T03:29:04Z [loadblk] [LoadMempool] Imported mempool transactions from disk: 0 succeeded, 0 failed, 0 expired, 0 already there, 0 waiting for initial broadcast 2020-08-25T03:29:04Z [dnsseed] [ThreadDNSAddressSeed] 0 addresses found from DNS seeds ``` ACKs for top commit: laanwj: Code review ACK b4511e2e2ed1a6077ae6826a9ee6b7a311293d08 MarcoFalke: review ACK b4511e2e2ed1a6077ae6826a9ee6b7a311293d08 🌃 Tree-SHA512: d100f5364630c323f31d275259864c597f7725e462d5f4bdedcc7033ea616d7fc0d16ef1b2af557e692f4deea73c6773ccfc681589e7bf6ba970b9ec169040c7
2021-02-18Merge #20429: refactor: replace (sizeof(a)/sizeof(a[0])) with C++17 std::sizeMarcoFalke
e829c9afbf75e930db6c3fe77a269b0af5e7a3ad refactor: replace sizeof(a)/sizeof(a[0]) by std::size (C++17) (Sebastian Falbesoner) 365539c84691d470b44d35df374d8c049f8c1192 refactor: init vectors via std::{begin,end} to avoid pointer arithmetic (Sebastian Falbesoner) 63d4ee1968144cc3d115f92baef95785abf813ac refactor: iterate arrays via C++11 range-based for loops if idx is not needed (Sebastian Falbesoner) Pull request description: This refactoring PR picks up the idea of #19626 and replaces all occurences of `sizeof(x)/sizeof(x[0])` (or `sizeof(x)/sizeof(*x)`, respectively) with the now-available C++17 [`std::size`](https://en.cppreference.com/w/cpp/iterator/size) (as [suggested by sipa](https://github.com/bitcoin/bitcoin/pull/19626#issuecomment-666487228)), making the macro `ARRAYLEN` obsolete. As preparation for this, two other changes are done to eliminate `sizeof(x)/sizeof(x[0])` usage: * all places where arrays are iterated via an index are changed to use C++11 range-based for loops If the index' only purpose is to access the array element (as [suggested by MarcoFalke](https://github.com/bitcoin/bitcoin/pull/19626#discussion_r463404541)). * `std::vector` initializations are done via `std::begin` and `std::end` rather than using pointer arithmetic to calculate the end (also [suggested by MarcoFalke](https://github.com/bitcoin/bitcoin/pull/20429#discussion_r567418821)). ACKs for top commit: practicalswift: cr ACK e829c9afbf75e930db6c3fe77a269b0af5e7a3ad: patch looks correct fanquake: ACK e829c9afbf75e930db6c3fe77a269b0af5e7a3ad MarcoFalke: review ACK e829c9afbf75e930db6c3fe77a269b0af5e7a3ad 🌩 Tree-SHA512: b01d32c04b9e04d562b7717cae00a651ec9a718645047a90761be6959e0cc2adbd67494e058fe894641076711bb09c3b47a047d0275c736f0b2218e1ce0d193d
2021-02-17Merge #21110: util: remove Boost posix_time usage from GetTime*Wladimir J. van der Laan
9266f7497f256d780178829e0f3a29ddaeb794ba util: Use std::chrono for time getters (MarcoFalke) 3c2e16be22ae04bf56663ee5ec1554d0d569741b time: add runtime sanity check (Cory Fields) Pull request description: I have a followup that should remove the last of our `boost:posix_time` usage in `ParseISO8601DateTime`, but that will likely need more cross-platform testing/discussion, so have just split them up as this change is straight forward. ACKs for top commit: practicalswift: Tested ACK 9266f7497f256d780178829e0f3a29ddaeb794ba laanwj: Code review ACK 9266f7497f256d780178829e0f3a29ddaeb794ba Tree-SHA512: 5471a60e65e9fa8ef48320743ef637f1d162724e717e0f5509118e1e5732fc0844656a9c09d3d1300eb657dcc7a1e1e67305d8c9ef959c63be67393607dd4ceb
2021-02-17util: Use std::chrono for time gettersMarcoFalke
2021-02-17time: add runtime sanity checkCory Fields
std::chrono::system_clock.time_since_epoch and time_t(0) are not guaranteed to use the Unix epoch timestamp, but in practice they almost certainly will. Any differing behavior will be assumed to be an error, unless certain platforms prove to consistently deviate, at which point we'll cope with it by adding offsets. Do a quick runtime check to verify that time_t(0) == std::chrono::system_clock's epoch time == unix epoch. Co-authored-by: Anthony Towns <aj@erisian.com.au>
2021-02-16[doc / util] Use comments to clarify time unit for int64_t type.Amiti Uttarwar
2021-02-16[util] Introduce a SetMockTime that takes chrono timeAmiti Uttarwar
2021-02-16[util] Change GetMockTime to return chrono type instead of intAmiti Uttarwar
2021-02-16faster HexStr => 13% faster blockToJSONMartin Ankerl
`std::string`'s push_back is rather slow because it needs to check & update the string size. For `HexStr` the output string size is already easily know, so we can initially create the string with the correct size and then just assign the data. `HexStr` is heavily usd in `blockToJSON`, so this change is a noticeable benefit. Benchmark on an i7-8700 @3.2GHz: * 71,315,461.00 ns/op master * 62,842,490.00 ns/op this commit So this little change makes `blockToJSON` about ~13% faster.
2021-02-11Merge #20788: net: add RAII socket and use it instead of bare SOCKETWladimir J. van der Laan
615ba0eb96cf131364c1ceca9d3dedf006fa1e1c test: add Sock unit tests (Vasil Dimov) 7bd21ce1efc363b3e8ea1d51dd1410ccd66820cb style: rename hSocket to sock (Vasil Dimov) 04ae8469049e1f14585aabfb618ae522150240a7 net: use Sock in InterruptibleRecv() and Socks5() (Vasil Dimov) ba9d73268f9585d4b9254adcf54708f88222798b net: add RAII socket and use it instead of bare SOCKET (Vasil Dimov) dec9b5e850c6aad989e814aea5b630b36f55d580 net: move CloseSocket() from netbase to util/sock (Vasil Dimov) aa17a44551c03b00a47854438afe9f2f89b6ea74 net: move MillisToTimeval() from netbase to util/time (Vasil Dimov) Pull request description: Introduce a class to manage the lifetime of a socket - when the object that contains the socket goes out of scope, the underlying socket will be closed. In addition, the new `Sock` class has a `Send()`, `Recv()` and `Wait()` methods that can be overridden by unit tests to mock the socket operations. The `Wait()` method also hides the `#ifdef USE_POLL poll() #else select() #endif` technique from higher level code. ACKs for top commit: laanwj: Re-ACK 615ba0eb96cf131364c1ceca9d3dedf006fa1e1c jonatack: re-ACK 615ba0eb96cf131364c1ceca9d3dedf006fa1e1c Tree-SHA512: 3003e6bc0259295ca0265ccdeb1522ee25b4abe66d32e6ceaa51b55e0a999df7ddee765f86ce558a788c1953ee2009bfa149b09d494593f7d799c0d7d930bee8
2021-02-11Merge #21043: net: Avoid UBSan warning in ProcessMessage(...)MarcoFalke
3ddbf22ed179a2db733af4b521bec5d2b13ebf4b util: Disallow negative mocktime (MarcoFalke) f5f2f9716885e7548809e77f46b493c896a019bf net: Avoid UBSan warning in ProcessMessage(...) (practicalswift) Pull request description: Avoid UBSan warning in `ProcessMessage(...)`. Context: https://github.com/bitcoin/bitcoin/pull/20380#issuecomment-770427182 (thanks Crypt-iQ!) ACKs for top commit: MarcoFalke: re-ACK 3ddbf22ed179a2db733af4b521bec5d2b13ebf4b only change is adding patch written by me ajtowns: ACK 3ddbf22ed179a2db733af4b521bec5d2b13ebf4b -- code review only Tree-SHA512: e8d7af0457ca86872b75a4e406c0a93aafd841c2962e244e147e748cc7ca118c56be0fdafe53765f4b291410030b2c3cc8f76f733b37a955d34fc885ab6037b9
2021-02-10net: add RAII socket and use it instead of bare SOCKETVasil Dimov
Introduce a class to manage the lifetime of a socket - when the object that contains the socket goes out of scope, the underlying socket will be closed. In addition, the new `Sock` class has a `Send()`, `Recv()` and `Wait()` methods that can be overridden by unit tests to mock the socket operations. The `Wait()` method also hides the `#ifdef USE_POLL poll() #else select() #endif` technique from higher level code.
2021-02-10net: move CloseSocket() from netbase to util/sockVasil Dimov
Move `CloseSocket()` (and `NetworkErrorString()` which it uses) from `netbase.{h,cpp}` to newly added `src/util/sock.{h,cpp}`. This is necessary in order to use `CloseSocket()` from a newly introduced Sock class (which will live in `src/util/sock.{h,cpp}`). `sock.{h,cpp}` cannot depend on netbase because netbase will depend on it.
2021-02-10net: move MillisToTimeval() from netbase to util/timeVasil Dimov
Move `MillisToTimeval()` from `netbase.{h,cpp}` to `src/util/system.{h,cpp}`. This is necessary in order to use `MillisToTimeval()` from a newly introduced `src/util/sock.{h,cpp}` which cannot depend on netbase because netbase will depend on it.
2021-02-09Merge #21052: refactor: Replace fs::unique_path with GetUniquePath(path) callsWladimir J. van der Laan
1bca2aa694cd85984c09699ae28daec313077462 Introduce GetUniquePath(base) helper method to replace boost::filesystem::unique_path() which is not available in std::filesystem. (Kiminuo) Pull request description: This PR makes it easier in #20744 to remove our dependency on the `boost::filesystem::unique_path()` function which does not have a direct equivalent in C++17. This PR attempts to re-implement `boost::filesystem::unique_path()` as `GetUniquePath(path)` but the implementations are not meant to be the same. Note: * Boost 1.75.0 implementation of `unique_path`: https://github.com/boostorg/filesystem/blob/9cab675b71e98706886a87afe7c19eb9da568961/src/unique_path.cpp#L235 * In the previous implementation, I attempted to add: ```cpp fs::path GetUniquePath(const fs::path& base) { FastRandomContext rnd; fs::path tmpFile = base / HexStr(rnd.randbytes(8)); return tmpFile; } ``` to `fs.cpp` but this leads to a circular dependency: "fs -> random -> logging -> fs". That is why the modified implementation adds a new file. ACKs for top commit: laanwj: Code review ACK 1bca2aa694cd85984c09699ae28daec313077462 ryanofsky: Code review ACK 1bca2aa694cd85984c09699ae28daec313077462. It's a simple change and extra test coverage is nice Tree-SHA512: f324bdf0e254160c616b5033c3ece33d87db23eb0135acee99346ade7b5cf0d30f3ceefe359a25a8e9b53ba8e4419f459c2bdd369e10fc0152ce95031d1f221c
2021-02-09[refactor] txmempool: split epoch logic into classAnthony Towns
2021-02-04refactor: Treat ArgsManager::Flags as uint32_t explicitlyMarcoFalke
2021-02-04Introduce GetUniquePath(base) helper method to replace ↵Kiminuo
boost::filesystem::unique_path() which is not available in std::filesystem.
2021-02-04Merge #20715: util: Add ArgsManager::GetCommand() and use it in bitcoin-walletMarcoFalke
fa61b9d1a68820758f9540653920deaeae6abe79 util: Add ArgsManager::GetCommand() and use it in bitcoin-wallet (MarcoFalke) 7777105a24a36b62df35d12ecf6c6370671568c8 refactor: Move all command dependend checks to ExecuteWalletToolFunc (MarcoFalke) fa06bce4ac17f93decd4ee38c956e7aa55983f0d test: Add tests (MarcoFalke) fac05ccdade8b34c969b9cd9b37b355bc0aabf9c wallet: [refactor] Pass ArgsManager to WalletAppInit (MarcoFalke) Pull request description: This not only moves the parsing responsibility out from the wallet tool, but it also makes it easier to implement bitcoin-util #19937 Fixes: #20902 ACKs for top commit: ajtowns: ACK fa61b9d1a68820758f9540653920deaeae6abe79 fjahr: Code review ACK fa61b9d1a68820758f9540653920deaeae6abe79 Tree-SHA512: 79622b806e8bf9dcd0dc24a8a6687345710df57720992e83a41cd8d6762a6dc112044ebc58fcf6e8fbf45de29a79b04873c5b8c2494a1eaaf902a2884703e47b
2021-02-02util: Disallow negative mocktimeMarcoFalke
Signed-off-by: practicalswift <practicalswift@users.noreply.github.com>
2021-02-01refactor: Move STRINGIZE macro to macros.hHennadii Stepanov
This is a move-only change.
2021-01-31refactor: replace sizeof(a)/sizeof(a[0]) by std::size (C++17)Sebastian Falbesoner
Removes the macro ARRAYLEN and also substitutes all other uses of the same "sizeof(a)/sizeof(a[0])" pattern by std::size, available since C++17.
2021-01-29refactor: remove boost::thread_group usagefanquake
2021-01-21util: Add ArgsManager::GetCommand() and use it in bitcoin-walletMarcoFalke
Co-Authored-by: Anthony Towns <aj@erisian.com.au>
2021-01-21Merge #20932: refactor: Replace fs::absolute calls with AbsPathJoin callsMarcoFalke
da9caa1cedd69702aea44cb44b2fd0a2d6d56916 Replace fs::absolute calls with AbsPathJoin calls (Kiminuo) 66576c4fd532ac18b8b355ea93d25581a2c15654 test: Clear forced -walletdir setting after wallet init_tests (Kiminuo) Pull request description: This adds better test coverage and will make it easier in #20744 to remove our dependency on the two-argument boost::filesystem::absolute() function which does not have a direct equivalent in C++17. This PR doesn't change behavior aside from adding an assert and fixing a test bug. ACKs for top commit: jonatack: Code review ACK da9caa1cedd69702aea44cb44b2fd0a2d6d56916 only doxygen improvements since my last review per `git diff d867d7a da9caa1` MarcoFalke: review ACK da9caa1cedd69702aea44cb44b2fd0a2d6d56916 📯 ryanofsky: Code review ACK da9caa1cedd69702aea44cb44b2fd0a2d6d56916. Just comment and test tweaks since previous review. Tree-SHA512: c940ee60f3ba374d4927cf34cf12d27c4c735c94af591fbc0ca408c641b30f8f8fbcfe521d66bfbddf9877a1fc8cd99bd8a47ebcd2fa59789de6bd87a7b9cf4d
2021-01-18Merge #19866: eBPF Linux tracepointsWladimir J. van der Laan
22eb7930a6ae021438aa0b8e750170534944f296 tracing: add tracing framework (William Casarin) 933ab8a720cb9b3341adec4109cffb6dc5b322a5 build: detect sys/sdt.h for eBPF tracing (William Casarin) Pull request description: Instead of writing ad-hoc logging everywhere (eg: #19509), we can take advantage of linux user static defined traces, aka. USDTs ( not the stablecoin :sweat_smile: ) The linux kernel can hook into these tracepoints at runtime, but otherwise they have little to no performance impact. Traces can pass data which can be printed externally via tools such as bpftrace. For example, here's one that prints incoming and outgoing network messages: # Examples ## Network Messages ``` #!/usr/bin/env bpftrace BEGIN { printf("bitcoin net msgs\n"); @start = nsecs; } usdt:./src/bitcoind:net:push_message { $ip = str(arg0); $peer_id = (int64)arg1; $command = str(arg2); $data_len = arg3; $data = buf(arg3,arg4); $t = (nsecs - @start) / 100000; printf("%zu outbound %s %s %zu %d %r\n", $t, $command, $ip, $peer_id, $data_len, $data); @outbound[$command]++; } usdt:./src/bitcoind:net:process_message { $ip = str(arg0); $peer_id = (int64)arg1; $command = str(arg2); $data_len = arg3; $data = buf(arg3,arg4); $t = (nsecs - @start) / 100000; printf("%zu inbound %s %s %zu %d %r\n", $t, $command, $ip, $peer_id, $data_len, $data); @inbound[$ip, $command]++; } ``` $ sudo bpftrace netmsg.bt output: https://jb55.com/s/b11312484b601fb3.txt if you look at the bottom of the output you can see a histogram of all the messages grouped by message type and IP. nice! ## IBD Benchmarking ``` #!/usr/bin/env bpftrace BEGIN { printf("IBD to 500,000 bench\n"); } usdt:./src/bitcoind:CChainState:ConnectBlock { $height = (uint32)arg0; if ($height == 1) { printf("block 1 found, starting benchmark\n"); @start = nsecs; } if ($height >= 500000) { @end = nsecs; @duration = @end - @start; exit(); } } END { printf("duration %d ms\n", @duration / 1000000) } ``` This one hooks into ConnectBlock and prints the IBD time to height 500,000 starting from the first call to ConnectBlock Userspace static tracepoints give lots of flexibility without invasive logging code. It's also more flexible than ad-hoc logging code, allowing you to instrument many different aspects of the system without having to enable per-subsystem logging. Other ideas: tracepoints for lock contention, threads, what else? Let me know what ya'll think and if this is worth adding to bitcoin. ## TODO - [ ] docs? - [x] Integrate systemtap-std-dev/libsystemtap into build (provides the <sys/sdt.h> header) - [x] ~dtrace macos support? (is this still a thing?)~ going to focus on linux for now ACKs for top commit: laanwj: Tested ACK 22eb7930a6ae021438aa0b8e750170534944f296 0xB10C: Tested ACK 22eb7930a6ae021438aa0b8e750170534944f296 Tree-SHA512: 69242242112b679c8a12a22b3bc50252c305894fb3055ae6e13d5f56221d858e58af1d698af55e23b69bdb7abedb5565ac6b45fa5144087b77a17acd04646a75
2021-01-15Replace fs::absolute calls with AbsPathJoin callsKiminuo
This adds better test coverage and will make it easier in #20744 to remove our dependency on the two-argument boost::filesystem::absolute() function which does not have a direct equivalent in C++17.
2021-01-15log: Prefix log messages with function name if -logsourcelocations is setpracticalswift
2021-01-13Merge #19935: Move SaltedHashers to separate file and add some new onesWladimir J. van der Laan
281fd1a4a032cded7f9ea9857e3e99fc793c714b Replace KeyIDHasher with SaltedSipHasher (Andrew Chow) 210b693db66e7c5b618014b5a287aee15af00045 Add generic SaltedSipHasher (Andrew Chow) 95e61c1cf2a91d041c8025306ba36f0ea2806894 Move Hashers to util/hasher.{cpp/h} (Andrew Chow) Pull request description: There are existing `SaltedOutPointHasher` and `SaltedTxidHasher` classes used for `std::unordered_map` and `std::unordered_set` that could be useful in other places in the codebase. So we these to their own `saltedhash.{cpp/h}` file. An existing `KeyIDHasher` is moved there too. Additionally, `ScriptIDHasher`, `SaltedPubkeyHasher`, and `SaltedScriptHasher` are added so that they can be used in future work. `KeyIDHasher` and `ScriptIDHasher` are not salted so that equality comparisons of maps and sets keyed by `CKeyID` and `CScriptID` will actually work. Split from #19602 (and a few other PRs/branches I have). ACKs for top commit: laanwj: Code review ACK 281fd1a4a032cded7f9ea9857e3e99fc793c714b jonatack: ACK 281fd1a4a032cded7f9ea9857e3e99fc793c714b, code review, debug build and ran bitcoind after rebasing to master @ dff0f6f753ea fjahr: utACK 281fd1a4a032cded7f9ea9857e3e99fc793c714b Tree-SHA512: bb03b231ccf3c9ecefc997b8da9c3770af4819f9be5b0a72997a103864e84046a2ac39b8eadf0dc9247bdccd53f86f433642e3a098882e6748341a9e7736271b
2021-01-11Merge #20480: Replace boost::variant with std::variantfanquake
faa8f68943615785a2855676cf96e0e96f3cc6bd Replace boost::variant with std::variant (MarcoFalke) Pull request description: Now that we can use std::variant from the vanilla standard library, drop the third-party boost variant dependency ACKs for top commit: fjahr: Code review ACK faa8f68943615785a2855676cf96e0e96f3cc6bd fanquake: ACK faa8f68943615785a2855676cf96e0e96f3cc6bd Tree-SHA512: 6e3aecd33b00c2e31a763f999247944d5b2ce5e3018f1965c516c1000cd08ff6703a8d50fb0be64883153da2925ae72986b8a6b96586db74057bd05d6f4986e6
2021-01-07Merge #14501: Fix possible data race when committing block filesWladimir J. van der Laan
ef712298c3f8bc2afdad783f05080443b72b3f77 util: Check for file being NULL in DirectoryCommit (Luke Dashjr) 457490403853321d308c6ca6aaa90d6f8f29b4cf Fix possible data race when committing block files (Evan Klitzke) 220bb16cbee5b91d0bc0fcc6c71560d631295fa5 util: Introduce DirectoryCommit commit function to sync a directory (Evan Klitzke) ce5cbaea63ad4ea78e533bdb14f47f414061ae7f util.h: Document FileCommit function (Evan Klitzke) 844d650eea3bd809884cc5dd996a388bdc58314e util: Prefer Mac-specific F_FULLSYNC over fdatasync in FileCommit (Evan Klitzke) f6cec0bcaf560fa310853ad3fe17022602b63d5f util: Refactor FileCommit from an #if sequence nested in #else, to a sequence of #elif (Evan Klitzke) Pull request description: Reviving #12696 ACKs for top commit: laanwj: Code review ACK ef712298c3f8bc2afdad783f05080443b72b3f77 Tree-SHA512: 07d650990ef4c18d645dee3f9a199a940683ad17557d79d93979a76c4e710d8d70e6eae01d1a5991494a24a7654eb7db868be0c34a31e70b2509945d95bc9cce
2021-01-05Replace boost::variant with std::variantMarcoFalke