aboutsummaryrefslogtreecommitdiff
path: root/ci
AgeCommit message (Collapse)Author
2022-12-02Merge bitcoin/bitcoin#26592: ci: only run USDT interface tests on CirrusCIMarcoFalke
2811f40f3099a22091870d6aa2efecd741d186ce ci: only run USDT interface tests on CirrusCI (0xb10c) Pull request description: As mentioned in #26571, the task running the USDT interface tests fail when run in docker. cc7335edc87c6ef34429b4df94f53973db520aac in #25528 added that the tests are run in a **VM** in Cirrus CI. Running them locally in docker containers might not work: - We use [bcc] as tracing toolkit which requires the kernel headers to compile the BPF bytecode. As docker containers use the hosts kernel and don't run their own, there is a potential for mismatches between kernel headers available in the container and the host kernel. This results in a failure loading the BPF byte code. - Privilges are required to load the BPF byte code into the kernel. Normally, the docker containers aren't run with these. - We currently use an untrusted third-party PPA to install the bpfcc-tools package on Ubuntu 22.04. Using this on a local dev system could be a security risk. To not hinder the ASan + LSan + UBSan part of the CI task, the USDT tests are disabled on non-CirrusCI runs. [bcc]: https://github.com/iovisor/bcc Top commit has no ACKs. Tree-SHA512: 7c159b59630b36d5ed927b501fa0ad6f54ff3d98d0902f975cc8122b4147a7f828175807d40a470a85f0f3b6eeb79307d3465a287dbd2f3d75b803769ad0b6e7
2022-11-28ci: only run USDT interface tests on CirrusCI0xb10c
As mentioned in #26571, the task running the USDT interface tests fail when run in docker. cc7335edc87c6ef34429b4df94f53973db520aac in #25528 added that the tests are run in a **VM** in Cirrus CI. Running them locally in docker containers might not work: - We use [bcc] as tracing toolkit which requires the kernel headers to compile the BPF bytecode. As docker containers use the hosts kernel and don't run their own, there is a potential for mismatches between kernel headers available in the container and the host kernel. This results in a failure loading the BPF byte code. - Privilges are required to load the BPF byte code into the kernel. Normally, the docker containers aren't run with these. - We currently use an untrusted third-party PPA to install the bpfcc-tools package on Ubuntu 22.04. Using this on a local dev system could be a security risk. To not hinder the ASan + LSan + UBSan part of the CI task, the USDT tests are disabled on non-CirrusCI runs. [bcc]: https://github.com/iovisor/bcc
2022-11-28Merge bitcoin/bitcoin#26588: ci: Skip COMMIT_RANGE if no CIRRUS_PRMarcoFalke
fad1c55301b9f2d091d3b0d8a75ff522ce8dae5a lint: Skip COMMIT_RANGE if no CIRRUS_PR (MarcoFalke) Pull request description: It doesn't make sense to run this for non-PRs, because: * There are known whitespace "violations" in previous commits, so the lint may fail * Once the changes are merged, it is too late to fix them up (force pushes are illegal) * It isn't possible to determine which commits to run on if there is no reference branch (target branch of the pull request) Moreover, the test fails on non-master: * https://github.com/bitcoin/bitcoin/runs/8664441400 Fix all issues by skipping it. ACKs for top commit: hebasto: ACK fad1c55301b9f2d091d3b0d8a75ff522ce8dae5a, also tested in my personal Cirrus account. Tree-SHA512: be15f00e2b2a9069583833545883e0e5968a33d2455dad59e6fb47c1102b4dd16ef932e9ba945e29e9d941e6c17bd531a02c66b0491097801be6bda476875537
2022-11-28lint: Skip COMMIT_RANGE if no CIRRUS_PRMarcoFalke
2022-11-28Merge bitcoin/bitcoin#26574: ci: use ci_exec_root for clang installMarcoFalke
54dd8f51ce8815a93a6aceb36dd72bd449d6b327 ci: use ci_exec_root for clang install (josibake) Pull request description: fixes a bug introduced in #25900 ; see https://github.com/bitcoin/bitcoin/pull/25900#issuecomment-1327311069 the general idea of #25900 was to use a non-root user as much as possible to avoid modifying the user's local filesystem. however, it appears the root user is needed to correctly install clang. ACKs for top commit: hebasto: ACK 54dd8f51ce8815a93a6aceb36dd72bd449d6b327, tested on Ubuntu 22.04. Tree-SHA512: beb01d4b6127fbba3c8d18e85cf7ec7d1b2ec93ea05c475ab51bcaa04ef1b0591d886f1a7e0732c5ae86806013f022c0b44027380d2b0cfb1bfdc843e40f99b4
2022-11-25ci: use ci_exec_root for clang installjosibake
2022-11-23fuzz: Move-only net utilsMarcoFalke
2022-11-22Merge bitcoin/bitcoin#25900: ci: run docker wrapper with a non-root userMarcoFalke
849f20a6d3e437631a07469d3c4af5faa0aa06ed ci: create and use non-root user for docker image (josibake) Pull request description: Previously, everything in the ci docker image ran as the root user. This would lead to certain directories (`ci/scratch`, `depends`) being owned by `root` after running the ci locally which would lead to annoying behavior such as subsequent guix builds failing due to `depends/` being owned by root. This PR adds a non-root user in the container and chowns the mounted working directory. All the `docker exec` commands now run as the non-root user, except for the few that still need to run as root (mainly, installing packages). To test this I checked out a fresh copy of the repo, applied my changes, ran the CI, and verified all the local file permissions were unchanged after the CI was finished running. ACKs for top commit: hebasto: ACK 849f20a6d3e437631a07469d3c4af5faa0aa06ed, tested on Ubuntu 22.04 by running commands as follows: Tree-SHA512: 734dca0f36157fce5fab243b4ff657fc17ba980e8e4e4644305f41002ff21bd5cef02c306ea1e0b5c841d4c07c095e8e4be16722e6a38c890717c60a3f5ec62a
2022-11-22Merge bitcoin/bitcoin#26292: util: move threadinterrupt into util/fanquake
b89530483d39f6a6a777df590b87ba2fad8c8b60 util: move threadinterrupt into util (fanquake) Pull request description: Alongside thread and threadnames. It's part of libbitcoin_util. ACKs for top commit: ryanofsky: Code review ACK b89530483d39f6a6a777df590b87ba2fad8c8b60. No changes since last review other than rebase theuni: ACK b89530483d39f6a6a777df590b87ba2fad8c8b60. Tree-SHA512: 0421f4d1881ec295272446804b27d16bf63e6b62b272f8bb52bfecde9ae6605e8109ed16294690d3e3ce4b15cc5e7c4046f99442df73adb10bdf069d3fb165aa
2022-11-21ci: create and use non-root user for docker imagejosibake
Running all commands as the root user in the docker image will change local file permissions in the ci and depends directory. Add a non-root user to the container and use this user whenever possible when running docker exec commands.
2022-11-16util: Move error message formatting of NonFatalCheckError to cppMacroFake
This allows to strip down the header file
2022-11-01util: move threadinterrupt into utilfanquake
2022-10-31refactor: move url.h/cpp from lib util to lib commonfanquake
2022-10-27Merge bitcoin/bitcoin#26388: ci: Use `macos-ventura-xcode:14.1` image for ↵MacroFake
"macOS native" task da168934741b776bce07d5503ca2344d300723b3 ci: Use `macos-ventura-xcode:14.1` image for "macOS native" task (Hennadii Stepanov) 702836530ffa351e863b1b1300fd2e559a14ef23 ci: Make `getopt` path architecture agnostic (Hennadii Stepanov) Pull request description: The "macOS native" CI task always uses the recent OS image. This PR updates it up to the recent macOS release. Cirrus Labs [stopped](https://github.com/bitcoin/bitcoin/pull/25160#issuecomment-1162829773) updating macOS images for `x86_64`, therefore, an `arm64` image been used. Also `make test-security-check` has been dropped as it ["isn't even expected to pass"](https://github.com/bitcoin/bitcoin/issues/26386#issuecomment-1290318628) on `arm64` in CI. ACKs for top commit: Sjors: utACK da16893 Tree-SHA512: 36785d33b7f11b3cdbc53bcfbf97d88bf821fad248c825982dd9f8e3413809a4ef11190eaf950e60fdf479b62ff66920c35d9ea42d534723f015742eec7e19b6
2022-10-25ci: Use `macos-ventura-xcode:14.1` image for "macOS native" taskHennadii Stepanov
2022-10-25ci: Make `getopt` path architecture agnosticHennadii Stepanov
2022-10-18iwyu: Add missing includesMacroFake
2022-10-10Merge bitcoin/bitcoin#26254: iwyu: Add zmq source filesMacroFake
13afcc0cd4c2975852924d2d9be5e96096147716 iwyu: Add zmq source files (Hennadii Stepanov) Pull request description: ACKs for top commit: fanquake: ACK 13afcc0cd4c2975852924d2d9be5e96096147716 Tree-SHA512: 7af95e991fc2782aeba2edfef0a2f75f9c361058295586adb062087aa31c47cfcce2425aee9dd5153e18e018cf1f9272c9617c671b7262db55f241526c3fcb15
2022-10-10iwyu: Add zmq source filesHennadii Stepanov
2022-10-05Remove clang-format from lint taskMacroFake
clang-format could be used in scripted diffs, but remained largely unused.
2022-10-04Merge bitcoin/bitcoin#26234: ci: Allow PIP_PACKAGES on centosfanquake
fa6054e952f4522b98dc89609033950a3cbfd06c ci: Allow PIP_PACKAGES on centos (MacroFake) fac085a05cc518b14271353128bb1fa830b3c612 ci: Remove unused package (MacroFake) Pull request description: This was added in 7fc5e865b93af59364e9c8bf75ec68b4decc7e5d but I can't see a reason why this should be forbidden. This is also needed for other changes (bumping the minimum python version). ACKs for top commit: hebasto: re-ACK fa6054e952f4522b98dc89609033950a3cbfd06c Tree-SHA512: e8ead9ee00079024eb1e8c6e7b31c78cf2a3392159b444765c2ea9a58bed2a7550bf71083210692a45bb8ed7896cb882b72bf70baa13a2384864b2b510b73005
2022-10-04ci: Allow PIP_PACKAGES on centosMacroFake
This was added in 7fc5e865b93af59364e9c8bf75ec68b4decc7e5d but I can't see a reason why this should be forbidden.
2022-10-04ci: Remove unused packageMacroFake
Address feedback from https://github.com/bitcoin/bitcoin/pull/24561/files#r985719812
2022-10-04ci: Move `git config` commands into script where they are usedHennadii Stepanov
2022-09-28ci: Use git2.34 for lint taskMacroFake
2022-09-15Bump codespell version to 2.2.1Jon Atack
2022-08-20Fix iwyuMacroFake
2022-08-05Merge bitcoin/bitcoin#25650: script: default to necessary tags in ↵MacroFake
`test/get_previous_releases.py` 21a9e94dbbc47939f19fbfe39d85cb8ff2ea206a ci: remove hardcoded tag list from ci scripts (josibake) d530ba390e7703433d3a77673c9a2b52bc446901 doc: update test/README.md (josibake) 614d4682badaadac74b825a45aaee9c2309a3e81 script: default to necessary tags in get_previous_releases.py (josibake) Pull request description: Almost every time I need to use this script, I forget the tag list is needed and that a specific set of tags is needed for the backwards compatibility tests to work. I end up wasting time reading through the script and googling to find the tag list before remembering it is in `test/README.md` I assume (hope) I'm not the only one this happens to, so I figured it would make more sense to have the script default to downloading/building the necessary tags. This has the added benefit of making the script the source of truth: the script already needs to be updated with the SHA256_SUM of the binary for every new tag that is added, so it makes sense to use `SHA256_SUMS` list as the necessary tag list. This means there is less risk of the README and the script drifting (i.e updating the readme with a new tag and forgetting to update the script, or updating the script and forgetting to update the README). Now all that needs to happen is to update the `SHA256_SUMS` list in the script and everything Just Works (TM) ACKs for top commit: Sjors: re-tACK 21a9e94dbbc47939f19fbfe39d85cb8ff2ea206a Tree-SHA512: 97b488227a89a6827584edd251820a7074fad75dfd7f26f1aa5f858e2521d2e02effd0f11e6dc4676e1155d3d5aba6ff94a4b58ffef80dc201376afd5927deb9
2022-08-03validationcaches: Add and use ValidationCacheSizesCarl Dong
Also: - Make DEFAULT_MAX_SIG_CACHE_SIZE into constexpr DEFAULT_MAX_SIG_CACHE_BYTES to utilize the compile-time integer arithmetic overflow checking available to constexpr. - Fix comment (MiB instead of MB) for DEFAULT_MAX_SIG_CACHE_BYTES. - Pass in max_size_bytes parameter to InitS*Cache(), modify log line to no longer allude to maxsigcachesize being split evenly between the two validation caches. - Fix possible integer truncation and add a comment. [META] I've kept the integer types as int64_t in order to not introduce unintended behaviour changes, in the next commit we will make them size_t.
2022-08-02sort after scripted-diffMacroFake
2022-08-02scripted-diff: Move mempool_args to src/nodeMacroFake
It is part of the node library. Also, it won't be moved to the kernel lib, as it will be pruned of ArgsManager. -BEGIN VERIFY SCRIPT- # Move module git mv src/mempool_args.cpp src/node/ git mv src/mempool_args.h src/node/ # Replacements sed -i 's:mempool_args\.h:node/mempool_args.h:g' $(git grep -l mempool_args) sed -i 's:mempool_args\.cpp:node/mempool_args.cpp:g' $(git grep -l mempool_args) sed -i 's:MEMPOOL_ARGS_H:NODE_MEMPOOL_ARGS_H:g' $(git grep -l MEMPOOL_ARGS_H) -END VERIFY SCRIPT-
2022-08-02Return optional error from ApplyArgsManOptionsMacroFake
Also pass in a (for now unused) reference to the params. Both changes are needed for the next commit.
2022-08-01Merge bitcoin/bitcoin#25528: ci: run USDT interface tests in the CIMacroFake
cc7335edc87c6ef34429b4df94f53973db520aac ci: run USDT interface test in a VM (0xb10c) dba6f8234217565957e37516a0ea655f1180d99c test: adopt USDT utxocache interface tests (0xb10c) 220a5a2841172a07d6d7849596316f0e0933e272 test: hook into PID in tracing tests (0xb10c) Pull request description: Changes a CI task that runs test the previously not run `test/functional/interface_usdt_*.py` functional tests (added in https://github.com/bitcoin/bitcoin/pull/24358). This task is run as CirussCI `compute_engine_instance` VM as hooking into the tracepoints is not possible in CirrusCI docker containers (https://github.com/bitcoin/bitcoin/issues/23296#issuecomment-1024920845). We use an unoffical PPA and untrusted `bpfcc-tools` package in the CI as the Ubuntu jammy and Debian bullseye packages are outdated. We hope use an official package when new Ubuntu/Debian releases are available for the use with Google Compute Engine. We make sure to hook into `bitcoind` binaries in USDT interface tests via their PID, instead of their path. This makes sure multiple functional tests running in parallel don't interfere with each other. The utxocache USDT interface tests is adopted to a change of the functional test framework that wasn't detected as the tests weren't run in the CI. As the tracepoints expose internals, it can happen that we need to adopt the interface test when internals change. This is a bit awkward, and if it happens to frequently, we should consider generalizing the tests a bit more. For now it's fine, I think. See the individual commit messages for more details on the changes. Fixes https://github.com/bitcoin/bitcoin/issues/24782 Fixes https://github.com/bitcoin/bitcoin/issues/23296 I'd like to hear from reviewers: - Are we OK with using the [`hadret/bpfcc`](https://launchpad.net/~hadret/+archive/ubuntu/bpfcc) PPA for now? There is a clear plan when to drop it and as is currently, it could only impact the newly added VM task. - ~~Adding a new task increases CI runtime and costs. Should an existing `container` CI task be ported to a VM and reused instead?~~ Yes, see https://github.com/bitcoin/bitcoin/pull/25528#issuecomment-1179509525 ACKs for top commit: MarcoFalke: cr ACK cc7335edc87c6ef34429b4df94f53973db520aac Tree-SHA512: b7fddccc0a77d82371229d048abe0bf2c4ccaa45906497ef3040cf99e7f05561890aef4c253c40e4afc96bb838c9787fae81c8454c6fd9db583276e005a4ccb3
2022-07-29tidy: run clang-tidy in quiet modefanquake
2022-07-26Merge bitcoin/bitcoin#24974: refactor: Make FEELER_SLEEP_WINDOW type safe ↵fanquake
(std::chrono) fa74e726c414f5f7a1e63126a69463491f66e0ec refactor: Make FEELER_SLEEP_WINDOW type safe (std::chrono) (MacroFake) fa3b3cb9b5d944d34b1d5ac3e102ac333482a475 Expose underlying clock in CThreadInterrupt (MacroFake) Pull request description: This gets rid of the `value*1000` manual conversion. ACKs for top commit: naumenkogs: utACK fa74e726c414f5f7a1e63126a69463491f66e0ec dergoegge: Code review ACK fa74e726c414f5f7a1e63126a69463491f66e0ec Tree-SHA512: 90409c05c25f0dd2f1c4dead78f707ebfd78b7d84ea4db9fcefd9c4958a1a3338ac657cd9e99eb8b47d52d4485fa3c947dce4ee1559fb56ae65878685e1ed9a3
2022-07-25refactor: Make CTransaction constructor explicitMacroFake
It involves calculating two hashes, so the performance impact should be made explicit. Also, add the module to iwyu.
2022-07-21ci: remove hardcoded tag list from ci scriptsjosibake
2022-07-21ci: better pin to dwarf4 in valgrind jobfanquake
Use `-gdwarf` and also set CFLAGS. I was seeing Valgrind issues otherwise.
2022-07-19ci: Enable IWYU in src/kernel directoryRyan Ofsky
Suggested https://github.com/bitcoin/bitcoin/pull/25308#discussion_r892505713
2022-07-19Remove unused includes from dbwrapper.hMacroFake
2022-07-19Add missing includes to node/chainstateMacroFake
This is needed for the next commit
2022-07-18Merge bitcoin/bitcoin#25487: [kernel 3b/n] Decouple `{Dump,Load}Mempool` ↵glozow
from `ArgsManager` cb3e9a1e3f8d72daaa361fc45dd853775e754b9d Move {Load,Dump}Mempool to kernel namespace (Carl Dong) aa306765419f7dbea12b12e15553039835ba0e4d Move DEFAULT_PERSIST_MEMPOOL out of libbitcoinkernel (Carl Dong) 06b88ffb8ae7f2b2a93a32908cd80e77fafd270c LoadMempool: Pass in load_path, stop using gArgs (Carl Dong) b857ac60d9a0433036519c26675378bbf56a1de1 test/fuzz: Invoke LoadMempool via CChainState (Carl Dong) b3267258b052557fc136b9a4dcb754afb9219470 Move FopenFn to fsbridge namespace (Carl Dong) ae1e8e37567fa603a5977d7d05105c682dd3f7db mempool: Use NodeClock+friends for LoadMempool (Carl Dong) f9e8e5719f28d84f68f7d75e26c8e7fccac8e7d3 mempool: Improve comments for [GS]etLoadTried (Carl Dong) 813962da0b17b918941c6849996845e35d84a451 scripted-diff: Rename m_is_loaded -> m_load_tried (Carl Dong) 413f4bb52b72e082ad8716664ede48352b8e7e5a DumpMempool: Pass in dump_path, stop using gArgs (Carl Dong) bd4407817e523e3c5b347bc6be25ed007cb27034 DumpMempool: Use std::chrono instead of weird int64_t arthmetics (Carl Dong) c84390b741ab7b61c9f702d8b447c8cadc1257c8 test/mempool_persist: Test manual savemempool when -persistmempool=0 (Carl Dong) Pull request description: This is part of the `libbitcoinkernel` project: #24303, https://github.com/bitcoin/bitcoin/projects/18 ----- This PR moves `{Dump,Load}Mempool` into its own `kernel/mempool_persist` module and introduces `ArgsManager` `node::` helpers in `node/mempool_persist_args`to remove the scattered calls to `GetBoolArg("-persistmempool", DEFAULT_PERSIST_MEMPOOL)`. More context can be gleaned from the commit messages. ----- One thing I was reflecting on as I wrote this was that in the long run, I think we should probably invert the validation <-> mempool relationship. Instead of mempool not depending on validation, it might make more sense to have validation not depend on mempool. Not super urgent since `libbitcoinkernel` will include both validation and mempool, but perhaps something for the future. ACKs for top commit: glozow: re ACK cb3e9a1e3f8d72daaa361fc45dd853775e754b9d via `git range-diff 7ae032e...cb3e9a1` MarcoFalke: ACK cb3e9a1e3f 🔒 ryanofsky: Code review ACK cb3e9a1e3f8d72daaa361fc45dd853775e754b9d Tree-SHA512: 979d7237c3abb5a1dd9b5ad3dbf3b954f906a6d8320ed7b923557f41a4472deccae3e8a6bca0018c8e7a3c4a93afecc502acd1e26756f2054f157f1c0edd939d
2022-07-18refactor: integrate no_nul into univalue unitesterfanquake
2022-07-15Move {Load,Dump}Mempool to kernel namespaceCarl Dong
Also: 1. Add the newly introduced kernel/mempool_persist.cpp to IWYU CI script 2. Add chrono mapping for iwyu
2022-07-13Expose underlying clock in CThreadInterruptMacroFake
Overloading sleep_for is not needed, as * seconds and minutes can be converted to milliseconds by the compiler, not needing a duration_cast * std::condition_variable::wait_for will convert milliseconds to the duration type of the underlying clock So simply expose the clock.
2022-07-08ci: run USDT interface test in a VM0xb10c
Our CI tasks are run by CirrusCI in Docker containers in a Google Compute Engine based Kubernetes environment. These containers have limited capabilities - especially CAP_SYS_ADMIN is missing. See https://github.com/bitcoin/bitcoin/issues/23296#issuecomment-1024920845 We need elevated privileges to hook into the USDT tracepoints. We use a CirrusCI "compute_engine_instance" (a VM, not a container) where we have the required privileges. The ubunut-mininmal-2204-lts was choosen with debian-11 being an alternative. Both pack an outdated 'bpfcc-tools' package (v0.18.0) from 2020. This version prints warnings to stderr during BPF bytecode compilation, which causes our functional test runner to fail. This is fixed in newer verison. Until debian-12 or a newer Ubuntu release is avaliable as image in GCE (https://cloud.google.com/compute/docs/images/os-details), we use a third-party and untrusted PPA that releases up-to-date versions of the package. The official iovisor (authors of BCC) PPA is outdated too. An alternative would be to compile BCC from source in the CI. Co-authored-by: MacroFake <falke.marco@gmail.com>
2022-07-08refactor: add most of src/util to iwyufanquake
These files change infrequently, and not much header shuffling is required. We don't add everything in src/util/ yet, because IWYU makes some dubious suggestions, which I'm going to follow up with upstream.
2022-07-07Merge bitcoin/bitcoin#25447: fuzz: add low-level target for txorphanageMacroFake
6eb0909cb7d5883a258f76ad6cf2c989fc6f892f fuzz: add low-level target for txorphanage (chinggg) Pull request description: This adds a low-level fuzz target for orphan transaction handling by creating random transactions and calling all functions in `TxOrphanage`. It cannot simulate real-world `orphan/unorphan` scenarios effectively since it does not maintain any state about the node and the chain. A high-level fuzz target which construct well-designed transaction graphs will be added later. ACKs for top commit: MarcoFalke: review ACK 6eb0909cb7d5883a258f76ad6cf2c989fc6f892f 🐈 Tree-SHA512: b4d64f5941df77d13981f75ec170cef6ffabe782797c982ede7f34134be01dc0026dd7c0bee614bc1d64715e90a933d2a8c95974d402e32eaba8e24cc928299e
2022-07-06fuzz: add low-level target for txorphanagechinggg
2022-07-04[ci] apply cache size limit and print ccache statistics in "ARM64 Android APK"sogoagain
generated by cgit v1.2.3 (git 2.26.2) at 2024-09-13 07:50:00 +0000