aboutsummaryrefslogtreecommitdiff
path: root/tests/docker/docker.py
AgeCommit message (Collapse)Author
2022-09-20tests/docker: remove FROM qemu/ support from docker.pyAlex Bennée
We want to migrate from docker.py to building our images directly with docker/podman. Before we get there we need to make sure we don't re-introduce our layered builds so bug out if we see FROM qemu/ in a Dockerfile. Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Acked-by: Thomas Huth <thuth@redhat.com> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Message-Id: <20220914155950.804707-30-alex.bennee@linaro.org>
2022-04-20tests/docker: remove SKIP_DOCKER_BUILDPaolo Bonzini
It is now unused. Signed-off-by: Paolo Bonzini <pbonzini@redhat.com> Message-Id: <20220401141326.1244422-17-pbonzini@redhat.com> Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Message-Id: <20220419091020.3008144-20-alex.bennee@linaro.org>
2021-07-14tests/docker: don't use BUILDKIT in GitLab eitherDaniel P. Berrangé
Using BUILDKIT breaks with certain container registries such as CentOS, with docker build reporting an error such as failed to solve with frontend dockerfile.v0: failed to build LLB: failed to load cache key: unexpected status code https://registry.centos.org/v2/centos/manifests/7: 403 Forbidden Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Reviewed-by: Willian Rampazzo <willianr@redhat.com> Message-Id: <20210623142245.307776-3-berrange@redhat.com> Message-Id: <20210709143005.1554-11-alex.bennee@linaro.org>
2021-05-18tests/docker: add "fetch" sub-commandAlex Bennée
This simply wraps up fetching a build from the registry and tagging it as the local build. Reviewed-by: Willian Rampazzo <willianr@redhat.com> Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Message-Id: <20210512102051.12134-6-alex.bennee@linaro.org>
2021-05-18tests/docker: allow "update" to add the current userAlex Bennée
The current user functionality is used for cross compiling to avoid complications with permissions when building test programs. However for images that come from the registry we still need the ability to add the user after the fact. Reviewed-by: Willian Rampazzo <willianr@redhat.com> Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Message-Id: <20210512102051.12134-5-alex.bennee@linaro.org>
2021-05-18tests/docker: make executable an optional argument to "update"Alex Bennée
We're going to extend the abilities of the command shortly. Reviewed-by: Willian Rampazzo <willianr@redhat.com> Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Message-Id: <20210512102051.12134-4-alex.bennee@linaro.org>
2021-05-18tests/docker: fix copying of executable in "update"Alex Bennée
We have the same symlink chasing problem when doing an "update" operation. Fix that. Based-on: 5e33f7fead ("tests/docker: better handle symlinked libs") Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Reviewed-by: Willian Rampazzo <willianr@redhat.com> Message-Id: <20210512102051.12134-3-alex.bennee@linaro.org>
2021-02-08tests/docker: add a docker-exec-copy-testAlex Bennée
This provides test machinery for checking the QEMU copying logic works properly. It takes considerably less time to run than starting a debootstrap only for it to fail later. I considered adding a remove command to docker.py but figured that might be gold plating given the relative size of the containers compared to the ones with actual stuff in them. Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Message-Id: <20210202134001.25738-7-alex.bennee@linaro.org>
2021-02-08tests/docker: preserve original name when copying libsAlex Bennée
While it is important we chase down the symlinks to copy the correct data we can confuse the kernel by renaming the interpreter to what is in the binary. Extend _copy_with_mkdir to preserve the original name of the file when asked. Fixes: 5e33f7fead ("tests/docker: better handle symlinked libs") Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Message-Id: <20210202134001.25738-5-alex.bennee@linaro.org>
2021-02-08tests/docker: make _copy_with_mkdir accept missing filesAlex Bennée
Depending on the linker/ldd setup we might get a file with no path. Typically this is the psuedo library linux-vdso.so which doesn't actually exist on the disk. Rather than try and catch these distro specific edge cases just shout about it and try and continue. Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org> Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org> Message-Id: <20210202134001.25738-4-alex.bennee@linaro.org>
2021-02-08tests/docker: Fix _get_so_libs() for docker-binfmt-imagePhilippe Mathieu-Daudé
Fix a variable rename mistake from commit 5e33f7fead5: Traceback (most recent call last): File "./tests/docker/docker.py", line 710, in <module> sys.exit(main()) File "./tests/docker/docker.py", line 706, in main return args.cmdobj.run(args, argv) File "./tests/docker/docker.py", line 489, in run _copy_binary_with_libs(args.include_executable, File "./tests/docker/docker.py", line 149, in _copy_binary_with_libs libs = _get_so_libs(src) File "./tests/docker/docker.py", line 123, in _get_so_libs libs.append(s.group(1)) NameError: name 's' is not defined Fixes: 5e33f7fead5 ("tests/docker: better handle symlinked libs") Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org> Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Message-Id: <20210119050149.516910-1-f4bug@amsat.org> Message-Id: <20210202134001.25738-2-alex.bennee@linaro.org>
2021-01-18Add newline when generating DockerfileAlessandro Di Federico
Signed-off-by: Alessandro Di Federico <ale@rev.ng> Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Reviewed-by: Willian Rampazzo <willianr@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com> Message-Id: <1610080146-14968-36-git-send-email-tsimpson@quicinc.com> Message-Id: <20210114165730.31607-5-alex.bennee@linaro.org>
2020-09-18docker.py: always use --rmPaolo Bonzini
Avoid that containers pile up. Signed-off-by: Paolo Bonzini <pbonzini@redhat.com> Acked-by: Philippe Mathieu-Daudé <f4bug@amsat.org> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
2020-07-27tests/docker: fix update command due to python3 str/bytes distinctionAlex Bennée
Does this seem convoluted to you? It feels a little complicated to me. Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Message-Id: <20200724064509.331-10-alex.bennee@linaro.org>
2020-07-15docker.py: fix fetching of FROM layersAlex Bennée
This worked on a system that was already bootstrapped because the stage 2 images already existed even if they wouldn't be used. What we should have pulled down was the FROM line containers first because building on gitlab doesn't have the advantage of using our build system to build the pre-requisite bits. We still pull the image we want to build just in case we can use the cached data. Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org> Message-Id: <20200713200415.26214-4-alex.bennee@linaro.org>
2020-07-11tests/docker: fall back more gracefully when pull failsAlex Bennée
I only spotted this in the small window between my testing with my registry while waiting for the gitlab PR to go in. As we pre-pull the registry image we know if that fails there isn't any point attempting to use the cache. Fall back to the way we used to do it at that point. Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org> Message-Id: <20200709141327.14631-12-alex.bennee@linaro.org>
2020-07-11tests/docker: add --registry support to toolingAlex Bennée
This allows us to point the tools towards a registry from which they can grab pre-built layers instead of doing everything from scratch each time. To enable this we need to be using the DOCKER_BUILDKIT engine. Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Message-Id: <20200701135652.1366-25-alex.bennee@linaro.org>
2020-07-11tests/docker: change tag naming scheme of our imagesAlex Bennée
We've been misusing the tag naming scheme for some time by overloading the post : section with the image type. Really it should be saved for the revision of that particular build. Move the details to the other side so we have: qemu/image-name with the implied :latest version added by the tooling. Suggested-by: Daniel P. Berrangé <berrange@redhat.com> Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com> Message-Id: <20200701135652.1366-18-alex.bennee@linaro.org>
2020-06-10docker.py/build: support binary files in --extra-filesPaolo Bonzini
Read the --extra-files in binary mode to avoid encoding errors. Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2020-06-10docker.py/build: support -t and -f argumentsPaolo Bonzini
The docker.py command line is subtly different from docker and podman's, in that the tag and Dockerfile are passed via positional arguments. Remove this gratuitous difference and just parse -f and -t. -f was previously used by --extra-files, only keep the long option. Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2020-05-31python: remove more instances of sys.version_infoJohn Snow
We guarantee 3.5+ everywhere; remove more dead checks. In general, try to avoid using version checks and instead prefer to attempt behavior when possible. Signed-off-by: John Snow <jsnow@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com> Message-Id: <20200514035230.25756-1-jsnow@redhat.com> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
2020-02-04tests/docker: better handle symlinked libsAlex Bennée
When we are copying we want to ensure we grab the first resolution (the found in path section). However even that binary might be a symlink so lets make sure we chase the symlinks to copy the right binary to where it can be found. Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com> Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com> Reviewed-by: Robert Foley <robert.foley@linaro.org> Message-Id: <20200203090932.19147-3-alex.bennee@linaro.org>
2019-09-26podman: fix command invocationJohn Snow
Oops; there's no argv here. Signed-off-by: John Snow <jsnow@redhat.com> Message-Id: <20190913193821.17756-1-jsnow@redhat.com> Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Reviewed-by: Cleber Rosa <crosa@redhat.com> Tested-by: Cleber Rosa <crosa@redhat.com> Reviewed-by: Thomas Huth <thuth@redhat.com> Tested-by: Thomas Huth <thuth@redhat.com>
2019-09-26tests/docker: reduce scary warnings by cleaning up clean upAlex Bennée
There was in the clean-up code caused by attempting to inspect images which finished before we got there. Clean up the clean up code by: - only track the one instance at a time - use --filter for docker ps instead of doing it by hand - just call docker rm -f to be done with it - use uuid.uuid4() for a random uid Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
2019-09-10tests/docker: fix final missing .encode when parsing solibsAlex Bennée
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
2019-09-10tests/docker: handle missing encoding keyword for subprocess.check_outputAlex Bennée
This was only added in Python 3.6 and not all the build hosts have that recent a python3. However we still need to ensure everything is returns as a unicode string so checks higher up the call chain don't barf. Signed-off-by: Alex Bennée <alex.bennee@linaro.org> fixup! tests/docker: handle missing encoding keyword for subprocess.check_output
2019-09-10tests/docker: fix "cc" command to work with podmanAlex Bennée
Podman requires a little bit of additional magic to the uid mapping which was already done for the normal RunCommand. We simplify the logic by pushing it directly into the Docker::run method to avoid instantiating an extra Docker() object and ensure the CC command always runs as the current user. Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
2019-09-10tests/docker: Use --userns=keep-id for podmanJohn Snow
The workaround that attempts to accomplish the same result as --userns=keep-id does not appear to work well with UIDs much above 1000 (like mine, which is above 20000.) Since we have official support for this "trick" now, use the supported method. Signed-off-by: John Snow <jsnow@redhat.com> Message-Id: <20190904232451.26466-1-jsnow@redhat.com> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
2019-09-10configure: clean-up container cross compile detectAlex Bennée
The introduction of podman support inadvertently broke configure's detect of the container support as the configure probe didn't specify an engine type. To fix this in docker.py: - only (re)set USE_ENGINE if --engine is specified - enhance the output so docker is no longer just yes In the configure script we can at least start cleaning up the detecting and naming of variables. To avoid too much churn the conversion of the various make DOCKER_foo variables has been left for future clean-ups. Fixes: 9459f754134b Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Tested-by: David Hildenbrand <david@redhat.com>
2019-09-02tests/docker: upgrade docker.py to python3Alex Bennée
The recent podman changes (9459f754134bb) imported enum which is part of the python3 standard library but only available as an external library for python2. This causes problems on the fairly restricted environment such as shippable. Lets bite the bullet and make the script a fully python3 one. To that end: - drop the from __future__ import (we are there now ;-) - avoid the StringIO import hack - be consistent with the mode we read/write dockerfiles - s/iteritems/items/ - ensure check_output returns strings for processing Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com> Cc: Marc-André Lureau <marcandre.lureau@redhat.com>
2019-08-22docker.py: add podman supportMarc-André Lureau
Add a --engine option to select either docker, podman or auto. Among other advantages, podman allows to run rootless & daemonless containers, fortunately sharing compatible CLI with docker. With current podman, we have to use a uidmap trick in order to be able to rw-share the ccache directory with the container user. With a user 1000, the default mapping is: 1000 (host) -> 0 (container). So write access to /var/tmp/ccache ends will end with permission denied error. With "--uidmap 1000:0:1 --uidmap 0:1:1000", the mapping is: 1000 (host) -> 0 (container, 1st namespace) -> 1000 (container, 2nd namespace). (the rest is mumbo jumbo to avoid holes in the range of UIDs) A future podman version may have an option such as --userns-keep-uid. Thanks to Debarshi Ray <rishi@redhat.com> for the help! Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com> Acked-by: Alex Bennée <alex.bennee@linaro.org> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
2019-08-22docker.py: add --run-as-current-userMarc-André Lureau
(podman will need further tweaks) Suggested-by: Paolo Bonzini <pbonzini@redhat.com> Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com> Reviewed-by: Alex Bennée <alex.bennee@linaro.org> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
2019-02-08tests: PEP8 cleanup of docker.py, mostly white spaceAlex Bennée
My editor keeps putting squiggly lines under a bunch of the python lines to remind me how non-PEP8 compliant it is. Clean that up so it's easier to spot new errors. Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
2019-02-08tests: docker.py be even smarter with persistent binfmt_miscAlex Bennée
If we have a persistent mapping we don't need the QEMU binary copied into the container as the kernel has already opened the file and will pass the fd in. However the support libraries will still need to be there. Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
2019-02-08tests: make docker.py check for persistent configsAlex Bennée
binfmt_misc configured with the "F" flag opens the interpreter at config time. This means it can use an already open file-descriptor to run QEMU so there is no point trying to copy the binary into a container. Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
2019-02-08tests: make docker.py update use configured binfmt pathAlex Bennée
When copying a QEMU binary into a linux-user docker image we should check what the current configured binfmt_misc path is rather than just assuming "/usr/bin/qemu-bin". Obviously if the user changes the configuration afterwards they will break their images again. Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
2018-10-26tests: Fix typos in comments and help message (found by codespell)Stefan Weil
Fix also a grammar issue. Signed-off-by: Stefan Weil <sw@weilnetz.de> Reviewed-by: Alex Bennée <alex.bennee@linaro.org> Message-Id: <20180713054755.23323-1-sw@weilnetz.de> Signed-off-by: Laurent Vivier <laurent@vivier.eu>
2018-10-03Revert "docker.py: Python 2.6 argparse compatibility"Eduardo Habkost
This reverts commit c2d3189667409561772e8c1e5615c5166cd8aa2c. scripts/argparse.py was removed from the tree, so we don't need this hack anymore. Signed-off-by: Eduardo Habkost <ehabkost@redhat.com> Message-Id: <20180618225131.13113-3-ehabkost@redhat.com> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com> Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
2018-07-24docker: perform basic binfmt_misc validation in docker.pyAlex Bennée
Setting up binfmt_misc is outside of the scope of the docker.py script but we can at least validate it with any given executable so we have a more useful error message than the sed line of deboostrap failing cryptically. Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Reported-by: Richard Henderson <richard.henderson@linaro.org>
2018-07-24docker: fail more gracefully on docker.py checkAlex Bennée
As this is called directly from the Makefile while determining dependencies and it is possible the user was configured in one window but not have credentials in the other. Let's catch the Exceptions and deal with it quietly. Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Reported-by: Peter Maydell <peter.maydell@linaro.org> Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
2018-07-05docker: add special handling for FROM:debian-%-user targetsAlex Bennée
These will have been build with debootstrap so we need to check against the debian-bootstrap dockerfile. This does mean sticking to debian-FOO-user as the naming conventions for boot-strapped images. The actual cross image is built on top. Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org> Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
2018-06-20docker: docker.py adding age check commandAlex Bennée
This is useful for querying if an image is too old. Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
2018-06-20docker: docker.py add check sub-commandAlex Bennée
This command allows you to check if we need to re-build a docker image. If the image isn't in the repository or the checksums don't match then we return false and some text (for processing in makefiles). Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
2018-06-20docker: docker.py don't conflate checksums for extra_filesAlex Bennée
This just gets confusing especially as the helper function doesn't even take into account any extra files (or the executable). Currently the actual check just ignores them and also passes the result through _dockerfile_preprocess so we fix that too. Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
2018-06-20docker: docker.py use "version" to probe usageAlex Bennée
The "images" command is a fairly heavyweight command to run as it involves searching the whole docker file-system inventory. On a machine with a lot of images this makes start-up fairly expensive. Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
2018-06-20docker: allow "cc" command to run in user contextAlex Bennée
Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Reviewed-by: Fam Zheng <famz@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org> Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
2018-06-20docker: extend "cc" command to accept compilerAlex Bennée
When calling our cross-compilation images we want to call something other than the default cc. Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org> Reviewed-by: Fam Zheng <famz@redhat.com>
2018-06-20docker: Add "cc" subcommandAlex Bennée
Signed-off-by: Fam Zheng <famz@redhat.com> [AJB: add if args.paths check] Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org> Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
2018-06-20docker: docker.py wrap StringIO import for python3Alex Bennée
Although the docker.py is nominally python2 we actually invoke it with the configured python from the configure script. Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
2018-06-08python: futurize -f libfuturize.fixes.fix_print_with_importEduardo Habkost
Change all Python code to use print as a function. This is necessary for Python 3 compatibility. Done using: $ py=$( (g grep -l -E '^#!.*python';find -name '*.py' -printf '%P\n';) | \ sort -u | grep -v README.sh4) $ futurize -w -f libfuturize.fixes.fix_print_with_import $py Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com> Acked-by: Fam Zheng <famz@redhat.com> Signed-off-by: Eduardo Habkost <ehabkost@redhat.com> Message-Id: <20180608122952.2009-2-ehabkost@redhat.com> [ehabkost: fixup tests/docker/docker.py] Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
generated by cgit v1.2.3 (git 2.26.2) at 2025-03-16 23:02:30 +0000