aboutsummaryrefslogtreecommitdiff
path: root/tests/docker/docker.py
AgeCommit message (Collapse)Author
2021-01-18Add newline when generating DockerfileAlessandro Di Federico
Signed-off-by: Alessandro Di Federico <ale@rev.ng> Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Reviewed-by: Willian Rampazzo <willianr@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com> Message-Id: <1610080146-14968-36-git-send-email-tsimpson@quicinc.com> Message-Id: <20210114165730.31607-5-alex.bennee@linaro.org>
2020-09-18docker.py: always use --rmPaolo Bonzini
Avoid that containers pile up. Signed-off-by: Paolo Bonzini <pbonzini@redhat.com> Acked-by: Philippe Mathieu-Daudé <f4bug@amsat.org> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
2020-07-27tests/docker: fix update command due to python3 str/bytes distinctionAlex Bennée
Does this seem convoluted to you? It feels a little complicated to me. Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Message-Id: <20200724064509.331-10-alex.bennee@linaro.org>
2020-07-15docker.py: fix fetching of FROM layersAlex Bennée
This worked on a system that was already bootstrapped because the stage 2 images already existed even if they wouldn't be used. What we should have pulled down was the FROM line containers first because building on gitlab doesn't have the advantage of using our build system to build the pre-requisite bits. We still pull the image we want to build just in case we can use the cached data. Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org> Message-Id: <20200713200415.26214-4-alex.bennee@linaro.org>
2020-07-11tests/docker: fall back more gracefully when pull failsAlex Bennée
I only spotted this in the small window between my testing with my registry while waiting for the gitlab PR to go in. As we pre-pull the registry image we know if that fails there isn't any point attempting to use the cache. Fall back to the way we used to do it at that point. Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org> Message-Id: <20200709141327.14631-12-alex.bennee@linaro.org>
2020-07-11tests/docker: add --registry support to toolingAlex Bennée
This allows us to point the tools towards a registry from which they can grab pre-built layers instead of doing everything from scratch each time. To enable this we need to be using the DOCKER_BUILDKIT engine. Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Message-Id: <20200701135652.1366-25-alex.bennee@linaro.org>
2020-07-11tests/docker: change tag naming scheme of our imagesAlex Bennée
We've been misusing the tag naming scheme for some time by overloading the post : section with the image type. Really it should be saved for the revision of that particular build. Move the details to the other side so we have: qemu/image-name with the implied :latest version added by the tooling. Suggested-by: Daniel P. Berrangé <berrange@redhat.com> Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com> Message-Id: <20200701135652.1366-18-alex.bennee@linaro.org>
2020-06-10docker.py/build: support binary files in --extra-filesPaolo Bonzini
Read the --extra-files in binary mode to avoid encoding errors. Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2020-06-10docker.py/build: support -t and -f argumentsPaolo Bonzini
The docker.py command line is subtly different from docker and podman's, in that the tag and Dockerfile are passed via positional arguments. Remove this gratuitous difference and just parse -f and -t. -f was previously used by --extra-files, only keep the long option. Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2020-05-31python: remove more instances of sys.version_infoJohn Snow
We guarantee 3.5+ everywhere; remove more dead checks. In general, try to avoid using version checks and instead prefer to attempt behavior when possible. Signed-off-by: John Snow <jsnow@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com> Message-Id: <20200514035230.25756-1-jsnow@redhat.com> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
2020-02-04tests/docker: better handle symlinked libsAlex Bennée
When we are copying we want to ensure we grab the first resolution (the found in path section). However even that binary might be a symlink so lets make sure we chase the symlinks to copy the right binary to where it can be found. Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com> Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com> Reviewed-by: Robert Foley <robert.foley@linaro.org> Message-Id: <20200203090932.19147-3-alex.bennee@linaro.org>
2019-09-26podman: fix command invocationJohn Snow
Oops; there's no argv here. Signed-off-by: John Snow <jsnow@redhat.com> Message-Id: <20190913193821.17756-1-jsnow@redhat.com> Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Reviewed-by: Cleber Rosa <crosa@redhat.com> Tested-by: Cleber Rosa <crosa@redhat.com> Reviewed-by: Thomas Huth <thuth@redhat.com> Tested-by: Thomas Huth <thuth@redhat.com>
2019-09-26tests/docker: reduce scary warnings by cleaning up clean upAlex Bennée
There was in the clean-up code caused by attempting to inspect images which finished before we got there. Clean up the clean up code by: - only track the one instance at a time - use --filter for docker ps instead of doing it by hand - just call docker rm -f to be done with it - use uuid.uuid4() for a random uid Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
2019-09-10tests/docker: fix final missing .encode when parsing solibsAlex Bennée
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
2019-09-10tests/docker: handle missing encoding keyword for subprocess.check_outputAlex Bennée
This was only added in Python 3.6 and not all the build hosts have that recent a python3. However we still need to ensure everything is returns as a unicode string so checks higher up the call chain don't barf. Signed-off-by: Alex Bennée <alex.bennee@linaro.org> fixup! tests/docker: handle missing encoding keyword for subprocess.check_output
2019-09-10tests/docker: fix "cc" command to work with podmanAlex Bennée
Podman requires a little bit of additional magic to the uid mapping which was already done for the normal RunCommand. We simplify the logic by pushing it directly into the Docker::run method to avoid instantiating an extra Docker() object and ensure the CC command always runs as the current user. Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
2019-09-10tests/docker: Use --userns=keep-id for podmanJohn Snow
The workaround that attempts to accomplish the same result as --userns=keep-id does not appear to work well with UIDs much above 1000 (like mine, which is above 20000.) Since we have official support for this "trick" now, use the supported method. Signed-off-by: John Snow <jsnow@redhat.com> Message-Id: <20190904232451.26466-1-jsnow@redhat.com> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
2019-09-10configure: clean-up container cross compile detectAlex Bennée
The introduction of podman support inadvertently broke configure's detect of the container support as the configure probe didn't specify an engine type. To fix this in docker.py: - only (re)set USE_ENGINE if --engine is specified - enhance the output so docker is no longer just yes In the configure script we can at least start cleaning up the detecting and naming of variables. To avoid too much churn the conversion of the various make DOCKER_foo variables has been left for future clean-ups. Fixes: 9459f754134b Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Tested-by: David Hildenbrand <david@redhat.com>
2019-09-02tests/docker: upgrade docker.py to python3Alex Bennée
The recent podman changes (9459f754134bb) imported enum which is part of the python3 standard library but only available as an external library for python2. This causes problems on the fairly restricted environment such as shippable. Lets bite the bullet and make the script a fully python3 one. To that end: - drop the from __future__ import (we are there now ;-) - avoid the StringIO import hack - be consistent with the mode we read/write dockerfiles - s/iteritems/items/ - ensure check_output returns strings for processing Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com> Cc: Marc-André Lureau <marcandre.lureau@redhat.com>
2019-08-22docker.py: add podman supportMarc-André Lureau
Add a --engine option to select either docker, podman or auto. Among other advantages, podman allows to run rootless & daemonless containers, fortunately sharing compatible CLI with docker. With current podman, we have to use a uidmap trick in order to be able to rw-share the ccache directory with the container user. With a user 1000, the default mapping is: 1000 (host) -> 0 (container). So write access to /var/tmp/ccache ends will end with permission denied error. With "--uidmap 1000:0:1 --uidmap 0:1:1000", the mapping is: 1000 (host) -> 0 (container, 1st namespace) -> 1000 (container, 2nd namespace). (the rest is mumbo jumbo to avoid holes in the range of UIDs) A future podman version may have an option such as --userns-keep-uid. Thanks to Debarshi Ray <rishi@redhat.com> for the help! Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com> Acked-by: Alex Bennée <alex.bennee@linaro.org> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
2019-08-22docker.py: add --run-as-current-userMarc-André Lureau
(podman will need further tweaks) Suggested-by: Paolo Bonzini <pbonzini@redhat.com> Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com> Reviewed-by: Alex Bennée <alex.bennee@linaro.org> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
2019-02-08tests: PEP8 cleanup of docker.py, mostly white spaceAlex Bennée
My editor keeps putting squiggly lines under a bunch of the python lines to remind me how non-PEP8 compliant it is. Clean that up so it's easier to spot new errors. Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
2019-02-08tests: docker.py be even smarter with persistent binfmt_miscAlex Bennée
If we have a persistent mapping we don't need the QEMU binary copied into the container as the kernel has already opened the file and will pass the fd in. However the support libraries will still need to be there. Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
2019-02-08tests: make docker.py check for persistent configsAlex Bennée
binfmt_misc configured with the "F" flag opens the interpreter at config time. This means it can use an already open file-descriptor to run QEMU so there is no point trying to copy the binary into a container. Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
2019-02-08tests: make docker.py update use configured binfmt pathAlex Bennée
When copying a QEMU binary into a linux-user docker image we should check what the current configured binfmt_misc path is rather than just assuming "/usr/bin/qemu-bin". Obviously if the user changes the configuration afterwards they will break their images again. Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
2018-10-26tests: Fix typos in comments and help message (found by codespell)Stefan Weil
Fix also a grammar issue. Signed-off-by: Stefan Weil <sw@weilnetz.de> Reviewed-by: Alex Bennée <alex.bennee@linaro.org> Message-Id: <20180713054755.23323-1-sw@weilnetz.de> Signed-off-by: Laurent Vivier <laurent@vivier.eu>
2018-10-03Revert "docker.py: Python 2.6 argparse compatibility"Eduardo Habkost
This reverts commit c2d3189667409561772e8c1e5615c5166cd8aa2c. scripts/argparse.py was removed from the tree, so we don't need this hack anymore. Signed-off-by: Eduardo Habkost <ehabkost@redhat.com> Message-Id: <20180618225131.13113-3-ehabkost@redhat.com> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com> Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
2018-07-24docker: perform basic binfmt_misc validation in docker.pyAlex Bennée
Setting up binfmt_misc is outside of the scope of the docker.py script but we can at least validate it with any given executable so we have a more useful error message than the sed line of deboostrap failing cryptically. Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Reported-by: Richard Henderson <richard.henderson@linaro.org>
2018-07-24docker: fail more gracefully on docker.py checkAlex Bennée
As this is called directly from the Makefile while determining dependencies and it is possible the user was configured in one window but not have credentials in the other. Let's catch the Exceptions and deal with it quietly. Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Reported-by: Peter Maydell <peter.maydell@linaro.org> Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
2018-07-05docker: add special handling for FROM:debian-%-user targetsAlex Bennée
These will have been build with debootstrap so we need to check against the debian-bootstrap dockerfile. This does mean sticking to debian-FOO-user as the naming conventions for boot-strapped images. The actual cross image is built on top. Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org> Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
2018-06-20docker: docker.py adding age check commandAlex Bennée
This is useful for querying if an image is too old. Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
2018-06-20docker: docker.py add check sub-commandAlex Bennée
This command allows you to check if we need to re-build a docker image. If the image isn't in the repository or the checksums don't match then we return false and some text (for processing in makefiles). Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
2018-06-20docker: docker.py don't conflate checksums for extra_filesAlex Bennée
This just gets confusing especially as the helper function doesn't even take into account any extra files (or the executable). Currently the actual check just ignores them and also passes the result through _dockerfile_preprocess so we fix that too. Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
2018-06-20docker: docker.py use "version" to probe usageAlex Bennée
The "images" command is a fairly heavyweight command to run as it involves searching the whole docker file-system inventory. On a machine with a lot of images this makes start-up fairly expensive. Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
2018-06-20docker: allow "cc" command to run in user contextAlex Bennée
Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Reviewed-by: Fam Zheng <famz@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org> Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
2018-06-20docker: extend "cc" command to accept compilerAlex Bennée
When calling our cross-compilation images we want to call something other than the default cc. Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org> Reviewed-by: Fam Zheng <famz@redhat.com>
2018-06-20docker: Add "cc" subcommandAlex Bennée
Signed-off-by: Fam Zheng <famz@redhat.com> [AJB: add if args.paths check] Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org> Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
2018-06-20docker: docker.py wrap StringIO import for python3Alex Bennée
Although the docker.py is nominally python2 we actually invoke it with the configured python from the configure script. Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
2018-06-08python: futurize -f libfuturize.fixes.fix_print_with_importEduardo Habkost
Change all Python code to use print as a function. This is necessary for Python 3 compatibility. Done using: $ py=$( (g grep -l -E '^#!.*python';find -name '*.py' -printf '%P\n';) | \ sort -u | grep -v README.sh4) $ futurize -w -f libfuturize.fixes.fix_print_with_import $py Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com> Acked-by: Fam Zheng <famz@redhat.com> Signed-off-by: Eduardo Habkost <ehabkost@redhat.com> Message-Id: <20180608122952.2009-2-ehabkost@redhat.com> [ehabkost: fixup tests/docker/docker.py] Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
2018-06-05docker: add "probe" command for configureAlex Bennée
This is a helper function for the configure script. It replies yes, sudo or no to inform the user if non-interactive docker support is available. We trap the Exception to fail gracefully. Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Reviewed-by: Fam Zheng <famz@redhat.com>
2017-11-08docker: Improved image checksumFam Zheng
When a base image locally defined by QEMU, such as in the debian images, is updated, the dockerfile checksum mechanism in docker.py still skips updating the derived image, because it only looks at the literal content of the dockerfile, without considering changes to the base image. For example we have a recent fix e58c1f9b35e81 that fixed debian-win64-cross by updating its base image, debian8-mxe, but due to above "feature" of docker.py the image in question is automatically NOT rebuilt unless you add NOCACHE=1. It is noticed on Shippable: https://app.shippable.com/github/qemu/qemu/runs/541/2/console because after the fix is merged, the error still occurs, and the log shows the container image is, as explained above, not updated. This is because at the time docker.py was written, there wasn't any dependencies between QEMU's docker images. Now improve this to preprocess any "FROM qemu:*" directives in the dockerfiles while doing checksum, and inline the base image's dockerfile content, recursively. This ensures any changes on the depended _QEMU_ images are taken into account. This means for external images that we expect to retrieve from docker registries, we still do it as before. It is not perfect, because registry images can get updated too. Technically we could substitute the image name with its hex ID as obtained with $(docker images $IMAGE --format="{{.Id}}"), but --format is not supported by RHEL 7, so leave it for now. Reported-by: Philippe Mathieu-Daudé <f4bug@amsat.org> Signed-off-by: Fam Zheng <famz@redhat.com> Message-Id: <20171103131229.4737-1-famz@redhat.com> Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org> Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org> Reviewed-by: Alex Bennée <alex.bennee@linaro.org> Signed-off-by: Fam Zheng <famz@redhat.com>
2017-09-22docker: docker.py make --no-cache skip checksum testAlex Bennée
If you invoke with NOCACHE=1 we pass --no-cache in the argv to docker.py but may still not force a rebuild if the dockerfile checksum hasn't changed. By testing for its presence we can force builds without having to manually remove the docker image. Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Message-Id: <20170725133425.436-5-alex.bennee@linaro.org> Signed-off-by: Fam Zheng <famz@redhat.com>
2017-08-30docker.py: Python 2.6 argparse compatibilityStefan Hajnoczi
Add the scripts/ directory to sys.path so Python 2.6 will be able to import argparse. Cc: Fam Zheng <famz@redhat.com> Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com> Acked-by: John Snow <jsnow@redhat.com> Acked-by: Fam Zheng <famz@redhat.com> Message-id: 20170825155732.15665-3-stefanha@redhat.com Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
2017-07-17docker.py: Improve subprocess exit code handlingFam Zheng
A few error handlings are missing because we ignore the subprocess exit code, for example "docker build" errors are currently ignored. Introduce _do_check() aside the existing _do() method and use it in a few places. Signed-off-by: Fam Zheng <famz@redhat.com> Message-Id: <20170712075528.22770-3-famz@redhat.com> Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org> Signed-off-by: Fam Zheng <famz@redhat.com>
2017-07-17docker.py: Drop infile parameterFam Zheng
The **kwargs can do this just well. Signed-off-by: Fam Zheng <famz@redhat.com> Message-Id: <20170712075528.22770-2-famz@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org> Signed-off-by: Fam Zheng <famz@redhat.com>
2017-06-21docker: rebuild image if 'extra files' checksum does not matchPhilippe Mathieu-Daudé
Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org> Reviewed-by: Alex Bennée <alex.bennee@linaro.org> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
2017-06-21docker: add --include-files argument to 'build' commandPhilippe Mathieu-Daudé
Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org> Reviewed-by: Alex Bennée <alex.bennee@linaro.org> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
2017-06-21docker: let _copy_with_mkdir() sub_path argument be optionalPhilippe Mathieu-Daudé
Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org> Reviewed-by: Alex Bennée <alex.bennee@linaro.org> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
2017-03-07tests/docker: support proxy / corporate firewallPhilippe Mathieu-Daudé
if ftp_proxy/http_proxy/https_proxy standard environment variables available, pass them to the docker daemon to build images. this is required when building behind corporate proxy/firewall, but also help when using local cache server (ie: apt/yum). Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org> Message-Id: <20170306205520.32311-1-f4bug@amsat.org> Reviewed-by: Fam Zheng <famz@redhat.com> Signed-off-by: Fam Zheng <famz@redhat.com>
2017-02-24tests/docker: add basic user mapping supportAlex Bennée
Currently all docker builds are done by exporting a tarball to the docker container and running the build as the containers root user. Other use cases are possible however and it is possible to map a part of users file-system to the container. This is useful for example for doing cross-builds of arbitrary source trees. For this to work smoothly the container needs to have a user created that maps cleanly to the host system. This adds a -u option to the docker script so that: DEB_ARCH=armhf DEB_TYPE=stable ./tests/docker/docker.py build \ -u --include-executable=arm-linux-user/qemu-arm \ debian:armhf ./tests/docker/dockerfiles/debian-bootstrap.docker Will build a container that can then be run like: docker run --rm -it -v /home/alex/lsrc/qemu/risu.git/:/src \ --user=alex:alex -w /src/ debian:armhf \ sh -c "make clean && ./configure -s && make" All docker containers built will add the current user unless explicitly disabled by specifying NOUSER when invoking the Makefile: make docker-image-debian-armhf-cross NOUSER=1 Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Reviewed-by: Fam Zheng <famz@redhat.com> Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org> Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org> Message-Id: <20170220105139.21581-2-alex.bennee@linaro.org> Signed-off-by: Fam Zheng <famz@redhat.com>
generated by cgit v1.2.3 (git 2.26.2) at 2025-02-17 18:34:03 +0000