aboutsummaryrefslogtreecommitdiff
AgeCommit message (Collapse)Author
2021-06-02Merge remote-tracking branch ↵Peter Maydell
'remotes/thuth-gitlab/tags/pull-request-2021-06-02' into staging * Update the references to some doc files (use *.rst instead of *.txt) * Bump minimum versions of some requirements after removing CentOS 7 support # gpg: Signature made Wed 02 Jun 2021 08:12:18 BST # gpg: using RSA key 27B88847EEE0250118F3EAB92ED9D774FE702DB5 # gpg: issuer "thuth@redhat.com" # gpg: Good signature from "Thomas Huth <th.huth@gmx.de>" [full] # gpg: aka "Thomas Huth <thuth@redhat.com>" [full] # gpg: aka "Thomas Huth <huth@tuxfamily.org>" [full] # gpg: aka "Thomas Huth <th.huth@posteo.de>" [unknown] # Primary key fingerprint: 27B8 8847 EEE0 2501 18F3 EAB9 2ED9 D774 FE70 2DB5 * remotes/thuth-gitlab/tags/pull-request-2021-06-02: configure: bump min required CLang to 6.0 / XCode 10.0 configure: bump min required GCC to 7.5.0 configure: bump min required glib version to 2.56 tests/docker: drop CentOS 7 container tests/vm: convert centos VM recipe to CentOS 8 crypto: drop used conditional check crypto: bump min gnutls to 3.5.18, dropping RHEL-7 support crypto: bump min gcrypt to 1.8.0, dropping RHEL-7 support crypto: drop back compatibility typedefs for nettle crypto: bump min nettle to 3.4, dropping RHEL-7 support patchew: move quick build job from CentOS 7 to CentOS 8 container block/ssh: Bump minimum libssh version to 0.8.7 docs: fix references to docs/devel/s390-dasd-ipl.rst docs: fix references to docs/specs/tpm.rst docs: fix references to docs/devel/build-system.rst docs: fix references to docs/devel/atomics.rst docs: fix references to docs/devel/tracing.rst Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
2021-06-02Merge remote-tracking branch 'remotes/jsnow-gitlab/tags/python-pull-request' ↵Peter Maydell
into staging Pull request V2: - Squashed in fixup for 'Python: add utility function for retrieving port redirection' - Rebased on today's upstream CI here: https://gitlab.com/jsnow/qemu/-/pipelines/313202814 # gpg: Signature made Wed 02 Jun 2021 00:29:55 BST # gpg: using RSA key F9B7ABDBBCACDF95BE76CBD07DEF8106AAFC390E # gpg: Good signature from "John Snow (John Huston) <jsnow@redhat.com>" [full] # Primary key fingerprint: FAEB 9711 A12C F475 812F 18F2 88A9 064D 1835 61EB # Subkey fingerprint: F9B7 ABDB BCAC DF95 BE76 CBD0 7DEF 8106 AAFC 390E * remotes/jsnow-gitlab/tags/python-pull-request: (44 commits) gitlab: add python linters to CI python: add tox support python: add .gitignore python: add Makefile for some common tasks python: add avocado-framework and tests python: add devel package requirements to setuptools python/qemu: add qemu package itself to pipenv python/qemu: add isort to pipenv python: move .isort.cfg into setup.cfg python: add mypy to pipenv python: move mypy.ini into setup.cfg python: Add flake8 to pipenv python: add excluded dirs to flake8 config python: move flake8 config to setup.cfg python: add pylint to pipenv python: move pylintrc into setup.cfg python: add pylint import exceptions python: Add pipenv support python: add MANIFEST.in python: add directory structure README.rst files ... Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
2021-06-02configure: bump min required CLang to 6.0 / XCode 10.0Daniel P. Berrangé
Several distros have been dropped since the last time we bumped the minimum required CLang version. Per repology, currently shipping versions are: RHEL-8: 10.0.1 Debian Buster: 7.0.1 openSUSE Leap 15.2: 9.0.1 Ubuntu LTS 18.04: 6.0.0 Ubuntu LTS 20.04: 10.0.0 FreeBSD 12: 8.0.1 Fedora 33: 11.0.0 Fedora 34: 11.1.0 With this list Ubuntu LTS 18.04 is the constraint at 6.0.0 An LLVM version of 6.0.0 corresponds to macOS XCode version of 10.0 which dates from Sept 2018. Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> Message-Id: <20210514120415.1368922-13-berrange@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com> Reviewed-by: Willian Rampazzo <willianr@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
2021-06-02configure: bump min required GCC to 7.5.0Daniel P. Berrangé
Several distros have been dropped since the last time we bumped the minimum required GCC version. Per repology, currently shipping versions are: RHEL-8: 8.3.1 Debian Buster: 8.3.0 openSUSE Leap 15.2: 7.5.0 Ubuntu LTS 18.04: 7.5.0 Ubuntu LTS 20.04: 9.3.0 FreeBSD: 10.3.0 Fedora 33: 9.2.0 Fedora 34: 11.0.1 OpenBSD: 8.4.0 macOS HomeBrew: 11.1.0 With this list Ubuntu LTS 18.04 / openSUSE Leap 15.2 are the constraint at 7.5.0 Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> Message-Id: <20210514120415.1368922-12-berrange@redhat.com> Reviewed-by: Thomas Huth <thuth@redhat.com> Reviewed-by: Willian Rampazzo <willianr@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
2021-06-02configure: bump min required glib version to 2.56Daniel P. Berrangé
The glib version was not previously constrained by RHEL-7 since it rebases fairly often. Instead SLES 12 and Ubuntu 16.04 were the constraints in 00f2cfbbec63fb6f5a7789797a62ccedd22466ea. Both of these are old enough that they are outside our platform support matrix now. Per repology, current shipping versions are: RHEL-8: 2.56.4 Debian Buster: 2.58.3 openSUSE Leap 15.2: 2.62.6 Ubuntu LTS 18.04: 2.56.4 Ubuntu LTS 20.04: 2.64.6 FreeBSD: 2.66.7 Fedora 33: 2.66.8 Fedora 34: 2.68.1 OpenBSD: 2.68.1 macOS HomeBrew: 2.68.1 Thus Ubuntu LTS 18.04 / RHEL-8 are the constraint for GLib version at 2.56 Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> Message-Id: <20210514120415.1368922-11-berrange@redhat.com> Reviewed-by: Thomas Huth <thuth@redhat.com> Reviewed-by: Willian Rampazzo <willianr@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
2021-06-02tests/docker: drop CentOS 7 containerDaniel P. Berrangé
It has been over two years since RHEL-8 was released, and thus per the platform build policy, we no longer need to support RHEL-7 as a build target. Reviewed-by: Thomas Huth <thuth@redhat.com> Acked-by: Alex Bennée <alex.bennee@linaro.org> Reviewed-by: Willian Rampazzo <willianr@redhat.com> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> Message-Id: <20210514120415.1368922-10-berrange@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
2021-06-02tests/vm: convert centos VM recipe to CentOS 8Daniel P. Berrangé
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> Message-Id: <20210514120415.1368922-9-berrange@redhat.com> Reviewed-by: Willian Rampazzo <willianr@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
2021-06-02crypto: drop used conditional checkDaniel P. Berrangé
The condition being tested has never been set since the day the code was first introduced. Reviewed-by: Willian Rampazzo <willianr@redhat.com> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> Message-Id: <20210514120415.1368922-8-berrange@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
2021-06-02crypto: bump min gnutls to 3.5.18, dropping RHEL-7 supportDaniel P. Berrangé
It has been over two years since RHEL-8 was released, and thus per the platform build policy, we no longer need to support RHEL-7 as a build target. This lets us increment the minimum required gnutls version Per repology, current shipping versions are: RHEL-8: 3.6.14 Debian Buster: 3.6.7 openSUSE Leap 15.2: 3.6.7 Ubuntu LTS 18.04: 3.5.18 Ubuntu LTS 20.04: 3.6.13 FreeBSD: 3.6.15 Fedora 33: 3.6.16 Fedora 34: 3.7.1 OpenBSD: 3.6.15 macOS HomeBrew: 3.6.15 Ubuntu LTS 18.04 has the oldest version and so 3.5.18 is the new minimum. Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> Message-Id: <20210514120415.1368922-7-berrange@redhat.com> Reviewed-by: Thomas Huth <thuth@redhat.com> Reviewed-by: Willian Rampazzo <willianr@redhat.com> [thuth: rebased to use .gitlab-ci.d/buildtest.yml] Signed-off-by: Thomas Huth <thuth@redhat.com>
2021-06-02crypto: bump min gcrypt to 1.8.0, dropping RHEL-7 supportDaniel P. Berrangé
It has been over two years since RHEL-8 was released, and thus per the platform build policy, we no longer need to support RHEL-7 as a build target. This lets us increment the minimum required gcrypt version and assume that HMAC is always supported Per repology, current shipping versions are: RHEL-8: 1.8.5 Debian Buster: 1.8.4 openSUSE Leap 15.2: 1.8.2 Ubuntu LTS 18.04: 1.8.1 Ubuntu LTS 20.04: 1.8.5 FreeBSD: 1.9.2 Fedora 33: 1.8.6 Fedora 34: 1.9.3 OpenBSD: 1.9.3 macOS HomeBrew: 1.9.3 Ubuntu LTS 18.04 has the oldest version and so 1.8.0 is the new minimum. Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> Message-Id: <20210514120415.1368922-6-berrange@redhat.com> Reviewed-by: Thomas Huth <thuth@redhat.com> [thuth: rebased to use .gitlab-ci.d/buildtest.yml] Signed-off-by: Thomas Huth <thuth@redhat.com>
2021-06-02crypto: drop back compatibility typedefs for nettleDaniel P. Berrangé
Now that we only support modern nettle, we don't need to have local typedefs to mask the real nettle types. Reviewed-by: Thomas Huth <thuth@redhat.com> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> Message-Id: <20210514120415.1368922-5-berrange@redhat.com> Reviewed-by: Willian Rampazzo <willianr@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
2021-06-02crypto: bump min nettle to 3.4, dropping RHEL-7 supportDaniel P. Berrangé
It has been over two years since RHEL-8 was released, and thus per the platform build policy, we no longer need to support RHEL-7 as a build target. This lets us increment the minimum required nettle version and drop a lot of backwards compatibility code for 2.x series of nettle. Per repology, current shipping versions are: RHEL-8: 3.4.1 Debian Buster: 3.4.1 openSUSE Leap 15.2: 3.4.1 Ubuntu LTS 18.04: 3.4 Ubuntu LTS 20.04: 3.5.1 FreeBSD: 3.7.2 Fedora 33: 3.5.1 Fedora 34: 3.7.2 OpenBSD: 3.7.2 macOS HomeBrew: 3.7.2 Ubuntu LTS 18.04 has the oldest version and so 3.4 is the new minimum. Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> Message-Id: <20210514120415.1368922-4-berrange@redhat.com> Reviewed-by: Thomas Huth <thuth@redhat.com> Reviewed-by: Willian Rampazzo <willianr@redhat.com> [thuth: rebased to use .gitlab-ci.d/buildtest.yml] Signed-off-by: Thomas Huth <thuth@redhat.com>
2021-06-02patchew: move quick build job from CentOS 7 to CentOS 8 containerDaniel P. Berrangé
It has been over two years since RHEL-8 was released, and thus per the platform build policy, we no longer need to support RHEL-7 as a build target. Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> Message-Id: <20210514120415.1368922-3-berrange@redhat.com> Reviewed-by: Thomas Huth <thuth@redhat.com> Reviewed-by: Willian Rampazzo <willianr@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
2021-06-02block/ssh: Bump minimum libssh version to 0.8.7Thomas Huth
It has been over two years since RHEL-8 was released, and thus per the platform build policy, we no longer need to support RHEL-7 as a build target. So from the RHEL-7 perspective, we do not have to support libssh v0.7 anymore now. Let's look at the versions from other distributions and operating systems - according to repology.org, current shipping versions are: RHEL-8: 0.9.4 Debian Buster: 0.8.7 openSUSE Leap 15.2: 0.8.7 Ubuntu LTS 18.04: 0.8.0 * Ubuntu LTS 20.04: 0.9.3 FreeBSD: 0.9.5 Fedora 33: 0.9.5 Fedora 34: 0.9.5 OpenBSD: 0.9.5 macOS HomeBrew: 0.9.5 HaikuPorts: 0.9.5 * The version of libssh in Ubuntu 18.04 claims to be 0.8.0 from the name of the package, but in reality it is a 0.7 patched up as a Frankenstein monster with patches from the 0.8 development branch. This gave us some headaches in the past already and so it never worked with QEMU. All attempts to get it supported have failed in the past, patches for QEMU have never been merged and a request to Ubuntu to fix it in their 18.04 distro has been ignored: https://bugs.launchpad.net/ubuntu/+source/libssh/+bug/1847514 Thus we really should ignore the libssh in Ubuntu 18.04 in QEMU, too. Fix it by bumping the minimum libssh version to something that is greater than 0.8.0 now. Debian Buster and openSUSE Leap have the oldest version and so 0.8.7 is the new minimum. Signed-off-by: Thomas Huth <thuth@redhat.com> Tested-by: Richard W.M. Jones <rjones@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> Acked-by: Richard W.M. Jones <rjones@redhat.com> Message-Id: <20210519155859.344569-1-thuth@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
2021-06-02docs: fix references to docs/devel/s390-dasd-ipl.rstStefano Garzarella
Commit cc3d15a5ea ("docs: rstfy s390 dasd ipl documentation") converted docs/devel/s390-dasd-ipl.txt to docs/devel/s390-dasd-ipl.rst. We still have several references to the old file, so let's fix them with the following command: sed -i s/s390-dasd-ipl.txt/s390-dasd-ipl.rst/ \ $(git grep -l docs/devel/s390-dasd-ipl.txt) Signed-off-by: Stefano Garzarella <sgarzare@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com> Acked-by: Cornelia Huck <cohuck@redhat.com> Message-Id: <20210517151702.109066-6-sgarzare@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
2021-06-02docs: fix references to docs/specs/tpm.rstStefano Garzarella
Commit 6e8a3ff6ed ("docs/specs/tpm: reST-ify TPM documentation") converted docs/specs/tpm.txt to docs/specs/tpm.rst. We still have several references to the old file, so let's fix them with the following command: sed -i s/tpm.txt/tpm.rst/ $(git grep -l docs/specs/tpm.txt) Signed-off-by: Stefano Garzarella <sgarzare@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com> Message-Id: <20210517151702.109066-5-sgarzare@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
2021-06-02docs: fix references to docs/devel/build-system.rstStefano Garzarella
Commit a14f0bf165 ("docs: convert build system documentation to rST") converted docs/devel/build-system.txt to docs/devel/build-system.rst. We still have several references to the old file, so let's fix them with the following command: sed -i s/build-system.txt/build-system.rst/ \ $(git grep -l docs/devel/build-system.txt) Signed-off-by: Stefano Garzarella <sgarzare@redhat.com> Message-Id: <20210517151702.109066-4-sgarzare@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
2021-06-02docs: fix references to docs/devel/atomics.rstStefano Garzarella
Commit 15e8699f00 ("atomics: convert to reStructuredText") converted docs/devel/atomics.txt to docs/devel/atomics.rst. We still have several references to the old file, so let's fix them with the following command: sed -i s/atomics.txt/atomics.rst/ $(git grep -l docs/devel/atomics.txt) Signed-off-by: Stefano Garzarella <sgarzare@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com> Message-Id: <20210517151702.109066-3-sgarzare@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
2021-06-02docs: fix references to docs/devel/tracing.rstStefano Garzarella
Commit e50caf4a5c ("tracing: convert documentation to rST") converted docs/devel/tracing.txt to docs/devel/tracing.rst. We still have several references to the old file, so let's fix them with the following command: sed -i s/tracing.txt/tracing.rst/ $(git grep -l docs/devel/tracing.txt) Signed-off-by: Stefano Garzarella <sgarzare@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com> Message-Id: <20210517151702.109066-2-sgarzare@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
2021-06-01Merge remote-tracking branch ↵Peter Maydell
'remotes/ehabkost-gl/tags/x86-next-pull-request' into staging x86 queue, 2021-06-01 Features: * Add CPU model versions supporting 'xsaves' (Vitaly Kuznetsov) * Support AVX512 ZMM regs dump (Robert Hoo) Bug fixes: * Use better matching family/model/stepping for generic CPUs (Daniel P. Berrangé) Cleanups: * Hyper-V feature initialization cleanup (Vitaly Kuznetsov) * SEV firmware error list touchups (Connor Kuehl) * Constify CPUCaches and X86CPUDefinition (Philippe Mathieu-Daudé) * Document when features can be added to kvm_default_props (Eduardo Habkost) # gpg: Signature made Tue 01 Jun 2021 19:08:33 BST # gpg: using RSA key 5A322FD5ABC4D3DBACCFD1AA2807936F984DC5A6 # gpg: issuer "ehabkost@redhat.com" # gpg: Good signature from "Eduardo Habkost <ehabkost@redhat.com>" [full] # Primary key fingerprint: 5A32 2FD5 ABC4 D3DB ACCF D1AA 2807 936F 984D C5A6 * remotes/ehabkost-gl/tags/x86-next-pull-request: (24 commits) sev: add missing firmware error conditions sev: use explicit indices for mapping firmware error codes to strings target/i386/sev: add support to query the attestation report i386: use global kvm_state in hyperv_enabled() check i386: prefer system KVM_GET_SUPPORTED_HV_CPUID ioctl over vCPU's one i386: adjust the expected KVM_GET_SUPPORTED_HV_CPUID array size i386: switch hyperv_expand_features() to using error_setg() i386: move eVMCS enablement to hyperv_init_vcpu() i386: split hyperv_handle_properties() into hyperv_expand_features()/hyperv_fill_cpuids() i386: introduce hv_cpuid_cache i386: drop FEAT_HYPERV feature leaves i386: introduce hv_cpuid_get_host() i386: introduce hyperv_feature_supported() i386: stop using env->features[] for filling Hyper-V CPUIDs i386: always fill Hyper-V CPUID feature leaves from X86CPU data i386: invert hyperv_spinlock_attempts setting logic with hv_passthrough i386: keep hyperv_vendor string up-to-date i386: use better matching family/model/stepping for 'max' CPU i386: use better matching family/model/stepping for 'qemu64' CPU i386/cpu_dump: support AVX512 ZMM regs dump ... Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
2021-06-01gitlab: add python linters to CIJohn Snow
Add a Python container that has just enough juice for us to run the Python code quality analysis tools. Base this container on Fedora, because Fedora has very convenient packaging for testing multiple Python versions. We need python3, pip (for pulling packages), pipenv and virtualenv for creating virtual environments, and tox for running tests. make is needed for running 'make check-tox' and 'make venv-check' targets. Python3.10 is needed explicitly because the tox package only pulls in 3.6-3.9, but we wish to test the forthcoming release of Python as well to help predict any problems. Lastly, we need gcc to compile PyPI packages that may not have a binary distribution available. Add two tests: check-python-pipenv uses pipenv to test a frozen, very explicit set of packages against our minimum supported python version, Python 3.6. This test is not allowed to fail. The dependencies this test uses do not change unless python/Pipfile.lock is changed. check-python-tox uses tox to install the latest versions of required python dependencies against a wide array of Python versions from 3.6 to 3.9, even including the yet-to-be-released Python 3.10. This test is allowed to fail with a warning. Signed-off-by: John Snow <jsnow@redhat.com> Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Reviewed-by: Alex Bennée <alex.bennee@linaro.org> Reviewed-by: Cleber Rosa <crosa@redhat.com> Message-id: 20210527211715.394144-32-jsnow@redhat.com [Fix rebase conflict over .gitlab-ci.yml --js] Signed-off-by: John Snow <jsnow@redhat.com>
2021-06-01python: add tox supportJohn Snow
This is intended to be a manually run, non-CI script. Use tox to test the linters against all python versions from 3.6 to 3.10. This will only work if you actually have those versions installed locally, but Fedora makes this easy: > sudo dnf install python3.6 python3.7 python3.8 python3.9 python3.10 Unlike the pipenv tests (make venv-check), this pulls "whichever" versions of the python packages, so they are unpinned and may break as time goes on. In the case that breakages are found, setup.cfg should be amended accordingly to avoid the bad dependant versions, or the code should be amended to work around the issue. With confidence that the tests pass on 3.6 through 3.10 inclusive, add the appropriate classifiers to setup.cfg to indicate which versions we claim to support. Tox 3.18.0 or above is required to use the 'allowlist_externals' option. Signed-off-by: John Snow <jsnow@redhat.com> Reviewed-by: Cleber Rosa <crosa@redhat.com> Tested-by: Cleber Rosa <crosa@redhat.com> Message-id: 20210527211715.394144-31-jsnow@redhat.com Signed-off-by: John Snow <jsnow@redhat.com>
2021-06-01python: add .gitignoreJohn Snow
Ignore *Python* build and package output (build, dist, qemu.egg-info); these files are not created as part of a QEMU build. They are created by running the commands 'python3 setup.py <sdist|bdist>' when preparing tarballs to upload to e.g. PyPI. Ignore miscellaneous cached python confetti (mypy, pylint, et al) Ignore .idea (pycharm) .vscode, and .venv (pipenv et al). Signed-off-by: John Snow <jsnow@redhat.com> Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Reviewed-by: Cleber Rosa <crosa@redhat.com> Message-id: 20210527211715.394144-30-jsnow@redhat.com Signed-off-by: John Snow <jsnow@redhat.com>
2021-06-01python: add Makefile for some common tasksJohn Snow
Add "make venv" to create the pipenv-managed virtual environment that contains our explicitly pinned dependencies. Add "make check" to run the python linters [in the host execution environment]. Add "make venv-check" which combines the above two: create/update the venv, then run the linters in that explicitly managed environment. Add "make develop" which canonizes the runes needed to get both the linting pre-requisites (the "[devel]" part), and the editable live-install (the "-e" part) of these python libraries. make clean: delete miscellaneous python packaging output possibly created by pipenv, pip, or other python packaging utilities make distclean: delete the above, the .venv, and the editable "qemu" package forwarder (qemu.egg-info) if there is one. Signed-off-by: John Snow <jsnow@redhat.com> Reviewed-by: Cleber Rosa <crosa@redhat.com> Tested-by: Cleber Rosa <crosa@redhat.com> Message-id: 20210527211715.394144-29-jsnow@redhat.com Signed-off-by: John Snow <jsnow@redhat.com>
2021-06-01python: add avocado-framework and testsJohn Snow
Try using avocado to manage our various tests; even though right now they're only invoking shell scripts and not really running any python-native code. Create tests/, and add shell scripts which call out to mypy, flake8, pylint and isort to enforce the standards in this directory. Add avocado-framework to the setup.cfg development dependencies, and add avocado.cfg to store some preferences for how we'd like the test output to look. Finally, add avocado-framework to the Pipfile environment and lock the new dependencies. We are using avocado >= 87.0 here to take advantage of some features that Cleber has helpfully added to make the test output here *very* friendly and easy to read for developers that might chance upon the output in Gitlab CI. [Note: ALL of the dependencies get updated to the most modern versions that exist at the time of this writing. No way around it that I have seen. Not ideal, but so it goes.] Provided you have the right development dependencies (mypy, flake8, isort, pylint, and now avocado-framework) You should be able to run "avocado --config avocado.cfg run tests/" from the python folder to run all of these linters with the correct arguments. (A forthcoming commit adds the much easier 'make check'.) Signed-off-by: John Snow <jsnow@redhat.com> Reviewed-by: Cleber Rosa <crosa@redhat.com> Tested-by: Cleber Rosa <crosa@redhat.com> Message-id: 20210527211715.394144-28-jsnow@redhat.com Signed-off-by: John Snow <jsnow@redhat.com>
2021-06-01python: add devel package requirements to setuptoolsJohn Snow
setuptools doesn't have a formal understanding of development requires, but it has an optional feataures section. Fine; add a "devel" feature and add the requirements to it. To avoid duplication, we can modify pipenv to install qemu[devel] instead. This enables us to run invocations like "pip install -e .[devel]" and test the package on bleeding-edge packages beyond those specified in Pipfile.lock. Importantly, this also allows us to install the qemu development packages in a non-networked mode: `pip3 install --no-index -e .[devel]` will now fail if the proper development dependencies are not already met. This can be useful for automated build scripts where fetching network packages may be undesirable. Signed-off-by: John Snow <jsnow@redhat.com> Reviewed-by: Cleber Rosa <crosa@redhat.com> Message-id: 20210527211715.394144-27-jsnow@redhat.com Signed-off-by: John Snow <jsnow@redhat.com>
2021-06-01python/qemu: add qemu package itself to pipenvJohn Snow
This adds the python qemu packages themselves to the pipenv manifest. 'pipenv sync' will create a virtual environment sufficient to use the SDK. 'pipenv sync --dev' will create a virtual environment sufficient to use and test the SDK (with pylint, mypy, isort, flake8, etc.) The qemu packages are installed in 'editable' mode; all changes made to the python package inside the git tree will be reflected in the installed package without reinstallation. This includes changes made via git pull and so on. Signed-off-by: John Snow <jsnow@redhat.com> Reviewed-by: Cleber Rosa <crosa@redhat.com> Tested-by: Cleber Rosa <crosa@redhat.com> Message-id: 20210527211715.394144-26-jsnow@redhat.com Signed-off-by: John Snow <jsnow@redhat.com>
2021-06-01python/qemu: add isort to pipenvJohn Snow
isort 5.0.0 through 5.0.4 has a bug that causes it to misinterpret certain "from ..." clauses that are not related to imports. isort < 5.1.1 has a bug where it does not handle comments near import statements correctly. Require 5.1.2 or greater. isort can be run (in "check" mode) with 'isort -c qemu' from the python root. isort can also be used to fix/rewrite import order automatically by using 'isort qemu'. Signed-off-by: John Snow <jsnow@redhat.com> Reviewed-by: Cleber Rosa <crosa@redhat.com> Message-id: 20210527211715.394144-25-jsnow@redhat.com Signed-off-by: John Snow <jsnow@redhat.com>
2021-06-01python: move .isort.cfg into setup.cfgJohn Snow
Signed-off-by: John Snow <jsnow@redhat.com> Reviewed-by: Cleber Rosa <crosa@redhat.com> Message-id: 20210527211715.394144-24-jsnow@redhat.com Signed-off-by: John Snow <jsnow@redhat.com>
2021-06-01python: add mypy to pipenvJohn Snow
0.730 appears to be about the oldest version that works with the features we want, including nice human readable output (to make sure iotest 297 passes), and type-parameterized Popen generics. 0.770, however, supports adding 'strict' to the config file, so require at least 0.770. Now that we are checking a namespace package, we need to tell mypy to allow PEP420 namespaces, so modify the mypy config as part of the move. mypy can now be run from the python root by typing 'mypy -p qemu'. A note on mypy invocation: Running it as "mypy qemu/" changes the import path detection mechanisms in mypy slightly, and it will fail. See https://github.com/python/mypy/issues/8584 for a decent entry point with more breadcrumbs on the various behaviors that contribute to this subtle difference. Signed-off-by: John Snow <jsnow@redhat.com> Reviewed-by: Cleber Rosa <crosa@redhat.com> Tested-by: Cleber Rosa <crosa@redhat.com> Message-id: 20210527211715.394144-23-jsnow@redhat.com Signed-off-by: John Snow <jsnow@redhat.com>
2021-06-01python: move mypy.ini into setup.cfgJohn Snow
mypy supports reading its configuration values from a central project configuration file; do so. Signed-off-by: John Snow <jsnow@redhat.com> Reviewed-by: Cleber Rosa <crosa@redhat.com> Message-id: 20210527211715.394144-22-jsnow@redhat.com Signed-off-by: John Snow <jsnow@redhat.com>
2021-06-01python: Add flake8 to pipenvJohn Snow
flake8 3.5.x does not support the --extend-ignore syntax used in the .flake8 file to gracefully extend default ignores, so 3.6.x is our minimum requirement. There is no known upper bound. flake8 can be run from the python/ directory with no arguments. Signed-off-by: John Snow <jsnow@redhat.com> Reviewed-by: Cleber Rosa <crosa@redhat.com> Tested-by: Cleber Rosa <crosa@redhat.com> Message-id: 20210527211715.394144-21-jsnow@redhat.com Signed-off-by: John Snow <jsnow@redhat.com>
2021-06-01python: add excluded dirs to flake8 configJohn Snow
Instruct flake8 to avoid certain well-known directories created by python tooling that it ought not check. Note that at-present, nothing actually creates a ".venv" directory; but it is in such widespread usage as a de-facto location for a developer's virtual environment that it should be excluded anyway. A forthcoming commit canonizes this with a "make venv" command. Signed-off-by: John Snow <jsnow@redhat.com> Reviewed-by: Cleber Rosa <crosa@redhat.com> Message-id: 20210527211715.394144-20-jsnow@redhat.com Signed-off-by: John Snow <jsnow@redhat.com>
2021-06-01python: move flake8 config to setup.cfgJohn Snow
Update the comment concerning the flake8 exception to match commit 42c0dd12, whose commit message stated: A note on the flake8 exception: flake8 will warn on *any* bare except, but pylint's is context-aware and will suppress the warning if you re-raise the exception. Signed-off-by: John Snow <jsnow@redhat.com> Reviewed-by: Cleber Rosa <crosa@redhat.com> Message-id: 20210527211715.394144-19-jsnow@redhat.com Signed-off-by: John Snow <jsnow@redhat.com>
2021-06-01python: add pylint to pipenvJohn Snow
We are specifying >= pylint 2.8.x for several reasons: 1. For setup.cfg support, added in pylint 2.5.x 2. To specify a version that has incompatibly dropped bad-whitespace checks (2.6.x) 3. 2.7.x fixes "unsubscriptable" warnings in Python 3.9 4. 2.8.x adds a new, incompatible 'consider-using-with' warning that must be disabled in some cases. These pragmas cause warnings themselves in 2.7.x. Signed-off-by: John Snow <jsnow@redhat.com> Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Reviewed-by: Cleber Rosa <crosa@redhat.com> Tested-by: Cleber Rosa <crosa@redhat.com> Message-id: 20210527211715.394144-18-jsnow@redhat.com Signed-off-by: John Snow <jsnow@redhat.com>
2021-06-01python: move pylintrc into setup.cfgJohn Snow
Delete the empty settings now that it's sharing a home with settings for other tools. pylint can now be run from this folder as "pylint qemu". Signed-off-by: John Snow <jsnow@redhat.com> Reviewed-by: Cleber Rosa <crosa@redhat.com> Tested-by: Cleber Rosa <crosa@redhat.com> Message-id: 20210527211715.394144-17-jsnow@redhat.com Signed-off-by: John Snow <jsnow@redhat.com>
2021-06-01python: add pylint import exceptionsJohn Snow
Pylint 2.5.x - 2.7.x have regressions that make import checking inconsistent, see: https://github.com/PyCQA/pylint/issues/3609 https://github.com/PyCQA/pylint/issues/3624 https://github.com/PyCQA/pylint/issues/3651 Pinning to 2.4.4 is worse, because it mandates versions of shared dependencies that are too old for features we want in isort and mypy. Oh well. Signed-off-by: John Snow <jsnow@redhat.com> Reviewed-by: Cleber Rosa <crosa@redhat.com> Message-id: 20210527211715.394144-16-jsnow@redhat.com Signed-off-by: John Snow <jsnow@redhat.com>
2021-06-01python: Add pipenv supportJohn Snow
pipenv is a tool used for managing virtual environments with pinned, explicit dependencies. It is used for precisely recreating python virtual environments. pipenv uses two files to do this: (1) Pipfile, which is similar in purpose and scope to what setup.cfg lists. It specifies the requisite minimum to get a functional environment for using this package. (2) Pipfile.lock, which is similar in purpose to `pip freeze > requirements.txt`. It specifies a canonical virtual environment used for deployment or testing. This ensures that all users have repeatable results. The primary benefit of using this tool is to ensure *rock solid* repeatable CI results with a known set of packages. Although I endeavor to support as many versions as I can, the fluid nature of the Python toolchain often means tailoring code for fairly specific versions. Note that pipenv is *not* required to install or use this module; this is purely for the sake of repeatable testing by CI or developers. Here, a "blank" pipfile is added with no dependencies, but specifies Python 3.6 for the virtual environment. Pipfile will specify our version minimums, while Pipfile.lock specifies an exact loadout of packages that were known to operate correctly. This latter file provides the real value for easy setup of container images and CI environments. Signed-off-by: John Snow <jsnow@redhat.com> Reviewed-by: Cleber Rosa <crosa@redhat.com> Message-id: 20210527211715.394144-15-jsnow@redhat.com Signed-off-by: John Snow <jsnow@redhat.com>
2021-06-01python: add MANIFEST.inJohn Snow
When creating a source or binary distribution via 'python3 setup.py <sdist|bdist>', the VERSION and PACKAGE.rst files aren't bundled by default. Create a MANIFEST.in file that instructs the build tools to include these so that installation from these files won't fail. This is required by 'tox', as well as by the tooling needed to upload packages to PyPI. Exclude the 'README.rst' file -- that's intended as a guidebook to our source tree, not a file that needs to be distributed. Signed-off-by: John Snow <jsnow@redhat.com> Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Reviewed-by: Cleber Rosa <crosa@redhat.com> Message-id: 20210527211715.394144-14-jsnow@redhat.com Signed-off-by: John Snow <jsnow@redhat.com>
2021-06-01python: add directory structure README.rst filesJohn Snow
Add short readmes to python/, python/qemu/, python/qemu/machine, python/qemu/qmp, and python/qemu/utils that explain the directory hierarchy. These readmes are visible when browsing the source on e.g. gitlab/github and are designed to help new developers/users quickly make sense of the source tree. They are not designed for inclusion in a published manual. Signed-off-by: John Snow <jsnow@redhat.com> Reviewed-by: Cleber Rosa <crosa@redhat.com> Message-id: 20210527211715.394144-13-jsnow@redhat.com Signed-off-by: John Snow <jsnow@redhat.com>
2021-06-01python: add VERSION fileJohn Snow
Python infrastructure as it exists today is not capable reliably of single-sourcing a package version from a parent directory. The authors of pip are working to correct this, but as of today this is not possible. The problem is that when using pip to build and install a python package, it copies files over to a temporary directory and performs its build there. This loses access to any information in the parent directory, including git itself. Further, Python versions have a standard (PEP 440) that may or may not follow QEMU's versioning. In general, it does; but naturally QEMU does not follow PEP 440. To avoid any automatically-generated conflict, a manual version file is preferred. I am proposing: - Python tooling follows the QEMU version, indirectly, but with a major version of 0 to indicate that the API is not expected to be stable. This would mean version 0.5.2.0, 0.5.1.1, 0.5.3.0, etc. - In the event that a Python package needs to be updated independently of the QEMU version, a pre-release alpha version should be preferred, but *only* after inclusion to the qemu development or stable branches. e.g. 0.5.2.0a1, 0.5.2.0a2, and so on should be preferred prior to 5.2.0's release. - The Python core tooling makes absolutely no version compatibility checks or constraints. It *may* work with releases of QEMU from the past or future, but it is not required to. i.e., "qemu.machine" will, for now, remain in lock-step with QEMU. - We reserve the right to split the qemu package into independently versioned subpackages at a later date. This might allow for us to begin versioning QMP independently from QEMU at a later date, if we so choose. Implement this versioning scheme by adding a VERSION file and setting it to 0.6.0.0a1. Signed-off-by: John Snow <jsnow@redhat.com> Reviewed-by: Cleber Rosa <crosa@redhat.com> Message-id: 20210527211715.394144-12-jsnow@redhat.com Signed-off-by: John Snow <jsnow@redhat.com>
2021-06-01python: add qemu package installerJohn Snow
Add setup.cfg and setup.py, necessary for installing a package via pip. Add a ReST document (PACKAGE.rst) explaining the basics of what this package is for and who to contact for more information. This document will be used as the landing page for the package on PyPI. List the subpackages we intend to package by name instead of using find_namespace because find_namespace will naively also packages tests, things it finds in the dist/ folder, etc. I could not figure out how to modify this behavior; adding allow/deny lists to setuptools kept changing the packaged hierarchy. This works, roll with it. I am not yet using a pyproject.toml style package manifest, because "editable" installs are not defined (yet?) by PEP-517/518. I consider editable installs crucial for development, though they have (apparently) always been somewhat poorly defined. Pip now (19.2 and later) now supports editable installs for projects using pyproject.toml manifests, but might require the use of the --no-use-pep517 flag, which somewhat defeats the point. Full support for setup.py-less editable installs was not introduced until pip 21.1.1: https://github.com/pypa/pip/pull/9547/commits/7a95720e796a5e56481c1cc20b6ce6249c50f357 For now, while the dust settles, stick with the de-facto setup.py/setup.cfg combination supported by setuptools. It will be worth re-evaluating this point again in the future when our supported build platforms all ship a fairly modern pip. Additional reading on this matter: https://github.com/pypa/packaging-problems/issues/256 https://github.com/pypa/pip/issues/6334 https://github.com/pypa/pip/issues/6375 https://github.com/pypa/pip/issues/6434 https://github.com/pypa/pip/issues/6438 Signed-off-by: John Snow <jsnow@redhat.com> Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Reviewed-by: Cleber Rosa <crosa@redhat.com> Message-id: 20210527211715.394144-11-jsnow@redhat.com Signed-off-by: John Snow <jsnow@redhat.com>
2021-06-01python: create qemu packagesJohn Snow
move python/qemu/*.py to python/qemu/[machine, qmp, utils]/*.py and update import directives across the tree. This is done to create a PEP420 namespace package, in which we may create subpackages. To do this, the namespace directory ("qemu") should not have any modules in it. Those files will go into new 'machine', 'qmp' and 'utils' subpackages instead. Implement machine/__init__.py making the top-level classes and functions from its various modules available directly inside the package. Change qmp.py to qmp/__init__.py similarly, such that all of the useful QMP library classes are available directly from "qemu.qmp" instead of "qemu.qmp.qmp". Signed-off-by: John Snow <jsnow@redhat.com> Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Reviewed-by: Cleber Rosa <crosa@redhat.com> Message-id: 20210527211715.394144-10-jsnow@redhat.com Signed-off-by: John Snow <jsnow@redhat.com>
2021-06-01iotests/297: add --namespace-packages to mypy argumentsJohn Snow
mypy is kind of weird about how it handles imports. For legacy reasons, it won't load PEP 420 namespaces, because of logic implemented prior to that becoming a standard. So, if you plan on using any, you have to pass --namespace-packages. Alright, fine. Signed-off-by: John Snow <jsnow@redhat.com> Reviewed-by: Cleber Rosa <crosa@redhat.com> Message-id: 20210527211715.394144-9-jsnow@redhat.com Signed-off-by: John Snow <jsnow@redhat.com>
2021-06-01python/machine: Trim line length to below 80 charsJohn Snow
One more little delinting fix that snuck in. Signed-off-by: John Snow <jsnow@redhat.com> Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Reviewed-by: Cleber Rosa <crosa@redhat.com> Message-id: 20210527211715.394144-8-jsnow@redhat.com Signed-off-by: John Snow <jsnow@redhat.com>
2021-06-01python/machine: disable warning for Popen in _launch()John Snow
We handle this resource rather meticulously in shutdown/kill/wait/__exit__ et al, through the laborious mechanisms in _do_shutdown(). Quiet this pylint warning here. Signed-off-by: John Snow <jsnow@redhat.com> Reviewed-by: Cleber Rosa <crosa@redhat.com> Message-id: 20210527211715.394144-7-jsnow@redhat.com Message-id: 20210517184808.3562549-7-jsnow@redhat.com Signed-off-by: John Snow <jsnow@redhat.com>
2021-06-01python/machine: Disable pylint warning for open() in _pre_launchJohn Snow
Shift the open() call later so that the pylint pragma applies *only* to that one open() call. Add a note that suggests why this is safe: the resource is unconditionally cleaned up in _post_shutdown(). _post_shutdown is called after failed launches (see launch()), and unconditionally after every call to shutdown(), and therefore also on __exit__. Signed-off-by: John Snow <jsnow@redhat.com> Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com> Reviewed-by: Cleber Rosa <crosa@redhat.com> Message-id: 20210527211715.394144-6-jsnow@redhat.com Message-id: 20210517184808.3562549-6-jsnow@redhat.com Signed-off-by: John Snow <jsnow@redhat.com>
2021-06-01python/console_socket: Add a pylint ignoreJohn Snow
We manage cleaning up this resource ourselves. Pylint should shush. Signed-off-by: John Snow <jsnow@redhat.com> Reviewed-by: Cleber Rosa <crosa@redhat.com> Message-id: 20210527211715.394144-5-jsnow@redhat.com Message-id: 20210517184808.3562549-5-jsnow@redhat.com Signed-off-by: John Snow <jsnow@redhat.com>
2021-06-01python/machine: use subprocess.run instead of subprocess.PopenJohn Snow
use run() instead of Popen() -- to assert to pylint that we are not forgetting to close a long-running program. Signed-off-by: John Snow <jsnow@redhat.com> Reviewed-by: Cleber Rosa <crosa@redhat.com> Tested-by: Cleber Rosa <crosa@redhat.com> Message-id: 20210527211715.394144-4-jsnow@redhat.com Message-id: 20210517184808.3562549-4-jsnow@redhat.com Signed-off-by: John Snow <jsnow@redhat.com>
2021-06-01python/machine: use subprocess.DEVNULL instead of open(os.path.devnull)John Snow
One less file resource to manage, and it helps quiet some pylint >= 2.8.0 warnings about not using a with-context manager for the open call. Signed-off-by: John Snow <jsnow@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com> Reviewed-by: Cleber Rosa <crosa@redhat.com> Message-id: 20210527211715.394144-3-jsnow@redhat.com Message-id: 20210517184808.3562549-3-jsnow@redhat.com Signed-off-by: John Snow <jsnow@redhat.com>
generated by cgit v1.2.3 (git 2.26.2) at 2024-12-20 07:46:46 +0000