aboutsummaryrefslogtreecommitdiff
path: root/python
AgeCommit message (Collapse)Author
2023-08-28Revert "tests: Use separate virtual environment for avocado"Paolo Bonzini
This reverts commit e8e4298feadae7924cf7600bb3bcc5b0a8d7cbe9. ensuregroup allows to specify both the acceptable versions of avocado, and a locked version to be used when avocado is not installed as a system pacakge. This lets us install avocado in pyvenv/ using "mkvenv.py" and reuse the distro package on Fedora and CentOS Stream (the only distros where it's available). ensuregroup's usage of "(>=..., <=...)" constraints when evaluating the distro package, and "==" constraints when installing it from PyPI, makes it possible to avoid conflicts between the known-good version and a package plugins included in the distro. This is because package plugins have "==" constraints on the version that is included in the distro, and, using "pip install avocado==88.1" on a venv that includes system packages will result in an error: avocado-framework-plugin-varianter-yaml-to-mux 98.0 requires avocado-framework==98.0, but you have avocado-framework 88.1 which is incompatible. avocado-framework-plugin-result-html 98.0 requires avocado-framework==98.0, but you have avocado-framework 88.1 which is incompatible. But at the same time, if the venv does not include a system distribution of avocado then we can install a known-good version and stick to LTS releases. Resolves: https://gitlab.com/qemu-project/qemu/-/issues/1663 Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2023-08-28python: use vendored tomliPaolo Bonzini
Debian only introduced tomli in the bookworm release. Use a vendored wheel to avoid requiring a package that is only in bullseye-backports and is also absent in Ubuntu 20.04. While at it, fix an issue in the vendor.py scripts which does not add a newline after each package and hash. Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2023-08-28python: mkvenv: add ensuregroup commandPaolo Bonzini
Introduce a new subcommand that retrieves the packages to be installed from a TOML file. This allows being more flexible in using the system version of a package, while at the same time using a known-good version when installing the package. This is important for packages that sometimes have backwards-incompatible changes or that depend on specific versions of their dependencies. Compared to JSON, TOML is more human readable and easier to edit. A parser is available in 3.11 but also available as a small (12k) package for older versions, tomli. While tomli is bundled with pip, this is only true of recent versions of pip. Of all the supported OSes pretty much only FreeBSD has a recent enough version of pip while staying on Python <3.11. So we cannot use the same trick that is in place for distlib. Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2023-08-28python: mkvenv: introduce TOML-like representation of dependenciesPaolo Bonzini
We would like to place all Python dependencies in the same file, so that we can add more information without having long and complex command lines. The plan is to have a TOML file with one entry per package, for example [avocado] avocado-framework = { accepted = "(>=88.1, <93.0)", installed = "88.1", canary = "avocado" } Each TOML section will thus be a dictionary of dictionaries. Modify mkvenv.py's workhorse function, _do_ensure, to already operate on such a data structure. The "ensure" subcommand is modified to separate the depspec into a name and a version part, and use the result (plus the --diagnose argument) to build a dictionary for each command line argument. Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2023-08-28python: mkvenv: tweak the matching of --diagnose to depspecsPaolo Bonzini
Move the matching between the "absent" array and dep_specs[0] inside the loop, preparing for the possibility of having multiple canaries among the installed packages. Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2023-07-07python: bump minimum requirements so they are compatible with 3.12Paolo Bonzini
There are many Python 3.12 issues right now, but a particularly problematic one when debugging them is that one cannot even use minreqs.txt in a Python 3.12 virtual environment to test with locked package versions. Bump the mypy and wrapt versions to fix this, while remaining within the realm of versions compatible with Python 3.7. This requires a workaround for a mypy false positive qemu/qmp/qmp_tui.py:350: error: Non-overlapping equality check (left operand type: "Literal[Runstate.DISCONNECTING]", right operand type: "Literal[Runstate.IDLE]") [comparison-overlap] where mypy does not realize that self.disconnect() could change the value of self.runstate. Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2023-06-06mkvenv: always pass locally-installed packages to pipPaolo Bonzini
Let pip decide whether a new version should be installed or the current one is okay. This ensures that the virtual environment is updated (either upgraded or downgraded) whenever a new version of a package is requested. The hardest part here is figuring out if a package is installed in the venv (which also has to be done twice to account for the presence of either setuptools in Python <3.8, or importlib in Python >=3.8). Suggested-by: Peter Maydell <peter.maydell@linaro.org> Cc: John Snow <jsnow@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2023-05-31Revert "python/qmp/protocol: add open_with_socket()"John Snow
This reverts commit a3cfea92e2030926e00a2519d299384ea648e36e. (It's being rolled back in favor of a different API, which brings the in-tree and out-of-tree versions of qemu.qmp back in sync.) Signed-off-by: John Snow <jsnow@redhat.com> Message-id: 20230517163406.2593480-6-jsnow@redhat.com Signed-off-by: John Snow <jsnow@redhat.com>
2023-05-31python/qmp/legacy: remove open_with_socket() callsJohn Snow
Favor using connect() when passing a socket instead of open_with_socket(). Simultaneously, update constructor calls to use the combined address argument for QEMUMonitorProtocol(). Signed-off-by: John Snow <jsnow@redhat.com> Message-id: 20230517163406.2593480-5-jsnow@redhat.com Signed-off-by: John Snow <jsnow@redhat.com>
2023-05-31python/machine: use connect-based interface for existing socketsJohn Snow
Instead of using accept() with sockets (which uses open_with_socket()), use calls to connect() to utilize existing sockets instead. A benefit of this is more robust error handling already present within the connect() call that isn't present in open_with_socket(). Signed-off-by: John Snow <jsnow@redhat.com> Message-id: 20230517163406.2593480-4-jsnow@redhat.com Signed-off-by: John Snow <jsnow@redhat.com>
2023-05-31python/qmp/legacy: allow using sockets for connect()John Snow
Instead of asserting that we have an address, allow the use of sockets instead of addresses during a call to connect(). Signed-off-by: John Snow <jsnow@redhat.com> Message-id: 20230517163406.2593480-3-jsnow@redhat.com Signed-off-by: John Snow <jsnow@redhat.com>
2023-05-31python/qmp: allow sockets to be passed to connect()John Snow
Allow existing sockets to be passed to connect(). The changes are pretty minimal, and this allows for far greater flexibility in setting up communications with an endpoint. Signed-off-by: John Snow <jsnow@redhat.com> Message-id: 20230517163406.2593480-2-jsnow@redhat.com Signed-off-by: John Snow <jsnow@redhat.com>
2023-05-22Merge tag 'pull-qapi-2023-05-17-v2' of https://repo.or.cz/qemu/armbru into ↵Richard Henderson
staging QAPI patches patches for 2023-05-17 # -----BEGIN PGP SIGNATURE----- # # iQJGBAABCAAwFiEENUvIs9frKmtoZ05fOHC0AOuRhlMFAmRrTcgSHGFybWJydUBy # ZWRoYXQuY29tAAoJEDhwtADrkYZTMycP/3sP6/U4kwOKMGGcB+n2pHJeioQS4xgF # 94NCW+KpewxApP0XzIC2nDGjUe/rPcUfQmBNUumvYbqHO91tq91wFwkllBv2UR0q # 6qfRji+e8+9H9hMDeVzzSNjlZZg/tSdIJlhkJDw1u4/3fpjfAmzVx6DO3wepSQ9Q # m5Af/+uhVZWyUXMZqcKr2Zq8qur6ZFEBNpXpPvT60Tvy2heuQ+vcoE3tl2ZRQbmj # b/jhtCu+NPjgOHtg9Gr2BPXqQiZBR4vFA7WBsB8wCf2xxULfTwHJvFz/e0vx5fUC # q0Fsyybf4USo2PRMsRFv2v4dEuVGHb3E1RIJY4NTAxQMqqm4zfOyK0BzOGNDkxCn # owNP4vKly0e/CfYDY74FHaPId295xyeo6S4Cj5ib9W23AAWUNt6f6vbjlDOLCLON # c7yXP/aJwhTb2w1t0mLTmsKum3DpLlrudPudTylVlmYfwchkvUGsWYbaxu6H6XWk # 49Ox/QPVwqG6elXNn3kTY4QqTAppXhE7QcPbioX9WOThVPf6aJCLdZSHEHu4HXkZ # 4FRu73Z2wcPNB789xOrQoXs24GdKmWXQ6K01KC4v7WNJQBXccec52yGxvktQRZBm # GL3zYdOOJEL+Y/8JrXTIo26M8HP/4kxV2VqB6KOuaGygMsW9w9jbG+ygLyjqUDQg # 3APV3hdmVOht # =6anf # -----END PGP SIGNATURE----- # gpg: Signature made Mon 22 May 2023 04:11:04 AM PDT # gpg: using RSA key 354BC8B3D7EB2A6B68674E5F3870B400EB918653 # gpg: issuer "armbru@redhat.com" # gpg: Good signature from "Markus Armbruster <armbru@redhat.com>" [undefined] # gpg: aka "Markus Armbruster <armbru@pond.sub.org>" [undefined] # gpg: WARNING: This key is not certified with a trusted signature! # gpg: There is no indication that the signature belongs to the owner. # Primary key fingerprint: 354B C8B3 D7EB 2A6B 6867 4E5F 3870 B400 EB91 8653 * tag 'pull-qapi-2023-05-17-v2' of https://repo.or.cz/qemu/armbru: docs/interop: Delete qmp-intro.txt docs/interop/qmp-spec: Update error description for parsing errors docs/interop: Convert qmp-spec.txt to rST qapi: Improve error message for description following section Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
2023-05-22docs/interop: Convert qmp-spec.txt to rSTPeter Maydell
Convert the qmp-spec.txt document to restructuredText. Notable points about the conversion: * numbers at the start of section headings are removed, to match the style of the rest of the manual * cross-references to other sections or documents are hyperlinked * various formatting tweaks (notably the examples, which need the -> and <- prefixed so the QMP code-block lexer will accept them) * English prose fixed in a few places Reviewed-by: Eric Blake <eblake@redhat.com> Signed-off-by: Peter Maydell <peter.maydell@linaro.org> Message-Id: <20230515162245.3964307-2-peter.maydell@linaro.org> Reviewed-by: Markus Armbruster <armbru@redhat.com> Signed-off-by: Markus Armbruster <armbru@redhat.com> [.. code-block:: dumbed down to :: to work around CI failure]
2023-05-19mkvenv: pass first missing package to diagnose()Paolo Bonzini
If sphinx is present but the theme is not, mkvenv will print an inaccurate diagnostic: ERROR: Could not find a version that satisfies the requirement sphinx-rtd-theme>=0.5.0 (from versions: none) ERROR: No matching distribution found for sphinx-rtd-theme>=0.5.0 'sphinx>=1.6.0' not found: • Python package 'sphinx' version '5.3.0' was found, but isn't suitable. • mkvenv was configured to operate offline and did not check PyPI. Instead, ignore the packages that were found to be present, and report an error based on the first absent package. Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2023-05-19mkvenv: replace distlib.database with importlib.metadata/pkg_resourcesPaolo Bonzini
importlib.metadata is just as good as distlib.database and a bit more battle-proven for "egg" based distributions, and in fact that is exactly why mkvenv.py is not using distlib.database to find entry points: it simply does not work for eggs. The only disadvantage of importlib.metadata is that it is not available by default before Python 3.8, so we need a fallback to pkg_resources (again, just like for the case of finding entry points). Do so to fix issues where incorrect egg metadata results in a JSONDecodeError. While at it, reuse the new _get_version function to diagnose an incorrect version of the package even if importlib.metadata is not available. Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2023-05-18meson: require 0.63.0Paolo Bonzini
This version allows cleanups in modinfo collection, but they only work with Ninja 1.9.x and 1.8.x is still supported. It also supports the equivalent of QEMU's --static option to configure. The wheel file is bumped to 0.63.3, the last release in the 0.63 branch. Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2023-05-18python: bump some of the dependenciesPaolo Bonzini
The version of pyflakes that is listed in python/tests/minreqs.txt breaks on Python 3.8 with the following message: AttributeError: 'FlakesChecker' object has no attribute 'CONSTANT' Now that we do not support EOL'd Python versions anymore, we can update to newer, fixed versions. It is a good time to do so, before Python packages start dropping support for Python 3.7 as well! The new mypy is also a bit smarter about which packages are actually being used, so remove the now-unnecessary sections from setup.cfg. Signed-off-by: Paolo Bonzini <pbonzini@redhat.com> Signed-off-by: John Snow <jsnow@redhat.com> Message-Id: <20230511035435.734312-27-jsnow@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2023-05-18mkvenv: mark command as requiredPaolo Bonzini
This is only available in Python 3.7+. Signed-off-by: Paolo Bonzini <pbonzini@redhat.com> Signed-off-by: John Snow <jsnow@redhat.com> Message-Id: <20230511035435.734312-26-jsnow@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2023-05-18Python: Drop support for Python 3.6Paolo Bonzini
Python 3.6 was EOL 2021-12-31. Newer versions of upstream libraries have begun dropping support for this version and it is becoming more cumbersome to support. Avocado-framework and qemu.qmp each have their own reasons for wanting to drop Python 3.6, but won't until QEMU does. Versions of Python available in our supported build platforms as of today, with optional versions available in parentheses: openSUSE Leap 15.4: 3.6.15 (3.9.10, 3.10.2) CentOS Stream 8: 3.6.8 (3.8.13, 3.9.16) CentOS Stream 9: 3.9.13 Fedora 36: 3.10 Fedora 37: 3.11 Debian 11: 3.9.2 Alpine 3.14, 3.15: 3.9.16 Alpine 3.16, 3.17: 3.10.10 Ubuntu 20.04 LTS: 3.8.10 Ubuntu 22.04 LTS: 3.10.4 NetBSD 9.3: 3.9.13* FreeBSD 12.4: 3.9.16 FreeBSD 13.1: 3.9.16 OpenBSD 7.2: 3.9.16 Note: Our VM tests install 3.9 explicitly for FreeBSD and 3.10 for NetBSD; the default for "python" or "python3" in FreeBSD is 3.9.16. NetBSD does not appear to have a default meta-package, but offers several options, the lowest of which is 3.7.15. "python39" appears to be a pre-requisite to one of the other packages we request in tests/vm/netbsd. pip, ensurepip and other Python essentials are currently only available for Python 3.10 for NetBSD. CentOS and OpenSUSE support parallel installation of multiple Python interpreters, and binaries in /usr/bin will always use Python 3.6. However, the newly introduced support for virtual environments ensures that all build steps that execute QEMU Python code use a single interpreter. Since it is safe to under our supported platform policy, bump our minimum supported version of Python to 3.7. Signed-off-by: John Snow <jsnow@redhat.com> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com> Message-Id: <20230511035435.734312-24-jsnow@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2023-05-18python/wheels: add vendored meson packageJohn Snow
In preference to vendoring meson source, vendor a built distributable ("bdist" in python parlance). This has some benefits: (1) We can get rid of a git submodule, (2) Installing built meson into a venv doesn't require any extra dependencies (the python "wheel" package, chiefly.) (3) We don't treat meson any differently than we would any other python package (we install it, end of story, done.) (4) All future tarball *and* developer checkouts will function offline; No git or PyPI connection needed to fetch meson. Note that because mkvenv prefers vendored packages to PyPI, as mkvenv is currently written we will never consult PyPI for meson. (Do keep in mind that your distribution's meson will be preferred above the vendored version, though.) ``` jsnow@scv ~/s/q/python (python-configure-venv)> python3 scripts/vendor.py pip download --dest /home/jsnow/src/qemu/python/wheels --require-hashes -r /tmp/tmpvo5qav7i Collecting meson==0.61.5 Using cached meson-0.61.5-py3-none-any.whl (862 kB) Saved ./wheels/meson-0.61.5-py3-none-any.whl Successfully downloaded meson ``` Signed-off-by: John Snow <jsnow@redhat.com> Message-Id: <20230511035435.734312-17-jsnow@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2023-05-18python: add vendor.py utilityJohn Snow
This is a teeny-tiny script that just downloads any packages we want to vendor from PyPI and stores them in qemu.git/python/wheels/. If I'm hit by a meteor, it'll be easy to replicate what I have done in order to udpate the vendored source. We don't really care which python runs it; it exists as a meta-utility with no external dependencies and we won't package or install it. It will be monitored by the linters/type checkers, though; so it's guaranteed safe on python 3.6+. Signed-off-by: John Snow <jsnow@redhat.com> Message-Id: <20230511035435.734312-15-jsnow@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2023-05-18mkvenv: work around broken pip installations on Debian 10John Snow
This is a workaround intended for Debian 10, where the debian-patched pip does not function correctly if accessed from within a virtual environment. We don't support Debian 10 as a build platform any longer, though we do still utilize it for our build-tricore-softmmu CI test. It's also possible that this bug might appear on other derivative platforms and this workaround may prove useful. Signed-off-by: Paolo Bonzini <pbonzini@redhat.com> Signed-off-by: John Snow <jsnow@redhat.com> Message-Id: <20230511035435.734312-11-jsnow@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2023-05-18mkvenv: avoid ensurepip if pip is installedJohn Snow
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2023-05-18mkvenv: use pip's vendored distlib as a fallbackJohn Snow
distlib is usually not installed on Linux distribution, but it is vendored into pip. Because the virtual environment has pip via ensurepip, we can piggy-back on pip's vendored version. This could break if they move our cheese in the future, but the fix would be simply to require distlib. If it is debundled, as it is on msys, it is simply available directly. Signed-off-by: John Snow <jsnow@redhat.com> [Move to toplevel. - Paolo] Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2023-05-18mkvenv: add console script entry point generationJohn Snow
When creating a virtual environment that inherits system packages, script entry points (like "meson", "sphinx-build", etc) are not re-generated with the correct shebang. When you are *inside* of the venv, this is not a problem, but if you are *outside* of it, you will not have a script that engages the virtual environment appropriately. Add a mechanism that generates new entry points for pre-existing packages so that we can use these scripts to run "meson", "sphinx-build", "pip", unambiguously inside the venv. Signed-off-by: Paolo Bonzini <pbonzini@redhat.com> Signed-off-by: John Snow <jsnow@redhat.com> Message-Id: <20230511035435.734312-9-jsnow@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2023-05-18mkvenv: add --diagnose option to explain "ensure" failuresJohn Snow
This is a routine that is designed to print some usable info for human beings back out to the terminal if/when "mkvenv ensure" fails to locate or install a package during configure time, such as meson or sphinx. Since we are requiring that "meson" and "sphinx" are installed to the same Python environment as QEMU is configured to build with, this can produce some surprising failures when things are mismatched. This method is here to try and ease that sting by offering some actionable diagnosis. Signed-off-by: John Snow <jsnow@redhat.com> Message-Id: <20230511035435.734312-8-jsnow@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2023-05-18mkvenv: add ensure subcommandJohn Snow
This command is to be used to add various packages (or ensure they're already present) into the configure-provided venv in a modular fashion. Examples: mkvenv ensure --online --dir "${source_dir}/python/wheels/" "meson>=0.61.5" mkvenv ensure --online "sphinx>=1.6.0" mkvenv ensure "qemu.qmp==0.0.2" It's designed to look for packages in three places, in order: (1) In system packages, if the version installed is already good enough. This way your distribution-provided meson, sphinx, etc are always used as first preference. (2) In a vendored packages directory. Here I am suggesting qemu.git/python/wheels/ as that directory. This is intended to serve as a replacement for vendoring the meson source for QEMU tarballs. It is also highly likely to be extremely useful for packaging the "qemu.qmp" package in source distributions for platforms that do not yet package qemu.qmp separately. (3) Online, via PyPI, ***only when "--online" is passed***. This is only ever used as a fallback if the first two sources do not have an appropriate package that meets the requirement. The ability to build QEMU and run tests *completely offline* is not impinged. Signed-off-by: Paolo Bonzini <pbonzini@redhat.com> Signed-off-by: John Snow <jsnow@redhat.com> Message-Id: <20230511035435.734312-7-jsnow@redhat.com> [Use distlib to lookup distributions. - Paolo] Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2023-05-18mkvenv: add nested venv workaroundJohn Snow
Python virtual environments do not typically nest; they may inherit from the top-level system packages or not at all. For our purposes, it would be convenient to emulate "nested" virtual environments to allow callers of the configure script to install specific versions of python utilities in order to test build system features, utility version compatibility, etc. While it is possible to install packages into the system environment (say, by using the --user flag), it's nicer to install test packages into a totally isolated environment instead. As detailed in https://www.qemu.org/2023/03/24/python/, Emulate a nested venv environment by using .pth files installed into the site-packages folder that points to the parent environment when appropriate. Signed-off-by: Paolo Bonzini <pbonzini@redhat.com> Signed-off-by: John Snow <jsnow@redhat.com> Message-Id: <20230511035435.734312-6-jsnow@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2023-05-18mkvenv: add better error message for broken or missing ensurepipJohn Snow
Debian debundles ensurepip for python; NetBSD debundles pyexpat but ensurepip needs pyexpat. Try our best to offer a helpful error message instead of just failing catastrophically. Signed-off-by: John Snow <jsnow@redhat.com> Message-Id: <20230511035435.734312-5-jsnow@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2023-05-18python: add mkvenv.pyJohn Snow
This script will be responsible for building a lightweight Python virtual environment at configure time. It works with Python 3.6 or newer. It has been designed to: - work *offline*, no PyPI required. - work *quickly*, The fast path is only ~65ms on my machine. - work *robustly*, with multiple fallbacks to keep things working. - work *cooperatively*, using system packages where possible. (You can use your distro's meson, no problem.) Due to its unique position in the build chain, it exists outside of the installable python packages in-tree and *must* be runnable without any third party dependencies. Under normal circumstances, the only dependency required to execute this script is Python 3.6+ itself. The script is *faster* by several seconds when setuptools and pip are installed in the host environment, which is probably the case for a typical multi-purpose developer workstation. In the event that pip/setuptools are missing or not usable, additional dependencies may be required on some distributions which remove certain Python stdlib modules to package them separately: - Debian may require python3-venv to provide "ensurepip" - NetBSD may require py310-expat to provide "pyexpat" * (* Or whichever version is current for NetBSD.) Signed-off-by: Paolo Bonzini <pbonzini@redhat.com> Signed-off-by: John Snow <jsnow@redhat.com> Message-Id: <20230511035435.734312-4-jsnow@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2023-05-18python: update pylint configurationJohn Snow
Pylint 2.17.x decided that SocketAddrT was a bad name for a Type Alias for some reason. Sure, fine, whatever. Signed-off-by: John Snow <jsnow@redhat.com> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> Message-Id: <20230511035435.734312-3-jsnow@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2023-05-18python: shut up "pip install" during "make check-minreqs"Paolo Bonzini
"make check-minreqs" runs pip without the --disable-pip-version-check option, which causes the obnoxious "A new release of pip available" message. Recent versions of pip also complain that some of the dependencies in our virtual environment rely on "setup.py install" instead of providing a pyproject.toml file; apparently it is deprecated to install them directly from pip instead of letting the "wheel" package take care of them. So, install "wheel" in the virtual environment. Signed-off-by: Paolo Bonzini <pbonzini@redhat.com> Signed-off-by: John Snow <jsnow@redhat.com> Message-Id: <20230511035435.734312-2-jsnow@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2023-02-22python: drop pipenvJohn Snow
The pipenv tool was nice in theory, but in practice it's just too hard to update selectively, and it makes using it a pain. The qemu.qmp repo dropped pipenv support a while back and it's been functioning just fine, so I'm backporting that change here to qemu.git. Signed-off-by: John Snow <jsnow@redhat.com> Message-id: 20230210003147.1309376-3-jsnow@redhat.com Signed-off-by: John Snow <jsnow@redhat.com>
2023-02-22python: support pylint 2.16John Snow
Pylint 2.16 adds a few new checks that cause the optional check-tox CI job to fail. 1. The superfluous-parens check seems to be a bit more aggressive, 2. broad-exception-raised is new; it discourages "raise Exception". Fix these minor issues and turn the lights green. Signed-off-by: John Snow <jsnow@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org> Reviewed-by: Beraldo Leal <bleal@redhat.com> Message-id: 20230210003147.1309376-2-jsnow@redhat.com Signed-off-by: John Snow <jsnow@redhat.com>
2023-01-24python/qemu/machine: use socketpair() for QMP by defaultMarc-André Lureau
When no monitor address is given, establish the QMP communication through a socketpair() (API is also supported on Windows since Python 3.5) Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> Message-id: 20230111080101.969151-4-marcandre.lureau@redhat.com [Resolved conflicts, fixed typing error. --js] Signed-off-by: John Snow <jsnow@redhat.com>
2023-01-24python/qmp/legacy: make QEMUMonitorProtocol accept a socketMarc-André Lureau
Teach QEMUMonitorProtocol to accept an exisiting socket. Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> Message-id: 20230111080101.969151-3-marcandre.lureau@redhat.com Signed-off-by: John Snow <jsnow@redhat.com>
2023-01-24python/qmp/protocol: add open_with_socket()Marc-André Lureau
Instead of listening for incoming connections with a SocketAddr, add a new method open_with_socket() that accepts an existing socket. Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> Message-id: 20230111080101.969151-2-marcandre.lureau@redhat.com Signed-off-by: John Snow <jsnow@redhat.com>
2023-01-24python/qmp: increase read buffer sizeMaksim Davydov
Current 256KB is not enough for some real cases. As a possible solution limit can be chosen to be the same as libvirt (10MB) Signed-off-by: Maksim Davydov <davydov-max@yandex-team.ru> Reviewed-by: John Snow <jsnow@redhat.com> Message-id: 20230112152805.33109-3-davydov-max@yandex-team.ru Signed-off-by: John Snow <jsnow@redhat.com>
2023-01-24python/machine: Fix AF_UNIX path too long on macOSPeter Delevoryas
On macOS, private $TMPDIR's are the default. These $TMPDIR's are generated from a user's unix UID and UUID [1], which can create a relatively long path: /var/folders/d7/rz20f6hd709c1ty8f6_6y_z40000gn/T/ QEMU's avocado tests create a temporary directory prefixed by "avo_qemu_sock_", and create QMP sockets within _that_ as well. The QMP socket is unnecessarily long, because a temporary directory is created for every QEMUMachine object. /avo_qemu_sock_uh3w_dgc/qemu-37331-10bacf110-monitor.sock The path limit for unix sockets on macOS is 104: [2] /* * [XSI] Definitions for UNIX IPC domain. */ struct sockaddr_un { unsigned char sun_len; /* sockaddr len including null */ sa_family_t sun_family; /* [XSI] AF_UNIX */ char sun_path[104]; /* [XSI] path name (gag) */ }; This results in avocado tests failing on macOS because the QMP unix socket can't be created, because the path is too long: ERROR| Failed to establish connection: OSError: AF_UNIX path too long This change resolves by reducing the size of the socket directory prefix and the suffix on the QMP and console socket names. The result is paths like this: pdel@pdel-mbp:/var/folders/d7/rz20f6hd709c1ty8f6_6y_z40000gn/T $ tree qemu* qemu_df4evjeq qemu_jbxel3gy qemu_ml9s_gg7 qemu_oc7h7f3u qemu_oqb1yf97 ├── 10a004050.con └── 10a004050.qmp [1] https://apple.stackexchange.com/questions/353832/why-is-mac-osx-temp-directory-in-weird-path [2] /Library/Developer/CommandLineTools/SDKs/MacOSX12.3.sdk/usr/include/sys/un.h Signed-off-by: Peter Delevoryas <peter@pjd.dev> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org> Message-id: 20230110082930.42129-2-peter@pjd.dev Signed-off-by: John Snow <jsnow@redhat.com>
2023-01-24python: QEMUMachine: enable qmp accept timeout by defaultVladimir Sementsov-Ogievskiy
I've spent much time trying to debug hanging pipeline in gitlab. I started from and idea that I have problem in code in my series (which has some timeouts). Finally I found that the problem is that I've used QEMUMachine class directly to avoid qtest, and didn't add necessary arguments. Qemu fails and we wait for qmp accept endlessly. In gitlab it's just stopped by timeout (one hour) with no sign of what's going wrong. With timeout enabled, gitlab don't wait for an hour and prints all needed information. Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru> Reviewed-by: John Snow <jsnow@redhat.com> Message-Id: <20220624195252.175249-1-vsementsov@yandex-team.ru> [Fixed typing. --js] Signed-off-by: John Snow <jsnow@redhat.com>
2023-01-24Fix some typosDongdong Zhang
Fix some typos in 'python' directory. Signed-off-by: Dongdong Zhang <zhangdongdong@eswincomputing.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org> Message-id: 20221130015358.6998-2-zhangdongdong@eswincomputing.com [Fixed additional typo spotted by Max Filippov. --js] Reviewed-by: John Snow <jsnow@redhat.com> Signed-off-by: John Snow <jsnow@redhat.com>
2023-01-04python: add 3.11 to supported listJohn Snow
Signed-off-by: John Snow <jsnow@redhat.com> Reviewed-by: Wilfred Mallawa <wilfred.mallawa@wdc.com> Message-id: 20221203005234.620788-4-jsnow@redhat.com Signed-off-by: John Snow <jsnow@redhat.com>
2023-01-04Python: fix flake8 configJohn Snow
Newer flake8 versions are a bit pickier about the config file, and my in-line comment confuses the parser. Fix it. Signed-off-by: John Snow <jsnow@redhat.com> Reviewed-by: Wilfred Mallawa <wilfred.mallawa@wdc.com> Message-id: 20221203005234.620788-2-jsnow@redhat.com Signed-off-by: John Snow <jsnow@redhat.com>
2023-01-04python/machine: Handle termination cases without QMPJohn Snow
If we request a shutdown of a VM without a QMP console, we'll just hang waiting. Not ideal. Add in code that attempts graceful termination in these cases. Tested lightly; it appears to work and I doubt we rely on this case anywhere, but it's a corner you're allowed to wedge yourself in, so it should be handled. Signed-off-by: John Snow <jsnow@redhat.com>
2023-01-04python/machine: Add debug logging to key state changesJohn Snow
When key decisions are made about the lifetime of the VM process being managed, there's no log entry. Juxtaposed with the very verbose runstate change logging of the QMP module, machine seems a bit too introverted now. Season the machine.py module with logging statements to taste to help make a tastier soup. Signed-off-by: John Snow <jsnow@redhat.com>
2022-08-01misc: fix commonly doubled up wordsDaniel P. Berrangé
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> Message-Id: <20220707163720.1421716-5-berrange@redhat.com> Reviewed-by: Peter Maydell <peter.maydell@linaro.org> Signed-off-by: Thomas Huth <thuth@redhat.com>
2022-07-18python/qemu/qmp/legacy: Replace 'returns-whitelist' with the correct typeThomas Huth
'returns-whitelist' has been renamed to 'command-returns-exceptions' in commit b86df3747848 ("qapi: Rename pragma *-whitelist to *-exceptions"). Message-Id: <20220711095721.61280-1-thuth@redhat.com> Reviewed-by: John Snow <jsnow@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
2022-06-08Fix 'writeable' typosPeter Maydell
We have about 30 instances of the typo/variant spelling 'writeable', and over 500 of the more common 'writable'. Standardize on the latter. Change produced with: sed -i -e 's/\([Ww][Rr][Ii][Tt]\)[Ee]\([Aa][Bb][Ll][Ee]\)/\1\2/g' $(git grep -il writeable) and then hand-undoing the instance in linux-headers/linux/kvm.h. Most of these changes are in comments or documentation; the exceptions are: * a local variable in accel/hvf/hvf-accel-ops.c * a local variable in accel/kvm/kvm-all.c * the PMCR_WRITABLE_MASK macro in target/arm/internals.h * the EPT_VIOLATION_GPA_WRITABLE macro in target/i386/hvf/vmcs.h (which is never used anywhere) * the AR_TYPE_WRITABLE_MASK macro in target/i386/hvf/vmx.h (which is never used anywhere) Signed-off-by: Peter Maydell <peter.maydell@linaro.org> Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org> Reviewed-by: Stefan Weil <sw@weilnetz.de> Message-id: 20220505095015.2714666-1-peter.maydell@linaro.org
2022-06-06python: update for mypy 0.950John Snow
typeshed (included in mypy) recently updated to improve the typing for WriteTransport objects. I was working around this, but now there's a version where I shouldn't work around it. Unfortunately this creates some minor ugliness if I want to support both pre- and post-0.950 versions. For now, for my sanity, just disable the unused-ignores warning. Signed-off-by: John Snow <jsnow@redhat.com> Reviewed-by: Paolo Bonzini <pbonzini@redhat.com> Message-Id: <20220526000921.1581503-2-jsnow@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
generated by cgit v1.2.3 (git 2.26.2) at 2024-11-29 21:15:15 +0000