aboutsummaryrefslogtreecommitdiff
path: root/scripts/qapi/gen.py
AgeCommit message (Collapse)Author
2022-12-13qapi: Start to elide redundant has_FOO in generated CMarkus Armbruster
In QAPI, absent optional members are distinct from any present value. We thus represent an optional schema member FOO as two C members: a FOO with the member's type, and a bool has_FOO. Likewise for function arguments. However, has_FOO is actually redundant for a pointer-valued FOO, which can be null only when has_FOO is false, i.e. has_FOO == !!FOO. Except for arrays, where we a null FOO can also be a present empty array. The redundant has_FOO are a nuisance to work with. Improve the generator to elide them. Uses of has_FOO need to be replaced as follows. Tests of has_FOO become the equivalent comparison of FOO with null. For brevity, this is commonly done by implicit conversion to bool. Assignments to has_FOO get dropped. Likewise for arguments to has_FOO parameters. Beware: code may violate the invariant has_FOO == !!FOO before the transformation, and get away with it. The above transformation can then break things. Two cases: * Absent: if code ignores FOO entirely when !has_FOO (except for freeing it if necessary), even non-null / uninitialized FOO works. Such code is known to exist. * Present: if code ignores FOO entirely when has_FOO, even null FOO works. Such code should not exist. In both cases, replacing tests of has_FOO by FOO reverts their sense. We have to fix the value of FOO then. To facilitate review of the necessary updates to handwritten code, add means to opt out of this change, and opt out for all QAPI schema modules where the change requires updates to handwritten code. The next few commits will remove these opt-outs in reviewable chunks, then drop the means to opt out. Signed-off-by: Markus Armbruster <armbru@redhat.com> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> Message-Id: <20221104160712.3005652-5-armbru@redhat.com>
2022-01-27qapi/gen: Add FOO.trace-events output moduleVladimir Sementsov-Ogievskiy
We are going to generate trace events for QMP commands. We should generate both trace_*() function calls and trace-events files listing events for trace generator. So, add an output module FOO.trace-events for each FOO schema module. Since we're going to add trace events only to command marshallers, make the trace-events output optional, so we don't generate so many useless empty files. Currently nobody set add_trace_events to True, so new functionality is disabled. It will be enabled for QAPISchemaGenCommandVisitor in a further commit. Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com> Message-Id: <20220126161130.3240892-2-vsementsov@virtuozzo.com> Reviewed-by: Markus Armbruster <armbru@redhat.com> Signed-off-by: Markus Armbruster <armbru@redhat.com>
2021-10-29qapi: Tools for sets of special feature flags in generated codeMarkus Armbruster
New enum QapiSpecialFeature enumerates the special feature flags. New helper gen_special_features() returns code to represent a collection of special feature flags as a bitset. The next few commits will put them to use. Signed-off-by: Markus Armbruster <armbru@redhat.com> Reviewed-by: John Snow <jsnow@redhat.com> Message-Id: <20211028102520.747396-5-armbru@redhat.com> Reviewed-by: Juan Quintela <quintela@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
2021-10-02qapi/gen: use dict.items() to iterate over _modulesJohn Snow
New pylint warning. I could silence it, but this is the only occurrence in the entire tree, including everything in iotests/ and python/. Easier to just change this one instance. (The warning is emitted in cases where you are fetching the values anyway, so you may as well just take advantage of the iterator to avoid redundant lookups.) Signed-off-by: John Snow <jsnow@redhat.com> Message-Id: <20210930205716.1148693-3-jsnow@redhat.com> Reviewed-by: Markus Armbruster <armbru@redhat.com> Signed-off-by: Markus Armbruster <armbru@redhat.com>
2021-09-03qapi: Simplify QAPISchemaIfCond's interface for generating CMarkus Armbruster
QAPISchemaIfCond.cgen() is only ever used like gen_if(ifcond.cgen()) and gen_endif(ifcond.cgen()) Simplify to ifcond.gen_if() and ifcond.gen_endif() Signed-off-by: Markus Armbruster <armbru@redhat.com> Message-Id: <20210831123809.1107782-2-armbru@redhat.com> Reviewed-by: Marc-André Lureau <marcandre.lureau@redhat.com> [Import statements tidied up with isort]
2021-08-26qapi: introduce QAPISchemaIfCond.cgen()Marc-André Lureau
Instead of building prepocessor conditions from a list of string, use the result generated from QAPISchemaIfCond.cgen() and hide the implementation details. Note: this patch introduces a minor regression, generating a redundant pair of parenthesis. This is mostly fixed in a later patch in this series ("qapi: replace if condition list with dict [..]") Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com> Message-Id: <20210804083105.97531-5-marcandre.lureau@redhat.com> Reviewed-by: Markus Armbruster <armbru@redhat.com> [Commit message tweaked] Signed-off-by: Markus Armbruster <armbru@redhat.com>
2021-08-26qapi: wrap Sequence[str] in an objectMarc-André Lureau
Mechanical change, except for a new assertion in QAPISchemaEntity.ifcond(). Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com> Message-Id: <20210804083105.97531-3-marcandre.lureau@redhat.com> Reviewed-by: Markus Armbruster <armbru@redhat.com> [Rebased with obvious conflicts, commit message adjusted] Signed-off-by: Markus Armbruster <armbru@redhat.com>
2021-02-18qapi: Replace List[str] with Sequence[str] for ifcondJohn Snow
It does happen to be a list (as of now), but we can describe it in more general terms with no loss in accuracy to allow tuples and other constructs. In the future, we can write "ifcond: Sequence[str] = ()" as a default parameter, which we could not do safely with a Mutable type like a List. Signed-off-by: John Snow <jsnow@redhat.com> Message-Id: <20210216021809.134886-2-jsnow@redhat.com> Reviewed-by: Markus Armbruster <armbru@redhat.com> [Commit message tweaked] Signed-off-by: Markus Armbruster <armbru@redhat.com>
2021-02-08qapi: type 'info' as Optional[QAPISourceInfo]John Snow
For everything typed so far, type this parameter as Optional[QAPISourceInfo]. In the most generic case, QAPISchemaEntity's info field may be None to represent types that come from built-in definitions. Although some Entity types may not currently have any built-in definitions, it is not easily possible to constrain the type except on an ad-hoc basis using assertions. It's easier and simpler, then, to just say it's always an Optional type. Signed-off-by: John Snow <jsnow@redhat.com> Message-Id: <20210201193747.2169670-16-jsnow@redhat.com> Reviewed-by: Markus Armbruster <armbru@redhat.com> Signed-off-by: Markus Armbruster <armbru@redhat.com>
2021-02-08qapi/gen: Drop support for QAPIGen without a file nameMarkus Armbruster
The previous commit removed the only user of QAPIGen(None). Tighten the type hint. Signed-off-by: Markus Armbruster <armbru@redhat.com> Signed-off-by: John Snow <jsnow@redhat.com> Message-Id: <20210201193747.2169670-15-jsnow@redhat.com>
2021-02-08qapi/gen: Support switching to another module temporarilyMarkus Armbruster
Signed-off-by: Markus Armbruster <armbru@redhat.com> Signed-off-by: John Snow <jsnow@redhat.com> Message-Id: <20210201193747.2169670-13-jsnow@redhat.com> [Commit message tweaked]
2021-02-08qapi/gen: write _genc/_genh access shimsJohn Snow
Many places assume they can access these fields without checking them first to ensure they are defined. Eliminating the _genc and _genh fields and replacing them with functional properties that check for correct state can ease the typing overhead by eliminating the Optional[T] return type. Signed-off-by: John Snow <jsnow@redhat.com> Message-Id: <20210201193747.2169670-12-jsnow@redhat.com> Reviewed-by: Markus Armbruster <armbru@redhat.com> Signed-off-by: Markus Armbruster <armbru@redhat.com>
2021-02-08qapi/gen: Combine ._add_[user|system]_moduleMarkus Armbruster
With callers to _add_system_module now explicitly using the './' prefix to indicate a system module, there is no longer any reason to have separate interfaces for adding system vs user modules; use a unified interface that differentiates based on the name. Signed-off-by: Markus Armbruster <armbru@redhat.com> Signed-off-by: John Snow <jsnow@redhat.com> Message-Id: <20210201193747.2169670-10-jsnow@redhat.com>
2021-02-08qapi: use './builtin' as the built-in module nameJohn Snow
Use './builtin' as the built-in module name instead of None. Clarify the typing that this is now always a string. Signed-off-by: John Snow <jsnow@redhat.com> Message-Id: <20210201193747.2169670-9-jsnow@redhat.com> Reviewed-by: Markus Armbruster <armbru@redhat.com> Signed-off-by: Markus Armbruster <armbru@redhat.com>
2021-02-08qapi: use explicitly internal module namesJohn Snow
QAPISchemaModularCVisitor._add_system_module() prefixes './' to its name argument to make it a module name. Pass the module name instead. This will allow us to coalesce the methods to add modules later on. Signed-off-by: John Snow <jsnow@redhat.com> Message-Id: <20210201193747.2169670-8-jsnow@redhat.com> Reviewed-by: Markus Armbruster <armbru@redhat.com> [Commit message reworded] Signed-off-by: Markus Armbruster <armbru@redhat.com>
2021-02-08qapi/gen: Replace ._begin_system_module()Markus Armbruster
QAPISchemaModularCVisitor._begin_system_module() is actually just for the builtin module. Rename it to ._begin_builtin_module() and drop its useless @name parameter. Clarify conditionals in visit_module to make this clear. Signed-off-by: Markus Armbruster <armbru@redhat.com> Signed-off-by: John Snow <jsnow@redhat.com> Message-Id: <20210201193747.2169670-7-jsnow@redhat.com>
2021-02-08qapi: centralize is_[user|system|builtin]_module methodsJohn Snow
Define what a module is and define what kind of a module it is once and for all, in one place. Signed-off-by: John Snow <jsnow@redhat.com> Message-Id: <20210201193747.2169670-6-jsnow@redhat.com> Reviewed-by: Markus Armbruster <armbru@redhat.com> Signed-off-by: Markus Armbruster <armbru@redhat.com>
2021-02-08qapi/gen: inline _wrap_ifcond into end_if()John Snow
We assert _start_if is not None in end_if, but that's opaque to mypy. By inlining _wrap_ifcond, that constraint becomes provable to mypy. Signed-off-by: John Snow <jsnow@redhat.com> Message-Id: <20210201193747.2169670-5-jsnow@redhat.com> Reviewed-by: Markus Armbruster <armbru@redhat.com> Signed-off-by: Markus Armbruster <armbru@redhat.com>
2020-10-10qapi/gen.py: delint with pylintJohn Snow
'fp' and 'fd' are self-evident in context, add them to the list of OK names. _top and _bottom also need to stay standard methods because some users override the method and need to use `self`. Tell pylint to shush. Signed-off-by: John Snow <jsnow@redhat.com> Reviewed-by: Eduardo Habkost <ehabkost@redhat.com> Reviewed-by: Cleber Rosa <crosa@redhat.com> Message-Id: <20201009161558.107041-32-jsnow@redhat.com> Reviewed-by: Markus Armbruster <armbru@redhat.com> Signed-off-by: Markus Armbruster <armbru@redhat.com>
2020-10-10qapi/gen.py: update write() to be more idiomaticJohn Snow
Make the file handling here just a tiny bit more idiomatic. (I realize this is heavily subjective.) Use exist_ok=True for os.makedirs and remove the exception, use fdopen() to wrap the file descriptor in a File-like object, and use a context manager for managing the file pointer. Signed-off-by: John Snow <jsnow@redhat.com> Reviewed-by: Eduardo Habkost <ehabkost@redhat.com> Reviewed-by: Cleber Rosa <crosa@redhat.com> Reviewed-by: Markus Armbruster <armbru@redhat.com> Message-Id: <20201009161558.107041-31-jsnow@redhat.com> Signed-off-by: Markus Armbruster <armbru@redhat.com>
2020-10-10qapi/gen.py: Remove unused parameterJohn Snow
_module_dirname doesn't use the 'what' argument, so remove it. Signed-off-by: John Snow <jsnow@redhat.com> Reviewed-by: Eduardo Habkost <ehabkost@redhat.com> Reviewed-by: Cleber Rosa <crosa@redhat.com> Message-Id: <20201009161558.107041-30-jsnow@redhat.com> Reviewed-by: Markus Armbruster <armbru@redhat.com> Signed-off-by: Markus Armbruster <armbru@redhat.com>
2020-10-10qapi/gen.py: add type hint annotationsJohn Snow
Annotations do not change runtime behavior. This commit *only* adds annotations. Signed-off-by: John Snow <jsnow@redhat.com> Reviewed-by: Eduardo Habkost <ehabkost@redhat.com> Reviewed-by: Cleber Rosa <crosa@redhat.com> Message-Id: <20201009161558.107041-28-jsnow@redhat.com> Message-Id: <20201009161558.107041-29-jsnow@redhat.com> [mypy.ini update squashed in] Reviewed-by: Markus Armbruster <armbru@redhat.com> Signed-off-by: Markus Armbruster <armbru@redhat.com>
2020-10-10qapi/gen: Make _is_user_module() return boolJohn Snow
_is_user_module() returns thruth values. The next commit wants it to return bool. Make it so. Signed-off-by: John Snow <jsnow@redhat.com> Reviewed-by: Cleber Rosa <crosa@redhat.com> Reviewed-by: Eduardo Habkost <ehabkost@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com> Message-Id: <20201009161558.107041-27-jsnow@redhat.com> Reviewed-by: Markus Armbruster <armbru@redhat.com> [Commit message rewritten] Signed-off-by: Markus Armbruster <armbru@redhat.com>
2020-10-10qapi/common.py: move build_params into gen.pyJohn Snow
Including it in common.py creates a circular import dependency; schema relies on common, but common.build_params requires a type annotation from schema. To type this properly, it needs to be moved outside the cycle. Signed-off-by: John Snow <jsnow@redhat.com> Reviewed-by: Eduardo Habkost <ehabkost@redhat.com> Reviewed-by: Cleber Rosa <crosa@redhat.com> Message-Id: <20201009161558.107041-18-jsnow@redhat.com> Reviewed-by: Markus Armbruster <armbru@redhat.com> Signed-off-by: Markus Armbruster <armbru@redhat.com>
2020-10-10qapi: Remove wildcard includesJohn Snow
Wildcard includes become hard to manage when refactoring and dealing with circular dependencies with strictly typed mypy. flake8 also flags each one as a warning, as it is not smart enough to know which names exist in the imported file. Remove them and include things explicitly by name instead. Signed-off-by: John Snow <jsnow@redhat.com> Reviewed-by: Eduardo Habkost <ehabkost@redhat.com> Reviewed-by: Cleber Rosa <crosa@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com> Message-Id: <20201009161558.107041-7-jsnow@redhat.com> Reviewed-by: Markus Armbruster <armbru@redhat.com> Signed-off-by: Markus Armbruster <armbru@redhat.com>
2020-10-10qapi: Prefer explicit relative importsJohn Snow
All of the QAPI include statements are changed to be package-aware, as explicit relative imports. A quirk of Python packages is that the name of the package exists only *outside* of the package. This means that to a module inside of the qapi folder, there is inherently no such thing as the "qapi" package. The reason these imports work is because the "qapi" package exists in the context of the caller -- the execution shim, where sys.path includes a directory that has a 'qapi' folder in it. When we write "from qapi import sibling", we are NOT referencing the folder 'qapi', but rather "any package named qapi in sys.path". If you should so happen to have a 'qapi' package in your path, it will use *that* package. When we write "from .sibling import foo", we always reference explicitly our sibling module; guaranteeing consistency in *where* we are importing these modules from. This can be useful when working with virtual environments and packages in development mode. In development mode, a package is installed as a series of symlinks that forwards to your same source files. The problem arises because code quality checkers will follow "import qapi.x" to the "installed" version instead of the sibling file and -- even though they are the same file -- they have different module paths, and this causes cyclic import problems, false positive type mismatch errors, and more. It can also be useful when dealing with hierarchical packages, e.g. if we allow qemu.core.qmp, qemu.qapi.parser, etc. Signed-off-by: John Snow <jsnow@redhat.com> Reviewed-by: Eduardo Habkost <ehabkost@redhat.com> Reviewed-by: Cleber Rosa <crosa@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com> Message-Id: <20201009161558.107041-6-jsnow@redhat.com> Reviewed-by: Markus Armbruster <armbru@redhat.com> Signed-off-by: Markus Armbruster <armbru@redhat.com>
2020-10-10qapi: modify docstrings to be sphinx-compatibleJohn Snow
A precise style guide and a package-wide overhaul is forthcoming pending further discussion and consensus. For now, merely avoid obvious errors that cause Sphinx documentation build problems, using a style loosely based on PEP 257 and Sphinx Autodoc. It is chosen for interoperability with our existing Sphinx framework, and because it has loose recognition in the Pycharm IDE. See also: https://www.python.org/dev/peps/pep-0257/ https://www.sphinx-doc.org/en/master/usage/restructuredtext/domains.html#info-field-lists Signed-off-by: John Snow <jsnow@redhat.com> Message-Id: <20201009161558.107041-3-jsnow@redhat.com> Reviewed-by: Markus Armbruster <armbru@redhat.com> Signed-off-by: Markus Armbruster <armbru@redhat.com>
2020-09-29scripts/qapi: Remove texinfo generation supportPeter Maydell
We no longer use the generated texinfo format documentation, so delete the code that generates it, and the test case for the generation. Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Signed-off-by: Peter Maydell <peter.maydell@linaro.org> Message-Id: <20200925162316.21205-17-peter.maydell@linaro.org> Reviewed-by: Markus Armbruster <armbru@redhat.com> Signed-off-by: Markus Armbruster <armbru@redhat.com>
2020-03-06qemu-storage-daemon: Add --monitor optionKevin Wolf
This adds and parses the --monitor option, so that a QMP monitor can be used in the storage daemon. The monitor offers commands defined in the QAPI schema at storage-daemon/qapi/qapi-schema.json. The --monitor options currently allows to create multiple monitors with the same ID. This part of the interface is considered unstable. We will reject such configurations as soon as we have a design for the monitor subsystem to perform these checks. (In the system emulator, we depend on QemuOpts rejecting duplicate IDs.) Signed-off-by: Kevin Wolf <kwolf@redhat.com> Message-Id: <20200224143008.13362-21-kwolf@redhat.com> Acked-by: Stefan Hajnoczi <stefanha@redhat.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2020-03-05qapi: Brush off some (py)lintMarkus Armbruster
Signed-off-by: Markus Armbruster <armbru@redhat.com> Message-Id: <20200304155932.20452-5-armbru@redhat.com> Reviewed-by: John Snow <jsnow@redhat.com>
2020-03-05qapi: Use super() now we have Python 3Markus Armbruster
Signed-off-by: Markus Armbruster <armbru@redhat.com> Message-Id: <20200304155932.20452-4-armbru@redhat.com> Reviewed-by: John Snow <jsnow@redhat.com>
2020-03-05qapi: Drop conditionals for Python 2Markus Armbruster
Signed-off-by: Markus Armbruster <armbru@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com> Message-Id: <20200304155932.20452-3-armbru@redhat.com> Reviewed-by: John Snow <jsnow@redhat.com>
2020-03-05qapi: Inheriting from object is pointless with Python 3, dropMarkus Armbruster
Signed-off-by: Markus Armbruster <armbru@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com> Message-Id: <20200304155932.20452-2-armbru@redhat.com> Reviewed-by: John Snow <jsnow@redhat.com>
2020-01-14qapi: Simplify QAPISchemaModularCVisitorMarkus Armbruster
Since the previous commit, QAPISchemaVisitor.visit_module() is called just once. Simplify QAPISchemaModularCVisitor accordingly. Signed-off-by: Markus Armbruster <armbru@redhat.com> Message-Id: <20191120182551.23795-7-armbru@redhat.com> Reviewed-by: Eric Blake <eblake@redhat.com>
2019-10-22qapi: Split up scripts/qapi/common.pyMarkus Armbruster
The QAPI code generator clocks in at some 3100 SLOC in 8 source files. Almost 60% of the code is in qapi/common.py. Split it into more focused modules: * Move QAPISchemaPragma and QAPISourceInfo to qapi/source.py. * Move QAPIError and its sub-classes to qapi/error.py. * Move QAPISchemaParser and QAPIDoc to parser.py. Use the opportunity to put QAPISchemaParser first. * Move check_expr() & friends to qapi/expr.py. Use the opportunity to put the code into a more sensible order. * Move QAPISchema & friends to qapi/schema.py * Move QAPIGen and its sub-classes, ifcontext, QAPISchemaModularCVisitor, and QAPISchemaModularCVisitor to qapi/gen.py * Delete camel_case(), it's unused since commit e98859a9b9 "qapi: Clean up after recent conversions to QAPISchemaVisitor" A number of helper functions remain in qapi/common.py. I considered moving the code generator helpers to qapi/gen.py, but decided not to. Perhaps we should rewrite them as methods of QAPIGen some day. Signed-off-by: Markus Armbruster <armbru@redhat.com> Reviewed-by: Eric Blake <eblake@redhat.com> Message-Id: <20191018074345.24034-7-armbru@redhat.com> [Add "# -*- coding: utf-8 -*-" lines]
generated by cgit v1.2.3 (git 2.26.2) at 2024-12-24 14:23:25 +0000