Merge remote-tracking branch 'remotes/ericb/tags/pull-qapi-2018-03-01-v4' into staging

qapi patches for 2018-03-01

- Markus Armbruster: Modularize generated QAPI code

# gpg: Signature made Fri 02 Mar 2018 19:50:16 GMT
# gpg:                using RSA key A7A16B4A2527436A
# gpg: Good signature from "Eric Blake <eblake@redhat.com>"
# gpg:                 aka "Eric Blake (Free Software Programmer) <ebb9@byu.net>"
# gpg:                 aka "[jpeg image of size 6874]"
# Primary key fingerprint: 71C2 CC22 B1C4 6029 27D2  F3AA A7A1 6B4A 2527 436A

* remotes/ericb/tags/pull-qapi-2018-03-01-v4: (30 commits)
  qapi: Don't create useless directory qapi-generated
  Fix up dangling references to qmp-commands.* in comment and doc
  qapi: Move qapi-schema.json to qapi/, rename generated files
  docs: Correct outdated information on QAPI
  docs/devel/writing-qmp-commands: Update for modular QAPI
  qapi: Empty out qapi-schema.json
  Include less of the generated modular QAPI headers
  qapi: Generate separate .h, .c for each module
  watchdog: Consolidate QAPI into single file
  qapi/common: Fix guardname() for funny filenames
  qapi/types qapi/visit: Generate built-in stuff into separate files
  qapi: Make code-generating visitors use QAPIGen more
  qapi: Rename generated qmp-marshal.c to qmp-commands.c
  qapi: Record 'include' directives in intermediate representation
  qapi: Generate in source order
  qapi: Record 'include' directives in parse tree
  qapi: Concentrate QAPISchemaParser.exprs updates in .__init__()
  qapi: Lift error reporting from QAPISchema.__init__() to callers
  qapi/common: Eliminate QAPISchema.exprs
  qapi: Improve include file name reporting in error messages
  ...

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

4 files changed, 78 insertions, 91 deletions

diff --git a/docs/devel/qapi-code-gen.txt b/docs/devel/qapi-code-gen.txt
index 5900b39b91..25b7180a18 100644
--- a/docs/devel/qapi-code-gen.txt
+++ b/docs/devel/qapi-code-gen.txt
@@ -647,7 +647,7 @@ name an event 'MAX', since the generator also produces a C enumeration
 of all event names with a generated _MAX value at the end.  When
 'data' is also specified, additional info will be included in the
 event, with similar semantics to a 'struct' expression.  Finally there
-will be C API generated in qapi-event.h; when called by QEMU code, a
+will be C API generated in qapi-events.h; when called by QEMU code, a
 message with timestamp will be emitted on the wire.
 
 An example event is:
@@ -899,12 +899,13 @@ the names of built-in types.  Clients should examine member
 
 == Code generation ==
 
-Schemas are fed into five scripts to generate all the code/files that,
-paired with the core QAPI libraries, comprise everything required to
-take JSON commands read in by a Client JSON Protocol server, unmarshal
-the arguments into the underlying C types, call into the corresponding
-C function, map the response back to a Client JSON Protocol response
-to be returned to the user, and introspect the commands.
+The QAPI code generator qapi-gen.py generates code and documentation
+from the schema.  Together with the core QAPI libraries, this code
+provides everything required to take JSON commands read in by a Client
+JSON Protocol server, unmarshal the arguments into the underlying C
+types, call into the corresponding C function, map the response back
+to a Client JSON Protocol response to be returned to the user, and
+introspect the commands.
 
 As an example, we'll use the following schema, which describes a
 single complex user-defined type, along with command which takes a
@@ -922,18 +923,23 @@ qmp_my_command(); everything else is produced by the generator.
 
     { 'event': 'MY_EVENT' }
 
+We run qapi-gen.py like this:
+
+    $ python scripts/qapi-gen.py --output-dir="qapi-generated" \
+    --prefix="example-" example-schema.json
+
 For a more thorough look at generated code, the testsuite includes
 tests/qapi-schema/qapi-schema-tests.json that covers more examples of
 what the generator will accept, and compiles the resulting C code as
 part of 'make check-unit'.
 
-=== scripts/qapi-types.py ===
+=== Code generated for QAPI types ===
 
-Used to generate the C types defined by a schema, along with
-supporting code. The following files are created:
+The following files are created:
 
 $(prefix)qapi-types.h - C types corresponding to types defined in
-                        the schema you pass in
+                        the schema
+
 $(prefix)qapi-types.c - Cleanup functions for the above C types
 
 The $(prefix) is an optional parameter used as a namespace to keep the
@@ -943,8 +949,6 @@ created code.
 
 Example:
 
-    $ python scripts/qapi-types.py --output-dir="qapi-generated" \
-    --prefix="example-" example-schema.json
     $ cat qapi-generated/example-qapi-types.h
 [Uninteresting stuff omitted...]
 
@@ -1008,28 +1012,26 @@ Example:
         visit_free(v);
     }
 
-=== scripts/qapi-visit.py ===
+=== Code generated for visiting QAPI types ===
 
-Used to generate the visitor functions used to walk through and
-convert between a native QAPI C data structure and some other format
-(such as QObject); the generated functions are named visit_type_FOO()
-and visit_type_FOO_members().
+These are the visitor functions used to walk through and convert
+between a native QAPI C data structure and some other format (such as
+QObject); the generated functions are named visit_type_FOO() and
+visit_type_FOO_members().
 
 The following files are generated:
 
-$(prefix)qapi-visit.c: visitor function for a particular C type, used
+$(prefix)qapi-visit.c: Visitor function for a particular C type, used
                        to automagically convert QObjects into the
                        corresponding C type and vice-versa, as well
                        as for deallocating memory for an existing C
                        type
 
-$(prefix)qapi-visit.h: declarations for previously mentioned visitor
+$(prefix)qapi-visit.h: Declarations for previously mentioned visitor
                        functions
 
 Example:
 
-    $ python scripts/qapi-visit.py --output-dir="qapi-generated"
-    --prefix="example-" example-schema.json
     $ cat qapi-generated/example-qapi-visit.h
 [Uninteresting stuff omitted...]
 
@@ -1137,31 +1139,23 @@ Example:
         error_propagate(errp, err);
     }
 
-=== scripts/qapi-commands.py ===
+=== Code generated for commands ===
+
+These are the marshaling/dispatch functions for the commands defined
+in the schema.  The generated code provides qmp_marshal_COMMAND(), and
+declares qmp_COMMAND() that the user must implement.
 
-Used to generate the marshaling/dispatch functions for the commands
-defined in the schema. The generated code implements
-qmp_marshal_COMMAND() (registered automatically), and declares
-qmp_COMMAND() that the user must implement.  The following files are
-generated:
+The following files are generated:
 
-$(prefix)qmp-marshal.c: command marshal/dispatch functions for each
-                        QMP command defined in the schema. Functions
-                        generated by qapi-visit.py are used to
-                        convert QObjects received from the wire into
-                        function parameters, and uses the same
-                        visitor functions to convert native C return
-                        values to QObjects from transmission back
-                        over the wire.
+$(prefix)qapi-commands.c: Command marshal/dispatch functions for each
+                          QMP command defined in the schema
 
-$(prefix)qmp-commands.h: Function prototypes for the QMP commands
-                         specified in the schema.
+$(prefix)qapi-commands.h: Function prototypes for the QMP commands
+                          specified in the schema
 
 Example:
 
-    $ python scripts/qapi-commands.py --output-dir="qapi-generated"
-    --prefix="example-" example-schema.json
-    $ cat qapi-generated/example-qmp-commands.h
+    $ cat qapi-generated/example-qapi-commands.h
 [Uninteresting stuff omitted...]
 
     #ifndef EXAMPLE_QMP_COMMANDS_H
@@ -1176,7 +1170,7 @@ Example:
     void qmp_marshal_my_command(QDict *args, QObject **ret, Error **errp);
 
     #endif
-    $ cat qapi-generated/example-qmp-marshal.c
+    $ cat qapi-generated/example-qapi-commands.c
 [Uninteresting stuff omitted...]
 
     static void qmp_marshal_output_UserDefOne(UserDefOne *ret_in, QObject **ret_out, Error **errp)
@@ -1242,21 +1236,21 @@ Example:
                              qmp_marshal_my_command, QCO_NO_OPTIONS);
     }
 
-=== scripts/qapi-event.py ===
+=== Code generated for events ===
 
-Used to generate the event-related C code defined by a schema, with
-implementations for qapi_event_send_FOO(). The following files are
-created:
+This is the code related to events defined in the schema, providing
+qapi_event_send_EVENT().
 
-$(prefix)qapi-event.h - Function prototypes for each event type, plus an
+The following files are created:
+
+$(prefix)qapi-events.h - Function prototypes for each event type, plus an
                         enumeration of all event names
-$(prefix)qapi-event.c - Implementation of functions to send an event
+
+$(prefix)qapi-events.c - Implementation of functions to send an event
 
 Example:
 
-    $ python scripts/qapi-event.py --output-dir="qapi-generated"
-    --prefix="example-" example-schema.json
-    $ cat qapi-generated/example-qapi-event.h
+    $ cat qapi-generated/example-qapi-events.h
 [Uninteresting stuff omitted...]
 
     #ifndef EXAMPLE_QAPI_EVENT_H
@@ -1279,7 +1273,7 @@ Example:
     extern const char *const example_QAPIEvent_lookup[];
 
     #endif
-    $ cat qapi-generated/example-qapi-event.c
+    $ cat qapi-generated/example-qapi-events.c
 [Uninteresting stuff omitted...]
 
     void qapi_event_send_my_event(Error **errp)
@@ -1301,25 +1295,25 @@ Example:
         QDECREF(qmp);
     }
 
-    const char *const example_QAPIEvent_lookup[] = {
-        [EXAMPLE_QAPI_EVENT_MY_EVENT] = "MY_EVENT",
-        [EXAMPLE_QAPI_EVENT__MAX] = NULL,
+    const QEnumLookup example_QAPIEvent_lookup = {
+        .array = (const char *const[]) {
+            [EXAMPLE_QAPI_EVENT_MY_EVENT] = "MY_EVENT",
+        },
+        .size = EXAMPLE_QAPI_EVENT__MAX
     };
 
-=== scripts/qapi-introspect.py ===
+=== Code generated for introspection ===
+
+The following files are created:
 
-Used to generate the introspection C code for a schema. The following
-files are created:
+$(prefix)qapi-introspect.c - Defines a string holding a JSON
+                            description of the schema
 
-$(prefix)qmp-introspect.c - Defines a string holding a JSON
-                            description of the schema.
-$(prefix)qmp-introspect.h - Declares the above string.
+$(prefix)qapi-introspect.h - Declares the above string
 
 Example:
 
-    $ python scripts/qapi-introspect.py --output-dir="qapi-generated"
-    --prefix="example-" example-schema.json
-    $ cat qapi-generated/example-qmp-introspect.h
+    $ cat qapi-generated/example-qapi-introspect.h
 [Uninteresting stuff omitted...]
 
     #ifndef EXAMPLE_QMP_INTROSPECT_H
@@ -1328,7 +1322,7 @@ Example:
     extern const char example_qmp_schema_json[];
 
     #endif
-    $ cat qapi-generated/example-qmp-introspect.c
+    $ cat qapi-generated/example-qapi-introspect.c
 [Uninteresting stuff omitted...]
 
     const char example_qmp_schema_json[] = "["
diff --git a/docs/devel/writing-qmp-commands.txt b/docs/devel/writing-qmp-commands.txt
index 4f5b24c0c4..9dfc62bf5a 100644
--- a/docs/devel/writing-qmp-commands.txt
+++ b/docs/devel/writing-qmp-commands.txt
@@ -15,8 +15,8 @@ start with docs/interop/qmp-intro.txt.
 Generally speaking, the following steps should be taken in order to write a
 new QMP command.
 
-1. Write the command's and type(s) specification in the QAPI schema file
-   (qapi-schema.json in the root source directory)
+1. Define the command and any types it needs in the appropriate QAPI
+   schema module.
 
 2. Write the QMP command itself, which is a regular C function. Preferably,
    the command should be exported by some QEMU subsystem. But it can also be
@@ -36,9 +36,9 @@ very simple and get more complex as we progress.
 For all the examples in the next sections, the test setup is the same and is
 shown here.
 
-First, QEMU should be started as:
+First, QEMU should be started like this:
 
-# /path/to/your/source/qemu [...] \
+# qemu-system-TARGET [...] \
     -chardev socket,id=qmp,port=4444,host=localhost,server \
     -mon chardev=qmp,mode=control,pretty=on
 
@@ -88,8 +88,9 @@ command carries some meaningful action in QEMU but here it will just print
 Our command will be called "hello-world". It takes no arguments, nor does it
 return any data.
 
-The first step is to add the following line to the bottom of the
-qapi-schema.json file:
+The first step is defining the command in the appropriate QAPI schema
+module.  We pick module qapi/misc.json, and add the following line at
+the bottom:
 
 { 'command': 'hello-world' }
 
@@ -178,7 +179,7 @@ described in the "Testing" section and then send two commands:
     }
 }
 
-You should see "Hello, world" and "we love qemu" in the terminal running qemu,
+You should see "Hello, world" and "We love qemu" in the terminal running qemu,
 if you don't see these strings, then something went wrong.
 
 === Errors ===
@@ -220,32 +221,25 @@ The QMP server's response should be:
     }
 }
 
-As a general rule, all QMP errors should use ERROR_CLASS_GENERIC_ERROR
-(done by default when using error_setg()). There are two exceptions to
-this rule:
+Note that error_setg() produces a "GenericError" class.  In general,
+all QMP errors should have that error class.  There are two exceptions
+to this rule:
 
- 1. A non-generic ErrorClass value exists* for the failure you want to report
-    (eg. DeviceNotFound)
+ 1. To support a management application's need to recognize a specific
+    error for special handling
 
- 2. Management applications have to take special action on the failure you
-    want to report, hence you have to add a new ErrorClass value so that they
-    can check for it
+ 2. Backward compatibility
 
 If the failure you want to report falls into one of the two cases above,
 use error_set() with a second argument of an ErrorClass value.
 
- * All existing ErrorClass values are defined in the qapi-schema.json file
-
 === Command Documentation ===
 
 There's only one step missing to make "hello-world"'s implementation complete,
 and that's its documentation in the schema file.
 
-This is very important. No QMP command will be accepted in QEMU without proper
-documentation.
-
 There are many examples of such documentation in the schema file already, but
-here goes "hello-world"'s new entry for the qapi-schema.json file:
+here goes "hello-world"'s new entry for qapi/misc.json:
 
 ##
 # @hello-world
@@ -425,8 +419,7 @@ There are a number of things to be noticed:
    allocated by the implementation. This is so because the QAPI also generates
    a function to free its types and it cannot distinguish between dynamically
    or statically allocated strings
-6. You have to include the "qmp-commands.h" header file in qemu-timer.c,
-   otherwise qemu won't build
+6. You have to include "qapi/qapi-commands-misc.h" in qemu-timer.c
 
 Time to test the new command. Build qemu, run it as described in the "Testing"
 section and try this:
diff --git a/docs/interop/qmp-intro.txt b/docs/interop/qmp-intro.txt
index adbc94abb1..900d69d612 100644
--- a/docs/interop/qmp-intro.txt
+++ b/docs/interop/qmp-intro.txt
@@ -78,7 +78,8 @@ Escape character is '^]'.
     }
 }
 
-Please, refer to the qapi-schema.json file for a complete command reference.
+Please refer to docs/interop/qemu-qmp-ref.* for a complete command
+reference, generated from qapi/qapi-schema.json.
 
 QMP wiki page
 -------------
diff --git a/docs/xen-save-devices-state.txt b/docs/xen-save-devices-state.txt
index a72ecc8081..1912ecad25 100644
--- a/docs/xen-save-devices-state.txt
+++ b/docs/xen-save-devices-state.txt
@@ -8,8 +8,7 @@ These operations are normally used with migration (see migration.txt),
 however it is also possible to save the state of all devices to file,
 without saving the RAM or the block devices of the VM.
 
-This operation is called "xen-save-devices-state" (see
-qmp-commands.txt)
+The save operation is available as QMP command xen-save-devices-state.
 
 
 The binary format used in the file is the following: