Merge remote-tracking branch 'remotes/thuth-gitlab/tags/pull-request-2021-03-09' into staging

* Add some missing gitlab-CI job dependencies
* Re-enable "make check SPEED=slow"
* Improve the gitlab-pipeline-status script
* Clean up inclusing of qtest.h headers
* Improve libqos/qgraph documentation
* Fix downloading problem in the acceptance tests
* Remove deprecated target tilegx
* Add new bsd-user maintainers

# gpg: Signature made Tue 09 Mar 2021 10:27:29 GMT
# gpg:                using RSA key 27B88847EEE0250118F3EAB92ED9D774FE702DB5
# gpg:                issuer "thuth@redhat.com"
# gpg: Good signature from "Thomas Huth <th.huth@gmx.de>" [full]
# gpg:                 aka "Thomas Huth <thuth@redhat.com>" [full]
# gpg:                 aka "Thomas Huth <huth@tuxfamily.org>" [full]
# gpg:                 aka "Thomas Huth <th.huth@posteo.de>" [unknown]
# Primary key fingerprint: 27B8 8847 EEE0 2501 18F3  EAB9 2ED9 D774 FE70 2DB5

* remotes/thuth-gitlab/tags/pull-request-2021-03-09:
  bsd-user: Add new maintainers
  Remove deprecated target tilegx
  Acceptance Tests: restore filtering of tests by target arch
  Acceptance Tests: restore downloading of VM images
  docs/devel/qgraph: improve qgraph documentation
  libqos/qgraph: format qgraph comments for sphinx documentation
  scripts/ci/gitlab-pipeline-status: give more info when pipeline not found
  scripts/ci/gitlab-pipeline-status: give more information on failures
  scripts/ci/gitlab-pipeline-status: split utlity function for HTTP GET
  meson: Re-enable the possibility to run "make check SPEED=slow"
  docker: OpenSBI build job depends on OpenSBI container
  docker: EDK2 build job depends on EDK2 container
  docker: Alpine build job depends on Alpine container
  qtest: delete superfluous inclusions of qtest.h

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

1 files changed, 568 insertions, 0 deletions

diff --git a/docs/devel/qgraph.rst b/docs/devel/qgraph.rst
new file mode 100644
index 0000000000..a9aff167ad
--- /dev/null
+++ b/docs/devel/qgraph.rst
@@ -0,0 +1,568 @@
+.. _qgraph:
+
+========================================
+Qtest Driver Framework
+========================================
+
+In order to test a specific driver, plain libqos tests need to
+take care of booting QEMU with the right machine and devices.
+This makes each test "hardcoded" for a specific configuration, reducing
+the possible coverage that it can reach.
+
+For example, the sdhci device is supported on both x86_64 and ARM boards,
+therefore a generic sdhci test should test all machines and drivers that
+support that device.
+Using only libqos APIs, the test has to manually take care of
+covering all the setups, and build the correct command line.
+
+This also introduces backward compability issues: if a device/driver command
+line name is changed, all tests that use that will not work
+properly anymore and need to be adjusted.
+
+The aim of qgraph is to create a graph of drivers, machines and tests such that
+a test aimed to a certain driver does not have to care of
+booting the right QEMU machine, pick the right device, build the command line
+and so on. Instead, it only defines what type of device it is testing
+(interface in qgraph terms) and the framework takes care of
+covering all supported types of devices and machine architectures.
+
+Following the above example, an interface would be ``sdhci``,
+so the sdhci-test should only care of linking its qgraph node with
+that interface. In this way, if the command line of a sdhci driver
+is changed, only the respective qgraph driver node has to be adjusted.
+
+The graph is composed by nodes that represent machines, drivers, tests
+and edges that define the relationships between them (``CONSUMES``, ``PRODUCES``, and
+``CONTAINS``).
+
+
+Nodes
+^^^^^^
+
+A node can be of four types:
+
+- **QNODE_MACHINE**:   for example ``arm/raspi2``
+- **QNODE_DRIVER**:    for example ``generic-sdhci``
+- **QNODE_INTERFACE**: for example ``sdhci`` (interface for all ``-sdhci``
+  drivers).
+  An interface is not explicitly created, it will be automatically
+  instantiated when a node consumes or produces it.
+  An interface is simply a struct that abstracts the various drivers
+  for the same type of device, and offers an API to the nodes that
+  use it ("consume" relation in qgraph terms) that is implemented/backed up by the drivers that implement it ("produce" relation in qgraph terms).
+- **QNODE_TEST**:      for example ``sdhci-test``. A test consumes an interface
+  and tests the functions provided by it.
+
+Notes for the nodes:
+
+- QNODE_MACHINE: each machine struct must have a ``QGuestAllocator`` and
+  implement ``get_driver()`` to return the allocator mapped to the interface
+  "memory". The function can also return ``NULL`` if the allocator
+  is not set.
+- QNODE_DRIVER:  driver names must be unique, and machines and nodes
+  planned to be "consumed" by other nodes must match QEMU
+  drivers name, otherwise they won't be discovered
+
+Edges
+^^^^^^
+
+An edge relation between two nodes (drivers or machines) `X` and `Y` can be:
+
+- ``X CONSUMES Y``: `Y` can be plugged into `X`
+- ``X PRODUCES Y``: `X` provides the interface `Y`
+- ``X CONTAINS Y``: `Y` is part of `X` component
+
+Execution steps
+^^^^^^^^^^^^^^^
+
+The basic framework steps are the following:
+
+- All nodes and edges are created in their respective
+  machine/driver/test files
+- The framework starts QEMU and asks for a list of available devices
+  and machines (note that only machines and "consumed" nodes are mapped
+  1:1 with QEMU devices)
+- The framework walks the graph starting from the available machines and
+  performs a Depth First Search for tests
+- Once a test is found, the path is walked again and all drivers are
+  allocated accordingly and the final interface is passed to the test
+- The test is executed
+- Unused objects are cleaned and the path discovery is continued
+
+Depending on the QEMU binary used, only some drivers/machines will be
+available and only test that are reached by them will be executed.
+
+Creating a new driver and its interface
+"""""""""""""""""""""""""""""""""""""""""
+
+Here we continue the ``sdhci`` use case, with the following scenario:
+
+- ``sdhci-test`` aims to test the ``read[q,w], writeq`` functions
+  offered by the ``sdhci`` drivers.
+- The current ``sdhci`` device is supported by both ``x86_64/pc`` and ``ARM``
+  (in this example we focus on the ``arm-raspi2``) machines.
+- QEMU offers 2 types of drivers: ``QSDHCI_MemoryMapped`` for ``ARM`` and
+  ``QSDHCI_PCI`` for ``x86_64/pc``. Both implement the
+  ``read[q,w], writeq`` functions.
+
+In order to implement such scenario in qgraph, the test developer needs to:
+
+- Create the ``x86_64/pc`` machine node. This machine uses the
+  ``pci-bus`` architecture so it ``contains`` a PCI driver,
+  ``pci-bus-pc``. The actual path is
+
+  ``x86_64/pc --contains--> 1440FX-pcihost --contains-->
+  pci-bus-pc --produces--> pci-bus``.
+
+  For the sake of this example,
+  we do not focus on the PCI interface implementation.
+- Create the ``sdhci-pci`` driver node, representing ``QSDHCI_PCI``.
+  The driver uses the PCI bus (and its API),
+  so it must ``consume`` the ``pci-bus`` generic interface (which abstracts
+  all the pci drivers available)
+
+  ``sdhci-pci --consumes--> pci-bus``
+- Create an ``arm/raspi2`` machine node. This machine ``contains``
+  a ``generic-sdhci`` memory mapped ``sdhci`` driver node, representing
+  ``QSDHCI_MemoryMapped``.
+
+  ``arm/raspi2 --contains--> generic-sdhci``
+- Create the ``sdhci`` interface node. This interface offers the
+  functions that are shared by all ``sdhci`` devices.
+  The interface is produced by ``sdhci-pci`` and ``generic-sdhci``,
+  the available architecture-specific drivers.
+
+  ``sdhci-pci --produces--> sdhci``
+
+  ``generic-sdhci --produces--> sdhci``
+- Create the ``sdhci-test`` test node. The test ``consumes`` the
+  ``sdhci`` interface, using its API. It doesn't need to look at
+  the supported machines or drivers.
+
+  ``sdhci-test --consumes--> sdhci``
+
+``arm-raspi2`` machine, simplified from
+``tests/qtest/libqos/arm-raspi2-machine.c``::
+
+    #include "qgraph.h"
+
+    struct QRaspi2Machine {
+        QOSGraphObject obj;
+        QGuestAllocator alloc;
+        QSDHCI_MemoryMapped sdhci;
+    };
+
+    static void *raspi2_get_driver(void *object, const char *interface)
+    {
+        QRaspi2Machine *machine = object;
+        if (!g_strcmp0(interface, "memory")) {
+            return &machine->alloc;
+        }
+
+        fprintf(stderr, "%s not present in arm/raspi2\n", interface);
+        g_assert_not_reached();
+    }
+
+    static QOSGraphObject *raspi2_get_device(void *obj,
+                                                const char *device)
+    {
+        QRaspi2Machine *machine = obj;
+        if (!g_strcmp0(device, "generic-sdhci")) {
+            return &machine->sdhci.obj;
+        }
+
+        fprintf(stderr, "%s not present in arm/raspi2\n", device);
+        g_assert_not_reached();
+    }
+
+    static void *qos_create_machine_arm_raspi2(QTestState *qts)
+    {
+        QRaspi2Machine *machine = g_new0(QRaspi2Machine, 1);
+
+        alloc_init(&machine->alloc, ...);
+
+        /* Get node(s) contained inside (CONTAINS) */
+        machine->obj.get_device = raspi2_get_device;
+
+        /* Get node(s) produced (PRODUCES) */
+        machine->obj.get_driver = raspi2_get_driver;
+
+        /* free the object */
+        machine->obj.destructor = raspi2_destructor;
+        qos_init_sdhci_mm(&machine->sdhci, ...);
+        return &machine->obj;
+    }
+
+    static void raspi2_register_nodes(void)
+    {
+        /* arm/raspi2 --contains--> generic-sdhci */
+        qos_node_create_machine("arm/raspi2",
+                                 qos_create_machine_arm_raspi2);
+        qos_node_contains("arm/raspi2", "generic-sdhci", NULL);
+    }
+
+    libqos_init(raspi2_register_nodes);
+
+``x86_64/pc`` machine, simplified from
+``tests/qtest/libqos/x86_64_pc-machine.c``::
+
+    #include "qgraph.h"
+
+    struct i440FX_pcihost {
+        QOSGraphObject obj;
+        QPCIBusPC pci;
+    };
+
+    struct QX86PCMachine {
+        QOSGraphObject obj;
+        QGuestAllocator alloc;
+        i440FX_pcihost bridge;
+    };
+
+    /* i440FX_pcihost */
+
+    static QOSGraphObject *i440FX_host_get_device(void *obj,
+                                                const char *device)
+    {
+        i440FX_pcihost *host = obj;
+        if (!g_strcmp0(device, "pci-bus-pc")) {
+            return &host->pci.obj;
+        }
+        fprintf(stderr, "%s not present in i440FX-pcihost\n", device);
+        g_assert_not_reached();
+    }
+
+    /* x86_64/pc machine */
+
+    static void *pc_get_driver(void *object, const char *interface)
+    {
+        QX86PCMachine *machine = object;
+        if (!g_strcmp0(interface, "memory")) {
+            return &machine->alloc;
+        }
+
+        fprintf(stderr, "%s not present in x86_64/pc\n", interface);
+        g_assert_not_reached();
+    }
+
+    static QOSGraphObject *pc_get_device(void *obj, const char *device)
+    {
+        QX86PCMachine *machine = obj;
+        if (!g_strcmp0(device, "i440FX-pcihost")) {
+            return &machine->bridge.obj;
+        }
+
+        fprintf(stderr, "%s not present in x86_64/pc\n", device);
+        g_assert_not_reached();
+    }
+
+    static void *qos_create_machine_pc(QTestState *qts)
+    {
+        QX86PCMachine *machine = g_new0(QX86PCMachine, 1);
+
+        /* Get node(s) contained inside (CONTAINS) */
+        machine->obj.get_device = pc_get_device;
+
+        /* Get node(s) produced (PRODUCES) */
+        machine->obj.get_driver = pc_get_driver;
+
+        /* free the object */
+        machine->obj.destructor = pc_destructor;
+        pc_alloc_init(&machine->alloc, qts, ALLOC_NO_FLAGS);
+
+        /* Get node(s) contained inside (CONTAINS) */
+        machine->bridge.obj.get_device = i440FX_host_get_device;
+
+        return &machine->obj;
+    }
+
+    static void pc_machine_register_nodes(void)
+    {
+        /* x86_64/pc --contains--> 1440FX-pcihost --contains-->
+         * pci-bus-pc [--produces--> pci-bus (in pci.h)] */
+        qos_node_create_machine("x86_64/pc", qos_create_machine_pc);
+        qos_node_contains("x86_64/pc", "i440FX-pcihost", NULL);
+
+        /* contained drivers don't need a constructor,
+         * they will be init by the parent */
+        qos_node_create_driver("i440FX-pcihost", NULL);
+        qos_node_contains("i440FX-pcihost", "pci-bus-pc", NULL);
+    }
+
+    libqos_init(pc_machine_register_nodes);
+
+``sdhci`` taken from ``tests/qtest/libqos/sdhci.c``::
+
+    /* Interface node, offers the sdhci API */
+    struct QSDHCI {
+        uint16_t (*readw)(QSDHCI *s, uint32_t reg);
+        uint64_t (*readq)(QSDHCI *s, uint32_t reg);
+        void (*writeq)(QSDHCI *s, uint32_t reg, uint64_t val);
+        /* other fields */
+    };
+
+    /* Memory Mapped implementation of QSDHCI */
+    struct QSDHCI_MemoryMapped {
+        QOSGraphObject obj;
+        QSDHCI sdhci;
+        /* other driver-specific fields */
+    };
+
+    /* PCI implementation of QSDHCI */
+    struct QSDHCI_PCI {
+        QOSGraphObject obj;
+        QSDHCI sdhci;
+        /* other driver-specific fields */
+    };
+
+    /* Memory mapped implementation of QSDHCI */
+
+    static void *sdhci_mm_get_driver(void *obj, const char *interface)
+    {
+        QSDHCI_MemoryMapped *smm = obj;
+        if (!g_strcmp0(interface, "sdhci")) {
+            return &smm->sdhci;
+        }
+        fprintf(stderr, "%s not present in generic-sdhci\n", interface);
+        g_assert_not_reached();
+    }
+
+    void qos_init_sdhci_mm(QSDHCI_MemoryMapped *sdhci, QTestState *qts,
+                        uint32_t addr, QSDHCIProperties *common)
+    {
+        /* Get node contained inside (CONTAINS) */
+        sdhci->obj.get_driver = sdhci_mm_get_driver;
+
+        /* SDHCI interface API */
+        sdhci->sdhci.readw = sdhci_mm_readw;
+        sdhci->sdhci.readq = sdhci_mm_readq;
+        sdhci->sdhci.writeq = sdhci_mm_writeq;
+        sdhci->qts = qts;
+    }
+
+    /* PCI implementation of QSDHCI */
+
+    static void *sdhci_pci_get_driver(void *object,
+                                      const char *interface)
+    {
+        QSDHCI_PCI *spci = object;
+        if (!g_strcmp0(interface, "sdhci")) {
+            return &spci->sdhci;
+        }
+
+        fprintf(stderr, "%s not present in sdhci-pci\n", interface);
+        g_assert_not_reached();
+    }
+
+    static void *sdhci_pci_create(void *pci_bus,
+                                  QGuestAllocator *alloc,
+                                  void *addr)
+    {
+        QSDHCI_PCI *spci = g_new0(QSDHCI_PCI, 1);
+        QPCIBus *bus = pci_bus;
+        uint64_t barsize;
+
+        qpci_device_init(&spci->dev, bus, addr);
+
+        /* SDHCI interface API */
+        spci->sdhci.readw = sdhci_pci_readw;
+        spci->sdhci.readq = sdhci_pci_readq;
+        spci->sdhci.writeq = sdhci_pci_writeq;
+
+        /* Get node(s) produced (PRODUCES) */
+        spci->obj.get_driver = sdhci_pci_get_driver;
+
+        spci->obj.start_hw = sdhci_pci_start_hw;
+        spci->obj.destructor = sdhci_destructor;
+        return &spci->obj;
+    }
+
+    static void qsdhci_register_nodes(void)
+    {
+        QOSGraphEdgeOptions opts = {
+            .extra_device_opts = "addr=04.0",
+        };
+
+        /* generic-sdhci */
+        /* generic-sdhci --produces--> sdhci */
+        qos_node_create_driver("generic-sdhci", NULL);
+        qos_node_produces("generic-sdhci", "sdhci");
+
+        /* sdhci-pci */
+        /* sdhci-pci --produces--> sdhci
+         * sdhci-pci --consumes--> pci-bus */
+        qos_node_create_driver("sdhci-pci", sdhci_pci_create);
+        qos_node_produces("sdhci-pci", "sdhci");
+        qos_node_consumes("sdhci-pci", "pci-bus", &opts);
+    }
+
+    libqos_init(qsdhci_register_nodes);
+
+In the above example, all possible types of relations are created::
+
+  x86_64/pc --contains--> 1440FX-pcihost --contains--> pci-bus-pc
+                                                            |
+               sdhci-pci --consumes--> pci-bus <--produces--+
+                  |
+                  +--produces--+
+                               |
+                               v
+                             sdhci
+                               ^
+                               |
+                               +--produces-- +
+                                             |
+               arm/raspi2 --contains--> generic-sdhci
+
+or inverting the consumes edge in consumed_by::
+
+  x86_64/pc --contains--> 1440FX-pcihost --contains--> pci-bus-pc
+                                                            |
+            sdhci-pci <--consumed by-- pci-bus <--produces--+
+                |
+                +--produces--+
+                             |
+                             v
+                            sdhci
+                             ^
+                             |
+                             +--produces-- +
+                                           |
+            arm/raspi2 --contains--> generic-sdhci
+
+Adding a new test
+"""""""""""""""""
+
+Given the above setup, adding a new test is very simple.
+``sdhci-test``, taken from ``tests/qtest/sdhci-test.c``::
+
+    static void check_capab_sdma(QSDHCI *s, bool supported)
+    {
+        uint64_t capab, capab_sdma;
+
+        capab = s->readq(s, SDHC_CAPAB);
+        capab_sdma = FIELD_EX64(capab, SDHC_CAPAB, SDMA);
+        g_assert_cmpuint(capab_sdma, ==, supported);
+    }
+
+    static void test_registers(void *obj, void *data,
+                                QGuestAllocator *alloc)
+    {
+        QSDHCI *s = obj;
+
+        /* example test */
+        check_capab_sdma(s, s->props.capab.sdma);
+    }
+
+    static void register_sdhci_test(void)
+    {
+        /* sdhci-test --consumes--> sdhci */
+        qos_add_test("registers", "sdhci", test_registers, NULL);
+    }
+
+    libqos_init(register_sdhci_test);
+
+Here a new test is created, consuming ``sdhci`` interface node
+and creating a valid path from both machines to a test.
+Final graph will be like this::
+
+  x86_64/pc --contains--> 1440FX-pcihost --contains--> pci-bus-pc
+                                                            |
+               sdhci-pci --consumes--> pci-bus <--produces--+
+                  |
+                  +--produces--+
+                               |
+                               v
+                             sdhci <--consumes-- sdhci-test
+                               ^
+                               |
+                               +--produces-- +
+                                             |
+               arm/raspi2 --contains--> generic-sdhci
+
+or inverting the consumes edge in consumed_by::
+
+  x86_64/pc --contains--> 1440FX-pcihost --contains--> pci-bus-pc
+                                                            |
+            sdhci-pci <--consumed by-- pci-bus <--produces--+
+                |
+                +--produces--+
+                             |
+                             v
+                            sdhci --consumed by--> sdhci-test
+                             ^
+                             |
+                             +--produces-- +
+                                           |
+            arm/raspi2 --contains--> generic-sdhci
+
+Assuming there the binary is
+``QTEST_QEMU_BINARY=./qemu-system-x86_64``
+a valid test path will be:
+``/x86_64/pc/1440FX-pcihost/pci-bus-pc/pci-bus/sdhci-pc/sdhci/sdhci-test``
+
+and for the binary ``QTEST_QEMU_BINARY=./qemu-system-arm``:
+
+``/arm/raspi2/generic-sdhci/sdhci/sdhci-test``
+
+Additional examples are also in ``test-qgraph.c``
+
+Command line:
+""""""""""""""
+
+Command line is built by using node names and optional arguments
+passed by the user when building the edges.
+
+There are three types of command line arguments:
+
+- ``in node``      : created from the node name. For example, machines will
+  have ``-M <machine>`` to its command line, while devices
+  ``-device <device>``. It is automatically done by the framework.
+- ``after node``   : added as additional argument to the node name.
+  This argument is added optionally when creating edges,
+  by setting the parameter ``after_cmd_line`` and
+  ``extra_edge_opts`` in ``QOSGraphEdgeOptions``.
+  The framework automatically adds
+  a comma before ``extra_edge_opts``,
+  because it is going to add attributes
+  after the destination node pointed by
+  the edge containing these options, and automatically
+  adds a space before ``after_cmd_line``, because it
+  adds an additional device, not an attribute.
+- ``before node``  : added as additional argument to the node name.
+  This argument is added optionally when creating edges,
+  by setting the parameter ``before_cmd_line`` in
+  ``QOSGraphEdgeOptions``. This attribute
+  is going to add attributes before the destination node
+  pointed by the edge containing these options. It is
+  helpful to commands that are not node-representable,
+  such as ``-fdsev`` or ``-netdev``.
+
+While adding command line in edges is always used, not all nodes names are
+used in every path walk: this is because the contained or produced ones
+are already added by QEMU, so only nodes that "consumes" will be used to
+build the command line. Also, nodes that will have ``{ "abstract" : true }``
+as QMP attribute will loose their command line, since they are not proper
+devices to be added in QEMU.
+
+Example::
+
+    QOSGraphEdgeOptions opts = {
+        .before_cmd_line = "-drive id=drv0,if=none,file=null-co://,"
+                           "file.read-zeroes=on,format=raw",
+        .after_cmd_line = "-device scsi-hd,bus=vs0.0,drive=drv0",
+
+        opts.extra_device_opts = "id=vs0";
+    };
+
+    qos_node_create_driver("virtio-scsi-device",
+                            virtio_scsi_device_create);
+    qos_node_consumes("virtio-scsi-device", "virtio-bus", &opts);
+
+Will produce the following command line:
+``-drive id=drv0,if=none,file=null-co://, -device virtio-scsi-device,id=vs0 -device scsi-hd,bus=vs0.0,drive=drv0``
+
+Qgraph API reference
+^^^^^^^^^^^^^^^^^^^^
+
+.. kernel-doc:: tests/qtest/libqos/qgraph.h