Merge remote-tracking branch 'remotes/kevin/tags/for-upstream' into staging

Block layer patches:

- qcow2: Support for external data files
- qcow2: Default to 4KB for the qcow2 cache entry size
- Apply block driver whitelist for -drive format=help
- Several qemu-iotests improvements

# gpg: Signature made Fri 08 Mar 2019 12:54:27 GMT
# gpg:                using RSA key 7F09B272C88F2FD6
# gpg: Good signature from "Kevin Wolf <kwolf@redhat.com>" [full]
# Primary key fingerprint: DC3D EB15 9A9A F95D 3D74  56FE 7F09 B272 C88F 2FD6

* remotes/kevin/tags/for-upstream: (33 commits)
  qcow2 spec: Describe string header extensions
  qemu-iotests: Add dependency to qemu-nbd tool
  ahci-test: Add dependency to qemu-img tool
  qemu-iotests: amend with external data file
  qemu-iotests: General tests for qcow2 with external data file
  qemu-iotests: Preallocation with external data file
  qcow2: Implement data-file-raw create option
  qcow2: Store data file name in the image
  qcow2: Creating images with external data file
  qcow2: Add basic data-file infrastructure
  qcow2: Support external data file in qemu-img check
  qcow2: Return error for snapshot operation with data file
  qcow2: External file I/O
  qcow2: Prepare qcow2_co_block_status() for data file
  qcow2: Return 0/-errno in qcow2_alloc_compressed_cluster_offset()
  qcow2: Don't assume 0 is an invalid cluster offset
  qcow2: Prepare count_contiguous_clusters() for external data file
  qcow2: Prepare qcow2_get_cluster_type() for external data file
  qcow2: Pass bs to qcow2_get_cluster_type()
  qcow2: Basic definitions for external data files
  ...

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

2 files changed, 60 insertions, 11 deletions

diff --git a/docs/interop/qcow2.txt b/docs/interop/qcow2.txt
index fb5cb47245..8c3098d8d9 100644
--- a/docs/interop/qcow2.txt
+++ b/docs/interop/qcow2.txt
@@ -97,7 +97,19 @@ in the description of a field.
                                 be written to (unless for regaining
                                 consistency).
 
-                    Bits 2-63:  Reserved (set to 0)
+                    Bit 2:      External data file bit.  If this bit is set, an
+                                external data file is used. Guest clusters are
+                                then stored in the external data file. For such
+                                images, clusters in the external data file are
+                                not refcounted. The offset field in the
+                                Standard Cluster Descriptor must match the
+                                guest offset and neither compressed clusters
+                                nor internal snapshots are supported.
+
+                                An External Data File Name header extension may
+                                be present if this bit is set.
+
+                    Bits 3-63:  Reserved (set to 0)
 
          80 -  87:  compatible_features
                     Bitmask of compatible features. An implementation can
@@ -126,7 +138,21 @@ in the description of a field.
                                 bit is unset, the bitmaps extension data must be
                                 considered inconsistent.
 
-                    Bits 1-63:  Reserved (set to 0)
+                    Bit 1:      If this bit is set, the external data file can
+                                be read as a consistent standalone raw image
+                                without looking at the qcow2 metadata.
+
+                                Setting this bit has a performance impact for
+                                some operations on the image (e.g. writing
+                                zeros requires writing to the data file instead
+                                of only setting the zero flag in the L2 table
+                                entry) and conflicts with backing files.
+
+                                This bit may only be set if the External Data
+                                File bit (incompatible feature bit 1) is also
+                                set.
+
+                    Bits 2-63:  Reserved (set to 0)
 
          96 -  99:  refcount_order
                     Describes the width of a reference count block entry (width
@@ -144,10 +170,11 @@ be stored. Each extension has a structure like the following:
 
     Byte  0 -  3:   Header extension type:
                         0x00000000 - End of the header extension area
-                        0xE2792ACA - Backing file format name
+                        0xE2792ACA - Backing file format name string
                         0x6803f857 - Feature name table
                         0x23852875 - Bitmaps extension
                         0x0537be77 - Full disk encryption header pointer
+                        0x44415441 - External data file name string
                         other      - Unknown header extension, can be safely
                                      ignored
 
@@ -169,6 +196,16 @@ data of compatible features that it doesn't support. Compatible features that
 need space for additional data can use a header extension.
 
 
+== String header extensions ==
+
+Some header extensions (such as the backing file format name and the external
+data file name) are just a single string. In this case, the header extension
+length is the string length and the string is not '\0' terminated. (The header
+extension padding can make it look like a string is '\0' terminated, but
+neither is padding always necessary nor is there a guarantee that zero bytes
+are used for padding.)
+
+
 == Feature name table ==
 
 The feature name table is an optional header extension that contains the name
@@ -437,6 +474,11 @@ L2 table entry:
                     This information is only accurate in L2 tables
                     that are reachable from the active L1 table.
 
+                    With external data files, all guest clusters have an
+                    implicit refcount of 1 (because of the fixed host = guest
+                    mapping for guest cluster offsets), so this bit should be 1
+                    for all allocated clusters.
+
 Standard Cluster Descriptor:
 
     Bit       0:    If set to 1, the cluster reads as all zeros. The host
@@ -450,8 +492,10 @@ Standard Cluster Descriptor:
          1 -  8:    Reserved (set to 0)
 
          9 - 55:    Bits 9-55 of host cluster offset. Must be aligned to a
-                    cluster boundary. If the offset is 0, the cluster is
-                    unallocated.
+                    cluster boundary. If the offset is 0 and bit 63 is clear,
+                    the cluster is unallocated. The offset may only be 0 with
+                    bit 63 set (indicating a host cluster offset of 0) when an
+                    external data file is used.
 
         56 - 61:    Reserved (set to 0)
 
diff --git a/docs/qcow2-cache.txt b/docs/qcow2-cache.txt
index c1e7751fea..d57f409861 100644
--- a/docs/qcow2-cache.txt
+++ b/docs/qcow2-cache.txt
@@ -158,10 +158,10 @@ refcount cache is as small as possible unless overridden by the user.
 
 Using smaller cache entries
 ---------------------------
-The qcow2 L2 cache stores complete tables by default. This means that
-if QEMU needs an entry from an L2 table then the whole table is read
-from disk and is kept in the cache. If the cache is full then a
-complete table needs to be evicted first.
+The qcow2 L2 cache can store complete tables. This means that if QEMU
+needs an entry from an L2 table then the whole table is read from disk
+and is kept in the cache. If the cache is full then a complete table
+needs to be evicted first.
 
 This can be inefficient with large cluster sizes since it results in
 more disk I/O and wastes more cache memory.
@@ -172,6 +172,9 @@ it smaller than the cluster size. This can be configured using the
 
    -drive file=hd.qcow2,l2-cache-size=2097152,l2-cache-entry-size=4096
 
+Since QEMU 4.0 the value of l2-cache-entry-size defaults to 4KB (or
+the cluster size if it's smaller).
+
 Some things to take into account:
 
  - The L2 cache entry size has the same restrictions as the cluster
@@ -185,7 +188,8 @@ Some things to take into account:
 
  - Try different entry sizes to see which one gives faster performance
    in your case. The block size of the host filesystem is generally a
-   good default (usually 4096 bytes in the case of ext4).
+   good default (usually 4096 bytes in the case of ext4, hence the
+   default).
 
  - Only the L2 cache can be configured this way. The refcount cache
    always uses the cluster size as the entry size.
@@ -194,7 +198,8 @@ Some things to take into account:
    (as explained in the "Choosing the right cache sizes" and "How to
    configure the cache sizes" sections in this document) then none of
    this is necessary and you can omit the "l2-cache-entry-size"
-   parameter altogether.
+   parameter altogether. In this case QEMU makes the entry size
+   equal to the cluster size by default.
 
 
 Reducing the memory usage