qcow2: introduce compression type feature

The patch adds some preparation parts for incompatible compression type
feature to qcow2 allowing the use different compression methods for
image clusters (de)compressing.

It is implied that the compression type is set on the image creation and
can be changed only later by image conversion, thus compression type
defines the only compression algorithm used for the image, and thus,
for all image clusters.

The goal of the feature is to add support of other compression methods
to qcow2. For example, ZSTD which is more effective on compression than ZLIB.

The default compression is ZLIB. Images created with ZLIB compression type
are backward compatible with older qemu versions.

Adding of the compression type breaks a number of tests because now the
compression type is reported on image creation and there are some changes
in the qcow2 header in size and offsets.

The tests are fixed in the following ways:
    * filter out compression_type for many tests
    * fix header size, feature table size and backing file offset
      affected tests: 031, 036, 061, 080
      header_size +=8: 1 byte compression type
                       7 bytes padding
      feature_table += 48: incompatible feature compression type
      backing_file_offset += 56 (8 + 48 -> header_change + feature_table_change)
    * add "compression type" for test output matching when it isn't filtered
      affected tests: 049, 060, 061, 065, 082, 085, 144, 182, 185, 198, 206,
                      242, 255, 274, 280

Signed-off-by: Denis Plotnikov <dplotnikov@virtuozzo.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
QAPI part:
Acked-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20200507082521.29210-2-dplotnikov@virtuozzo.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>

1 files changed, 41 insertions, 7 deletions

diff --git a/tests/qemu-iotests/082.out b/tests/qemu-iotests/082.out
index 9d4ed4dc9d..529a1214e1 100644
--- a/tests/qemu-iotests/082.out
+++ b/tests/qemu-iotests/082.out
@@ -3,38 +3,40 @@ QA output created by 082
 === create: Options specified more than once ===
 
 Testing: create -f foo -f qcow2 TEST_DIR/t.qcow2 128M
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=134217728 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=134217728 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
 image: TEST_DIR/t.IMGFMT
 file format: IMGFMT
 virtual size: 128 MiB (134217728 bytes)
 cluster_size: 65536
 
 Testing: create -f qcow2 -o cluster_size=4k -o lazy_refcounts=on TEST_DIR/t.qcow2 128M
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=134217728 cluster_size=4096 lazy_refcounts=on refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=134217728 cluster_size=4096 lazy_refcounts=on refcount_bits=16 compression_type=zlib
 image: TEST_DIR/t.IMGFMT
 file format: IMGFMT
 virtual size: 128 MiB (134217728 bytes)
 cluster_size: 4096
 Format specific information:
     compat: 1.1
+    compression type: zlib
     lazy refcounts: true
     refcount bits: 16
     corrupt: false
 
 Testing: create -f qcow2 -o cluster_size=4k -o lazy_refcounts=on -o cluster_size=8k TEST_DIR/t.qcow2 128M
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=134217728 cluster_size=8192 lazy_refcounts=on refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=134217728 cluster_size=8192 lazy_refcounts=on refcount_bits=16 compression_type=zlib
 image: TEST_DIR/t.IMGFMT
 file format: IMGFMT
 virtual size: 128 MiB (134217728 bytes)
 cluster_size: 8192
 Format specific information:
     compat: 1.1
+    compression type: zlib
     lazy refcounts: true
     refcount bits: 16
     corrupt: false
 
 Testing: create -f qcow2 -o cluster_size=4k,cluster_size=8k TEST_DIR/t.qcow2 128M
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=134217728 cluster_size=8192 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=134217728 cluster_size=8192 lazy_refcounts=off refcount_bits=16 compression_type=zlib
 image: TEST_DIR/t.IMGFMT
 file format: IMGFMT
 virtual size: 128 MiB (134217728 bytes)
@@ -48,6 +50,7 @@ Supported options:
   backing_fmt=<str>      - Image format of the base image
   cluster_size=<size>    - qcow2 cluster size
   compat=<str>           - Compatibility level (v2 [0.10] or v3 [1.1])
+  compression_type=<str> - Compression method used for image cluster compression
   data_file=<str>        - File name of an external data file
   data_file_raw=<bool (on/off)> - The external data file must stay valid as a raw image
   encrypt.cipher-alg=<str> - Name of encryption cipher algorithm
@@ -71,6 +74,7 @@ Supported options:
   backing_fmt=<str>      - Image format of the base image
   cluster_size=<size>    - qcow2 cluster size
   compat=<str>           - Compatibility level (v2 [0.10] or v3 [1.1])
+  compression_type=<str> - Compression method used for image cluster compression
   data_file=<str>        - File name of an external data file
   data_file_raw=<bool (on/off)> - The external data file must stay valid as a raw image
   encrypt.cipher-alg=<str> - Name of encryption cipher algorithm
@@ -94,6 +98,7 @@ Supported options:
   backing_fmt=<str>      - Image format of the base image
   cluster_size=<size>    - qcow2 cluster size
   compat=<str>           - Compatibility level (v2 [0.10] or v3 [1.1])
+  compression_type=<str> - Compression method used for image cluster compression
   data_file=<str>        - File name of an external data file
   data_file_raw=<bool (on/off)> - The external data file must stay valid as a raw image
   encrypt.cipher-alg=<str> - Name of encryption cipher algorithm
@@ -117,6 +122,7 @@ Supported options:
   backing_fmt=<str>      - Image format of the base image
   cluster_size=<size>    - qcow2 cluster size
   compat=<str>           - Compatibility level (v2 [0.10] or v3 [1.1])
+  compression_type=<str> - Compression method used for image cluster compression
   data_file=<str>        - File name of an external data file
   data_file_raw=<bool (on/off)> - The external data file must stay valid as a raw image
   encrypt.cipher-alg=<str> - Name of encryption cipher algorithm
@@ -140,6 +146,7 @@ Supported options:
   backing_fmt=<str>      - Image format of the base image
   cluster_size=<size>    - qcow2 cluster size
   compat=<str>           - Compatibility level (v2 [0.10] or v3 [1.1])
+  compression_type=<str> - Compression method used for image cluster compression
   data_file=<str>        - File name of an external data file
   data_file_raw=<bool (on/off)> - The external data file must stay valid as a raw image
   encrypt.cipher-alg=<str> - Name of encryption cipher algorithm
@@ -163,6 +170,7 @@ Supported options:
   backing_fmt=<str>      - Image format of the base image
   cluster_size=<size>    - qcow2 cluster size
   compat=<str>           - Compatibility level (v2 [0.10] or v3 [1.1])
+  compression_type=<str> - Compression method used for image cluster compression
   data_file=<str>        - File name of an external data file
   data_file_raw=<bool (on/off)> - The external data file must stay valid as a raw image
   encrypt.cipher-alg=<str> - Name of encryption cipher algorithm
@@ -186,6 +194,7 @@ Supported options:
   backing_fmt=<str>      - Image format of the base image
   cluster_size=<size>    - qcow2 cluster size
   compat=<str>           - Compatibility level (v2 [0.10] or v3 [1.1])
+  compression_type=<str> - Compression method used for image cluster compression
   data_file=<str>        - File name of an external data file
   data_file_raw=<bool (on/off)> - The external data file must stay valid as a raw image
   encrypt.cipher-alg=<str> - Name of encryption cipher algorithm
@@ -209,6 +218,7 @@ Supported options:
   backing_fmt=<str>      - Image format of the base image
   cluster_size=<size>    - qcow2 cluster size
   compat=<str>           - Compatibility level (v2 [0.10] or v3 [1.1])
+  compression_type=<str> - Compression method used for image cluster compression
   data_file=<str>        - File name of an external data file
   data_file_raw=<bool (on/off)> - The external data file must stay valid as a raw image
   encrypt.cipher-alg=<str> - Name of encryption cipher algorithm
@@ -227,10 +237,10 @@ Supported options:
   size=<size>            - Virtual disk size
 
 Testing: create -f qcow2 -u -o backing_file=TEST_DIR/t.qcow2,,help TEST_DIR/t.qcow2 128M
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=134217728 backing_file=TEST_DIR/t.qcow2,,help cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=134217728 backing_file=TEST_DIR/t.qcow2,,help cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
 
 Testing: create -f qcow2 -u -o backing_file=TEST_DIR/t.qcow2,,? TEST_DIR/t.qcow2 128M
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=134217728 backing_file=TEST_DIR/t.qcow2,,? cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=134217728 backing_file=TEST_DIR/t.qcow2,,? cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
 
 Testing: create -f qcow2 -o backing_file=TEST_DIR/t.qcow2, -o help TEST_DIR/t.qcow2 128M
 qemu-img: Invalid option list: backing_file=TEST_DIR/t.qcow2,
@@ -247,6 +257,7 @@ Supported qcow2 options:
   backing_fmt=<str>      - Image format of the base image
   cluster_size=<size>    - qcow2 cluster size
   compat=<str>           - Compatibility level (v2 [0.10] or v3 [1.1])
+  compression_type=<str> - Compression method used for image cluster compression
   data_file=<str>        - File name of an external data file
   data_file_raw=<bool (on/off)> - The external data file must stay valid as a raw image
   encrypt.cipher-alg=<str> - Name of encryption cipher algorithm
@@ -279,7 +290,7 @@ qemu-img: Format driver 'bochs' does not support image creation
 === convert: Options specified more than once ===
 
 Testing: create -f qcow2 TEST_DIR/t.qcow2 128M
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=134217728 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=134217728 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
 
 Testing: convert -f foo -f qcow2 TEST_DIR/t.qcow2 TEST_DIR/t.qcow2.base
 image: TEST_DIR/t.IMGFMT.base
@@ -299,6 +310,7 @@ virtual size: 128 MiB (134217728 bytes)
 cluster_size: 4096
 Format specific information:
     compat: 1.1
+    compression type: zlib
     lazy refcounts: true
     refcount bits: 16
     corrupt: false
@@ -310,6 +322,7 @@ virtual size: 128 MiB (134217728 bytes)
 cluster_size: 8192
 Format specific information:
     compat: 1.1
+    compression type: zlib
     lazy refcounts: true
     refcount bits: 16
     corrupt: false
@@ -328,6 +341,7 @@ Supported options:
   backing_fmt=<str>      - Image format of the base image
   cluster_size=<size>    - qcow2 cluster size
   compat=<str>           - Compatibility level (v2 [0.10] or v3 [1.1])
+  compression_type=<str> - Compression method used for image cluster compression
   data_file=<str>        - File name of an external data file
   data_file_raw=<bool (on/off)> - The external data file must stay valid as a raw image
   encrypt.cipher-alg=<str> - Name of encryption cipher algorithm
@@ -351,6 +365,7 @@ Supported options:
   backing_fmt=<str>      - Image format of the base image
   cluster_size=<size>    - qcow2 cluster size
   compat=<str>           - Compatibility level (v2 [0.10] or v3 [1.1])
+  compression_type=<str> - Compression method used for image cluster compression
   data_file=<str>        - File name of an external data file
   data_file_raw=<bool (on/off)> - The external data file must stay valid as a raw image
   encrypt.cipher-alg=<str> - Name of encryption cipher algorithm
@@ -374,6 +389,7 @@ Supported options:
   backing_fmt=<str>      - Image format of the base image
   cluster_size=<size>    - qcow2 cluster size
   compat=<str>           - Compatibility level (v2 [0.10] or v3 [1.1])
+  compression_type=<str> - Compression method used for image cluster compression
   data_file=<str>        - File name of an external data file
   data_file_raw=<bool (on/off)> - The external data file must stay valid as a raw image
   encrypt.cipher-alg=<str> - Name of encryption cipher algorithm
@@ -397,6 +413,7 @@ Supported options:
   backing_fmt=<str>      - Image format of the base image
   cluster_size=<size>    - qcow2 cluster size
   compat=<str>           - Compatibility level (v2 [0.10] or v3 [1.1])
+  compression_type=<str> - Compression method used for image cluster compression
   data_file=<str>        - File name of an external data file
   data_file_raw=<bool (on/off)> - The external data file must stay valid as a raw image
   encrypt.cipher-alg=<str> - Name of encryption cipher algorithm
@@ -420,6 +437,7 @@ Supported options:
   backing_fmt=<str>      - Image format of the base image
   cluster_size=<size>    - qcow2 cluster size
   compat=<str>           - Compatibility level (v2 [0.10] or v3 [1.1])
+  compression_type=<str> - Compression method used for image cluster compression
   data_file=<str>        - File name of an external data file
   data_file_raw=<bool (on/off)> - The external data file must stay valid as a raw image
   encrypt.cipher-alg=<str> - Name of encryption cipher algorithm
@@ -443,6 +461,7 @@ Supported options:
   backing_fmt=<str>      - Image format of the base image
   cluster_size=<size>    - qcow2 cluster size
   compat=<str>           - Compatibility level (v2 [0.10] or v3 [1.1])
+  compression_type=<str> - Compression method used for image cluster compression
   data_file=<str>        - File name of an external data file
   data_file_raw=<bool (on/off)> - The external data file must stay valid as a raw image
   encrypt.cipher-alg=<str> - Name of encryption cipher algorithm
@@ -466,6 +485,7 @@ Supported options:
   backing_fmt=<str>      - Image format of the base image
   cluster_size=<size>    - qcow2 cluster size
   compat=<str>           - Compatibility level (v2 [0.10] or v3 [1.1])
+  compression_type=<str> - Compression method used for image cluster compression
   data_file=<str>        - File name of an external data file
   data_file_raw=<bool (on/off)> - The external data file must stay valid as a raw image
   encrypt.cipher-alg=<str> - Name of encryption cipher algorithm
@@ -489,6 +509,7 @@ Supported options:
   backing_fmt=<str>      - Image format of the base image
   cluster_size=<size>    - qcow2 cluster size
   compat=<str>           - Compatibility level (v2 [0.10] or v3 [1.1])
+  compression_type=<str> - Compression method used for image cluster compression
   data_file=<str>        - File name of an external data file
   data_file_raw=<bool (on/off)> - The external data file must stay valid as a raw image
   encrypt.cipher-alg=<str> - Name of encryption cipher algorithm
@@ -527,6 +548,7 @@ Supported qcow2 options:
   backing_fmt=<str>      - Image format of the base image
   cluster_size=<size>    - qcow2 cluster size
   compat=<str>           - Compatibility level (v2 [0.10] or v3 [1.1])
+  compression_type=<str> - Compression method used for image cluster compression
   data_file=<str>        - File name of an external data file
   data_file_raw=<bool (on/off)> - The external data file must stay valid as a raw image
   encrypt.cipher-alg=<str> - Name of encryption cipher algorithm
@@ -579,6 +601,7 @@ virtual size: 128 MiB (134217728 bytes)
 cluster_size: 65536
 Format specific information:
     compat: 1.1
+    compression type: zlib
     lazy refcounts: true
     refcount bits: 16
     corrupt: false
@@ -590,6 +613,7 @@ virtual size: 130 MiB (136314880 bytes)
 cluster_size: 65536
 Format specific information:
     compat: 1.1
+    compression type: zlib
     lazy refcounts: false
     refcount bits: 16
     corrupt: false
@@ -601,6 +625,7 @@ virtual size: 132 MiB (138412032 bytes)
 cluster_size: 65536
 Format specific information:
     compat: 1.1
+    compression type: zlib
     lazy refcounts: true
     refcount bits: 16
     corrupt: false
@@ -619,6 +644,7 @@ Creation options for 'qcow2':
   backing_fmt=<str>      - Image format of the base image
   cluster_size=<size>    - qcow2 cluster size
   compat=<str>           - Compatibility level (v2 [0.10] or v3 [1.1])
+  compression_type=<str> - Compression method used for image cluster compression
   data_file=<str>        - File name of an external data file
   data_file_raw=<bool (on/off)> - The external data file must stay valid as a raw image
   encrypt.cipher-alg=<str> - Name of encryption cipher algorithm
@@ -643,6 +669,7 @@ Creation options for 'qcow2':
   backing_fmt=<str>      - Image format of the base image
   cluster_size=<size>    - qcow2 cluster size
   compat=<str>           - Compatibility level (v2 [0.10] or v3 [1.1])
+  compression_type=<str> - Compression method used for image cluster compression
   data_file=<str>        - File name of an external data file
   data_file_raw=<bool (on/off)> - The external data file must stay valid as a raw image
   encrypt.cipher-alg=<str> - Name of encryption cipher algorithm
@@ -667,6 +694,7 @@ Creation options for 'qcow2':
   backing_fmt=<str>      - Image format of the base image
   cluster_size=<size>    - qcow2 cluster size
   compat=<str>           - Compatibility level (v2 [0.10] or v3 [1.1])
+  compression_type=<str> - Compression method used for image cluster compression
   data_file=<str>        - File name of an external data file
   data_file_raw=<bool (on/off)> - The external data file must stay valid as a raw image
   encrypt.cipher-alg=<str> - Name of encryption cipher algorithm
@@ -691,6 +719,7 @@ Creation options for 'qcow2':
   backing_fmt=<str>      - Image format of the base image
   cluster_size=<size>    - qcow2 cluster size
   compat=<str>           - Compatibility level (v2 [0.10] or v3 [1.1])
+  compression_type=<str> - Compression method used for image cluster compression
   data_file=<str>        - File name of an external data file
   data_file_raw=<bool (on/off)> - The external data file must stay valid as a raw image
   encrypt.cipher-alg=<str> - Name of encryption cipher algorithm
@@ -715,6 +744,7 @@ Creation options for 'qcow2':
   backing_fmt=<str>      - Image format of the base image
   cluster_size=<size>    - qcow2 cluster size
   compat=<str>           - Compatibility level (v2 [0.10] or v3 [1.1])
+  compression_type=<str> - Compression method used for image cluster compression
   data_file=<str>        - File name of an external data file
   data_file_raw=<bool (on/off)> - The external data file must stay valid as a raw image
   encrypt.cipher-alg=<str> - Name of encryption cipher algorithm
@@ -739,6 +769,7 @@ Creation options for 'qcow2':
   backing_fmt=<str>      - Image format of the base image
   cluster_size=<size>    - qcow2 cluster size
   compat=<str>           - Compatibility level (v2 [0.10] or v3 [1.1])
+  compression_type=<str> - Compression method used for image cluster compression
   data_file=<str>        - File name of an external data file
   data_file_raw=<bool (on/off)> - The external data file must stay valid as a raw image
   encrypt.cipher-alg=<str> - Name of encryption cipher algorithm
@@ -763,6 +794,7 @@ Creation options for 'qcow2':
   backing_fmt=<str>      - Image format of the base image
   cluster_size=<size>    - qcow2 cluster size
   compat=<str>           - Compatibility level (v2 [0.10] or v3 [1.1])
+  compression_type=<str> - Compression method used for image cluster compression
   data_file=<str>        - File name of an external data file
   data_file_raw=<bool (on/off)> - The external data file must stay valid as a raw image
   encrypt.cipher-alg=<str> - Name of encryption cipher algorithm
@@ -787,6 +819,7 @@ Creation options for 'qcow2':
   backing_fmt=<str>      - Image format of the base image
   cluster_size=<size>    - qcow2 cluster size
   compat=<str>           - Compatibility level (v2 [0.10] or v3 [1.1])
+  compression_type=<str> - Compression method used for image cluster compression
   data_file=<str>        - File name of an external data file
   data_file_raw=<bool (on/off)> - The external data file must stay valid as a raw image
   encrypt.cipher-alg=<str> - Name of encryption cipher algorithm
@@ -828,6 +861,7 @@ Creation options for 'qcow2':
   backing_fmt=<str>      - Image format of the base image
   cluster_size=<size>    - qcow2 cluster size
   compat=<str>           - Compatibility level (v2 [0.10] or v3 [1.1])
+  compression_type=<str> - Compression method used for image cluster compression
   data_file=<str>        - File name of an external data file
   data_file_raw=<bool (on/off)> - The external data file must stay valid as a raw image
   encrypt.cipher-alg=<str> - Name of encryption cipher algorithm