aboutsummaryrefslogtreecommitdiff
path: root/docs/interop/vhost-user.rst
blob: 401652397cac6dcb736df745d180502002eabbd7 (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
1001
1002
1003
1004
1005
1006
1007
1008
1009
1010
1011
1012
1013
1014
1015
1016
1017
1018
1019
1020
1021
1022
1023
1024
1025
1026
1027
1028
1029
1030
1031
1032
1033
1034
1035
1036
1037
1038
1039
1040
1041
1042
1043
1044
1045
1046
1047
1048
1049
1050
1051
1052
1053
1054
1055
1056
1057
1058
1059
1060
1061
1062
1063
1064
1065
1066
1067
1068
1069
1070
1071
1072
1073
1074
1075
1076
1077
1078
1079
1080
1081
1082
1083
1084
1085
1086
1087
1088
1089
1090
1091
1092
1093
1094
1095
1096
1097
1098
1099
1100
1101
1102
1103
1104
1105
1106
1107
1108
1109
1110
1111
1112
1113
1114
1115
1116
1117
1118
1119
1120
1121
1122
1123
1124
1125
1126
1127
1128
1129
1130
1131
1132
1133
1134
1135
1136
1137
1138
1139
1140
1141
1142
1143
1144
1145
1146
1147
1148
1149
1150
1151
1152
1153
1154
1155
1156
1157
1158
1159
1160
1161
1162
1163
1164
1165
1166
1167
1168
1169
1170
1171
1172
1173
1174
1175
1176
1177
1178
1179
1180
1181
1182
1183
1184
1185
1186
1187
1188
1189
1190
1191
1192
1193
1194
1195
1196
1197
1198
1199
1200
1201
1202
1203
1204
1205
1206
1207
1208
1209
1210
1211
1212
1213
1214
1215
1216
1217
1218
1219
1220
1221
1222
1223
1224
1225
1226
1227
1228
1229
1230
1231
1232
1233
1234
1235
1236
1237
1238
1239
1240
1241
1242
1243
1244
1245
1246
1247
1248
1249
1250
1251
1252
1253
1254
1255
1256
1257
1258
1259
1260
1261
1262
1263
1264
1265
1266
1267
1268
1269
1270
1271
1272
1273
1274
1275
1276
1277
1278
1279
1280
1281
1282
1283
1284
1285
1286
1287
1288
1289
1290
1291
1292
1293
1294
1295
1296
1297
1298
1299
1300
1301
1302
1303
1304
1305
1306
1307
1308
1309
1310
1311
1312
1313
1314
1315
1316
1317
1318
1319
1320
1321
1322
1323
1324
1325
1326
1327
1328
1329
1330
1331
1332
1333
1334
1335
1336
1337
1338
1339
1340
1341
1342
1343
1344
1345
1346
1347
1348
1349
1350
1351
1352
1353
1354
1355
1356
1357
1358
1359
1360
1361
1362
1363
1364
1365
1366
1367
1368
1369
1370
1371
1372
1373
1374
1375
1376
1377
1378
1379
1380
1381
1382
1383
1384
1385
1386
1387
1388
1389
1390
1391
1392
1393
1394
1395
1396
1397
1398
1399
1400
1401
1402
1403
1404
1405
1406
1407
1408
1409
1410
1411
1412
1413
1414
1415
1416
1417
1418
1419
1420
1421
1422
1423
1424
1425
1426
1427
1428
1429
1430
1431
1432
1433
1434
1435
1436
1437
1438
1439
1440
1441
1442
1443
1444
1445
1446
1447
1448
1449
1450
1451
1452
1453
1454
1455
1456
1457
1458
1459
1460
1461
1462
1463
1464
1465
1466
1467
1468
1469
1470
1471
1472
1473
1474
1475
1476
1477
1478
1479
1480
1481
1482
1483
1484
1485
1486
1487
1488
1489
1490
1491
1492
1493
1494
1495
1496
===================
Vhost-user Protocol
===================
:Copyright: 2014 Virtual Open Systems Sarl.
:Copyright: 2019 Intel Corporation
:Licence: This work is licensed under the terms of the GNU GPL,
          version 2 or later. See the COPYING file in the top-level
          directory.

.. contents:: Table of Contents

Introduction
============

This protocol is aiming to complement the ``ioctl`` interface used to
control the vhost implementation in the Linux kernel. It implements
the control plane needed to establish virtqueue sharing with a user
space process on the same host. It uses communication over a Unix
domain socket to share file descriptors in the ancillary data of the
message.

The protocol defines 2 sides of the communication, *master* and
*slave*. *Master* is the application that shares its virtqueues, in
our case QEMU. *Slave* is the consumer of the virtqueues.

In the current implementation QEMU is the *master*, and the *slave* is
the external process consuming the virtio queues, for example a
software Ethernet switch running in user space, such as Snabbswitch,
or a block device backend processing read & write to a virtual
disk. In order to facilitate interoperability between various backend
implementations, it is recommended to follow the :ref:`Backend program
conventions <backend_conventions>`.

*Master* and *slave* can be either a client (i.e. connecting) or
server (listening) in the socket communication.

Message Specification
=====================

.. Note:: All numbers are in the machine native byte order.

A vhost-user message consists of 3 header fields and a payload.

+---------+-------+------+---------+
| request | flags | size | payload |
+---------+-------+------+---------+

Header
------

:request: 32-bit type of the request

:flags: 32-bit bit field

- Lower 2 bits are the version (currently 0x01)
- Bit 2 is the reply flag - needs to be sent on each reply from the slave
- Bit 3 is the need_reply flag - see :ref:`REPLY_ACK <reply_ack>` for
  details.

:size: 32-bit size of the payload

Payload
-------

Depending on the request type, **payload** can be:

A single 64-bit integer
^^^^^^^^^^^^^^^^^^^^^^^

+-----+
| u64 |
+-----+

:u64: a 64-bit unsigned integer

A vring state description
^^^^^^^^^^^^^^^^^^^^^^^^^

+-------+-----+
| index | num |
+-------+-----+

:index: a 32-bit index

:num: a 32-bit number

A vring address description
^^^^^^^^^^^^^^^^^^^^^^^^^^^

+-------+-------+------+------------+------+-----------+-----+
| index | flags | size | descriptor | used | available | log |
+-------+-------+------+------------+------+-----------+-----+

:index: a 32-bit vring index

:flags: a 32-bit vring flags

:descriptor: a 64-bit ring address of the vring descriptor table

:used: a 64-bit ring address of the vring used ring

:available: a 64-bit ring address of the vring available ring

:log: a 64-bit guest address for logging

Note that a ring address is an IOVA if ``VIRTIO_F_IOMMU_PLATFORM`` has
been negotiated. Otherwise it is a user address.

Memory regions description
^^^^^^^^^^^^^^^^^^^^^^^^^^

+-------------+---------+---------+-----+---------+
| num regions | padding | region0 | ... | region7 |
+-------------+---------+---------+-----+---------+

:num regions: a 32-bit number of regions

:padding: 32-bit

A region is:

+---------------+------+--------------+-------------+
| guest address | size | user address | mmap offset |
+---------------+------+--------------+-------------+

:guest address: a 64-bit guest address of the region

:size: a 64-bit size

:user address: a 64-bit user address

:mmap offset: 64-bit offset where region starts in the mapped memory

Log description
^^^^^^^^^^^^^^^

+----------+------------+
| log size | log offset |
+----------+------------+

:log size: size of area used for logging

:log offset: offset from start of supplied file descriptor where
             logging starts (i.e. where guest address 0 would be
             logged)

An IOTLB message
^^^^^^^^^^^^^^^^

+------+------+--------------+-------------------+------+
| iova | size | user address | permissions flags | type |
+------+------+--------------+-------------------+------+

:iova: a 64-bit I/O virtual address programmed by the guest

:size: a 64-bit size

:user address: a 64-bit user address

:permissions flags: an 8-bit value:
  - 0: No access
  - 1: Read access
  - 2: Write access
  - 3: Read/Write access

:type: an 8-bit IOTLB message type:
  - 1: IOTLB miss
  - 2: IOTLB update
  - 3: IOTLB invalidate
  - 4: IOTLB access fail

Virtio device config space
^^^^^^^^^^^^^^^^^^^^^^^^^^

+--------+------+-------+---------+
| offset | size | flags | payload |
+--------+------+-------+---------+

:offset: a 32-bit offset of virtio device's configuration space

:size: a 32-bit configuration space access size in bytes

:flags: a 32-bit value:
  - 0: Vhost master messages used for writeable fields
  - 1: Vhost master messages used for live migration

:payload: Size bytes array holding the contents of the virtio
          device's configuration space

Vring area description
^^^^^^^^^^^^^^^^^^^^^^

+-----+------+--------+
| u64 | size | offset |
+-----+------+--------+

:u64: a 64-bit integer contains vring index and flags

:size: a 64-bit size of this area

:offset: a 64-bit offset of this area from the start of the
         supplied file descriptor

Inflight description
^^^^^^^^^^^^^^^^^^^^

+-----------+-------------+------------+------------+
| mmap size | mmap offset | num queues | queue size |
+-----------+-------------+------------+------------+

:mmap size: a 64-bit size of area to track inflight I/O

:mmap offset: a 64-bit offset of this area from the start
              of the supplied file descriptor

:num queues: a 16-bit number of virtqueues

:queue size: a 16-bit size of virtqueues

C structure
-----------

In QEMU the vhost-user message is implemented with the following struct:

.. code:: c

  typedef struct VhostUserMsg {
      VhostUserRequest request;
      uint32_t flags;
      uint32_t size;
      union {
          uint64_t u64;
          struct vhost_vring_state state;
          struct vhost_vring_addr addr;
          VhostUserMemory memory;
          VhostUserLog log;
          struct vhost_iotlb_msg iotlb;
          VhostUserConfig config;
          VhostUserVringArea area;
          VhostUserInflight inflight;
      };
  } QEMU_PACKED VhostUserMsg;

Communication
=============

The protocol for vhost-user is based on the existing implementation of
vhost for the Linux Kernel. Most messages that can be sent via the
Unix domain socket implementing vhost-user have an equivalent ioctl to
the kernel implementation.

The communication consists of *master* sending message requests and
*slave* sending message replies. Most of the requests don't require
replies. Here is a list of the ones that do:

* ``VHOST_USER_GET_FEATURES``
* ``VHOST_USER_GET_PROTOCOL_FEATURES``
* ``VHOST_USER_GET_VRING_BASE``
* ``VHOST_USER_SET_LOG_BASE`` (if ``VHOST_USER_PROTOCOL_F_LOG_SHMFD``)
* ``VHOST_USER_GET_INFLIGHT_FD`` (if ``VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD``)

.. seealso::

   :ref:`REPLY_ACK <reply_ack>`
       The section on ``REPLY_ACK`` protocol extension.

There are several messages that the master sends with file descriptors passed
in the ancillary data:

* ``VHOST_USER_SET_MEM_TABLE``
* ``VHOST_USER_SET_LOG_BASE`` (if ``VHOST_USER_PROTOCOL_F_LOG_SHMFD``)
* ``VHOST_USER_SET_LOG_FD``
* ``VHOST_USER_SET_VRING_KICK``
* ``VHOST_USER_SET_VRING_CALL``
* ``VHOST_USER_SET_VRING_ERR``
* ``VHOST_USER_SET_SLAVE_REQ_FD``
* ``VHOST_USER_SET_INFLIGHT_FD`` (if ``VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD``)

If *master* is unable to send the full message or receives a wrong
reply it will close the connection. An optional reconnection mechanism
can be implemented.

If *slave* detects some error such as incompatible features, it may also
close the connection. This should only happen in exceptional circumstances.

Any protocol extensions are gated by protocol feature bits, which
allows full backwards compatibility on both master and slave.  As
older slaves don't support negotiating protocol features, a feature
bit was dedicated for this purpose::

  #define VHOST_USER_F_PROTOCOL_FEATURES 30

Starting and stopping rings
---------------------------

Client must only process each ring when it is started.

Client must only pass data between the ring and the backend, when the
ring is enabled.

If ring is started but disabled, client must process the ring without
talking to the backend.

For example, for a networking device, in the disabled state client
must not supply any new RX packets, but must process and discard any
TX packets.

If ``VHOST_USER_F_PROTOCOL_FEATURES`` has not been negotiated, the
ring is initialized in an enabled state.

If ``VHOST_USER_F_PROTOCOL_FEATURES`` has been negotiated, the ring is
initialized in a disabled state. Client must not pass data to/from the
backend until ring is enabled by ``VHOST_USER_SET_VRING_ENABLE`` with
parameter 1, or after it has been disabled by
``VHOST_USER_SET_VRING_ENABLE`` with parameter 0.

Each ring is initialized in a stopped state, client must not process
it until ring is started, or after it has been stopped.

Client must start ring upon receiving a kick (that is, detecting that
file descriptor is readable) on the descriptor specified by
``VHOST_USER_SET_VRING_KICK`` or receiving the in-band message
``VHOST_USER_VRING_KICK`` if negotiated, and stop ring upon receiving
``VHOST_USER_GET_VRING_BASE``.

While processing the rings (whether they are enabled or not), client
must support changing some configuration aspects on the fly.

Multiple queue support
----------------------

Many devices have a fixed number of virtqueues.  In this case the master
already knows the number of available virtqueues without communicating with the
slave.

Some devices do not have a fixed number of virtqueues.  Instead the maximum
number of virtqueues is chosen by the slave.  The number can depend on host
resource availability or slave implementation details.  Such devices are called
multiple queue devices.

Multiple queue support allows the slave to advertise the maximum number of
queues.  This is treated as a protocol extension, hence the slave has to
implement protocol features first. The multiple queues feature is supported
only when the protocol feature ``VHOST_USER_PROTOCOL_F_MQ`` (bit 0) is set.

The max number of queues the slave supports can be queried with message
``VHOST_USER_GET_QUEUE_NUM``. Master should stop when the number of requested
queues is bigger than that.

As all queues share one connection, the master uses a unique index for each
queue in the sent message to identify a specified queue.

The master enables queues by sending message ``VHOST_USER_SET_VRING_ENABLE``.
vhost-user-net has historically automatically enabled the first queue pair.

Slaves should always implement the ``VHOST_USER_PROTOCOL_F_MQ`` protocol
feature, even for devices with a fixed number of virtqueues, since it is simple
to implement and offers a degree of introspection.

Masters must not rely on the ``VHOST_USER_PROTOCOL_F_MQ`` protocol feature for
devices with a fixed number of virtqueues.  Only true multiqueue devices
require this protocol feature.

Migration
---------

During live migration, the master may need to track the modifications
the slave makes to the memory mapped regions. The client should mark
the dirty pages in a log. Once it complies to this logging, it may
declare the ``VHOST_F_LOG_ALL`` vhost feature.

To start/stop logging of data/used ring writes, server may send
messages ``VHOST_USER_SET_FEATURES`` with ``VHOST_F_LOG_ALL`` and
``VHOST_USER_SET_VRING_ADDR`` with ``VHOST_VRING_F_LOG`` in ring's
flags set to 1/0, respectively.

All the modifications to memory pointed by vring "descriptor" should
be marked. Modifications to "used" vring should be marked if
``VHOST_VRING_F_LOG`` is part of ring's flags.

Dirty pages are of size::

  #define VHOST_LOG_PAGE 0x1000

The log memory fd is provided in the ancillary data of
``VHOST_USER_SET_LOG_BASE`` message when the slave has
``VHOST_USER_PROTOCOL_F_LOG_SHMFD`` protocol feature.

The size of the log is supplied as part of ``VhostUserMsg`` which
should be large enough to cover all known guest addresses. Log starts
at the supplied offset in the supplied file descriptor.  The log
covers from address 0 to the maximum of guest regions. In pseudo-code,
to mark page at ``addr`` as dirty::

  page = addr / VHOST_LOG_PAGE
  log[page / 8] |= 1 << page % 8

Where ``addr`` is the guest physical address.

Use atomic operations, as the log may be concurrently manipulated.

Note that when logging modifications to the used ring (when
``VHOST_VRING_F_LOG`` is set for this ring), ``log_guest_addr`` should
be used to calculate the log offset: the write to first byte of the
used ring is logged at this offset from log start. Also note that this
value might be outside the legal guest physical address range
(i.e. does not have to be covered by the ``VhostUserMemory`` table), but
the bit offset of the last byte of the ring must fall within the size
supplied by ``VhostUserLog``.

``VHOST_USER_SET_LOG_FD`` is an optional message with an eventfd in
ancillary data, it may be used to inform the master that the log has
been modified.

Once the source has finished migration, rings will be stopped by the
source. No further update must be done before rings are restarted.

In postcopy migration the slave is started before all the memory has
been received from the source host, and care must be taken to avoid
accessing pages that have yet to be received.  The slave opens a
'userfault'-fd and registers the memory with it; this fd is then
passed back over to the master.  The master services requests on the
userfaultfd for pages that are accessed and when the page is available
it performs WAKE ioctl's on the userfaultfd to wake the stalled
slave.  The client indicates support for this via the
``VHOST_USER_PROTOCOL_F_PAGEFAULT`` feature.

Memory access
-------------

The master sends a list of vhost memory regions to the slave using the
``VHOST_USER_SET_MEM_TABLE`` message.  Each region has two base
addresses: a guest address and a user address.

Messages contain guest addresses and/or user addresses to reference locations
within the shared memory.  The mapping of these addresses works as follows.

User addresses map to the vhost memory region containing that user address.

When the ``VIRTIO_F_IOMMU_PLATFORM`` feature has not been negotiated:

* Guest addresses map to the vhost memory region containing that guest
  address.

When the ``VIRTIO_F_IOMMU_PLATFORM`` feature has been negotiated:

* Guest addresses are also called I/O virtual addresses (IOVAs).  They are
  translated to user addresses via the IOTLB.

* The vhost memory region guest address is not used.

IOMMU support
-------------

When the ``VIRTIO_F_IOMMU_PLATFORM`` feature has been negotiated, the
master sends IOTLB entries update & invalidation by sending
``VHOST_USER_IOTLB_MSG`` requests to the slave with a ``struct
vhost_iotlb_msg`` as payload. For update events, the ``iotlb`` payload
has to be filled with the update message type (2), the I/O virtual
address, the size, the user virtual address, and the permissions
flags. Addresses and size must be within vhost memory regions set via
the ``VHOST_USER_SET_MEM_TABLE`` request. For invalidation events, the
``iotlb`` payload has to be filled with the invalidation message type
(3), the I/O virtual address and the size. On success, the slave is
expected to reply with a zero payload, non-zero otherwise.

The slave relies on the slave communcation channel (see :ref:`Slave
communication <slave_communication>` section below) to send IOTLB miss
and access failure events, by sending ``VHOST_USER_SLAVE_IOTLB_MSG``
requests to the master with a ``struct vhost_iotlb_msg`` as
payload. For miss events, the iotlb payload has to be filled with the
miss message type (1), the I/O virtual address and the permissions
flags. For access failure event, the iotlb payload has to be filled
with the access failure message type (4), the I/O virtual address and
the permissions flags.  For synchronization purpose, the slave may
rely on the reply-ack feature, so the master may send a reply when
operation is completed if the reply-ack feature is negotiated and
slaves requests a reply. For miss events, completed operation means
either master sent an update message containing the IOTLB entry
containing requested address and permission, or master sent nothing if
the IOTLB miss message is invalid (invalid IOVA or permission).

The master isn't expected to take the initiative to send IOTLB update
messages, as the slave sends IOTLB miss messages for the guest virtual
memory areas it needs to access.

.. _slave_communication:

Slave communication
-------------------

An optional communication channel is provided if the slave declares
``VHOST_USER_PROTOCOL_F_SLAVE_REQ`` protocol feature, to allow the
slave to make requests to the master.

The fd is provided via ``VHOST_USER_SET_SLAVE_REQ_FD`` ancillary data.

A slave may then send ``VHOST_USER_SLAVE_*`` messages to the master
using this fd communication channel.

If ``VHOST_USER_PROTOCOL_F_SLAVE_SEND_FD`` protocol feature is
negotiated, slave can send file descriptors (at most 8 descriptors in
each message) to master via ancillary data using this fd communication
channel.

Inflight I/O tracking
---------------------

To support reconnecting after restart or crash, slave may need to
resubmit inflight I/Os. If virtqueue is processed in order, we can
easily achieve that by getting the inflight descriptors from
descriptor table (split virtqueue) or descriptor ring (packed
virtqueue). However, it can't work when we process descriptors
out-of-order because some entries which store the information of
inflight descriptors in available ring (split virtqueue) or descriptor
ring (packed virtqueue) might be overrided by new entries. To solve
this problem, slave need to allocate an extra buffer to store this
information of inflight descriptors and share it with master for
persistent. ``VHOST_USER_GET_INFLIGHT_FD`` and
``VHOST_USER_SET_INFLIGHT_FD`` are used to transfer this buffer
between master and slave. And the format of this buffer is described
below:

+---------------+---------------+-----+---------------+
| queue0 region | queue1 region | ... | queueN region |
+---------------+---------------+-----+---------------+

N is the number of available virtqueues. Slave could get it from num
queues field of ``VhostUserInflight``.

For split virtqueue, queue region can be implemented as:

.. code:: c

  typedef struct DescStateSplit {
      /* Indicate whether this descriptor is inflight or not.
       * Only available for head-descriptor. */
      uint8_t inflight;

      /* Padding */
      uint8_t padding[5];

      /* Maintain a list for the last batch of used descriptors.
       * Only available when batching is used for submitting */
      uint16_t next;

      /* Used to preserve the order of fetching available descriptors.
       * Only available for head-descriptor. */
      uint64_t counter;
  } DescStateSplit;

  typedef struct QueueRegionSplit {
      /* The feature flags of this region. Now it's initialized to 0. */
      uint64_t features;

      /* The version of this region. It's 1 currently.
       * Zero value indicates an uninitialized buffer */
      uint16_t version;

      /* The size of DescStateSplit array. It's equal to the virtqueue
       * size. Slave could get it from queue size field of VhostUserInflight. */
      uint16_t desc_num;

      /* The head of list that track the last batch of used descriptors. */
      uint16_t last_batch_head;

      /* Store the idx value of used ring */
      uint16_t used_idx;

      /* Used to track the state of each descriptor in descriptor table */
      DescStateSplit desc[0];
  } QueueRegionSplit;

To track inflight I/O, the queue region should be processed as follows:

When receiving available buffers from the driver:

#. Get the next available head-descriptor index from available ring, ``i``

#. Set ``desc[i].counter`` to the value of global counter

#. Increase global counter by 1

#. Set ``desc[i].inflight`` to 1

When supplying used buffers to the driver:

1. Get corresponding used head-descriptor index, i

2. Set ``desc[i].next`` to ``last_batch_head``

3. Set ``last_batch_head`` to ``i``

#. Steps 1,2,3 may be performed repeatedly if batching is possible

#. Increase the ``idx`` value of used ring by the size of the batch

#. Set the ``inflight`` field of each ``DescStateSplit`` entry in the batch to 0

#. Set ``used_idx`` to the ``idx`` value of used ring

When reconnecting:

#. If the value of ``used_idx`` does not match the ``idx`` value of
   used ring (means the inflight field of ``DescStateSplit`` entries in
   last batch may be incorrect),

   a. Subtract the value of ``used_idx`` from the ``idx`` value of
      used ring to get last batch size of ``DescStateSplit`` entries

   #. Set the ``inflight`` field of each ``DescStateSplit`` entry to 0 in last batch
      list which starts from ``last_batch_head``

   #. Set ``used_idx`` to the ``idx`` value of used ring

#. Resubmit inflight ``DescStateSplit`` entries in order of their
   counter value

For packed virtqueue, queue region can be implemented as:

.. code:: c

  typedef struct DescStatePacked {
      /* Indicate whether this descriptor is inflight or not.
       * Only available for head-descriptor. */
      uint8_t inflight;

      /* Padding */
      uint8_t padding;

      /* Link to the next free entry */
      uint16_t next;

      /* Link to the last entry of descriptor list.
       * Only available for head-descriptor. */
      uint16_t last;

      /* The length of descriptor list.
       * Only available for head-descriptor. */
      uint16_t num;

      /* Used to preserve the order of fetching available descriptors.
       * Only available for head-descriptor. */
      uint64_t counter;

      /* The buffer id */
      uint16_t id;

      /* The descriptor flags */
      uint16_t flags;

      /* The buffer length */
      uint32_t len;

      /* The buffer address */
      uint64_t addr;
  } DescStatePacked;

  typedef struct QueueRegionPacked {
      /* The feature flags of this region. Now it's initialized to 0. */
      uint64_t features;

      /* The version of this region. It's 1 currently.
       * Zero value indicates an uninitialized buffer */
      uint16_t version;

      /* The size of DescStatePacked array. It's equal to the virtqueue
       * size. Slave could get it from queue size field of VhostUserInflight. */
      uint16_t desc_num;

      /* The head of free DescStatePacked entry list */
      uint16_t free_head;

      /* The old head of free DescStatePacked entry list */
      uint16_t old_free_head;

      /* The used index of descriptor ring */
      uint16_t used_idx;

      /* The old used index of descriptor ring */
      uint16_t old_used_idx;

      /* Device ring wrap counter */
      uint8_t used_wrap_counter;

      /* The old device ring wrap counter */
      uint8_t old_used_wrap_counter;

      /* Padding */
      uint8_t padding[7];

      /* Used to track the state of each descriptor fetched from descriptor ring */
      DescStatePacked desc[0];
  } QueueRegionPacked;

To track inflight I/O, the queue region should be processed as follows:

When receiving available buffers from the driver:

#. Get the next available descriptor entry from descriptor ring, ``d``

#. If ``d`` is head descriptor,

   a. Set ``desc[old_free_head].num`` to 0

   #. Set ``desc[old_free_head].counter`` to the value of global counter

   #. Increase global counter by 1

   #. Set ``desc[old_free_head].inflight`` to 1

#. If ``d`` is last descriptor, set ``desc[old_free_head].last`` to
   ``free_head``

#. Increase ``desc[old_free_head].num`` by 1

#. Set ``desc[free_head].addr``, ``desc[free_head].len``,
   ``desc[free_head].flags``, ``desc[free_head].id`` to ``d.addr``,
   ``d.len``, ``d.flags``, ``d.id``

#. Set ``free_head`` to ``desc[free_head].next``

#. If ``d`` is last descriptor, set ``old_free_head`` to ``free_head``

When supplying used buffers to the driver:

1. Get corresponding used head-descriptor entry from descriptor ring,
   ``d``

2. Get corresponding ``DescStatePacked`` entry, ``e``

3. Set ``desc[e.last].next`` to ``free_head``

4. Set ``free_head`` to the index of ``e``

#. Steps 1,2,3,4 may be performed repeatedly if batching is possible

#. Increase ``used_idx`` by the size of the batch and update
   ``used_wrap_counter`` if needed

#. Update ``d.flags``

#. Set the ``inflight`` field of each head ``DescStatePacked`` entry
   in the batch to 0

#. Set ``old_free_head``,  ``old_used_idx``, ``old_used_wrap_counter``
   to ``free_head``, ``used_idx``, ``used_wrap_counter``

When reconnecting:

#. If ``used_idx`` does not match ``old_used_idx`` (means the
   ``inflight`` field of ``DescStatePacked`` entries in last batch may
   be incorrect),

   a. Get the next descriptor ring entry through ``old_used_idx``, ``d``

   #. Use ``old_used_wrap_counter`` to calculate the available flags

   #. If ``d.flags`` is not equal to the calculated flags value (means
      slave has submitted the buffer to guest driver before crash, so
      it has to commit the in-progres update), set ``old_free_head``,
      ``old_used_idx``, ``old_used_wrap_counter`` to ``free_head``,
      ``used_idx``, ``used_wrap_counter``

#. Set ``free_head``, ``used_idx``, ``used_wrap_counter`` to
   ``old_free_head``, ``old_used_idx``, ``old_used_wrap_counter``
   (roll back any in-progress update)

#. Set the ``inflight`` field of each ``DescStatePacked`` entry in
   free list to 0

#. Resubmit inflight ``DescStatePacked`` entries in order of their
   counter value

In-band notifications
---------------------

In some limited situations (e.g. for simulation) it is desirable to
have the kick, call and error (if used) signals done via in-band
messages instead of asynchronous eventfd notifications. This can be
done by negotiating the ``VHOST_USER_PROTOCOL_F_INBAND_NOTIFICATIONS``
protocol feature.

Note that due to the fact that too many messages on the sockets can
cause the sending application(s) to block, it is not advised to use
this feature unless absolutely necessary. It is also considered an
error to negotiate this feature without also negotiating
``VHOST_USER_PROTOCOL_F_SLAVE_REQ`` and ``VHOST_USER_PROTOCOL_F_REPLY_ACK``,
the former is necessary for getting a message channel from the slave
to the master, while the latter needs to be used with the in-band
notification messages to block until they are processed, both to avoid
blocking later and for proper processing (at least in the simulation
use case.) As it has no other way of signalling this error, the slave
should close the connection as a response to a
``VHOST_USER_SET_PROTOCOL_FEATURES`` message that sets the in-band
notifications feature flag without the other two.

Protocol features
-----------------

.. code:: c

  #define VHOST_USER_PROTOCOL_F_MQ                    0
  #define VHOST_USER_PROTOCOL_F_LOG_SHMFD             1
  #define VHOST_USER_PROTOCOL_F_RARP                  2
  #define VHOST_USER_PROTOCOL_F_REPLY_ACK             3
  #define VHOST_USER_PROTOCOL_F_MTU                   4
  #define VHOST_USER_PROTOCOL_F_SLAVE_REQ             5
  #define VHOST_USER_PROTOCOL_F_CROSS_ENDIAN          6
  #define VHOST_USER_PROTOCOL_F_CRYPTO_SESSION        7
  #define VHOST_USER_PROTOCOL_F_PAGEFAULT             8
  #define VHOST_USER_PROTOCOL_F_CONFIG                9
  #define VHOST_USER_PROTOCOL_F_SLAVE_SEND_FD        10
  #define VHOST_USER_PROTOCOL_F_HOST_NOTIFIER        11
  #define VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD       12
  #define VHOST_USER_PROTOCOL_F_RESET_DEVICE         13
  #define VHOST_USER_PROTOCOL_F_INBAND_NOTIFICATIONS 14

Master message types
--------------------

``VHOST_USER_GET_FEATURES``
  :id: 1
  :equivalent ioctl: ``VHOST_GET_FEATURES``
  :master payload: N/A
  :slave payload: ``u64``

  Get from the underlying vhost implementation the features bitmask.
  Feature bit ``VHOST_USER_F_PROTOCOL_FEATURES`` signals slave support
  for ``VHOST_USER_GET_PROTOCOL_FEATURES`` and
  ``VHOST_USER_SET_PROTOCOL_FEATURES``.

``VHOST_USER_SET_FEATURES``
  :id: 2
  :equivalent ioctl: ``VHOST_SET_FEATURES``
  :master payload: ``u64``

  Enable features in the underlying vhost implementation using a
  bitmask.  Feature bit ``VHOST_USER_F_PROTOCOL_FEATURES`` signals
  slave support for ``VHOST_USER_GET_PROTOCOL_FEATURES`` and
  ``VHOST_USER_SET_PROTOCOL_FEATURES``.

``VHOST_USER_GET_PROTOCOL_FEATURES``
  :id: 15
  :equivalent ioctl: ``VHOST_GET_FEATURES``
  :master payload: N/A
  :slave payload: ``u64``

  Get the protocol feature bitmask from the underlying vhost
  implementation.  Only legal if feature bit
  ``VHOST_USER_F_PROTOCOL_FEATURES`` is present in
  ``VHOST_USER_GET_FEATURES``.

.. Note::
   Slave that reported ``VHOST_USER_F_PROTOCOL_FEATURES`` must
   support this message even before ``VHOST_USER_SET_FEATURES`` was
   called.

``VHOST_USER_SET_PROTOCOL_FEATURES``
  :id: 16
  :equivalent ioctl: ``VHOST_SET_FEATURES``
  :master payload: ``u64``

  Enable protocol features in the underlying vhost implementation.

  Only legal if feature bit ``VHOST_USER_F_PROTOCOL_FEATURES`` is present in
  ``VHOST_USER_GET_FEATURES``.

.. Note::
   Slave that reported ``VHOST_USER_F_PROTOCOL_FEATURES`` must support
   this message even before ``VHOST_USER_SET_FEATURES`` was called.

``VHOST_USER_SET_OWNER``
  :id: 3
  :equivalent ioctl: ``VHOST_SET_OWNER``
  :master payload: N/A

  Issued when a new connection is established. It sets the current
  *master* as an owner of the session. This can be used on the *slave*
  as a "session start" flag.

``VHOST_USER_RESET_OWNER``
  :id: 4
  :master payload: N/A

.. admonition:: Deprecated

   This is no longer used. Used to be sent to request disabling all
   rings, but some clients interpreted it to also discard connection
   state (this interpretation would lead to bugs).  It is recommended
   that clients either ignore this message, or use it to disable all
   rings.

``VHOST_USER_SET_MEM_TABLE``
  :id: 5
  :equivalent ioctl: ``VHOST_SET_MEM_TABLE``
  :master payload: memory regions description
  :slave payload: (postcopy only) memory regions description

  Sets the memory map regions on the slave so it can translate the
  vring addresses. In the ancillary data there is an array of file
  descriptors for each memory mapped region. The size and ordering of
  the fds matches the number and ordering of memory regions.

  When ``VHOST_USER_POSTCOPY_LISTEN`` has been received,
  ``SET_MEM_TABLE`` replies with the bases of the memory mapped
  regions to the master.  The slave must have mmap'd the regions but
  not yet accessed them and should not yet generate a userfault
  event.

.. Note::
   ``NEED_REPLY_MASK`` is not set in this case.  QEMU will then
   reply back to the list of mappings with an empty
   ``VHOST_USER_SET_MEM_TABLE`` as an acknowledgement; only upon
   reception of this message may the guest start accessing the memory
   and generating faults.

``VHOST_USER_SET_LOG_BASE``
  :id: 6
  :equivalent ioctl: ``VHOST_SET_LOG_BASE``
  :master payload: u64
  :slave payload: N/A

  Sets logging shared memory space.

  When slave has ``VHOST_USER_PROTOCOL_F_LOG_SHMFD`` protocol feature,
  the log memory fd is provided in the ancillary data of
  ``VHOST_USER_SET_LOG_BASE`` message, the size and offset of shared
  memory area provided in the message.

``VHOST_USER_SET_LOG_FD``
  :id: 7
  :equivalent ioctl: ``VHOST_SET_LOG_FD``
  :master payload: N/A

  Sets the logging file descriptor, which is passed as ancillary data.

``VHOST_USER_SET_VRING_NUM``
  :id: 8
  :equivalent ioctl: ``VHOST_SET_VRING_NUM``
  :master payload: vring state description

  Set the size of the queue.

``VHOST_USER_SET_VRING_ADDR``
  :id: 9
  :equivalent ioctl: ``VHOST_SET_VRING_ADDR``
  :master payload: vring address description
  :slave payload: N/A

  Sets the addresses of the different aspects of the vring.

``VHOST_USER_SET_VRING_BASE``
  :id: 10
  :equivalent ioctl: ``VHOST_SET_VRING_BASE``
  :master payload: vring state description

  Sets the base offset in the available vring.

``VHOST_USER_GET_VRING_BASE``
  :id: 11
  :equivalent ioctl: ``VHOST_USER_GET_VRING_BASE``
  :master payload: vring state description
  :slave payload: vring state description

  Get the available vring base offset.

``VHOST_USER_SET_VRING_KICK``
  :id: 12
  :equivalent ioctl: ``VHOST_SET_VRING_KICK``
  :master payload: ``u64``

  Set the event file descriptor for adding buffers to the vring. It is
  passed in the ancillary data.

  Bits (0-7) of the payload contain the vring index. Bit 8 is the
  invalid FD flag. This flag is set when there is no file descriptor
  in the ancillary data. This signals that polling should be used
  instead of waiting for the kick. Note that if the protocol feature
  ``VHOST_USER_PROTOCOL_F_INBAND_NOTIFICATIONS`` has been negotiated
  this message isn't necessary as the ring is also started on the
  ``VHOST_USER_VRING_KICK`` message, it may however still be used to
  set an event file descriptor (which will be preferred over the
  message) or to enable polling.

``VHOST_USER_SET_VRING_CALL``
  :id: 13
  :equivalent ioctl: ``VHOST_SET_VRING_CALL``
  :master payload: ``u64``

  Set the event file descriptor to signal when buffers are used. It is
  passed in the ancillary data.

  Bits (0-7) of the payload contain the vring index. Bit 8 is the
  invalid FD flag. This flag is set when there is no file descriptor
  in the ancillary data. This signals that polling will be used
  instead of waiting for the call. Note that if the protocol features
  ``VHOST_USER_PROTOCOL_F_INBAND_NOTIFICATIONS`` and
  ``VHOST_USER_PROTOCOL_F_SLAVE_REQ`` have been negotiated this message
  isn't necessary as the ``VHOST_USER_SLAVE_VRING_CALL`` message can be
  used, it may however still be used to set an event file descriptor
  or to enable polling.

``VHOST_USER_SET_VRING_ERR``
  :id: 14
  :equivalent ioctl: ``VHOST_SET_VRING_ERR``
  :master payload: ``u64``

  Set the event file descriptor to signal when error occurs. It is
  passed in the ancillary data.

  Bits (0-7) of the payload contain the vring index. Bit 8 is the
  invalid FD flag. This flag is set when there is no file descriptor
  in the ancillary data. Note that if the protocol features
  ``VHOST_USER_PROTOCOL_F_INBAND_NOTIFICATIONS`` and
  ``VHOST_USER_PROTOCOL_F_SLAVE_REQ`` have been negotiated this message
  isn't necessary as the ``VHOST_USER_SLAVE_VRING_ERR`` message can be
  used, it may however still be used to set an event file descriptor
  (which will be preferred over the message).

``VHOST_USER_GET_QUEUE_NUM``
  :id: 17
  :equivalent ioctl: N/A
  :master payload: N/A
  :slave payload: u64

  Query how many queues the backend supports.

  This request should be sent only when ``VHOST_USER_PROTOCOL_F_MQ``
  is set in queried protocol features by
  ``VHOST_USER_GET_PROTOCOL_FEATURES``.

``VHOST_USER_SET_VRING_ENABLE``
  :id: 18
  :equivalent ioctl: N/A
  :master payload: vring state description

  Signal slave to enable or disable corresponding vring.

  This request should be sent only when
  ``VHOST_USER_F_PROTOCOL_FEATURES`` has been negotiated.

``VHOST_USER_SEND_RARP``
  :id: 19
  :equivalent ioctl: N/A
  :master payload: ``u64``

  Ask vhost user backend to broadcast a fake RARP to notify the migration
  is terminated for guest that does not support GUEST_ANNOUNCE.

  Only legal if feature bit ``VHOST_USER_F_PROTOCOL_FEATURES`` is
  present in ``VHOST_USER_GET_FEATURES`` and protocol feature bit
  ``VHOST_USER_PROTOCOL_F_RARP`` is present in
  ``VHOST_USER_GET_PROTOCOL_FEATURES``.  The first 6 bytes of the
  payload contain the mac address of the guest to allow the vhost user
  backend to construct and broadcast the fake RARP.

``VHOST_USER_NET_SET_MTU``
  :id: 20
  :equivalent ioctl: N/A
  :master payload: ``u64``

  Set host MTU value exposed to the guest.

  This request should be sent only when ``VIRTIO_NET_F_MTU`` feature
  has been successfully negotiated, ``VHOST_USER_F_PROTOCOL_FEATURES``
  is present in ``VHOST_USER_GET_FEATURES`` and protocol feature bit
  ``VHOST_USER_PROTOCOL_F_NET_MTU`` is present in
  ``VHOST_USER_GET_PROTOCOL_FEATURES``.

  If ``VHOST_USER_PROTOCOL_F_REPLY_ACK`` is negotiated, slave must
  respond with zero in case the specified MTU is valid, or non-zero
  otherwise.

``VHOST_USER_SET_SLAVE_REQ_FD``
  :id: 21
  :equivalent ioctl: N/A
  :master payload: N/A

  Set the socket file descriptor for slave initiated requests. It is passed
  in the ancillary data.

  This request should be sent only when
  ``VHOST_USER_F_PROTOCOL_FEATURES`` has been negotiated, and protocol
  feature bit ``VHOST_USER_PROTOCOL_F_SLAVE_REQ`` bit is present in
  ``VHOST_USER_GET_PROTOCOL_FEATURES``.  If
  ``VHOST_USER_PROTOCOL_F_REPLY_ACK`` is negotiated, slave must
  respond with zero for success, non-zero otherwise.

``VHOST_USER_IOTLB_MSG``
  :id: 22
  :equivalent ioctl: N/A (equivalent to ``VHOST_IOTLB_MSG`` message type)
  :master payload: ``struct vhost_iotlb_msg``
  :slave payload: ``u64``

  Send IOTLB messages with ``struct vhost_iotlb_msg`` as payload.

  Master sends such requests to update and invalidate entries in the
  device IOTLB. The slave has to acknowledge the request with sending
  zero as ``u64`` payload for success, non-zero otherwise.

  This request should be send only when ``VIRTIO_F_IOMMU_PLATFORM``
  feature has been successfully negotiated.

``VHOST_USER_SET_VRING_ENDIAN``
  :id: 23
  :equivalent ioctl: ``VHOST_SET_VRING_ENDIAN``
  :master payload: vring state description

  Set the endianness of a VQ for legacy devices. Little-endian is
  indicated with state.num set to 0 and big-endian is indicated with
  state.num set to 1. Other values are invalid.

  This request should be sent only when
  ``VHOST_USER_PROTOCOL_F_CROSS_ENDIAN`` has been negotiated.
  Backends that negotiated this feature should handle both
  endiannesses and expect this message once (per VQ) during device
  configuration (ie. before the master starts the VQ).

``VHOST_USER_GET_CONFIG``
  :id: 24
  :equivalent ioctl: N/A
  :master payload: virtio device config space
  :slave payload: virtio device config space

  When ``VHOST_USER_PROTOCOL_F_CONFIG`` is negotiated, this message is
  submitted by the vhost-user master to fetch the contents of the
  virtio device configuration space, vhost-user slave's payload size
  MUST match master's request, vhost-user slave uses zero length of
  payload to indicate an error to vhost-user master. The vhost-user
  master may cache the contents to avoid repeated
  ``VHOST_USER_GET_CONFIG`` calls.

``VHOST_USER_SET_CONFIG``
  :id: 25
  :equivalent ioctl: N/A
  :master payload: virtio device config space
  :slave payload: N/A

  When ``VHOST_USER_PROTOCOL_F_CONFIG`` is negotiated, this message is
  submitted by the vhost-user master when the Guest changes the virtio
  device configuration space and also can be used for live migration
  on the destination host. The vhost-user slave must check the flags
  field, and slaves MUST NOT accept SET_CONFIG for read-only
  configuration space fields unless the live migration bit is set.

``VHOST_USER_CREATE_CRYPTO_SESSION``
  :id: 26
  :equivalent ioctl: N/A
  :master payload: crypto session description
  :slave payload: crypto session description

  Create a session for crypto operation. The server side must return
  the session id, 0 or positive for success, negative for failure.
  This request should be sent only when
  ``VHOST_USER_PROTOCOL_F_CRYPTO_SESSION`` feature has been
  successfully negotiated.  It's a required feature for crypto
  devices.

``VHOST_USER_CLOSE_CRYPTO_SESSION``
  :id: 27
  :equivalent ioctl: N/A
  :master payload: ``u64``

  Close a session for crypto operation which was previously
  created by ``VHOST_USER_CREATE_CRYPTO_SESSION``.

  This request should be sent only when
  ``VHOST_USER_PROTOCOL_F_CRYPTO_SESSION`` feature has been
  successfully negotiated.  It's a required feature for crypto
  devices.

``VHOST_USER_POSTCOPY_ADVISE``
  :id: 28
  :master payload: N/A
  :slave payload: userfault fd

  When ``VHOST_USER_PROTOCOL_F_PAGEFAULT`` is supported, the master
  advises slave that a migration with postcopy enabled is underway,
  the slave must open a userfaultfd for later use.  Note that at this
  stage the migration is still in precopy mode.

``VHOST_USER_POSTCOPY_LISTEN``
  :id: 29
  :master payload: N/A

  Master advises slave that a transition to postcopy mode has
  happened.  The slave must ensure that shared memory is registered
  with userfaultfd to cause faulting of non-present pages.

  This is always sent sometime after a ``VHOST_USER_POSTCOPY_ADVISE``,
  and thus only when ``VHOST_USER_PROTOCOL_F_PAGEFAULT`` is supported.

``VHOST_USER_POSTCOPY_END``
  :id: 30
  :slave payload: ``u64``

  Master advises that postcopy migration has now completed.  The slave
  must disable the userfaultfd. The response is an acknowledgement
  only.

  When ``VHOST_USER_PROTOCOL_F_PAGEFAULT`` is supported, this message
  is sent at the end of the migration, after
  ``VHOST_USER_POSTCOPY_LISTEN`` was previously sent.

  The value returned is an error indication; 0 is success.

``VHOST_USER_GET_INFLIGHT_FD``
  :id: 31
  :equivalent ioctl: N/A
  :master payload: inflight description

  When ``VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD`` protocol feature has
  been successfully negotiated, this message is submitted by master to
  get a shared buffer from slave. The shared buffer will be used to
  track inflight I/O by slave. QEMU should retrieve a new one when vm
  reset.

``VHOST_USER_SET_INFLIGHT_FD``
  :id: 32
  :equivalent ioctl: N/A
  :master payload: inflight description

  When ``VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD`` protocol feature has
  been successfully negotiated, this message is submitted by master to
  send the shared inflight buffer back to slave so that slave could
  get inflight I/O after a crash or restart.

``VHOST_USER_GPU_SET_SOCKET``
  :id: 33
  :equivalent ioctl: N/A
  :master payload: N/A

  Sets the GPU protocol socket file descriptor, which is passed as
  ancillary data. The GPU protocol is used to inform the master of
  rendering state and updates. See vhost-user-gpu.rst for details.

``VHOST_USER_RESET_DEVICE``
  :id: 34
  :equivalent ioctl: N/A
  :master payload: N/A
  :slave payload: N/A

  Ask the vhost user backend to disable all rings and reset all
  internal device state to the initial state, ready to be
  reinitialized. The backend retains ownership of the device
  throughout the reset operation.

  Only valid if the ``VHOST_USER_PROTOCOL_F_RESET_DEVICE`` protocol
  feature is set by the backend.

``VHOST_USER_VRING_KICK``
  :id: 35
  :equivalent ioctl: N/A
  :slave payload: vring state description
  :master payload: N/A

  When the ``VHOST_USER_PROTOCOL_F_INBAND_NOTIFICATIONS`` protocol
  feature has been successfully negotiated, this message may be
  submitted by the master to indicate that a buffer was added to
  the vring instead of signalling it using the vring's kick file
  descriptor or having the slave rely on polling.

  The state.num field is currently reserved and must be set to 0.

Slave message types
-------------------

``VHOST_USER_SLAVE_IOTLB_MSG``
  :id: 1
  :equivalent ioctl: N/A (equivalent to ``VHOST_IOTLB_MSG`` message type)
  :slave payload: ``struct vhost_iotlb_msg``
  :master payload: N/A

  Send IOTLB messages with ``struct vhost_iotlb_msg`` as payload.
  Slave sends such requests to notify of an IOTLB miss, or an IOTLB
  access failure. If ``VHOST_USER_PROTOCOL_F_REPLY_ACK`` is
  negotiated, and slave set the ``VHOST_USER_NEED_REPLY`` flag, master
  must respond with zero when operation is successfully completed, or
  non-zero otherwise.  This request should be send only when
  ``VIRTIO_F_IOMMU_PLATFORM`` feature has been successfully
  negotiated.

``VHOST_USER_SLAVE_CONFIG_CHANGE_MSG``
  :id: 2
  :equivalent ioctl: N/A
  :slave payload: N/A
  :master payload: N/A

  When ``VHOST_USER_PROTOCOL_F_CONFIG`` is negotiated, vhost-user
  slave sends such messages to notify that the virtio device's
  configuration space has changed, for those host devices which can
  support such feature, host driver can send ``VHOST_USER_GET_CONFIG``
  message to slave to get the latest content. If
  ``VHOST_USER_PROTOCOL_F_REPLY_ACK`` is negotiated, and slave set the
  ``VHOST_USER_NEED_REPLY`` flag, master must respond with zero when
  operation is successfully completed, or non-zero otherwise.

``VHOST_USER_SLAVE_VRING_HOST_NOTIFIER_MSG``
  :id: 3
  :equivalent ioctl: N/A
  :slave payload: vring area description
  :master payload: N/A

  Sets host notifier for a specified queue. The queue index is
  contained in the ``u64`` field of the vring area description. The
  host notifier is described by the file descriptor (typically it's a
  VFIO device fd) which is passed as ancillary data and the size
  (which is mmap size and should be the same as host page size) and
  offset (which is mmap offset) carried in the vring area
  description. QEMU can mmap the file descriptor based on the size and
  offset to get a memory range. Registering a host notifier means
  mapping this memory range to the VM as the specified queue's notify
  MMIO region. Slave sends this request to tell QEMU to de-register
  the existing notifier if any and register the new notifier if the
  request is sent with a file descriptor.

  This request should be sent only when
  ``VHOST_USER_PROTOCOL_F_HOST_NOTIFIER`` protocol feature has been
  successfully negotiated.

``VHOST_USER_SLAVE_VRING_CALL``
  :id: 4
  :equivalent ioctl: N/A
  :slave payload: vring state description
  :master payload: N/A

  When the ``VHOST_USER_PROTOCOL_F_INBAND_NOTIFICATIONS`` protocol
  feature has been successfully negotiated, this message may be
  submitted by the slave to indicate that a buffer was used from
  the vring instead of signalling this using the vring's call file
  descriptor or having the master relying on polling.

  The state.num field is currently reserved and must be set to 0.

``VHOST_USER_SLAVE_VRING_ERR``
  :id: 5
  :equivalent ioctl: N/A
  :slave payload: vring state description
  :master payload: N/A

  When the ``VHOST_USER_PROTOCOL_F_INBAND_NOTIFICATIONS`` protocol
  feature has been successfully negotiated, this message may be
  submitted by the slave to indicate that an error occurred on the
  specific vring, instead of signalling the error file descriptor
  set by the master via ``VHOST_USER_SET_VRING_ERR``.

  The state.num field is currently reserved and must be set to 0.

.. _reply_ack:

VHOST_USER_PROTOCOL_F_REPLY_ACK
-------------------------------

The original vhost-user specification only demands replies for certain
commands. This differs from the vhost protocol implementation where
commands are sent over an ``ioctl()`` call and block until the client
has completed.

With this protocol extension negotiated, the sender (QEMU) can set the
``need_reply`` [Bit 3] flag to any command. This indicates that the
client MUST respond with a Payload ``VhostUserMsg`` indicating success
or failure. The payload should be set to zero on success or non-zero
on failure, unless the message already has an explicit reply body.

The response payload gives QEMU a deterministic indication of the result
of the command. Today, QEMU is expected to terminate the main vhost-user
loop upon receiving such errors. In future, qemu could be taught to be more
resilient for selective requests.

For the message types that already solicit a reply from the client,
the presence of ``VHOST_USER_PROTOCOL_F_REPLY_ACK`` or need_reply bit
being set brings no behavioural change. (See the Communication_
section for details.)

.. _backend_conventions:

Backend program conventions
===========================

vhost-user backends can provide various devices & services and may
need to be configured manually depending on the use case. However, it
is a good idea to follow the conventions listed here when
possible. Users, QEMU or libvirt, can then rely on some common
behaviour to avoid heterogenous configuration and management of the
backend programs and facilitate interoperability.

Each backend installed on a host system should come with at least one
JSON file that conforms to the vhost-user.json schema. Each file
informs the management applications about the backend type, and binary
location. In addition, it defines rules for management apps for
picking the highest priority backend when multiple match the search
criteria (see ``@VhostUserBackend`` documentation in the schema file).

If the backend is not capable of enabling a requested feature on the
host (such as 3D acceleration with virgl), or the initialization
failed, the backend should fail to start early and exit with a status
!= 0. It may also print a message to stderr for further details.

The backend program must not daemonize itself, but it may be
daemonized by the management layer. It may also have a restricted
access to the system.

File descriptors 0, 1 and 2 will exist, and have regular
stdin/stdout/stderr usage (they may have been redirected to /dev/null
by the management layer, or to a log handler).

The backend program must end (as quickly and cleanly as possible) when
the SIGTERM signal is received. Eventually, it may receive SIGKILL by
the management layer after a few seconds.

The following command line options have an expected behaviour. They
are mandatory, unless explicitly said differently:

--socket-path=PATH

  This option specify the location of the vhost-user Unix domain socket.
  It is incompatible with --fd.

--fd=FDNUM

  When this argument is given, the backend program is started with the
  vhost-user socket as file descriptor FDNUM. It is incompatible with
  --socket-path.

--print-capabilities

  Output to stdout the backend capabilities in JSON format, and then
  exit successfully. Other options and arguments should be ignored, and
  the backend program should not perform its normal function.  The
  capabilities can be reported dynamically depending on the host
  capabilities.

The JSON output is described in the ``vhost-user.json`` schema, by
```@VHostUserBackendCapabilities``.  Example:

.. code:: json

  {
    "type": "foo",
    "features": [
      "feature-a",
      "feature-b"
    ]
  }

vhost-user-input
----------------

Command line options:

--evdev-path=PATH

  Specify the linux input device.

  (optional)

--no-grab

  Do no request exclusive access to the input device.

  (optional)

vhost-user-gpu
--------------

Command line options:

--render-node=PATH

  Specify the GPU DRM render node.

  (optional)

--virgl

  Enable virgl rendering support.

  (optional)

vhost-user-blk
--------------

Command line options:

--blk-file=PATH

  Specify block device or file path.

  (optional)

--read-only

  Enable read-only.

  (optional)