Donate to e Foundation | Murena handsets with /e/OS | Own a part of Murena! Learn more

Commit edc5f445 authored by Linus Torvalds's avatar Linus Torvalds
Browse files

Merge tag 'vfio-v4.10-rc1' of git://github.com/awilliam/linux-vfio 

Pull VFIO updates from Alex Williamson:

 - VFIO updates for v4.10 primarily include a new Mediated Device
   interface, which essentially allows software defined devices to be
   exposed to users through VFIO. The host vendor driver providing this
   virtual device polices, or mediates user access to the device.

   These devices often incorporate portions of real devices, for
   instance the primary initial users of this interface expose vGPUs
   which allow the user to map mediated devices, or mdevs, to a portion
   of a physical GPU. QEMU composes these mdevs into PCI representations
   using the existing VFIO user API. This enables both Intel KVM-GT
   support, which is also expected to arrive into Linux mainline during
   the v4.10 merge window, as well as NVIDIA vGPU, and also Channel I/O
   devices (aka CCW devices) for s390 virtualization support. (Kirti
   Wankhede, Neo Jia)

 - Drop unnecessary uses of pcibios_err_to_errno() (Cao Jin)

 - Fixes to VFIO capability chain handling (Eric Auger)

 - Error handling fixes for fallout from mdev (Christophe JAILLET)

 - Notifiers to expose struct kvm to mdev vendor drivers (Jike Song)

 - type1 IOMMU model search fixes (Kirti Wankhede, Neo Jia)

* tag 'vfio-v4.10-rc1' of git://github.com/awilliam/linux-vfio: (30 commits)
  vfio iommu type1: Fix size argument to vfio_find_dma() in pin_pages/unpin_pages
  vfio iommu type1: Fix size argument to vfio_find_dma() during DMA UNMAP.
  vfio iommu type1: WARN_ON if notifier block is not unregistered
  kvm: set/clear kvm to/from vfio_group when group add/delete
  vfio: support notifier chain in vfio_group
  vfio: vfio_register_notifier: classify iommu notifier
  vfio: Fix handling of error returned by 'vfio_group_get_from_dev()'
  vfio: fix vfio_info_cap_add/shift
  vfio/pci: Drop unnecessary pcibios_err_to_errno()
  MAINTAINERS: Add entry VFIO based Mediated device drivers
  docs: Sample driver to demonstrate how to use Mediated device framework.
  docs: Sysfs ABI for mediated device framework
  docs: Add Documentation for Mediated devices
  vfio: Define device_api strings
  vfio_platform: Updated to use vfio_set_irqs_validate_and_prepare()
  vfio_pci: Updated to use vfio_set_irqs_validate_and_prepare()
  vfio: Introduce vfio_set_irqs_validate_and_prepare()
  vfio_pci: Update vfio_pci to use vfio_info_add_capability()
  vfio: Introduce common function to add capabilities
  vfio iommu: Add blocking notifier to notify DMA_UNMAP
  ...
parents 22d8262c 2b8bb1d7

Documentation/ABI/testing/sysfs-bus-vfio-mdev

0 → 100644
+111 −0
Original line number Diff line number Diff line 
What:           /sys/.../<device>/mdev_supported_types/

Date:           October 2016

Contact:        Kirti Wankhede <kwankhede@nvidia.com>

Description:

                This directory contains list of directories of currently

		supported mediated device types and their details for

		<device>. Supported type attributes are defined by the

		vendor driver who registers with Mediated device framework.

		Each supported type is a directory whose name is created

		by adding the device driver string as a prefix to the

		string provided by the vendor driver.



What:           /sys/.../<device>/mdev_supported_types/<type-id>/

Date:           October 2016

Contact:        Kirti Wankhede <kwankhede@nvidia.com>

Description:

                This directory gives details of supported type, like name,

		description, available_instances, device_api etc.

		'device_api' and 'available_instances' are mandatory

		attributes to be provided by vendor driver. 'name',

		'description' and other vendor driver specific attributes

		are optional.



What:           /sys/.../mdev_supported_types/<type-id>/create

Date:           October 2016

Contact:        Kirti Wankhede <kwankhede@nvidia.com>

Description:

		Writing UUID to this file will create mediated device of

		type <type-id> for parent device <device>. This is a

		write-only file.

		For example:

		# echo "83b8f4f2-509f-382f-3c1e-e6bfe0fa1001" >	\

		       /sys/devices/foo/mdev_supported_types/foo-1/create



What:           /sys/.../mdev_supported_types/<type-id>/devices/

Date:           October 2016

Contact:        Kirti Wankhede <kwankhede@nvidia.com>

Description:

		This directory contains symbolic links pointing to mdev

		devices sysfs entries which are created of this <type-id>.



What:           /sys/.../mdev_supported_types/<type-id>/available_instances

Date:           October 2016

Contact:        Kirti Wankhede <kwankhede@nvidia.com>

Description:

		Reading this attribute will show the number of mediated

		devices of type <type-id> that can be created. This is a

		readonly file.

Users:

		Userspace applications interested in creating mediated

		device of that type. Userspace application should check

		the number of available instances could be created before

		creating mediated device of this type.



What:           /sys/.../mdev_supported_types/<type-id>/device_api

Date:           October 2016

Contact:        Kirti Wankhede <kwankhede@nvidia.com>

Description:

		Reading this attribute will show VFIO device API supported

		by this type. For example, "vfio-pci" for a PCI device,

		"vfio-platform" for platform device.



What:           /sys/.../mdev_supported_types/<type-id>/name

Date:           October 2016

Contact:        Kirti Wankhede <kwankhede@nvidia.com>

Description:

		Reading this attribute will show human readable name of the

		mediated device that will get created of type <type-id>.

		This is optional attribute. For example: "Grid M60-0Q"

Users:

		Userspace applications interested in knowing the name of

		a particular <type-id> that can help in understanding the

		type of mediated device.



What:           /sys/.../mdev_supported_types/<type-id>/description

Date:           October 2016

Contact:        Kirti Wankhede <kwankhede@nvidia.com>

Description:

		Reading this attribute will show description of the type of

		mediated device that will get created of type <type-id>.

		This is optional attribute. For example:

		"2 heads, 512M FB, 2560x1600 maximum resolution"

Users:

		Userspace applications interested in knowing the details of

		a particular <type-id> that can help in understanding the

		features provided by that type of mediated device.



What:           /sys/.../<device>/<UUID>/

Date:           October 2016

Contact:        Kirti Wankhede <kwankhede@nvidia.com>

Description:

		This directory represents device directory of mediated

		device. It contains all the attributes related to mediated

		device.



What:           /sys/.../<device>/<UUID>/mdev_type

Date:           October 2016

Contact:        Kirti Wankhede <kwankhede@nvidia.com>

Description:

		This is symbolic link pointing to supported type, <type-id>

		directory of which this mediated device is created.



What:           /sys/.../<device>/<UUID>/remove

Date:           October 2016

Contact:        Kirti Wankhede <kwankhede@nvidia.com>

Description:

		Writing '1' to this file destroys the mediated device. The

		vendor driver can fail the remove() callback if that device

		is active and the vendor driver doesn't support hot unplug.

		Example:

		# echo 1 > /sys/bus/mdev/devices/<UUID>/remove

Documentation/vfio-mediated-device.txt

0 → 100644
+398 −0
Original line number Diff line number Diff line 
/*

 * VFIO Mediated devices

 *

 * Copyright (c) 2016, NVIDIA CORPORATION. All rights reserved.

 *     Author: Neo Jia <cjia@nvidia.com>

 *             Kirti Wankhede <kwankhede@nvidia.com>

 *

 * This program is free software; you can redistribute it and/or modify

 * it under the terms of the GNU General Public License version 2 as

 * published by the Free Software Foundation.

 */



Virtual Function I/O (VFIO) Mediated devices[1]

===============================================



The number of use cases for virtualizing DMA devices that do not have built-in

SR_IOV capability is increasing. Previously, to virtualize such devices,

developers had to create their own management interfaces and APIs, and then

integrate them with user space software. To simplify integration with user space

software, we have identified common requirements and a unified management

interface for such devices.



The VFIO driver framework provides unified APIs for direct device access. It is

an IOMMU/device-agnostic framework for exposing direct device access to user

space in a secure, IOMMU-protected environment. This framework is used for

multiple devices, such as GPUs, network adapters, and compute accelerators. With

direct device access, virtual machines or user space applications have direct

access to the physical device. This framework is reused for mediated devices.



The mediated core driver provides a common interface for mediated device

management that can be used by drivers of different devices. This module

provides a generic interface to perform these operations:



* Create and destroy a mediated device

* Add a mediated device to and remove it from a mediated bus driver

* Add a mediated device to and remove it from an IOMMU group



The mediated core driver also provides an interface to register a bus driver.

For example, the mediated VFIO mdev driver is designed for mediated devices and

supports VFIO APIs. The mediated bus driver adds a mediated device to and

removes it from a VFIO group.



The following high-level block diagram shows the main components and interfaces

in the VFIO mediated driver framework. The diagram shows NVIDIA, Intel, and IBM

devices as examples, as these devices are the first devices to use this module.



     +---------------+

     |               |

     | +-----------+ |  mdev_register_driver() +--------------+

     | |           | +<------------------------+              |

     | |  mdev     | |                         |              |

     | |  bus      | +------------------------>+ vfio_mdev.ko |<-> VFIO user

     | |  driver   | |     probe()/remove()    |              |    APIs

     | |           | |                         +--------------+

     | +-----------+ |

     |               |

     |  MDEV CORE    |

     |   MODULE      |

     |   mdev.ko     |

     | +-----------+ |  mdev_register_device() +--------------+

     | |           | +<------------------------+              |

     | |           | |                         |  nvidia.ko   |<-> physical

     | |           | +------------------------>+              |    device

     | |           | |        callbacks        +--------------+

     | | Physical  | |

     | |  device   | |  mdev_register_device() +--------------+

     | | interface | |<------------------------+              |

     | |           | |                         |  i915.ko     |<-> physical

     | |           | +------------------------>+              |    device

     | |           | |        callbacks        +--------------+

     | |           | |

     | |           | |  mdev_register_device() +--------------+

     | |           | +<------------------------+              |

     | |           | |                         | ccw_device.ko|<-> physical

     | |           | +------------------------>+              |    device

     | |           | |        callbacks        +--------------+

     | +-----------+ |

     +---------------+





Registration Interfaces

=======================



The mediated core driver provides the following types of registration

interfaces:



* Registration interface for a mediated bus driver

* Physical device driver interface



Registration Interface for a Mediated Bus Driver

------------------------------------------------



The registration interface for a mediated bus driver provides the following

structure to represent a mediated device's driver:



     /*

      * struct mdev_driver [2] - Mediated device's driver

      * @name: driver name

      * @probe: called when new device created

      * @remove: called when device removed

      * @driver: device driver structure

      */

     struct mdev_driver {

	     const char *name;

	     int  (*probe)  (struct device *dev);

	     void (*remove) (struct device *dev);

	     struct device_driver    driver;

     };



A mediated bus driver for mdev should use this structure in the function calls

to register and unregister itself with the core driver:



* Register:



  extern int  mdev_register_driver(struct mdev_driver *drv,

				   struct module *owner);



* Unregister:



  extern void mdev_unregister_driver(struct mdev_driver *drv);



The mediated bus driver is responsible for adding mediated devices to the VFIO

group when devices are bound to the driver and removing mediated devices from

the VFIO when devices are unbound from the driver.





Physical Device Driver Interface

--------------------------------



The physical device driver interface provides the parent_ops[3] structure to

define the APIs to manage work in the mediated core driver that is related to

the physical device.



The structures in the parent_ops structure are as follows:



* dev_attr_groups: attributes of the parent device

* mdev_attr_groups: attributes of the mediated device

* supported_config: attributes to define supported configurations



The functions in the parent_ops structure are as follows:



* create: allocate basic resources in a driver for a mediated device

* remove: free resources in a driver when a mediated device is destroyed



The callbacks in the parent_ops structure are as follows:



* open: open callback of mediated device

* close: close callback of mediated device

* ioctl: ioctl callback of mediated device

* read : read emulation callback

* write: write emulation callback

* mmap: mmap emulation callback



A driver should use the parent_ops structure in the function call to register

itself with the mdev core driver:



extern int  mdev_register_device(struct device *dev,

                                 const struct parent_ops *ops);



However, the parent_ops structure is not required in the function call that a

driver should use to unregister itself with the mdev core driver:



extern void mdev_unregister_device(struct device *dev);





Mediated Device Management Interface Through sysfs

==================================================



The management interface through sysfs enables user space software, such as

libvirt, to query and configure mediated devices in a hardware-agnostic fashion.

This management interface provides flexibility to the underlying physical

device's driver to support features such as:



* Mediated device hot plug

* Multiple mediated devices in a single virtual machine

* Multiple mediated devices from different physical devices



Links in the mdev_bus Class Directory

-------------------------------------

The /sys/class/mdev_bus/ directory contains links to devices that are registered

with the mdev core driver.



Directories and files under the sysfs for Each Physical Device

--------------------------------------------------------------



|- [parent physical device]

|--- Vendor-specific-attributes [optional]

|--- [mdev_supported_types]

|     |--- [<type-id>]

|     |   |--- create

|     |   |--- name

|     |   |--- available_instances

|     |   |--- device_api

|     |   |--- description

|     |   |--- [devices]

|     |--- [<type-id>]

|     |   |--- create

|     |   |--- name

|     |   |--- available_instances

|     |   |--- device_api

|     |   |--- description

|     |   |--- [devices]

|     |--- [<type-id>]

|          |--- create

|          |--- name

|          |--- available_instances

|          |--- device_api

|          |--- description

|          |--- [devices]



* [mdev_supported_types]



  The list of currently supported mediated device types and their details.



  [<type-id>], device_api, and available_instances are mandatory attributes

  that should be provided by vendor driver.



* [<type-id>]



  The [<type-id>] name is created by adding the the device driver string as a

  prefix to the string provided by the vendor driver. This format of this name

  is as follows:



	sprintf(buf, "%s-%s", dev_driver_string(parent->dev), group->name);



* device_api



  This attribute should show which device API is being created, for example,

  "vfio-pci" for a PCI device.



* available_instances



  This attribute should show the number of devices of type <type-id> that can be

  created.



* [device]



  This directory contains links to the devices of type <type-id> that have been

created.



* name



  This attribute should show human readable name. This is optional attribute.



* description



  This attribute should show brief features/description of the type. This is

  optional attribute.



Directories and Files Under the sysfs for Each mdev Device

----------------------------------------------------------



|- [parent phy device]

|--- [$MDEV_UUID]

         |--- remove

         |--- mdev_type {link to its type}

         |--- vendor-specific-attributes [optional]



* remove (write only)

Writing '1' to the 'remove' file destroys the mdev device. The vendor driver can

fail the remove() callback if that device is active and the vendor driver

doesn't support hot unplug.



Example:

	# echo 1 > /sys/bus/mdev/devices/$mdev_UUID/remove



Mediated device Hot plug:

------------------------



Mediated devices can be created and assigned at runtime. The procedure to hot

plug a mediated device is the same as the procedure to hot plug a PCI device.



Translation APIs for Mediated Devices

=====================================



The following APIs are provided for translating user pfn to host pfn in a VFIO

driver:



extern int vfio_pin_pages(struct device *dev, unsigned long *user_pfn,

                          int npage, int prot, unsigned long *phys_pfn);



extern int vfio_unpin_pages(struct device *dev, unsigned long *user_pfn,

                            int npage);



These functions call back into the back-end IOMMU module by using the pin_pages

and unpin_pages callbacks of the struct vfio_iommu_driver_ops[4]. Currently

these callbacks are supported in the TYPE1 IOMMU module. To enable them for

other IOMMU backend modules, such as PPC64 sPAPR module, they need to provide

these two callback functions.



Using the Sample Code

=====================



mtty.c in samples/vfio-mdev/ directory is a sample driver program to

demonstrate how to use the mediated device framework.



The sample driver creates an mdev device that simulates a serial port over a PCI

card.



1. Build and load the mtty.ko module.



   This step creates a dummy device, /sys/devices/virtual/mtty/mtty/



   Files in this device directory in sysfs are similar to the following:



   # tree /sys/devices/virtual/mtty/mtty/

      /sys/devices/virtual/mtty/mtty/

      |-- mdev_supported_types

      |   |-- mtty-1

      |   |   |-- available_instances

      |   |   |-- create

      |   |   |-- device_api

      |   |   |-- devices

      |   |   `-- name

      |   `-- mtty-2

      |       |-- available_instances

      |       |-- create

      |       |-- device_api

      |       |-- devices

      |       `-- name

      |-- mtty_dev

      |   `-- sample_mtty_dev

      |-- power

      |   |-- autosuspend_delay_ms

      |   |-- control

      |   |-- runtime_active_time

      |   |-- runtime_status

      |   `-- runtime_suspended_time

      |-- subsystem -> ../../../../class/mtty

      `-- uevent



2. Create a mediated device by using the dummy device that you created in the

   previous step.



   # echo "83b8f4f2-509f-382f-3c1e-e6bfe0fa1001" >	\

              /sys/devices/virtual/mtty/mtty/mdev_supported_types/mtty-2/create



3. Add parameters to qemu-kvm.



   -device vfio-pci,\

    sysfsdev=/sys/bus/mdev/devices/83b8f4f2-509f-382f-3c1e-e6bfe0fa1001



4. Boot the VM.



   In the Linux guest VM, with no hardware on the host, the device appears

   as  follows:



   # lspci -s 00:05.0 -xxvv

   00:05.0 Serial controller: Device 4348:3253 (rev 10) (prog-if 02 [16550])

           Subsystem: Device 4348:3253

           Physical Slot: 5

           Control: I/O+ Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr-

   Stepping- SERR- FastB2B- DisINTx-

           Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort-

   <TAbort- <MAbort- >SERR- <PERR- INTx-

           Interrupt: pin A routed to IRQ 10

           Region 0: I/O ports at c150 [size=8]

           Region 1: I/O ports at c158 [size=8]

           Kernel driver in use: serial

   00: 48 43 53 32 01 00 00 02 10 02 00 07 00 00 00 00

   10: 51 c1 00 00 59 c1 00 00 00 00 00 00 00 00 00 00

   20: 00 00 00 00 00 00 00 00 00 00 00 00 48 43 53 32

   30: 00 00 00 00 00 00 00 00 00 00 00 00 0a 01 00 00



   In the Linux guest VM, dmesg output for the device is as follows:



   serial 0000:00:05.0: PCI INT A -> Link[LNKA] -> GSI 10 (level, high) -> IRQ

10

   0000:00:05.0: ttyS1 at I/O 0xc150 (irq = 10) is a 16550A

   0000:00:05.0: ttyS2 at I/O 0xc158 (irq = 10) is a 16550A





5. In the Linux guest VM, check the serial ports.



   # setserial -g /dev/ttyS*

   /dev/ttyS0, UART: 16550A, Port: 0x03f8, IRQ: 4

   /dev/ttyS1, UART: 16550A, Port: 0xc150, IRQ: 10

   /dev/ttyS2, UART: 16550A, Port: 0xc158, IRQ: 10



6. Using a minicom or any terminal enulation program, open port /dev/ttyS1 or

   /dev/ttyS2 with hardware flow control disabled.



7. Type data on the minicom terminal or send data to the terminal emulation

   program and read the data.



   Data is loop backed from hosts mtty driver.



8. Destroy the mediated device that you created.



   # echo 1 > /sys/bus/mdev/devices/83b8f4f2-509f-382f-3c1e-e6bfe0fa1001/remove



References

==========



[1] See Documentation/vfio.txt for more information on VFIO.

[2] struct mdev_driver in include/linux/mdev.h

[3] struct parent_ops in include/linux/mdev.h

[4] struct vfio_iommu_driver_ops in include/linux/vfio.h

MAINTAINERS

+9 −0
Original line number Diff line number Diff line
@@ -12791,6 +12791,15 @@ F: drivers/vfio/ 
F:	include/linux/vfio.h

F:	include/uapi/linux/vfio.h



VFIO MEDIATED DEVICE DRIVERS

M:	Kirti Wankhede <kwankhede@nvidia.com>

L:	kvm@vger.kernel.org

S:	Maintained

F:	Documentation/vfio-mediated-device.txt

F:	drivers/vfio/mdev/

F:	include/linux/mdev.h

F:	samples/vfio-mdev/



VFIO PLATFORM DRIVER

M:	Baptiste Reynal <b.reynal@virtualopensystems.com>

L:	kvm@vger.kernel.org

drivers/vfio/Kconfig

+1 −0
Original line number Diff line number Diff line
@@ -48,4 +48,5 @@ menuconfig VFIO_NOIOMMU 


source "drivers/vfio/pci/Kconfig"

source "drivers/vfio/platform/Kconfig"

source "drivers/vfio/mdev/Kconfig"

source "virt/lib/Kconfig"

drivers/vfio/Makefile

+1 −0
Original line number Diff line number Diff line
@@ -7,3 +7,4 @@ obj-$(CONFIG_VFIO_IOMMU_SPAPR_TCE) += vfio_iommu_spapr_tce.o 
obj-$(CONFIG_VFIO_SPAPR_EEH) += vfio_spapr_eeh.o

obj-$(CONFIG_VFIO_PCI) += pci/

obj-$(CONFIG_VFIO_PLATFORM) += platform/

obj-$(CONFIG_VFIO_MDEV) += mdev/