mdevctl - a mediated device management utility for Linux

Description

mdevctl is a utility for managing and persisting devices in the mediated device device framework of the Linux kernel. Mediated devices are sub-devices of a parent device (ex. a vGPU) which can be dynamically created and potentially used by drivers like vfio-mdev for assignment to virtual machines.

License

Licensed under the GNU Lesser General Public License aka LGPL v2.1. See COPYING for details.

Source repository

https://github.com/mdevctl/mdevctl

Installation

mdevctl is built with rust's cargo tool. To build the executable, run cargo build. This will compile the code and also generate a Makefile that can be used for installing the executable and all support files into your system. On RPM based systems, you can run make rpm then install the resulting package. Otherwise, run make install.

Architecture

mdevctl stores defined mediated devices in /etc/mdevctl.d/ with directories matching the parent device name and config files named by the UUID of the mdev device itself. The format used is JSON; a configuration file for an mdev device looks like follows:

  {
   "mdev_type": "$VENDOR_TYPE",
   "start": "auto|manual",
   "attrs": [
    ...optional list of device-specific attributes...
   ]
  }

When a known parent device add udev event occurs (or, for more recent kernels, change events with MDEV_STATE values), mdevctl is called by a udev rule to create defined devices with "start": "auto" configured.

mdevctl defines three classes of commands, those that manage device config files, those that manage the device itself, and listing commands for showing defined, active, or potential mdev devices.

Starting with the latter, mdevctl is able to manage mdev devices created either with mdevctl or externally, such as through direct sysfs interactions. Likewise, when generating a list of currently active mdev devices via the list command, all mdevs are included. When provided with the --defined option, the list command will show mdev device configs defined on the system, regardless of whether they are currently active. The types command provides details of the mdev types supported on the system, including the number of instances of each that may be created, the API exposed for each, as well as vendor provided name and description as available.

Mediated device definitions can be created with the define command, which not only accepts a fully specified configuration via options, but can also create a config for a currently running mdev. Thus a transient device created either through mdevctl or sysfs can be promoted to a defined device. The undefine command simply removes the config definition without modifying the running device, while the modify command allows device config to be modified. Config modifications to a running device to not take effect until the device is stopped and restarted.

This leads to the final class of commands, which provides the start and stop functionality. The start command can operate either on a previously defined mdev or the mdev can be fully specified via options to create a transient device, ie. a running device with no persistence.

Usage

List running mdev devices:

# mdevctl list
85006552-1b4b-45ef-ad62-de05be9171df 0000:00:02.0 i915-GVTg_V4_4
83c32df7-d52e-4ec1-9668-1f3c7e4df107 0000:00:02.0 i915-GVTg_V4_8 (defined)

List defined mdev devices:

# mdevctl list -d
83c32df7-d52e-4ec1-9668-1f3c7e4df107 0000:00:02.0 i915-GVTg_V4_8 auto
b0a3989f-8138-4d49-b63a-59db28ec8b48 0000:00:02.0 i915-GVTg_V4_8 auto
5cf14a12-a437-4c82-a13f-70e945782d7b 0000:00:02.0 i915-GVTg_V4_4 manual

List mdev types supported on the host system:

# mdevctl types
0000:00:02.0
  i915-GVTg_V4_2
    Available instances: 1
    Device API: vfio-pci
    Description: low_gm_size: 256MB high_gm_size: 1024MB fence: 4 resolution: 1920x1200 weight: 8 
  i915-GVTg_V4_1
    Available instances: 0
    Device API: vfio-pci
    Description: low_gm_size: 512MB high_gm_size: 2048MB fence: 4 resolution: 1920x1200 weight: 16 
  i915-GVTg_V4_8
    Available instances: 4
    Device API: vfio-pci
    Description: low_gm_size: 64MB high_gm_size: 384MB fence: 4 resolution: 1024x768 weight: 2 
  i915-GVTg_V4_4
    Available instances: 3
    Device API: vfio-pci
    Description: low_gm_size: 128MB high_gm_size: 512MB fence: 4 resolution: 1920x1200 weight: 4 

Modify a defined device from automatic start to manual:

# mdevctl modify --uuid 83c32df7-d52e-4ec1-9668-1f3c7e4df107 --manual
# mdevctl list -d
83c32df7-d52e-4ec1-9668-1f3c7e4df107 0000:00:02.0 i915-GVTg_V4_8 manual
b0a3989f-8138-4d49-b63a-59db28ec8b48 0000:00:02.0 i915-GVTg_V4_8 auto
5cf14a12-a437-4c82-a13f-70e945782d7b 0000:00:02.0 i915-GVTg_V4_4 manual

Stop a running mdev device:

# mdevctl stop -u 83c32df7-d52e-4ec1-9668-1f3c7e4df107

Start an mdev device that is not defined

# uuidgen
6eba5b41-176e-40db-b93e-7f18e04e0b93
# mdevctl start -u 6eba5b41-176e-40db-b93e-7f18e04e0b93 -p 0000:00:02.0 --type i915-GVTg_V4_1
# mdevctl list
85006552-1b4b-45ef-ad62-de05be9171df 0000:00:02.0 i915-GVTg_V4_4
6eba5b41-176e-40db-b93e-7f18e04e0b93 0000:00:02.0 i915-GVTg_V4_1

Promote the new created mdev to a defined device:

# mdevctl define --uuid 6eba5b41-176e-40db-b93e-7f18e04e0b93
# mdevctl list -d
83c32df7-d52e-4ec1-9668-1f3c7e4df107 0000:00:02.0 i915-GVTg_V4_8 manual
6eba5b41-176e-40db-b93e-7f18e04e0b93 0000:00:02.0 i915-GVTg_V4_1 manual
b0a3989f-8138-4d49-b63a-59db28ec8b48 0000:00:02.0 i915-GVTg_V4_8 auto
5cf14a12-a437-4c82-a13f-70e945782d7b 0000:00:02.0 i915-GVTg_V4_4 manual

Advanced usage (attributes and JSON)

mdevctl provides support for specifying additional configuration via device-specific attributes. It also provides support for inspecting and modifying its internal JSON representation of the configuration directly.

Example:

# mdevctl list -d
783e6dbb-ea0e-411f-94e2-717eaad438bf matrix vfio_ap-passthrough manual

Add some attributes:

# mdevctl modify -u 783e6dbb-ea0e-411f-94e2-717eaad438bf --addattr=assign_adapter --value=5
# mdevctl modify -u 783e6dbb-ea0e-411f-94e2-717eaad438bf --addattr=assign_adapter --value=6
# mdevctl modify -u 783e6dbb-ea0e-411f-94e2-717eaad438bf --addattr=assign_domain --value=0xab
# mdevctl modify -u 783e6dbb-ea0e-411f-94e2-717eaad438bf --addattr=assign_control_domain --value=0xab
# mdevctl modify -u 783e6dbb-ea0e-411f-94e2-717eaad438bf --addattr=assign_domain --value=4
# mdevctl modify -u 783e6dbb-ea0e-411f-94e2-717eaad438bf --addattr=assign_control_domain --value=4
# mdevctl list -dv
783e6dbb-ea0e-411f-94e2-717eaad438bf matrix vfio_ap-passthrough manual
  Attrs:
    @{0}: {"assign_adapter":"5"}
    @{1}: {"assign_adapter":"6"}
    @{2}: {"assign_domain":"0xab"}
    @{3}: {"assign_control_domain":"0xab"}
    @{4}: {"assign_domain":"4"}
    @{5}: {"assign_control_domain":"4"}

Dump the JSON configuration:

# mdevctl list -d -u 783e6dbb-ea0e-411f-94e2-717eaad438bf --dumpjson
{
  "mdev_type": "vfio_ap-passthrough",
  "start": "manual",
  "attrs": [
    {
      "assign_adapter": "5"
    },
    {
      "assign_adapter": "6"
    },
    {
      "assign_domain": "0xab"
    },
    {
      "assign_control_domain": "0xab"
    },
    {
      "assign_domain": "4"
    },
    {
      "assign_control_domain": "4"
    }
  ]
}

Remove some attributes:

# mdevctl modify -u 783e6dbb-ea0e-411f-94e2-717eaad438bf --delattr --index=5
# mdevctl modify -u 783e6dbb-ea0e-411f-94e2-717eaad438bf --delattr --index=4
# mdevctl list -dv
783e6dbb-ea0e-411f-94e2-717eaad438bf matrix vfio_ap-passthrough manual
  Attrs:
    @{0}: {"assign_adapter":"5"}
    @{1}: {"assign_adapter":"6"}
    @{2}: {"assign_domain":"0xab"}
    @{3}: {"assign_control_domain":"0xab"}

Define an mdev device from a file:

# cat vfio_ap_device.json
{
  "mdev_type": "vfio_ap-passthrough",
  "start": "manual",
  "attrs": [
    {
      "assign_adapter": "5"
    },
    {
      "assign_domain": "0x47"
    },
    {
      "assign_domain": "0xff"
    }
  ]
}
# mdevctl define -p matrix --jsonfile vfio_ap_device.json
e2e73122-cc39-40ee-89eb-b0a47d334cae
# mdevctl list -dv
783e6dbb-ea0e-411f-94e2-717eaad438bf matrix vfio_ap-passthrough manual
  Attrs:
    @{0}: {"assign_adapter":"5"}
    @{1}: {"assign_adapter":"6"}
    @{2}: {"assign_domain":"0xab"}
    @{3}: {"assign_control_domain":"0xab"}
e2e73122-cc39-40ee-89eb-b0a47d334cae matrix vfio_ap-passthrough manual
  Attrs:
    @{0}: {"assign_adapter":"5"}
    @{1}: {"assign_domain":"0x47"}
    @{2}: {"assign_domain":"0xff"}

See mdevctl --help or the manpage for more information.