vmtest

vmtest enables you to quickly and programmatically run tests inside a virtual machine.

This can be useful in the following, non-exhaustive, list of scenarios:

• You ship a virtual machine image and you want to programmatically test the image during development, both locally and in CI.
• You develop eBPF-powered applications and you want to run your application tests on a variety of kernels your application supports, both locally and in CI.
• You are a kernel developer and you want to quickly iterate on changes.

A key feature is that the root host userspace can-be/is transparently mapped into the guest VM. This makes dropping vmtest into existing CI workflows easy, as dependencies installed on the root host can also be effortlessly reused inside the guest VM.

Dependencies

The following are required dependencies, grouped by location:

Host machine:

• qemu
• qemu-guest-agent
• OVMF

Virtual machine image:

• qemu-guest-agent
• Kernel 9p filesystem support, either compiled in or as modules (see kernel dependencies)
  • Most (if not all) distros already ship support as modules or better

Kernel:

• CONFIG_VIRTIO=y
• CONFIG_VIRTIO_PCI=y
• CONFIG_VIRTIO_CONSOLE=y
• CONFIG_NET_9P=y
• CONFIG_NET_9P_VIRTIO=y
• CONFIG_9P_FS=y

Note the virtual machine image dependencies are only required if you're using the image target parameter. Likewise, the same applies for kernel dependencies.

Installation

Assuming you have a rust toolchain installed, simply run:

$ cargo install vmtest

Alternatively, vmtest publishes statically linked binaries in its release assets. Currently only x86-64-linux is published.

Usage

One-liner interface

The config file interface is more powerful and unlocks all vmtest features. However it can be a bit heavyweight if you're just trying to do something one-off. For such lighter-weight cases, vmtest has a one-liner interface.

For example, to run an arbitrary command in the guest VM with a different kernel:

$ vmtest -k ./bzImage-v6.2 "uname -r"
=> bzImage-v6.2
===> Booting
===> Setting up VM
===> Running command
6.2.0

To run an arbitrary command in a guest VM with a different kernel and rootfs:

$ vmtest -k ./bzImage-v6.2 -r ./rootfs "uname -r"
=> bzImage-v6.2
===> Booting
===> Setting up VM
===> Running command
6.2.0

To run an arbitrary command from a kernel from another architecture in a guest VM:

$ vmtest -k ./kernels/Image-arm64 -r ./rootfs/ubuntu-lunar-arm64 -a aarch64 "uname -r"
=> Image-arm64
===> Booting
===> Setting up VM
===> Running command
6.6.0-rc5-ga4a0c99f10ca-dirty

It is also possible to get an interactive shell prompt in the guest by using the command -:

vmtest -k ./bzImage-v6.2 "-"
...
...
root@(none):/#

See vmtest --help for all options and flags.

Config file interface

vmtest by default reads from vmtest.toml in the current working directory. vmtest.toml, in turn, describes which targets should be run.

For example, consider the following vmtest.toml:

[[target]]
name = "AWS kernel"
kernel = "./bzImage-5.15.0-1022-aws"
command = "uname -r | grep -e aws$"

[[target]]
name = "OCI image"
image = "./oci-stage-6/oci-stage-6-disk001.qcow2"
command = "ls -l /mnt/vmtest && cat /proc/thiswillfail"

[[target]]
name = "Foreign Architecture"
kernel = "./kernels/Image-arm64"
arch = "aarch64"
rootfs = "./rootfs/ubuntu-lunar-arm64"
command = "uname -m | grep aarch64"

In the above config, two see two defined targets: "AWS kernel" and "OCI image".

In plain english, the "AWS kernel" target tells vmtest to run command in a VM with the same userspace environment as the host, except with the specified kernel.

"OCI image", on the other hand, tells vmtest to run command inside the provided VM image. The image completely defines the environment command is run in with the exception of /mnt/vmtest. /mnt/vmtest (as we will see below) contains the full directory tree of the host machine rooted at the directory containing vmtest.toml. This directory tree is shared - not copied - with both readable and writable permissions.

Running vmtest with the above config yields the following results:

$ vmtest
=> AWS kernel
PASS
=> OCI image
===> Booting
===> Setting up VM
===> Running command
total 2057916
drwxr-xr-x 1 ubuntu ubuntu        200 Nov 14 20:41 avx-gateway-oci-stage-6
-rw-r--r-- 1 ubuntu ubuntu   11631520 Feb  1 00:33 bzImage-5.15.0-1022-aws
-rw-r--r-- 1 ubuntu ubuntu        359 Feb  4 01:41 vmtest.toml
cat: /proc/thiswillfail: No such file or directory
Command failed with exit code: 1
FAILED
=> Foreign Architecture
===> Booting
===> Setting up VM
===> Running command
aarch64

For full configuration documentation, see config.md.

Usage in Github CI

vmtest-action is a convenient wrapper around vmtest that is designed to run inside Github Actions. See vmtest-action documentation for more details.

Technical details

For general architecture notes, see architecture.md.

Acknowledgements

Many thanks to drgn's vmtest by Omar Sandoval and Andy Lutomirski's most excellent virtme for providing both ideas and technical exploration.