GPU passthrough via VFIO

A retrospective note of a multi-weekend project from October 2021 - Feb 2022.

Background

I always needed both Windows and Linux operating systems for various reasons since the beginning of University.

GNU/Linux offers tiling window managers, theming, native Emacs support, and a wonderful ecosystem of terminals, providing comfortable developement environments. It's not the most polished place when you first install it, but you can pretty much customize it to look like anything you want. The only downside is gaming (as of 2021, although Proton is changing the field rapidly) and it's limited integration with popular commercial and academic software.

Windows, on the other hand, is more polished on the surface but just never felt like home. It has intrusive promotions/ads and bundled software/games, aggressive telemetry, and forced updates. However, it works well in organisations such as hospitals and universities with its software ecosystem such as Office, Endnote and other proprietary software. And did I mention that gaming just works?

I tried dual booting on my laptop for a while, but it's never been a satisfying experience. For example, switching between the 2 systems requires a full reboot. Not only it takes time to shut down (and sometimes install updates) but also requires me to close all the apps and files I have opened. This means I have to think twice before switching over each time.

After I started working, I quickly assembled two desktops towers, one runs Windows and the other Linux. This has been a big step up from dual boot, because I can now switch between the two systems easily. The only catch is switching between keyboard and mous. But that was relieved by the built-in KVM in my Dell monitor.

There are still minor issues, though. For example, there's a bug with the windows box, that the KVM constantly disconnects and reconnects my USB hub after each switching from Linux. Also, having two boxes under the desk can be quite noisy, espcially when the Windows box was a few years older (couldn't afford good fans initially).

Then I tried virtual machines with solutions such as Virtualbox and VMware. They are really good CPU-wise, except for the heavy graphics performance penalty with emulated GPU.

There is one last solution I have yet to try, that is called "VFIO GPU passthrough". It is where the guest operating system runs in a virtual machine, but with the graphics performance of a native computer. I actually first attempted this with a single GPU (a second hand Radeon HD 7950 with a broken fan) in 2016 on Arch Linux, but failed despite multiple weekends of tinkering.

I reckon it's worth giving it another try 5 years later, when the documentation and technology support is far more mature.

My goals are:

  • Seamless switching between Linux and Windows
  • High graphics performance on Windows, enough for gaming
  • Single computer with minimal interruption to switch

Requirements

Hardware

CPU Ryzen 5950
Motherboard Asus Dark Hero X570
Ram 64 GB
Host storage Samsung 980Pro 1T
Guest storage Samsung 960Pro 1T
GPU (host) RX 5700XT
GPU (guest) GTX 1070ti
Display 1 Dell 4k
Display 2 Dell 2k

Software

  • Host: KDE Neon (Ubuntu 20.04 LTS)
  • Guest: Windows 10
  • Enable UEFI options for Amd-V and IOMMU
  • Add various kernel parameters on Grub2
  • Install qemu, kvm, libvirt, vir-manager
  • Install guest additions

    • VirtIO guest tools for better performing drivers
    • Spice guest tools for mouse, keyboard, clipboard and sound

Configuration

Network

  • I just passed through one of my two ethernet ports on the motherboard
  • Added a built-in bridge just-in-case

Input devices (uvdev)

Add uvdev rules

  • How to add uvdev rules
  • Permission issues to /dev/input/by-id/ (very common problem)

    • Add user to group: input
    • Add ACL rules and change user/group to root in qemu.conf
    • Modify AppArmour rules

Passthrough USB controllers

  • The USB port and id map in the post was very helpful
  • Passed 2 usb controllers for windows including the USB hub
  • Now the Logitech Brio and Steam Controller is working

Persistent evdev

  • Because I lose mouse and keyboard after switching between USB-C Macbook to Windows/Linux
  • Ubuntu LTS is v6, so can only use qemu parameters

Update 2023

  • Purchased a USB KVM for hardware switch
  • Built-in SPICE controller has minimal delay for casual use

Audio

Arch wiki: Slowed down audio pumped through HDMI on the video card

  • Fixed by installing MSI Utility and configure the GPU as MSI
  • Alternatively use scream audio

    • Requires vfio brige setup
    • Not as fast as Pipewire

Passthrough motherboard audio port

  • Saves my DAC for linux (easier to manage with my side monitor)
  • Easy to switch to windows guest if I needed by plugging the DAC USB to the passed through port
  • I mostly use windows for work, meetings and gaming anyways
  • Gaming actually benefits from the bass with my Logitech setup, and I use my monitor speakers with Linux for music only

Update 2023

  • Looking glass B6 supports pipewire passthrough
  • I also have a USB KVM to change over my Scarlett 2i2 over as well, useful when needing to use the virtual amplifiers

TPM2 for Windows 11

  • Turned on fTPM module on motherboard
  • Add permission to apparmour.d
  • changed setting to tlt

Looking glass

High performance guest display as a host window with minimal overhead

First attempt on 01/05/2022

  • Configuration was simple, just followed the website
  • Nixpkgs looking-glass-client unable to find EGL display
  • Client unable to be built on neither Ubuntu (cannot find library) or Nix toolchain (target spec issue)

Second attempt on 02/05/2022

  • Turns out my suspicion was correct, nix overriden the ubuntu path.
  • Compliation succeeded with PATH=/bin:/usr/bin:$PATH make
  • I've now disabled evdev by disabling persistent-evdev.

Update on 17/05/2022

  • Turns out the reason nix packages couldn't find EGL display was to do with nix and the graphics driver
  • Fixed by installing nixGL
  • Also learned how to override pkgs with Nix

Update 04/2023

  • After playing around with disk emulation, iothreads and ROM BAR, unfortunately Windows boot drive corrupted and I had to reset.
  • I retained my file, but had to reinstall everything
  • For whatever reason, no more IO problems…
  • I've tried to unset ROM BAR and iothreads but nothing changed
  • Ongoing mystery

Optimisations

CPU optimisation

For Ryzen 5950 this is the optimal setup according to multiple sources such as this reddit post.

<cputune>
  <vcpupin vcpu="0" cpuset="8"/>
  <vcpupin vcpu="1" cpuset="24"/>
  <vcpupin vcpu="2" cpuset="9"/>
  <vcpupin vcpu="3" cpuset="25"/>
  <vcpupin vcpu="4" cpuset="10"/>
  <vcpupin vcpu="5" cpuset="26"/>
  <vcpupin vcpu="6" cpuset="11"/>
  <vcpupin vcpu="7" cpuset="27"/>
  <vcpupin vcpu="8" cpuset="12"/>
  <vcpupin vcpu="9" cpuset="28"/>
  <vcpupin vcpu="10" cpuset="13"/>
  <vcpupin vcpu="11" cpuset="29"/>
  <vcpupin vcpu="12" cpuset="14"/>
  <vcpupin vcpu="13" cpuset="30"/>
  <vcpupin vcpu="14" cpuset="15"/>
  <vcpupin vcpu="15" cpuset="31"/>
  <emulatorpin cpuset="7,23"/>
</cputune>

Huge page memory optimisation

Easily done following online tutorials.

Result

Seamless switching of the input devices

With looking glass, windows is like a normal window. I can use the USB KVM to switch keyboard and mouse. Game controller is connected via USB passthrough.

Great performance (with 4090 in 2023)

  • Trialed Wasteland 2, Spin Rhythm, Halo, Cyberpunk 2077
  • All works wonderfully

Userbench was amazing

Near 100% native machine performance with userbench

Windows 11 compatible

Passed hardware requirements for Windows 11 assessed

Removed the old windows box

Reduced noise!

References

Bryan Steiner tutorial

  • Initial attempt
  • Failed to work for me, freezes host due to unable to dynamically bind and unbind guest GPU
  • Showed a nice shortcut for mounting NVME SSDs via PCI

Arch Wiki

  • Works well enough
  • Not specific for Ubuntu

Mathais Shueber tutorial

  • Tailored for Ubuntu 20.04, works well in combination with Arch Wiki