GPU passthrough via VFIO
A retrospective note of a multi-weekend project from October 2021 - Feb 2022.
Background
I always needed both Windows and Linux operating systems for various reasons since the beginning of University.
GNU/Linux offers tiling window managers, theming, native Emacs support, and a wonderful ecosystem of terminals, providing comfortable developement environments. It's not the most polished place when you first install it, but you can pretty much customize it to look like anything you want. The only downside is gaming (as of 2021, although Proton is changing the field rapidly) and it's limited integration with popular commercial and academic software.
Windows, on the other hand, is more polished on the surface but just never felt like home. It has intrusive promotions/ads and bundled software/games, aggressive telemetry, and forced updates. However, it works well in organisations such as hospitals and universities with its software ecosystem such as Office, Endnote and other proprietary software. And did I mention that gaming just works?
I tried dual booting on my laptop for a while, but it's never been a satisfying experience. For example, switching between the 2 systems requires a full reboot. Not only it takes time to shut down (and sometimes install updates) but also requires me to close all the apps and files I have opened. This means I have to think twice before switching over each time.
After I started working, I quickly assembled two desktops towers, one runs Windows and the other Linux. This has been a big step up from dual boot, because I can now switch between the two systems easily. The only catch is switching between keyboard and mous. But that was relieved by the built-in KVM in my Dell monitor.
There are still minor issues, though. For example, there's a bug with the windows box, that the KVM constantly disconnects and reconnects my USB hub after each switching from Linux. Also, having two boxes under the desk can be quite noisy, espcially when the Windows box was a few years older (couldn't afford good fans initially).
Then I tried virtual machines with solutions such as Virtualbox and VMware. They are really good CPU-wise, except for the heavy graphics performance penalty with emulated GPU.
There is one last solution I have yet to try, that is called "VFIO GPU passthrough". It is where the guest operating system runs in a virtual machine, but with the graphics performance of a native computer. I actually first attempted this with a single GPU (a second hand Radeon HD 7950 with a broken fan) in 2016 on Arch Linux, but failed despite multiple weekends of tinkering.
I reckon it's worth giving it another try 5 years later, when the documentation and technology support is far more mature.
My goals are:
- Seamless switching between Linux and Windows
- High graphics performance on Windows, enough for gaming
- Single computer with minimal interruption to switch
Requirements
Hardware
CPU | Ryzen 5950 |
Motherboard | Asus Dark Hero X570 |
Ram | 64 GB |
Host storage | Samsung 980Pro 1T |
Guest storage | Samsung 960Pro 1T |
GPU (host) | RX 5700XT |
GPU (guest) | GTX 1070ti |
Display 1 | Dell 4k |
Display 2 | Dell 2k |
Software
- Host: KDE Neon (Ubuntu 20.04 LTS)
- Guest: Windows 10
- Enable UEFI options for Amd-V and IOMMU
- Add various kernel parameters on Grub2
- Install qemu, kvm, libvirt, vir-manager
-
Install guest additions
- VirtIO guest tools for better performing drivers
- Spice guest tools for mouse, keyboard, clipboard and sound
Configuration
Network
- I just passed through one of my two ethernet ports on the motherboard
- Added a built-in bridge just-in-case
Input devices (uvdev)
Add uvdev rules
- How to add uvdev rules
-
Permission issues to
/dev/input/by-id/
(very common problem)- Add user to group:
input
- Add ACL rules and change user/group to
root
in qemu.conf - Modify AppArmour rules
- Add user to group:
Passthrough USB controllers
- The USB port and id map in the post was very helpful
- Passed 2 usb controllers for windows including the USB hub
- Now the Logitech Brio and Steam Controller is working
Persistent evdev
- Because I lose mouse and keyboard after switching between USB-C Macbook to Windows/Linux
- Ubuntu LTS is v6, so can only use
qemu
parameters
Update 2023
- Purchased a USB KVM for hardware switch
- Built-in SPICE controller has minimal delay for casual use
Shared folder
Audio
Arch wiki: Slowed down audio pumped through HDMI on the video card
- Fixed by installing MSI Utility and configure the GPU as MSI
-
Alternatively use scream audio
- Requires vfio brige setup
- Not as fast as Pipewire
Passthrough motherboard audio port
- Saves my DAC for linux (easier to manage with my side monitor)
- Easy to switch to windows guest if I needed by plugging the DAC USB to the passed through port
- I mostly use windows for work, meetings and gaming anyways
- Gaming actually benefits from the bass with my Logitech setup, and I use my monitor speakers with Linux for music only
Update 2023
- Looking glass B6 supports pipewire passthrough
- I also have a USB KVM to change over my Scarlett 2i2 over as well, useful when needing to use the virtual amplifiers
TPM2 for Windows 11
- Turned on fTPM module on motherboard
- Add permission to apparmour.d
- changed setting to tlt
Looking glass
High performance guest display as a host window with minimal overhead
First attempt on 01/05/2022
- Configuration was simple, just followed the website
- Nixpkgs looking-glass-client unable to find
EGL display
- Client unable to be built on neither Ubuntu (cannot find library) or Nix toolchain (target spec issue)
Second attempt on 02/05/2022
- Turns out my suspicion was correct,
nix
overriden the ubuntu path. - Compliation succeeded with
PATH=/bin:/usr/bin:$PATH make
- I've now disabled
evdev
by disablingpersistent-evdev
.
Update on 17/05/2022
- Turns out the reason nix packages couldn't find
EGL display
was to do with nix and the graphics driver - Fixed by installing nixGL
- Also learned how to override pkgs with Nix
Update 04/2023
- After playing around with
disk emulation
,iothreads
andROM BAR
, unfortunately Windows boot drive corrupted and I had to reset. - I retained my file, but had to reinstall everything
- For whatever reason, no more IO problems…
- I've tried to unset
ROM BAR
andiothreads
but nothing changed - Ongoing mystery
Optimisations
CPU optimisation
For Ryzen 5950 this is the optimal setup according to multiple sources such as this reddit post.
<cputune>
<vcpupin vcpu="0" cpuset="8"/>
<vcpupin vcpu="1" cpuset="24"/>
<vcpupin vcpu="2" cpuset="9"/>
<vcpupin vcpu="3" cpuset="25"/>
<vcpupin vcpu="4" cpuset="10"/>
<vcpupin vcpu="5" cpuset="26"/>
<vcpupin vcpu="6" cpuset="11"/>
<vcpupin vcpu="7" cpuset="27"/>
<vcpupin vcpu="8" cpuset="12"/>
<vcpupin vcpu="9" cpuset="28"/>
<vcpupin vcpu="10" cpuset="13"/>
<vcpupin vcpu="11" cpuset="29"/>
<vcpupin vcpu="12" cpuset="14"/>
<vcpupin vcpu="13" cpuset="30"/>
<vcpupin vcpu="14" cpuset="15"/>
<vcpupin vcpu="15" cpuset="31"/>
<emulatorpin cpuset="7,23"/>
</cputune>
Huge page memory optimisation
Easily done following online tutorials.
AVIC setup
Result
Seamless switching of the input devices
With looking glass, windows is like a normal window. I can use the USB KVM to switch keyboard and mouse. Game controller is connected via USB passthrough.
Great performance (with 4090 in 2023)
- Trialed Wasteland 2, Spin Rhythm, Halo, Cyberpunk 2077
- All works wonderfully
Userbench was amazing
Near 100% native machine performance with userbench
Windows 11 compatible
Passed hardware requirements for Windows 11 assessed
Removed the old windows box
Reduced noise!
References
Bryan Steiner tutorial
- Initial attempt
- Failed to work for me, freezes host due to unable to dynamically bind and unbind guest GPU
- Showed a nice shortcut for mounting NVME SSDs via PCI
Arch Wiki
- Works well enough
- Not specific for Ubuntu
Mathais Shueber tutorial
- Tailored for Ubuntu 20.04, works well in combination with Arch Wiki