DKVM - Desktop KVM

Desktop KVM Operating System

I have never really liked the idea of having Windows running native on my machines - so for about the last year QEMU/KVM + VFIO has been my solution to running Windows applications.

While many others are currently using QEMU/KVM + VFIO Passthrough to run Windows at near native speed - I was unable to find a dedicated "KVM OS" for my needs.
My "daily-driver" is Linux - and, I wanted an option to allocate more resources to the Windows VM, without scarifying resources to my main Linux OS.
A way to accomplish this, is to have both the "daily driver" and the "gaming os" being virtualized, and not running at the same time.

So, my journey began to find a KVM hypervisor OS focused on running desktop VM's.

My KVM Desktop OS whishes

• VFIO support
• Be able to use my one "good" graphics-card for both Linux and Windows
• Run-From-RAM style OS.
• Boot from USB stick
• Minimal memory foot-print
• Support for software raid (preferable mdadm)
• Support for bcache

The failed attempts

• I gave oVirt a go, which honestly is quite nice. But, making this a Run-From-RAM style OS gave me quite some issues. A lot of tweaking was required - and maintaining all my tweaks has too much of a hazzle.
• VMware ESXi recently has good pass-through support, and qorks really well booting from a USB stick (And it is already a Run-From-RAM style OS). Sadly ESXi has no support for software raid, and the SSD cache functionality is limited compared to bcache.
• LinuxFromScratch was next in line, and I scriptet most of the build process - and got a fairly good working system. Though, I lacked the "easy maintenance" - and put the code on the shelf for now.

DKVM - My Desktop KVM OS

Lastly, I have AlpineLinux a go - as this has builtin support for Run-From-RAM style booting - including booting from a USB stick.

In short, I use QEMU to build a AlpineLinux bootable image, which can be flashed to a USB stick.

As of 3.8.0 release of AlpineLinux, both UEFI and Legacy booting now works out-of-the-box.

The build process

The setup.sh script is a wrapper for starting the build process - and is also creating a new empty disk-image file.

Here we will also be creating two addition ISO's used for stage01 and stage02.

stage01

Here we setup the base of the operating system.

I have chosen to include an "answer.txt" file for the alpine-setup program. This allowed the setup to run non-interactive.
The result of stage01 is a bootable disk image in a Run-From-RAM style

stage02

Here we setup the more custom part of the OS. Basic configuration like support for saving changes to USB disk is done here - among tweaks to varies aspects of the OS
We also do things like setting up both UEFI and Legacy bootloaders with vfio-pci.ids, and other KVM optimization arguments.

In my case, I have chosen to go for a setup with CPU isolation to minimize latency inside the running guests.

This is done using the isolcpus, nohz_full and rcu_nocbs arguments to the kernel - in combination with taskset to make affinity-rules for the vCPU threads.

vm_start

When the DKVM OS is loaded, i am presented with a simple menu to start my configured VMs.

At the time of writing, I only have two VM's configured - but I would imagine that will change.

This is done fairly simple with a wrapper-script around QEMU. 

vCPU affinity and isolation

Isolation is done using isolcpus - this will ensure no other threads are running on those cores.

The affinity is done using QEMU's builtin QMP system. This allowed the user to query for the threads used by QEMU to serve the vCPUs.

NOTE: In my case, my hyperthreaded cores is interleaved with the physical cores when looking at the processor ID. This can be checked using something similar to: 

# cat /proc/cpuinfo | grep "core id\|processor"

The CORELIST and isolcpu will need to reflect this.

All of the sources are available at my github - though, keep in mind this is written for personal use only!

Any recommendations/patches are welcome though :)

Other useful information / links

Linux and Wendell teamed up to test gaming on Linux. The different options are all quite interesting.

• DKVM build scripts
• VFIO tips and tricks - really good blog about VFIO
• PCI passthrough via OVMF - good documentation about passthrough.
• forum.level1techs.com - Great forums for Linux / VFIO