GPU Passthrough

Using a physical GPU in a virtual machine

IOMMU preperation (Intel CPU Only)

Add the kernel boot parameter to enable Intel IOMMU.

Edit /etc/default/grub and insert intel_iommu=on as an argument on the GRUB_CMDLINE_LINUX_DEFAULT line - separate other arguments with space. Save the file and run update-grub

Make sure that the Proxmox linux installation does not attempt to use the nVidia card by blacklisting the drivers and updating initramfs afterwards:

echo "blacklist nouveau" >> /etc/modprobe.d/blacklist.conf
echo "blacklist nvidia" >> /etc/modprobe.d/blacklist.conf
update-initramfs -u

Configure vfio - add the following to /etc/modules:

vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

Now we need to attach the PCI devices to the vfio driver. First find your PCI device with lspci.

lspci -v

Now scroll up to find your GPU - my output has this section:

00:1f.6 Signal processing controller: Intel Corporation C600/X79 series chipset Thermal Management Controller (rev 06)
    Subsystem: Super Micro Computer Inc C600/X79 series chipset Thermal Management Controller
    Flags: bus master, fast devsel, latency 0, IRQ 5, NUMA node 0
    Memory at fed08000 (64-bit, non-prefetchable) [size=4K]
    Capabilities: [50] Power Management version 3

04:00.0 VGA compatible controller: NVIDIA Corporation GP107 [GeForce GTX 1050 Ti] (rev a1) (prog-if 00 [VGA controller])
    Subsystem: NVIDIA Corporation GP107 [GeForce GTX 1050 Ti]
    Flags: bus master, fast devsel, latency 0, IRQ 62, NUMA node 0
    Memory at dd000000 (32-bit, non-prefetchable) [size=16M]
    Memory at c0000000 (64-bit, prefetchable) [size=256M]
    Memory at d0000000 (64-bit, prefetchable) [size=32M]
    I/O ports at 8000 [size=128]
    Expansion ROM at de000000 [disabled] [size=512K]
    Capabilities: [60] Power Management version 3
    Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
    Capabilities: [78] Express Legacy Endpoint, MSI 00
    Capabilities: [100] Virtual Channel
    Capabilities: [250] Latency Tolerance Reporting
    Capabilities: [128] Power Budgeting <?>
    Capabilities: [420] Advanced Error Reporting
    Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
    Capabilities: [900] #19
    Kernel driver in use: vfio-pci
    Kernel modules: nvidiafb, nouveau

04:00.1 Audio device: NVIDIA Corporation Device 0fb9 (rev a1)
    Subsystem: NVIDIA Corporation Device 11bf
    Flags: bus master, fast devsel, latency 0, IRQ 63, NUMA node 0
    Memory at de080000 (32-bit, non-prefetchable) [size=16K]
    Capabilities: [60] Power Management version 3
    Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
    Capabilities: [78] Express Endpoint, MSI 00
    Capabilities: [100] Advanced Error Reporting
    Kernel driver in use: vfio-pci
    Kernel modules: snd_hda_intel

Note the Nvidia device on 04:00.0, but also a matching sound device on 04:00.1. We want to pass both to our vm.

We can find the device ID of both with lspci by just referring to the top part of the address 04:00 In my case I can find the IDs here:

lspci -n -s 04:00
04:00.0 0300: 10de:1c82 (rev a1)
04:00.1 0403: 10de:0fb9 (rev a1)

Add the device IDs in /etc/modprobe/vfio.conf as options vfio-pci ids=XX:XX,XX:XX In my case:

echo "options vfio-pci ids=10de:1c82,10de:0fb9" > /etc/modprobe.d/vfio.conf

Reboot the Proxmox host before using the device on a vm.

Proxmox 6

Not sure why, but now I need the following in the [vmid].conf args: -no-hpet -rtc driftfix=slew -global kvm-pit.lost_tick_policy=discard -cpu host,hv_time,kvm=off,+kvm_pv_unhalt,+kvm_pv_eoi,hv_spinlocks=0x1fff,hv_vapic,hv_time,hv_reset,hv_vpindex,hv_runtime,hv_relaxed,hv_synic,hv_stimer,hv_vendor_id=proxmox -machine kernel_irqchip=on

Also, only seems to work after fresh DDU removal

Alternatively, read up at: https://forum.proxmox.com/threads/fix-for-pci-passthrough-and-issues-with-proxmox-6-gui-regex-that-needs-resolving-after-upgrade.56507/