Add the kernel boot parameter to enable Intel IOMMU.
Edit /etc/default/grub and insert intel_iommu=on as an argument on the GRUB_CMDLINE_LINUX_DEFAULT line - separate other arguments with space. Save the file and run update-grub
Make sure that the Proxmox linux installation does not attempt to use the nVidia card by blacklisting the drivers and updating initramfs afterwards:
echo "blacklist nouveau" >> /etc/modprobe.d/blacklist.conf
echo "blacklist nvidia" >> /etc/modprobe.d/blacklist.conf
update-initramfs -u
Configure vfio - add the following to /etc/modules:
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd
Now we need to attach the PCI devices to the vfio driver. First find your PCI device with lspci.
lspci -v
Now scroll up to find your GPU - my output has this section:
You'll be able to scroll back with shift+pgup/pgdown in most terminals.
00:1f.6 Signal processing controller: Intel Corporation C600/X79 series chipset Thermal Management Controller (rev 06)
Subsystem: Super Micro Computer Inc C600/X79 series chipset Thermal Management Controller
Flags: bus master, fast devsel, latency 0, IRQ 5, NUMA node 0
Memory at fed08000 (64-bit, non-prefetchable) [size=4K]
Capabilities: [50] Power Management version 3
04:00.0 VGA compatible controller: NVIDIA Corporation GP107 [GeForce GTX 1050 Ti] (rev a1) (prog-if 00 [VGA controller])
Subsystem: NVIDIA Corporation GP107 [GeForce GTX 1050 Ti]
Flags: bus master, fast devsel, latency 0, IRQ 62, NUMA node 0
Memory at dd000000 (32-bit, non-prefetchable) [size=16M]
Memory at c0000000 (64-bit, prefetchable) [size=256M]
Memory at d0000000 (64-bit, prefetchable) [size=32M]
I/O ports at 8000 [size=128]
Expansion ROM at de000000 [disabled] [size=512K]
Capabilities: [60] Power Management version 3
Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
Capabilities: [78] Express Legacy Endpoint, MSI 00
Capabilities: [100] Virtual Channel
Capabilities: [250] Latency Tolerance Reporting
Capabilities: [128] Power Budgeting <?>
Capabilities: [420] Advanced Error Reporting
Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
Capabilities: [900] #19
Kernel driver in use: vfio-pci
Kernel modules: nvidiafb, nouveau
04:00.1 Audio device: NVIDIA Corporation Device 0fb9 (rev a1)
Subsystem: NVIDIA Corporation Device 11bf
Flags: bus master, fast devsel, latency 0, IRQ 63, NUMA node 0
Memory at de080000 (32-bit, non-prefetchable) [size=16K]
Capabilities: [60] Power Management version 3
Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
Capabilities: [78] Express Endpoint, MSI 00
Capabilities: [100] Advanced Error Reporting
Kernel driver in use: vfio-pci
Kernel modules: snd_hda_intel
Note the Nvidia device on 04:00.0, but also a matching sound device on 04:00.1. We want to pass both to our vm.
We can find the device ID of both with lspci by just referring to the top part of the address 04:00 In my case I can find the IDs here:
lspci -n -s 04:00
04:00.0 0300: 10de:1c82 (rev a1)
04:00.1 0403: 10de:0fb9 (rev a1)
Add the device IDs in /etc/modprobe/vfio.conf as options vfio-pci ids=XX:XX,XX:XX In my case:
echo "options vfio-pci ids=10de:1c82,10de:0fb9" > /etc/modprobe.d/vfio.conf
Reboot the Proxmox host before using the device on a vm.
Not sure why, but now I need the following in the [vmid].conf args: -no-hpet -rtc driftfix=slew -global kvm-pit.lost_tick_policy=discard -cpu host,hv_time,kvm=off,+kvm_pv_unhalt,+kvm_pv_eoi,hv_spinlocks=0x1fff,hv_vapic,hv_time,hv_reset,hv_vpindex,hv_runtime,hv_relaxed,hv_synic,hv_stimer,hv_vendor_id=proxmox -machine kernel_irqchip=on
Also, only seems to work after fresh DDU removal
Alternatively, read up at: https://forum.proxmox.com/threads/fix-for-pci-passthrough-and-issues-with-proxmox-6-gui-regex-that-needs-resolving-after-upgrade.56507/