GPU passthrough staged: vfio claim + q35 + hostpci0 on VM 5111 (#606)

vfio-pci owns 10de:06dd/10de:0be5 at next boot (IOMMU group 20 clean); VM 5111 gains machine q35 + hostpci0 pcie=1, live at the human's pre-Friday roll. Revert + verification steps in staged/gpu-passthrough.md. Details: https://projects.knownelement.com/issues/606
2026-09-01 13:42:11 -05:00
parent 46fe0cb6a5
commit 534c9340b8
2 changed files with 60 additions and 1 deletions
@@ -0,0 +1,58 @@
+# GPU passthrough — STAGED (Quadro 4000 -> VM 5111) — #606
+
+Staged 2026-09-01 by crush via host ssh. Activates at the NEXT VM 5111
+start (the human's pre-Friday roll). Nothing below touches the running
+VM or other guests.
+
+## What is staged
+
+**On pfv-tsys5 (host):**
+
+- `/etc/modprobe.d/vfio-quadro4000.conf`
+  - `options vfio-pci ids=10de:06dd,10de:0be5` (GPU + HDMI audio)
+  - `softdep nouveau/nvidia/snd_hda_intel pre: vfio-pci`
+  - initramfs rebuilt for all kernels.
+  - Result at next host boot: vfio-pci claims 03:00.0 + 03:00.1
+    (IOMMU group 20, verified isolated).
+- Runtime note: nouveau was unbound from 03:00.0 during staging
+  (driverless until next boot — host console on that card is dark
+  already; ssh/PVE web unaffected).
+
+**On VM 5111 (pending config, live at next start):**
+
+    hostpci0: 0000:03:00,pcie=1
+    machine: q35        (was default pc/i440fx; virtio disks unaffected)
+
+## What to expect after the roll
+
+- Guest: `lspci -nnk -d 10de:06dd` → NVIDIA GF100GL, kernel 6.12 binds
+  **nouveau** (Fermi-class). There is NO proprietary NVIDIA driver for
+  Fermi on kernel 6.x — no CUDA, no NVENC. Usable for KMS display /
+  capture (Sunshine KMS backend) and basic GL. Real GPU compute waits
+  for the #601 hardware (new cards get their own staging pass).
+- Host console: permanently headless once vfio owns the card (this box
+  is ssh/PVE-web managed; console access = the pfv-tsys5 console infra).
+- GPU-as-primary variant (drops std VGA): change hostpci0 to
+  `0000:03:00,pcie=1,x-vga=1` — only if the guest should boot WITH the
+  Quadro as its only display.
+
+## Verify after the roll
+
+    # host (ssh root@100.87.135.12)
+    lspci -nnk -s 03:00 | grep "driver in use"   # -> vfio-pci (both fns)
+    # guest
+    lspci -nnk -d 10de:06dd                       # -> nouveau
+    dmesg | grep -i vfio                          # assignment trace
+
+## Revert (if the roll goes sideways)
+
+    # host: stop claiming the card
+    rm /etc/modprobe.d/vfio-quadro4000.conf && update-initramfs -u -k all
+    # VM 5111: back to pre-staging machine + no passthrough
+    qm set 5111 --delete hostpci0 --delete machine
+    # then roll VM 5111 again; host reclaims the card at its next reboot
+
+## Roll procedure
+
+Same as any 5111 roll: stop crush sessions, `qm shutdown 5111`, wait,
+`qm start 5111` (onboot order=10 also covers a host reboot).