Files

T

mrcharles 534c9340b8 GPU passthrough staged: vfio claim + q35 + hostpci0 on VM 5111 (#606 )

vfio-pci owns 10de:06dd/10de:0be5 at next boot (IOMMU group 20 clean);
VM 5111 gains machine q35 + hostpci0 pcie=1, live at the human's
pre-Friday roll. Revert + verification steps in staged/gpu-passthrough.md.

Details: https://projects.knownelement.com/issues/606

2026-09-01 13:42:11 -05:00

2.3 KiB

Raw Blame History

GPU passthrough — STAGED (Quadro 4000 -> VM 5111) — #606

Staged 2026-09-01 by crush via host ssh. Activates at the NEXT VM 5111 start (the human's pre-Friday roll). Nothing below touches the running VM or other guests.

What is staged

On pfv-tsys5 (host):

/etc/modprobe.d/vfio-quadro4000.conf
- options vfio-pci ids=10de:06dd,10de:0be5 (GPU + HDMI audio)
- softdep nouveau/nvidia/snd_hda_intel pre: vfio-pci
- initramfs rebuilt for all kernels.
- Result at next host boot: vfio-pci claims 03:00.0 + 03:00.1 (IOMMU group 20, verified isolated).
Runtime note: nouveau was unbound from 03:00.0 during staging (driverless until next boot — host console on that card is dark already; ssh/PVE web unaffected).

On VM 5111 (pending config, live at next start):

hostpci0: 0000:03:00,pcie=1
machine: q35        (was default pc/i440fx; virtio disks unaffected)

What to expect after the roll

Guest: lspci -nnk -d 10de:06dd → NVIDIA GF100GL, kernel 6.12 binds nouveau (Fermi-class). There is NO proprietary NVIDIA driver for Fermi on kernel 6.x — no CUDA, no NVENC. Usable for KMS display / capture (Sunshine KMS backend) and basic GL. Real GPU compute waits for the #601 hardware (new cards get their own staging pass).
Host console: permanently headless once vfio owns the card (this box is ssh/PVE-web managed; console access = the pfv-tsys5 console infra).
GPU-as-primary variant (drops std VGA): change hostpci0 to 0000:03:00,pcie=1,x-vga=1 — only if the guest should boot WITH the Quadro as its only display.

Verify after the roll

# host (ssh root@100.87.135.12)
lspci -nnk -s 03:00 | grep "driver in use"   # -> vfio-pci (both fns)
# guest
lspci -nnk -d 10de:06dd                       # -> nouveau
dmesg | grep -i vfio                          # assignment trace

Revert (if the roll goes sideways)

# host: stop claiming the card
rm /etc/modprobe.d/vfio-quadro4000.conf && update-initramfs -u -k all
# VM 5111: back to pre-staging machine + no passthrough
qm set 5111 --delete hostpci0 --delete machine
# then roll VM 5111 again; host reclaims the card at its next reboot

Roll procedure

Same as any 5111 roll: stop crush sessions, qm shutdown 5111, wait, qm start 5111 (onboot order=10 also covers a host reboot).

2.3 KiB Raw Blame History