Regular-VM lane proven: libvirt q35 works nested, ops note added (#615)

Full VMs run on the workstation (q35 + host-passthrough + KVM): Debian cloud guest reached DHCP + ssh + running systemd. REGULAR-VMS.md holds the recipe and the three workstation gotchas (disk perms under images/, /boot-only kernels for virt-aa-helper, console=ttyS0 for serial). Details: https://projects.knownelement.com/issues/615
2026-09-01 14:00:20 -05:00
parent 534c9340b8
commit 0b54273fd5
3 changed files with 64 additions and 1 deletions
@@ -0,0 +1,60 @@
+# REGULAR-VMS.md — full (non-micro) VMs on the workstation (#615)
+
+**Proven 2026-09-01**: regular libvirt/qemu VMs run on this workstation
+(nested KVM). Test record at the bottom. `microvm` covers the fast
+disposable lane; this is the standard lane — virt-manager over RDP works
+for GUI management too.
+
+## The three workstation-specific rules (learned the hard way)
+
+1. **Disks must live under `/var/lib/libvirt/images/reachableceo/`**
+   (home dirs are 0700 by house rule — `libvirt-qemu` cannot traverse
+   `$HOME`). Dir is `770 reachableceo:libvirt-qemu`; make new images
+   `660` with group `libvirt-qemu`:
+       chgrp libvirt-qemu FILE && chmod 660 FILE
+2. **Direct-kernel-boot kernels/initrds must be under `/boot`**
+   (libvirt's AppArmor `virt-aa-helper` whitelist). Staged copies:
+   `/boot/microvm-vmlinuz`, `/boot/microvm-initrd` (from the microvm
+   base; refresh if the base image changes). Out-of-tree paths fail
+   with `error: skipped restricted file` / `cannot load AppArmor
+   profile`.
+3. **Serial visibility needs `console=ttyS0`** — the Debian cloud image
+   grub does not set it. Use direct kernel boot (below) or expect no
+   serial output. (Quirk: even with a serial-file chardev, output was
+   empty in the test; `domifaddr --source lease` + ssh is the reliable
+   health check.)
+
+## Working recipe (import a cloud image)
+
+    # seed: same vfat cidata trick as microvm (user-data + meta-data)
+    truncate -s 4M seed.img && mkfs.vfat -n CIDATA seed.img
+    mcopy -i seed.img user-data meta-data ::
+
+    virt-install --connect qemu:///system \
+      --name NAME --memory 1024 --vcpus 1 --import --os-variant debian12 \
+      --disk path=/var/lib/libvirt/images/reachableceo/NAME.qcow2,format=qcow2,bus=virtio \
+      --disk path=/var/lib/libvirt/images/reachableceo/seed.img,format=raw,bus=virtio,readonly=on \
+      --network network=default,model=virtio \
+      --graphics none --noautoconsole
+
+    virsh -c qemu:///system domifaddr NAME --source lease   # wait for IP
+    ssh debian@<ip>                                          # key from seed
+
+Defaults in play: machine `pc-q35`, cpu `host-passthrough`, KVM accel,
+`default` NAT network (192.168.122.0/24, now `autostart`ed).
+For console debugging add to the domain XML `<os>`:
+`<kernel>/boot/microvm-vmlinuz</kernel><initrd>/boot/microvm-initrd</initrd>
+<cmdline>console=ttyS0 root=/dev/vda1 rw</cmdline>`.
+
+## Test record (2026-09-01, #615)
+
+- `regvm-test`: Debian 13 cloud image import, 1 vCPU/1G, q35 +
+  host-passthrough, direct-kernel-boot cmdline.
+- Result: DHCP lease (`192.168.122.229`, hostname `regvm`) within ~60s
+  of start, ssh OK (key from NoCloud seed), `systemctl is-system-running`
+  = `running`, 6 virtio-PCI devices, root auto-grew to 9.7G.
+- Nested virt: guest CPU shows E5620 with `vmx` (VMs-in-VMs possible).
+- Teardown verified: destroy + undefine --remove-all-storage clean.
+- Compare: microvm lane boots to ssh in ~28s first boot / ~5-8s warm;
+  the full libvirt lane is slower but gives q35/PCI, virt-manager,
+  snapshots, standard tooling.