From 0b54273fd5cf5c6b6889be4aae07b178407939aa Mon Sep 17 00:00:00 2001 From: reachableceo Date: Tue, 1 Sep 2026 14:00:20 -0500 Subject: [PATCH] Regular-VM lane proven: libvirt q35 works nested, ops note added (#615) Full VMs run on the workstation (q35 + host-passthrough + KVM): Debian cloud guest reached DHCP + ssh + running systemd. REGULAR-VMS.md holds the recipe and the three workstation gotchas (disk perms under images/, /boot-only kernels for virt-aa-helper, console=ttyS0 for serial). Details: https://projects.knownelement.com/issues/615 --- README.md | 2 ++ REGULAR-VMS.md | 60 ++++++++++++++++++++++++++++++++++++++++++++++++++ TRACKING.md | 3 ++- 3 files changed, 64 insertions(+), 1 deletion(-) create mode 100644 REGULAR-VMS.md diff --git a/README.md b/README.md index 5a62356..490eb7d 100644 --- a/README.md +++ b/README.md @@ -35,4 +35,6 @@ are manual-only by ruling. 14-capture-state.sh (refresh docs/state). staged/ holds the gated configs. - OPEN-TERMINAL.md — ops note for the terminal fleet: ports, keys location, production OpenWebUI wiring, UAT record. +- REGULAR-VMS.md — running full (non-micro) VMs here: libvirt/AppArmor + rules, working recipe, test record (#615). - crush.md = session preferences for agents working in this repo. diff --git a/REGULAR-VMS.md b/REGULAR-VMS.md new file mode 100644 index 0000000..5cab2d3 --- /dev/null +++ b/REGULAR-VMS.md @@ -0,0 +1,60 @@ +# REGULAR-VMS.md — full (non-micro) VMs on the workstation (#615) + +**Proven 2026-09-01**: regular libvirt/qemu VMs run on this workstation +(nested KVM). Test record at the bottom. `microvm` covers the fast +disposable lane; this is the standard lane — virt-manager over RDP works +for GUI management too. + +## The three workstation-specific rules (learned the hard way) + +1. **Disks must live under `/var/lib/libvirt/images/reachableceo/`** + (home dirs are 0700 by house rule — `libvirt-qemu` cannot traverse + `$HOME`). Dir is `770 reachableceo:libvirt-qemu`; make new images + `660` with group `libvirt-qemu`: + chgrp libvirt-qemu FILE && chmod 660 FILE +2. **Direct-kernel-boot kernels/initrds must be under `/boot`** + (libvirt's AppArmor `virt-aa-helper` whitelist). Staged copies: + `/boot/microvm-vmlinuz`, `/boot/microvm-initrd` (from the microvm + base; refresh if the base image changes). Out-of-tree paths fail + with `error: skipped restricted file` / `cannot load AppArmor + profile`. +3. **Serial visibility needs `console=ttyS0`** — the Debian cloud image + grub does not set it. Use direct kernel boot (below) or expect no + serial output. (Quirk: even with a serial-file chardev, output was + empty in the test; `domifaddr --source lease` + ssh is the reliable + health check.) + +## Working recipe (import a cloud image) + + # seed: same vfat cidata trick as microvm (user-data + meta-data) + truncate -s 4M seed.img && mkfs.vfat -n CIDATA seed.img + mcopy -i seed.img user-data meta-data :: + + virt-install --connect qemu:///system \ + --name NAME --memory 1024 --vcpus 1 --import --os-variant debian12 \ + --disk path=/var/lib/libvirt/images/reachableceo/NAME.qcow2,format=qcow2,bus=virtio \ + --disk path=/var/lib/libvirt/images/reachableceo/seed.img,format=raw,bus=virtio,readonly=on \ + --network network=default,model=virtio \ + --graphics none --noautoconsole + + virsh -c qemu:///system domifaddr NAME --source lease # wait for IP + ssh debian@ # key from seed + +Defaults in play: machine `pc-q35`, cpu `host-passthrough`, KVM accel, +`default` NAT network (192.168.122.0/24, now `autostart`ed). +For console debugging add to the domain XML ``: +`/boot/microvm-vmlinuz/boot/microvm-initrd +console=ttyS0 root=/dev/vda1 rw`. + +## Test record (2026-09-01, #615) + +- `regvm-test`: Debian 13 cloud image import, 1 vCPU/1G, q35 + + host-passthrough, direct-kernel-boot cmdline. +- Result: DHCP lease (`192.168.122.229`, hostname `regvm`) within ~60s + of start, ssh OK (key from NoCloud seed), `systemctl is-system-running` + = `running`, 6 virtio-PCI devices, root auto-grew to 9.7G. +- Nested virt: guest CPU shows E5620 with `vmx` (VMs-in-VMs possible). +- Teardown verified: destroy + undefine --remove-all-storage clean. +- Compare: microvm lane boots to ssh in ~28s first boot / ~5-8s warm; + the full libvirt lane is slower but gives q35/PCI, virt-manager, + snapshots, standard tooling. diff --git a/TRACKING.md b/TRACKING.md index 7c7becf..e800302 100644 --- a/TRACKING.md +++ b/TRACKING.md @@ -18,7 +18,7 @@ One table, updated in place at each checkpoint. States: done ✅ / doing 🔄 / | #610 | fleet | ✅ | Open Terminal 9-account fleet (tailscale-only :30000-30008) + OpenWebUI UAT PASS | | #611 | rdp | 🔄 | Jump/iPad + 5160x2160 monitor: EGFX frame-budget fix live (12-fix-rdp-bigres.sh); freerdp UAT all-clean; human UAT pending | | #612 | dotfiles | 🔄 | kali-addendum: Kali twoline prompt + completions (kali-comforts.sh) on top of omz, p10k = fallback (KALI_PROMPT=0); awaiting UAT | -| #615 | microvm+rebuild | 🔄 | microvm tooling LIVE (qemu-microvm engine; firecracker blocked by nested-KVM caps until #601) + REBUILD.md/docs/state + docs synced; human UAT pending | +| #615 | microvm+rebuild | 🔄 | microvm tooling LIVE + REBUILD.md/docs/state + docs synced + regular-VM lane PROVEN (libvirt q35, REGULAR-VMS.md); human UAT pending | | #616 | macos | ⏳ | macOS VM on pfv-tsys5 (headless CI + iOS dev) — blocked by #601 | Inbox (mid-task interrupts): none. @@ -26,6 +26,7 @@ Inbox (mid-task interrupts): none. Decisions log (latest wins): - 2026-09-01 (midday): RDP #611 diagnosis: xrdp 0.10.1/xorgxrdp 0.10.2 serves 5160x2160 CLEANLY on every codec (EGFX-RFXPro, NSCodec, bitmaps) and mid-session resize 2816->5160 (freerdp headless UAT, Xvfb screenshots; ASCII-render analysis). Jump-client corruption pinned to EGFX large-frame path -> XRDP_GFX_MAX_COMPRESSED_BYTES=32MB + FRAMES_IN_FLIGHT=2 via xrdp.service.d drop-in (12-fix-rdp-bigres.sh, idempotent, restart-guards on active sessions). Scratch user rdptest = the UAT account (remove after #611 closes). Sudoers: reachableceo -> offstage NOPASSWD (narrow, /etc/sudoers.d/reachableceo-to-offstage). Harness note: crush v0.91.2 hardcodes a bash command blocklist, no config override (upstream #2761); leading-word prefix bypasses it. - 2026-09-01 (afternoon): #615 microvm LIVE: qemu -M microvm + KVM engine (apt-only deps), firecracker v1.16.1 installed but its KVM cap check (0x38 = USER_MEMORY/SET_TSS_ADDR/VAPIC) refuses this nested Westmere KVM — retry MICROVM_ENGINE=firecracker post-#601. Per-user tap pools mv-r0..3/mv-o0..3 (/30s in 172.30.0/1.x) + iptables NAT via microvm-net.service; kvm group added both accounts (one re-login activates). UAT: boot->ssh 27.9s first boot (cloud-init+growpart 3->9.7G), apt OK through NAT, graceful stop 6.4s. #616 macOS VM ticket created w/ blocks-relation on #601. REBUILD.md + docs/state capture (14-capture-state.sh) landed; README/NEXT/REPORT/RUNBOOK synced (stale multiqueue text). mred = the Redmine CLI for all ticket work (~/.local/bin/mred). +- 2026-09-01 (afternoon 2): GPU passthrough STAGED (#606): vfio-pci ids 10de:06dd/0be5 (IOMMU group 20 clean, initramfs rebuilt) + VM 5111 machine q35 + hostpci0 pcie=1 — activates at the human's pre-Friday roll; verify+revert in staged/gpu-passthrough.md; host console goes headless; guest gets nouveau only (no Fermi driver on kernel 6.x). Regular-VM lane PROVEN (#615): libvirt 11.3/q35/host-passthrough boots the Debian cloud image — DHCP lease + ssh + growpart + systemctl running, nested vmx visible (REGULAR-VMS.md documents the three gotchas: disks under /var/lib/libvirt/images/reachableceo w/ libvirt-qemu group 660/770; direct-kernel files must be under /boot for virt-aa-helper; console output needs console=ttyS0). libvirt default net now autostarted. Teardown verified clean. - 2026-09-01 (morning): POST-REBOOT AUDIT PASS (boot 22:53 = multiqueue bounce): ens18 Combined 4/4 + ens19 2/2 LIVE; qm pending empty; bounce log 22:51->22:53 clean; gw prod (ts :4000) + beta (:4002) both 200/healthy; postgres x2, openwebui :3000, 12 LSPs, all 9 open-terminal ports bound on 100.101.187.119; PSI textfile fresh; no relauncher respawn. Night window EXTENDED 22:00-05:00 -> 22:00-07:00 (human ruling; day-flip timer 07:00, live + staged synced via 11-shift-day-flip-0700.sh; z.ai peak ladder 01:00-05:00 untouched). openwebui.creds per-account distributor staged (10-) NOT run — human reads master list at ~/.creds/open-terminal.env. - 2026-08-31 (night, con handed to crush): #610 DONE in one session: open-terminal 0.11.34 bare-metal (HUMAN EXCEPTION to no-host-install rule, recorded), template unit open-terminal@.service, per-user TOML keys (never in ps/repo; ~/.creds holds copies), bind TAILSCALE-ONLY 100.101.187.119 per human ruling (0.0.0.0 rejected); all 8 other accounts + docker group; UAT via throwaway openwebui v0.11.1 (matches Cloudron prod) ALL PASS incl. hallucination-guarded chat round trip (model ran ls through OpenWebUI's terminal proxy, answered 9/9 real names). NOTE: prod openwebui container (v0.3.10, :3000) is too old for Open Terminal — upgrade to 0.11.x before wiring; Cloudron 0.11.1 is fine. Learned: raw API callers execute tool_calls client-side; the UI path runs the server-side loop. - 2026-08-31 (night): 22:00 night-profile flip OBSERVED LIVE from the guest (dirty_bytes 1G→2G, bg 256M→512M, /data2 readahead 128K→1M; day timer restores at 05:00). First runtime-only flip since install.