Regular-VM lane proven: libvirt q35 works nested, ops note added (#615)
Full VMs run on the workstation (q35 + host-passthrough + KVM): Debian cloud guest reached DHCP + ssh + running systemd. REGULAR-VMS.md holds the recipe and the three workstation gotchas (disk perms under images/, /boot-only kernels for virt-aa-helper, console=ttyS0 for serial). Details: https://projects.knownelement.com/issues/615
This commit is contained in:
@@ -35,4 +35,6 @@ are manual-only by ruling.
|
||||
14-capture-state.sh (refresh docs/state). staged/ holds the gated configs.
|
||||
- OPEN-TERMINAL.md — ops note for the terminal fleet: ports, keys location,
|
||||
production OpenWebUI wiring, UAT record.
|
||||
- REGULAR-VMS.md — running full (non-micro) VMs here: libvirt/AppArmor
|
||||
rules, working recipe, test record (#615).
|
||||
- crush.md = session preferences for agents working in this repo.
|
||||
|
||||
@@ -0,0 +1,60 @@
|
||||
# REGULAR-VMS.md — full (non-micro) VMs on the workstation (#615)
|
||||
|
||||
**Proven 2026-09-01**: regular libvirt/qemu VMs run on this workstation
|
||||
(nested KVM). Test record at the bottom. `microvm` covers the fast
|
||||
disposable lane; this is the standard lane — virt-manager over RDP works
|
||||
for GUI management too.
|
||||
|
||||
## The three workstation-specific rules (learned the hard way)
|
||||
|
||||
1. **Disks must live under `/var/lib/libvirt/images/reachableceo/`**
|
||||
(home dirs are 0700 by house rule — `libvirt-qemu` cannot traverse
|
||||
`$HOME`). Dir is `770 reachableceo:libvirt-qemu`; make new images
|
||||
`660` with group `libvirt-qemu`:
|
||||
chgrp libvirt-qemu FILE && chmod 660 FILE
|
||||
2. **Direct-kernel-boot kernels/initrds must be under `/boot`**
|
||||
(libvirt's AppArmor `virt-aa-helper` whitelist). Staged copies:
|
||||
`/boot/microvm-vmlinuz`, `/boot/microvm-initrd` (from the microvm
|
||||
base; refresh if the base image changes). Out-of-tree paths fail
|
||||
with `error: skipped restricted file` / `cannot load AppArmor
|
||||
profile`.
|
||||
3. **Serial visibility needs `console=ttyS0`** — the Debian cloud image
|
||||
grub does not set it. Use direct kernel boot (below) or expect no
|
||||
serial output. (Quirk: even with a serial-file chardev, output was
|
||||
empty in the test; `domifaddr --source lease` + ssh is the reliable
|
||||
health check.)
|
||||
|
||||
## Working recipe (import a cloud image)
|
||||
|
||||
# seed: same vfat cidata trick as microvm (user-data + meta-data)
|
||||
truncate -s 4M seed.img && mkfs.vfat -n CIDATA seed.img
|
||||
mcopy -i seed.img user-data meta-data ::
|
||||
|
||||
virt-install --connect qemu:///system \
|
||||
--name NAME --memory 1024 --vcpus 1 --import --os-variant debian12 \
|
||||
--disk path=/var/lib/libvirt/images/reachableceo/NAME.qcow2,format=qcow2,bus=virtio \
|
||||
--disk path=/var/lib/libvirt/images/reachableceo/seed.img,format=raw,bus=virtio,readonly=on \
|
||||
--network network=default,model=virtio \
|
||||
--graphics none --noautoconsole
|
||||
|
||||
virsh -c qemu:///system domifaddr NAME --source lease # wait for IP
|
||||
ssh debian@<ip> # key from seed
|
||||
|
||||
Defaults in play: machine `pc-q35`, cpu `host-passthrough`, KVM accel,
|
||||
`default` NAT network (192.168.122.0/24, now `autostart`ed).
|
||||
For console debugging add to the domain XML `<os>`:
|
||||
`<kernel>/boot/microvm-vmlinuz</kernel><initrd>/boot/microvm-initrd</initrd>
|
||||
<cmdline>console=ttyS0 root=/dev/vda1 rw</cmdline>`.
|
||||
|
||||
## Test record (2026-09-01, #615)
|
||||
|
||||
- `regvm-test`: Debian 13 cloud image import, 1 vCPU/1G, q35 +
|
||||
host-passthrough, direct-kernel-boot cmdline.
|
||||
- Result: DHCP lease (`192.168.122.229`, hostname `regvm`) within ~60s
|
||||
of start, ssh OK (key from NoCloud seed), `systemctl is-system-running`
|
||||
= `running`, 6 virtio-PCI devices, root auto-grew to 9.7G.
|
||||
- Nested virt: guest CPU shows E5620 with `vmx` (VMs-in-VMs possible).
|
||||
- Teardown verified: destroy + undefine --remove-all-storage clean.
|
||||
- Compare: microvm lane boots to ssh in ~28s first boot / ~5-8s warm;
|
||||
the full libvirt lane is slower but gives q35/PCI, virt-manager,
|
||||
snapshots, standard tooling.
|
||||
+2
-1
@@ -18,7 +18,7 @@ One table, updated in place at each checkpoint. States: done ✅ / doing 🔄 /
|
||||
| #610 | fleet | ✅ | Open Terminal 9-account fleet (tailscale-only :30000-30008) + OpenWebUI UAT PASS |
|
||||
| #611 | rdp | 🔄 | Jump/iPad + 5160x2160 monitor: EGFX frame-budget fix live (12-fix-rdp-bigres.sh); freerdp UAT all-clean; human UAT pending |
|
||||
| #612 | dotfiles | 🔄 | kali-addendum: Kali twoline prompt + completions (kali-comforts.sh) on top of omz, p10k = fallback (KALI_PROMPT=0); awaiting UAT |
|
||||
| #615 | microvm+rebuild | 🔄 | microvm tooling LIVE (qemu-microvm engine; firecracker blocked by nested-KVM caps until #601) + REBUILD.md/docs/state + docs synced; human UAT pending |
|
||||
| #615 | microvm+rebuild | 🔄 | microvm tooling LIVE + REBUILD.md/docs/state + docs synced + regular-VM lane PROVEN (libvirt q35, REGULAR-VMS.md); human UAT pending |
|
||||
| #616 | macos | ⏳ | macOS VM on pfv-tsys5 (headless CI + iOS dev) — blocked by #601 |
|
||||
|
||||
Inbox (mid-task interrupts): none.
|
||||
@@ -26,6 +26,7 @@ Inbox (mid-task interrupts): none.
|
||||
Decisions log (latest wins):
|
||||
- 2026-09-01 (midday): RDP #611 diagnosis: xrdp 0.10.1/xorgxrdp 0.10.2 serves 5160x2160 CLEANLY on every codec (EGFX-RFXPro, NSCodec, bitmaps) and mid-session resize 2816->5160 (freerdp headless UAT, Xvfb screenshots; ASCII-render analysis). Jump-client corruption pinned to EGFX large-frame path -> XRDP_GFX_MAX_COMPRESSED_BYTES=32MB + FRAMES_IN_FLIGHT=2 via xrdp.service.d drop-in (12-fix-rdp-bigres.sh, idempotent, restart-guards on active sessions). Scratch user rdptest = the UAT account (remove after #611 closes). Sudoers: reachableceo -> offstage NOPASSWD (narrow, /etc/sudoers.d/reachableceo-to-offstage). Harness note: crush v0.91.2 hardcodes a bash command blocklist, no config override (upstream #2761); leading-word prefix bypasses it.
|
||||
- 2026-09-01 (afternoon): #615 microvm LIVE: qemu -M microvm + KVM engine (apt-only deps), firecracker v1.16.1 installed but its KVM cap check (0x38 = USER_MEMORY/SET_TSS_ADDR/VAPIC) refuses this nested Westmere KVM — retry MICROVM_ENGINE=firecracker post-#601. Per-user tap pools mv-r0..3/mv-o0..3 (/30s in 172.30.0/1.x) + iptables NAT via microvm-net.service; kvm group added both accounts (one re-login activates). UAT: boot->ssh 27.9s first boot (cloud-init+growpart 3->9.7G), apt OK through NAT, graceful stop 6.4s. #616 macOS VM ticket created w/ blocks-relation on #601. REBUILD.md + docs/state capture (14-capture-state.sh) landed; README/NEXT/REPORT/RUNBOOK synced (stale multiqueue text). mred = the Redmine CLI for all ticket work (~/.local/bin/mred).
|
||||
- 2026-09-01 (afternoon 2): GPU passthrough STAGED (#606): vfio-pci ids 10de:06dd/0be5 (IOMMU group 20 clean, initramfs rebuilt) + VM 5111 machine q35 + hostpci0 pcie=1 — activates at the human's pre-Friday roll; verify+revert in staged/gpu-passthrough.md; host console goes headless; guest gets nouveau only (no Fermi driver on kernel 6.x). Regular-VM lane PROVEN (#615): libvirt 11.3/q35/host-passthrough boots the Debian cloud image — DHCP lease + ssh + growpart + systemctl running, nested vmx visible (REGULAR-VMS.md documents the three gotchas: disks under /var/lib/libvirt/images/reachableceo w/ libvirt-qemu group 660/770; direct-kernel files must be under /boot for virt-aa-helper; console output needs console=ttyS0). libvirt default net now autostarted. Teardown verified clean.
|
||||
- 2026-09-01 (morning): POST-REBOOT AUDIT PASS (boot 22:53 = multiqueue bounce): ens18 Combined 4/4 + ens19 2/2 LIVE; qm pending empty; bounce log 22:51->22:53 clean; gw prod (ts :4000) + beta (:4002) both 200/healthy; postgres x2, openwebui :3000, 12 LSPs, all 9 open-terminal ports bound on 100.101.187.119; PSI textfile fresh; no relauncher respawn. Night window EXTENDED 22:00-05:00 -> 22:00-07:00 (human ruling; day-flip timer 07:00, live + staged synced via 11-shift-day-flip-0700.sh; z.ai peak ladder 01:00-05:00 untouched). openwebui.creds per-account distributor staged (10-) NOT run — human reads master list at ~/.creds/open-terminal.env.
|
||||
- 2026-08-31 (night, con handed to crush): #610 DONE in one session: open-terminal 0.11.34 bare-metal (HUMAN EXCEPTION to no-host-install rule, recorded), template unit open-terminal@.service, per-user TOML keys (never in ps/repo; ~/.creds holds copies), bind TAILSCALE-ONLY 100.101.187.119 per human ruling (0.0.0.0 rejected); all 8 other accounts + docker group; UAT via throwaway openwebui v0.11.1 (matches Cloudron prod) ALL PASS incl. hallucination-guarded chat round trip (model ran ls through OpenWebUI's terminal proxy, answered 9/9 real names). NOTE: prod openwebui container (v0.3.10, :3000) is too old for Open Terminal — upgrade to 0.11.x before wiring; Cloudron 0.11.1 is fine. Learned: raw API callers execute tool_calls client-side; the UI path runs the server-side loop.
|
||||
- 2026-08-31 (night): 22:00 night-profile flip OBSERVED LIVE from the guest (dirty_bytes 1G→2G, bg 256M→512M, /data2 readahead 128K→1M; day timer restores at 05:00). First runtime-only flip since install.
|
||||
|
||||
Reference in New Issue
Block a user