ultix/REBUILD.md

REBUILD.md — ultix-streaming (VM 5111) full-rebuild runbook (#615)

How to rebuild this workstation from bare metal up. The mechanism is this repo's numbered idempotent scripts, run in order; this file is the map + the parts scripts don't cover. Fleet-wide ansible will consume this later (human goal, 2026-09-01); until then: shell-first.

• Live state snapshots: docs/state/ — regenerate any time with ./14-capture-state.sh (read-only; guest + PVE).
• Secrets live ONLY in ~/.creds (human restores from backup); never in this repo, images, or compose files.
• Docker/registry content is repo-owned (repos rebuild their own containers) — out of scope here.
• PVE-side DR path for total loss: existing vzdump/backup policy, not this runbook (this runbook is for a clean, deliberate rebuild).

0. PVE side (host pfv-tsys5)

Prereqs on the host: nested KVM enabled (/sys/module/kvm_intel/parameters/nested = Y), dir storages NVME, ssd2, SSD, bridges vmbr0 (LAN) + datanet.

VM definition (live copy: docs/state/pve-5111.txt):

qm create 5111 --name ultix-streaming --ostype l26 \
  --cores 4 --sockets 2 --cpu host,flags=+nested-virt \
  --memory 50000 --balloon 0 \
  --scsihw virtio-scsi-single \
  --net0 virtio,bridge=vmbr0,queues=4 \
  --net1 virtio,bridge=datanet,queues=2 \
  --agent 1 --onboot 1 --startup order=10,up=180 --cpuunits 9000
# disks (each with discard=on,iothread=1,ssd=1):
#   scsi0 NVME:438G (OS)   scsi1 ssd2:400G (/data2)   scsi2 SSD:200G (/data1)

Flags why: cpu host+nested-virt = microvm/Firecracker inside; virtio-scsi-single + iothread = per-disk IO threads; queues 4/2 match guest multiqueue (activated in live qm config 2026-09-01); cpuunits 9000 keeps the workstation above the fleet; onboot+order=10 = first VM up after host boot (gateway auto-recovers).

1. Guest OS

• Debian 13 trixie amd64 netinst, KDE desktop, standard guided partitioning on the NVME (ext4 + swap; ends up / 279G + swap 8.8G).
• Mount the data disks per docs/state/guest-system.txt FSTAB section: /data1 (SSD 200G) and /data2 (ssd2 400G), ext4 noatime.
• Hostname ultix-streaming; ssh enabled.
• KDE policy (2026-09-01 ruling, supersedes 08-31): NO sddm autologin — an always-alive console session holds the user's single plasmashell and black-screens RDP (Plasma 6 = one shell per user, #611). Console shows the sddm greeter at boot; RDP is the primary desktop. When Sunshine lands with the GPU it gets a dedicated kiosk account or XFCE.

2. Accounts + secrets + tools

• Accounts: reachableceo (uid 1001), reachableceo-offstage; shells /usr/bin/zsh; groups: sudo, kvm, docker, libvirt (see docs/state/guest-users.txt).
• ~/.creds/* restored by the human (redmine, discourse, gateway, open-terminal env files...). Nothing secret is in this repo.
• ssh keys: human restores (bitwarden ssh-agent pattern).
• /usr/local/bin/chezmoi v2.72.1 (single binary), /usr/local/bin/ firecracker + jailer v1.16.1 (installed by 13-microvm-setup.sh).

3. Repo scripts, in order

script	what / when
1-guest-prep.sh	guest base prep (perf pass)
2-host-one-shot.sh	PVE-side flags via ssh (run once at build)
3-post-reboot-fixes.sh	ethtool + multiqueue oneshot bits
4-host-netcheck.sh	verify host flags + multiqueue (read-only)
5-activate-multiqueue.sh	HISTORICAL: activated multiqueue 2026-09-01; a fresh VM gets queues from the qm config in step 0 + the oneshot unit
6-remove-agent-stacks.sh	HISTORICAL teardown; N/A on fresh build
7-open-terminal-install.sh + 8-open-terminal-users.sh	Open Terminal fleet (#610)
9-uat-openwebui.sh	OpenWebUI UAT harness (uat/ compose)
12-fix-rdp-bigres.sh	xrdp EGFX budget drop-in for 5160x2160 (#611)
13-microvm-setup.sh	microvm tooling: qemu microvm engine, firecracker binary, tap pools + NAT unit, cloud assets (#615)
14-capture-state.sh	refresh docs/state after any drift
15-docker-to-data2.sh	STAGED (not run): moves docker data-root to /data2 (drain-aware, reversible)
16-grow-root-online.sh	RAN 2026-09-01: / grown online to 431G, swap removed by design (#603)
15-docker-to-data2.sh	STAGED (not run): moves docker data-root
to /data2 (drain-aware, reversible)
16-grow-root-online.sh	RAN 2026-09-01: / grown online to
431G, swap removed by design (#603)

4. Dotfiles (both accounts) — #612

1. gitea ssh access (human key) → clone reachableceo/dotfiles (private) to ~/projects/dotfiles.
2. ln -s ~/projects/dotfiles ~/.local/share/chezmoi
3. chezmoi apply (fetches pinned omz/p10k/plugin externals itself).
4. offstage: root-copies the source snapshot and applies with -S ~reachableceo-offstage/projects/dotfiles (no gitea creds there) — see the dotfiles README.

5. Standing services (beyond the scripts)

• Gateway lanes via compose + ukrrs-gateway-ensure.service (boot-race fix; must be enabled on BOTH lanes before any host bounce).
• Day/night cgroup flips: ukrrs-dayprofile.timer 07:00 / 22:00-07:00 night window (runtime-only cgroup flips).
• cgroup slices: ukrrs-acct-*.slice (per-account), ukrrs-gateway, ukrrs-builder-prune.timer, ukrrs-lsp.slice (#608).
• sysctl profile /etc/sysctl.d/60-ukrrs-vm.conf (measured rationale in REPORT.md §4.1; day/night variants raise dirty thresholds).
• xrdp (+ bigres drop-in from 12-), microvm-net.service, open-terminal@.service fleet, beszel-agent.
• Docker Engine: docker.com apt repo (docker-ce, currently 29.x — see docs/state/guest-docker-pkg.txt); compose v2 plugin.

6. Package baseline

docs/state/guest-apt-manual.txt = apt-mark showmanual snapshot (2026-09-01). On a rebuild, after all scripts:

apt-mark showmanual | sort | diff - docs/state/guest-apt-manual.txt

Notables installed by hand over time: zsh fzf ripgrep bat eza fd-find tmux, qemu-system-x86 qemu-utils dosfstools mtools, libvirt/clt tools, ethtool, imageMagick, KDE extras. Full list is in the snapshot.

7. Deliberately NOT captured here

• Secrets (~/.creds), ssh keys, browser profiles.
• Docker images/volumes (repos own their stacks; gateway lanes pull their own images from the gateway registry/promote pipeline).
• The scratch UAT harnesses (/tmp/rdptest) and scratch users (rdptest, removed when #611 closes).
• PVE host itself (that is the fleet-ansible goal; docs/state/pve-* is the reference capture of today's host facts).

8. Post-rebuild verification checklist

1. 4-host-netcheck.sh sweep clean (queues 4/4 + 2/2 live).
2. Gateway /status = normal on both lanes after a bounce.
3. RDP from iPad + 5160x2160 monitor clean (#611 UAT).
4. Fresh zsh on both accounts: Kali-style prompt + omz (#612 UAT).
5. microvm create t && microvm start t && microvm ssh t then rm.
6. Open Terminal ports 30000-30008 answer (tailscale-only).
7. Rerun 14-capture-state.sh, commit drift to docs/state.