# REBUILD.md — ultix-streaming (VM 5111) full-rebuild runbook (#615)

How to rebuild this workstation from bare metal up. The mechanism is
this repo's numbered idempotent scripts, run in order; this file is the
map + the parts scripts don't cover. Fleet-wide ansible will consume
this later (human goal, 2026-09-01); until then: shell-first.

- Live state snapshots: `docs/state/` — regenerate any time with
  `./14-capture-state.sh` (read-only; guest + PVE).
- Secrets live ONLY in `~/.creds` (human restores from backup); never
  in this repo, images, or compose files.
- Docker/registry content is repo-owned (repos rebuild their own
  containers) — out of scope here.
- PVE-side DR path for total loss: existing vzdump/backup policy, not
  this runbook (this runbook is for a clean, deliberate rebuild).

## 0. PVE side (host pfv-tsys5)

Prereqs on the host: nested KVM enabled
(`/sys/module/kvm_intel/parameters/nested` = Y), dir storages `NVME`,
`ssd2`, `SSD`, bridges `vmbr0` (LAN) + `datanet`.

VM definition (live copy: `docs/state/pve-5111.txt`):

    qm create 5111 --name ultix-streaming --ostype l26 \
      --cores 4 --sockets 2 --cpu host,flags=+nested-virt \
      --memory 50000 --balloon 0 \
      --scsihw virtio-scsi-single \
      --net0 virtio,bridge=vmbr0,queues=4 \
      --net1 virtio,bridge=datanet,queues=2 \
      --agent 1 --onboot 1 --startup order=10,up=180 --cpuunits 9000
    # disks (each with discard=on,iothread=1,ssd=1):
    #   scsi0 NVME:438G (OS)   scsi1 ssd2:400G (/data2)   scsi2 SSD:200G (/data1)

Flags why: `cpu host`+nested-virt = microvm/Firecracker inside;
virtio-scsi-single + iothread = per-disk IO threads; queues 4/2 match
guest multiqueue (activated in live qm config 2026-09-01); cpuunits
9000 keeps the workstation above the fleet; onboot+order=10 = first VM
up after host boot (gateway auto-recovers).

## 1. Guest OS

- Debian 13 trixie amd64 netinst, KDE desktop, standard guided
  partitioning on the NVME (ext4 + swap; ends up / 279G + swap 8.8G).
- Mount the data disks per `docs/state/guest-system.txt` FSTAB section:
  `/data1` (SSD 200G) and `/data2` (ssd2 400G), ext4 `noatime`.
- Hostname `ultix-streaming`; ssh enabled.
- KDE policy (2026-08-31 ruling): autologin into an UNLOCKED
  always-alive session (sddm Relogin=true, autolock OFF) — applied by
  `fix-kde.sh` in this repo.

## 2. Accounts + secrets + tools

- Accounts: `reachableceo` (uid 1001), `reachableceo-offstage`;
  shells `/usr/bin/zsh`; groups: sudo, kvm, docker, libvirt (see
  `docs/state/guest-users.txt`).
- `~/.creds/*` restored by the human (redmine, discourse, gateway,
  open-terminal env files...). Nothing secret is in this repo.
- ssh keys: human restores (bitwarden ssh-agent pattern).
- `/usr/local/bin/chezmoi` v2.72.1 (single binary), `/usr/local/bin/
  firecracker` + `jailer` v1.16.1 (installed by 13-microvm-setup.sh).

## 3. Repo scripts, in order

| script | what / when |
|---|---|
| `1-guest-prep.sh` | guest base prep (perf pass) |
| `2-host-one-shot.sh` | PVE-side flags via ssh (run once at build) |
| `3-post-reboot-fixes.sh` | ethtool + multiqueue oneshot bits |
| `4-host-netcheck.sh` | verify host flags + multiqueue (read-only) |
| `5-activate-multiqueue.sh` | HISTORICAL: activated multiqueue 2026-09-01; a fresh VM gets queues from the qm config in step 0 + the oneshot unit |
| `6-remove-agent-stacks.sh` | HISTORICAL teardown; N/A on fresh build |
| `7-open-terminal-install.sh` + `8-open-terminal-users.sh` | Open Terminal fleet (#610) |
| `9-uat-openwebui.sh` | OpenWebUI UAT harness (uat/ compose) |
| `12-fix-rdp-bigres.sh` | xrdp EGFX budget drop-in for 5160x2160 (#611) |
| `13-microvm-setup.sh` | microvm tooling: qemu microvm engine, firecracker binary, tap pools + NAT unit, cloud assets (#615) |
| `14-capture-state.sh` | refresh docs/state after any drift |

## 4. Dotfiles (both accounts) — #612

1. gitea ssh access (human key) → clone `reachableceo/dotfiles`
   (private) to `~/projects/dotfiles`.
2. `ln -s ~/projects/dotfiles ~/.local/share/chezmoi`
3. `chezmoi apply` (fetches pinned omz/p10k/plugin externals itself).
4. offstage: root-copies the source snapshot and applies with
   `-S ~reachableceo-offstage/projects/dotfiles` (no gitea creds there)
   — see the dotfiles README.

## 5. Standing services (beyond the scripts)

- Gateway lanes via compose + `ukrrs-gateway-ensure.service` (boot-race
  fix; must be enabled on BOTH lanes before any host bounce).
- Day/night cgroup flips: `ukrrs-dayprofile.timer` 07:00 / 22:00-07:00
  night window (runtime-only cgroup flips).
- cgroup slices: `ukrrs-acct-*.slice` (per-account), `ukrrs-gateway`,
  `ukrrs-builder-prune.timer`, `ukrrs-lsp.slice` (#608).
- sysctl profile `/etc/sysctl.d/60-ukrrs-vm.conf` (measured rationale
  in REPORT.md §4.1; day/night variants raise dirty thresholds).
- xrdp (+ bigres drop-in from `12-`), `microvm-net.service`,
  `open-terminal@.service` fleet, beszel-agent.
- Docker Engine: docker.com apt repo (`docker-ce`, currently 29.x — see
  `docs/state/guest-docker-pkg.txt`); compose v2 plugin.

## 6. Package baseline

`docs/state/guest-apt-manual.txt` = `apt-mark showmanual` snapshot
(2026-09-01). On a rebuild, after all scripts:

    apt-mark showmanual | sort | diff - docs/state/guest-apt-manual.txt

Notables installed by hand over time: zsh fzf ripgrep bat eza fd-find
tmux, qemu-system-x86 qemu-utils dosfstools mtools, libvirt/clt tools,
ethtool, imageMagick, KDE extras. Full list is in the snapshot.

## 7. Deliberately NOT captured here

- Secrets (`~/.creds`), ssh keys, browser profiles.
- Docker images/volumes (repos own their stacks; gateway lanes pull
  their own images from the gateway registry/promote pipeline).
- The scratch UAT harnesses (`/tmp/rdptest`) and scratch users
  (`rdptest`, removed when #611 closes).
- PVE host itself (that is the fleet-ansible goal; `docs/state/pve-*`
  is the reference capture of today's host facts).

## 8. Post-rebuild verification checklist

1. `4-host-netcheck.sh` sweep clean (queues 4/4 + 2/2 live).
2. Gateway `/status` = normal on both lanes after a bounce.
3. RDP from iPad + 5160x2160 monitor clean (#611 UAT).
4. Fresh zsh on both accounts: Kali-style prompt + omz (#612 UAT).
5. `microvm create t && microvm start t && microvm ssh t` then rm.
6. Open Terminal ports 30000-30008 answer (tailscale-only).
7. Rerun `14-capture-state.sh`, commit drift to docs/state.