mrcharles
f815149472
Run outputs moved to logs/ (incl. sda partition-table backup). New from the 15:57-16:03 parallel session: 16-grow-root-online.sh (RAN: / = 431G online, swap removed by design — GROW-ROOT-RUNBOOK superseded) and 15-docker-to-data2.sh (staged, not run). README/NEXT/RUNBOOK/REBUILD reflect current reality. Moot rdp-testprep timer removed from the VM. Details: https://projects.knownelement.com/issues/615
142 lines
6.9 KiB
Markdown
142 lines
6.9 KiB
Markdown
# REBUILD.md — ultix-streaming (VM 5111) full-rebuild runbook (#615)
|
|
|
|
How to rebuild this workstation from bare metal up. The mechanism is
|
|
this repo's numbered idempotent scripts, run in order; this file is the
|
|
map + the parts scripts don't cover. Fleet-wide ansible will consume
|
|
this later (human goal, 2026-09-01); until then: shell-first.
|
|
|
|
- Live state snapshots: `docs/state/` — regenerate any time with
|
|
`./14-capture-state.sh` (read-only; guest + PVE).
|
|
- Secrets live ONLY in `~/.creds` (human restores from backup); never
|
|
in this repo, images, or compose files.
|
|
- Docker/registry content is repo-owned (repos rebuild their own
|
|
containers) — out of scope here.
|
|
- PVE-side DR path for total loss: existing vzdump/backup policy, not
|
|
this runbook (this runbook is for a clean, deliberate rebuild).
|
|
|
|
## 0. PVE side (host pfv-tsys5)
|
|
|
|
Prereqs on the host: nested KVM enabled
|
|
(`/sys/module/kvm_intel/parameters/nested` = Y), dir storages `NVME`,
|
|
`ssd2`, `SSD`, bridges `vmbr0` (LAN) + `datanet`.
|
|
|
|
VM definition (live copy: `docs/state/pve-5111.txt`):
|
|
|
|
qm create 5111 --name ultix-streaming --ostype l26 \
|
|
--cores 4 --sockets 2 --cpu host,flags=+nested-virt \
|
|
--memory 50000 --balloon 0 \
|
|
--scsihw virtio-scsi-single \
|
|
--net0 virtio,bridge=vmbr0,queues=4 \
|
|
--net1 virtio,bridge=datanet,queues=2 \
|
|
--agent 1 --onboot 1 --startup order=10,up=180 --cpuunits 9000
|
|
# disks (each with discard=on,iothread=1,ssd=1):
|
|
# scsi0 NVME:438G (OS) scsi1 ssd2:400G (/data2) scsi2 SSD:200G (/data1)
|
|
|
|
Flags why: `cpu host`+nested-virt = microvm/Firecracker inside;
|
|
virtio-scsi-single + iothread = per-disk IO threads; queues 4/2 match
|
|
guest multiqueue (activated in live qm config 2026-09-01); cpuunits
|
|
9000 keeps the workstation above the fleet; onboot+order=10 = first VM
|
|
up after host boot (gateway auto-recovers).
|
|
|
|
## 1. Guest OS
|
|
|
|
- Debian 13 trixie amd64 netinst, KDE desktop, standard guided
|
|
partitioning on the NVME (ext4 + swap; ends up / 279G + swap 8.8G).
|
|
- Mount the data disks per `docs/state/guest-system.txt` FSTAB section:
|
|
`/data1` (SSD 200G) and `/data2` (ssd2 400G), ext4 `noatime`.
|
|
- Hostname `ultix-streaming`; ssh enabled.
|
|
- KDE policy (2026-09-01 ruling, supersedes 08-31): **NO sddm autologin**
|
|
— an always-alive console session holds the user's single plasmashell
|
|
and black-screens RDP (Plasma 6 = one shell per user, #611). Console
|
|
shows the sddm greeter at boot; RDP is the primary desktop. When
|
|
Sunshine lands with the GPU it gets a dedicated kiosk account or XFCE.
|
|
|
|
## 2. Accounts + secrets + tools
|
|
|
|
- Accounts: `reachableceo` (uid 1001), `reachableceo-offstage`;
|
|
shells `/usr/bin/zsh`; groups: sudo, kvm, docker, libvirt (see
|
|
`docs/state/guest-users.txt`).
|
|
- `~/.creds/*` restored by the human (redmine, discourse, gateway,
|
|
open-terminal env files...). Nothing secret is in this repo.
|
|
- ssh keys: human restores (bitwarden ssh-agent pattern).
|
|
- `/usr/local/bin/chezmoi` v2.72.1 (single binary), `/usr/local/bin/
|
|
firecracker` + `jailer` v1.16.1 (installed by 13-microvm-setup.sh).
|
|
|
|
## 3. Repo scripts, in order
|
|
|
|
| script | what / when |
|
|
|---|---|
|
|
| `1-guest-prep.sh` | guest base prep (perf pass) |
|
|
| `2-host-one-shot.sh` | PVE-side flags via ssh (run once at build) |
|
|
| `3-post-reboot-fixes.sh` | ethtool + multiqueue oneshot bits |
|
|
| `4-host-netcheck.sh` | verify host flags + multiqueue (read-only) |
|
|
| `5-activate-multiqueue.sh` | HISTORICAL: activated multiqueue 2026-09-01; a fresh VM gets queues from the qm config in step 0 + the oneshot unit |
|
|
| `6-remove-agent-stacks.sh` | HISTORICAL teardown; N/A on fresh build |
|
|
| `7-open-terminal-install.sh` + `8-open-terminal-users.sh` | Open Terminal fleet (#610) |
|
|
| `9-uat-openwebui.sh` | OpenWebUI UAT harness (uat/ compose) |
|
|
| `12-fix-rdp-bigres.sh` | xrdp EGFX budget drop-in for 5160x2160 (#611) |
|
|
| `13-microvm-setup.sh` | microvm tooling: qemu microvm engine, firecracker binary, tap pools + NAT unit, cloud assets (#615) |
|
|
| `14-capture-state.sh` | refresh docs/state after any drift |
|
|
| `15-docker-to-data2.sh` | STAGED (not run): moves docker data-root to /data2 (drain-aware, reversible) |
|
|
| `16-grow-root-online.sh` | RAN 2026-09-01: / grown online to 431G, swap removed by design (#603) |
|
|
| `15-docker-to-data2.sh` | STAGED (not run): moves docker data-root
|
|
to /data2 (drain-aware, reversible) |
|
|
| `16-grow-root-online.sh` | RAN 2026-09-01: / grown online to
|
|
431G, swap removed by design (#603) |
|
|
|
|
## 4. Dotfiles (both accounts) — #612
|
|
|
|
1. gitea ssh access (human key) → clone `reachableceo/dotfiles`
|
|
(private) to `~/projects/dotfiles`.
|
|
2. `ln -s ~/projects/dotfiles ~/.local/share/chezmoi`
|
|
3. `chezmoi apply` (fetches pinned omz/p10k/plugin externals itself).
|
|
4. offstage: root-copies the source snapshot and applies with
|
|
`-S ~reachableceo-offstage/projects/dotfiles` (no gitea creds there)
|
|
— see the dotfiles README.
|
|
|
|
## 5. Standing services (beyond the scripts)
|
|
|
|
- Gateway lanes via compose + `ukrrs-gateway-ensure.service` (boot-race
|
|
fix; must be enabled on BOTH lanes before any host bounce).
|
|
- Day/night cgroup flips: `ukrrs-dayprofile.timer` 07:00 / 22:00-07:00
|
|
night window (runtime-only cgroup flips).
|
|
- cgroup slices: `ukrrs-acct-*.slice` (per-account), `ukrrs-gateway`,
|
|
`ukrrs-builder-prune.timer`, `ukrrs-lsp.slice` (#608).
|
|
- sysctl profile `/etc/sysctl.d/60-ukrrs-vm.conf` (measured rationale
|
|
in REPORT.md §4.1; day/night variants raise dirty thresholds).
|
|
- xrdp (+ bigres drop-in from `12-`), `microvm-net.service`,
|
|
`open-terminal@.service` fleet, beszel-agent.
|
|
- Docker Engine: docker.com apt repo (`docker-ce`, currently 29.x — see
|
|
`docs/state/guest-docker-pkg.txt`); compose v2 plugin.
|
|
|
|
## 6. Package baseline
|
|
|
|
`docs/state/guest-apt-manual.txt` = `apt-mark showmanual` snapshot
|
|
(2026-09-01). On a rebuild, after all scripts:
|
|
|
|
apt-mark showmanual | sort | diff - docs/state/guest-apt-manual.txt
|
|
|
|
Notables installed by hand over time: zsh fzf ripgrep bat eza fd-find
|
|
tmux, qemu-system-x86 qemu-utils dosfstools mtools, libvirt/clt tools,
|
|
ethtool, imageMagick, KDE extras. Full list is in the snapshot.
|
|
|
|
## 7. Deliberately NOT captured here
|
|
|
|
- Secrets (`~/.creds`), ssh keys, browser profiles.
|
|
- Docker images/volumes (repos own their stacks; gateway lanes pull
|
|
their own images from the gateway registry/promote pipeline).
|
|
- The scratch UAT harnesses (`/tmp/rdptest`) and scratch users
|
|
(`rdptest`, removed when #611 closes).
|
|
- PVE host itself (that is the fleet-ansible goal; `docs/state/pve-*`
|
|
is the reference capture of today's host facts).
|
|
|
|
## 8. Post-rebuild verification checklist
|
|
|
|
1. `4-host-netcheck.sh` sweep clean (queues 4/4 + 2/2 live).
|
|
2. Gateway `/status` = normal on both lanes after a bounce.
|
|
3. RDP from iPad + 5160x2160 monitor clean (#611 UAT).
|
|
4. Fresh zsh on both accounts: Kali-style prompt + omz (#612 UAT).
|
|
5. `microvm create t && microvm start t && microvm ssh t` then rm.
|
|
6. Open Terminal ports 30000-30008 answer (tailscale-only).
|
|
7. Rerun `14-capture-state.sh`, commit drift to docs/state.
|