feat: full re-audit of all 7 hosts with fresh ground truth

Deployed check.sh to all 7 hosts at 21:50 CDT. Captures the live state
after the user's PDM migrations:

Cnode movements since last audit:
- cnode1: tsys1 -> tsys9
- cnode2: tsys6 -> tsys7
- cnode5: tsys6 -> tsys7, storage D5(tsys4) -> S2(tsys5)

Wnode changes:
- wnode-tsys1 (102): new VM on S2, stopped
- wnode-tsys3: RAM bumped 20 -> 28 GB
- wnode-tsys6: now running (was stopped)
- wnode-tsys9: storage moved S3 -> S2

Storage distribution improved from 90/10 to 73/27 (tsys4/tsys5).
Still need 2 more cnode moves for etcd quorum survival.

Updated executive summary, k8s distribution tables, storage
utilization, and open items with the fresh data. Captured future k8s
requirements: vcluster + Rancher, OIDC to Keycloak, workload isolation
(RackRental/Suborbital ITAR/non-ITAR/SLP), and solar-aware scale-out
with PowerEdge 19xx/2950 systems.

Added tsys9 to deploy-check.sh host list.
This commit is contained in:
2026-07-27 22:04:05 -05:00
parent 53954d8494
commit 291a7ebc27
21 changed files with 29261 additions and 21654 deletions
+14 -15
View File
@@ -51,22 +51,21 @@ Cnodes weighted to tsys1/9 (lightweight hosts, keep heavy hosts free for
workers). Wnodes: one per hypervisor host across the fleet. Production lives
on a VPS in Reston VA (Cloudron) -- this cluster is R&D only.
## Critical VM-layer findings (see PROJECT.md + K8S.md for full detail)
## Critical VM-layer findings (re-audited 2026-07-27 21:50)
1. **ALL 10 pfv-k8s nodes store disks on tsys4 NFS.** tsys4 failure = entire
k8s cluster dead. etcd quorum lost.
2. **Both -01/-02 pairs (netinfra, UCS) on tsys4 NFS only.** Both halves die
if tsys4 goes down. Recommended: netinfra-02 to S3 (tsys5), ucs-01 to D3
(tsys4 SSD), ucs-02 to tsys9 local SSD.
3. **No k8s node uses SSD or NVMe.** All on NFS-over-HDD. tsys9 has 136 GB
unused local SSD; tsys3 has 349 GB unused local **NVMe** (Samsung PM961).
4. **D3 SSD (tsys4) is 99% empty (445 GB free).** Nobody using the only SSD
NFS export. Should host ucs-01 (LDAP/AD latency benefit).
5. **S2/S3/S4 (tsys5) are all 99% empty.** Secondary storage server barely
used. VMs should be rebalanced to reduce tsys4 blast radius.
6. **tsys6/7 local-lvm is USB 2.0 portable HDD** (WD My Passport, ~30 MB/s).
Slower than NFS-HDD. Do NOT use for VM storage. Entire Proxmox OS on these
hosts boots from USB 2.0 -- user accepts this risk.
1. **4 of 5 cnodes still on tsys4 NFS** (cnode5 moved to tsys5 S2). Need 2
more moves (cnode3→S3, cnode4→S2) for etcd quorum survival. User has been
actively rebalancing via PDM -- storage distribution improved 90/10 to 73/27.
2. **Both -01/-02 pairs (netinfra, UCS) on tsys4 NFS.** TODO today: move
netinfra-02 to S3, ucs-02 to S2 (both tsys5 HDD).
3. **No k8s node uses SSD/NVMe yet.** tsys3 has 349 GB unused local NVMe
(Samsung PM961), tsys9 has 136 GB local SSD. Deferred to k8s session.
4. **tsys6/7 local-lvm is USB 2.0 portable HDD** (~30 MB/s). Slower than NFS.
Do NOT use for VM storage. User accepts OS-on-USB for these hosts.
5. **Storage philosophy:** NVMe/SSD = k8s scratch + ultix-streaming (dev
workstation). Spinning rust = all infrastructure VMs.
6. **Hosts are standalone, managed via PDM** (Proxmox Datacenter Manager).
VM migration between nodes is done through PDM UI, not manual disk copy.
## Storage network IPs (VLAN1000, 10.100.100.0/24)