feat: full re-audit of all 7 hosts with fresh ground truth
Deployed check.sh to all 7 hosts at 21:50 CDT. Captures the live state after the user's PDM migrations: Cnode movements since last audit: - cnode1: tsys1 -> tsys9 - cnode2: tsys6 -> tsys7 - cnode5: tsys6 -> tsys7, storage D5(tsys4) -> S2(tsys5) Wnode changes: - wnode-tsys1 (102): new VM on S2, stopped - wnode-tsys3: RAM bumped 20 -> 28 GB - wnode-tsys6: now running (was stopped) - wnode-tsys9: storage moved S3 -> S2 Storage distribution improved from 90/10 to 73/27 (tsys4/tsys5). Still need 2 more cnode moves for etcd quorum survival. Updated executive summary, k8s distribution tables, storage utilization, and open items with the fresh data. Captured future k8s requirements: vcluster + Rancher, OIDC to Keycloak, workload isolation (RackRental/Suborbital ITAR/non-ITAR/SLP), and solar-aware scale-out with PowerEdge 19xx/2950 systems. Added tsys9 to deploy-check.sh host list.
This commit is contained in:
@@ -51,22 +51,21 @@ Cnodes weighted to tsys1/9 (lightweight hosts, keep heavy hosts free for
|
||||
workers). Wnodes: one per hypervisor host across the fleet. Production lives
|
||||
on a VPS in Reston VA (Cloudron) -- this cluster is R&D only.
|
||||
|
||||
## Critical VM-layer findings (see PROJECT.md + K8S.md for full detail)
|
||||
## Critical VM-layer findings (re-audited 2026-07-27 21:50)
|
||||
|
||||
1. **ALL 10 pfv-k8s nodes store disks on tsys4 NFS.** tsys4 failure = entire
|
||||
k8s cluster dead. etcd quorum lost.
|
||||
2. **Both -01/-02 pairs (netinfra, UCS) on tsys4 NFS only.** Both halves die
|
||||
if tsys4 goes down. Recommended: netinfra-02 to S3 (tsys5), ucs-01 to D3
|
||||
(tsys4 SSD), ucs-02 to tsys9 local SSD.
|
||||
3. **No k8s node uses SSD or NVMe.** All on NFS-over-HDD. tsys9 has 136 GB
|
||||
unused local SSD; tsys3 has 349 GB unused local **NVMe** (Samsung PM961).
|
||||
4. **D3 SSD (tsys4) is 99% empty (445 GB free).** Nobody using the only SSD
|
||||
NFS export. Should host ucs-01 (LDAP/AD latency benefit).
|
||||
5. **S2/S3/S4 (tsys5) are all 99% empty.** Secondary storage server barely
|
||||
used. VMs should be rebalanced to reduce tsys4 blast radius.
|
||||
6. **tsys6/7 local-lvm is USB 2.0 portable HDD** (WD My Passport, ~30 MB/s).
|
||||
Slower than NFS-HDD. Do NOT use for VM storage. Entire Proxmox OS on these
|
||||
hosts boots from USB 2.0 -- user accepts this risk.
|
||||
1. **4 of 5 cnodes still on tsys4 NFS** (cnode5 moved to tsys5 S2). Need 2
|
||||
more moves (cnode3→S3, cnode4→S2) for etcd quorum survival. User has been
|
||||
actively rebalancing via PDM -- storage distribution improved 90/10 to 73/27.
|
||||
2. **Both -01/-02 pairs (netinfra, UCS) on tsys4 NFS.** TODO today: move
|
||||
netinfra-02 to S3, ucs-02 to S2 (both tsys5 HDD).
|
||||
3. **No k8s node uses SSD/NVMe yet.** tsys3 has 349 GB unused local NVMe
|
||||
(Samsung PM961), tsys9 has 136 GB local SSD. Deferred to k8s session.
|
||||
4. **tsys6/7 local-lvm is USB 2.0 portable HDD** (~30 MB/s). Slower than NFS.
|
||||
Do NOT use for VM storage. User accepts OS-on-USB for these hosts.
|
||||
5. **Storage philosophy:** NVMe/SSD = k8s scratch + ultix-streaming (dev
|
||||
workstation). Spinning rust = all infrastructure VMs.
|
||||
6. **Hosts are standalone, managed via PDM** (Proxmox Datacenter Manager).
|
||||
VM migration between nodes is done through PDM UI, not manual disk copy.
|
||||
|
||||
## Storage network IPs (VLAN1000, 10.100.100.0/24)
|
||||
|
||||
|
||||
Reference in New Issue
Block a user