Files

T

mrcharles edb34a75ec docs(status): sync action items with audit, fix stale disk refs

Rewrite STATUS Pending section to match the audit's phased action plan (§11)
exactly. Remove stale "wnode-tsys6 D5->S4, wnode-tsys7 D5->S1" line (wnode6
already on S4, wnode7 on S3, S1 is not a target). Fix section reference (§12 ->
§11). Correct actual-usage percentages (73%/37%/62%) and CPU idle range (0-5%).
Add D2 IOPS hotspot as a known issue. Fix sectestbed suite count in audit (x9 ->
x10). Verified all VM counts and host data against fresh ground truth.

💘 Generated with Crush

Assisted-by: Crush:glm-5.2

2026-07-29 16:29:17 -05:00

8.0 KiB

Raw Blame History

Project Status

Human read-only. Agents maintain this file automatically after each work session. Do not edit by hand — the next agent run will overwrite it.

Last updated: 2026-07-29 by Crush (GLM-5.2)

Capacity targets are now codified in proxmox/docs/CAPACITY-GOALS.md: 80% RAM steady-state, ~50% idle CPU. The current placement analysis lives in proxmox/docs/AUDIT-2026-07-29.md.

Current State: STABLE

The merged PFVCluster repo is fully operational across provisioning, DNS infrastructure, Proxmox cluster ops, and k8s control plane.

Completed Work

Server Provisioning (validated on sectestbed-sandbox)

SetupNewSystem.sh deploys end-to-end to rc=0 (Debian 13 trixie)
Security hardening: SSH, SCAP-STIG, Wazuh, 2FA (SSH+Cockpit+Webmin)
NTP: redundant pfv-netinfra-01/02 (192.168.3.252/253), synced stratum 3
DNS resolv.conf: managed static file pointing at netinfra pair
Test suite: 5 tests (framework, safe-download, 2fa, https, system-req)
VM validation harness: git-based deploy + auto-rollback + guest-agent access

DNS Cluster (pfv-netinfra-01/02)

Production Technitium config replicated from tailscale-router (read-only)
124 zones on both nodes (knel.net + reverse DNS)
pfv-netinfra-01 = PRIMARY, pfv-netinfra-02 = SECONDARY
Zone replication via rsync systemd timer (every 60s)
Credentials + 2FA replicated identically to production
Both LAN IPs resolve knel.net device names + recurse externally

Proxmox Cluster Ops

5 of 7 hosts fully performance-tuned (tsys1/3/6/7/9)
Fleet audit refreshed 2026-07-29 (consolidated, storage-corrected)
Memory goal (≤80% actual) met fleet-wide — highest host is tsys1 at 73%
Capacity goals codified (80% RAM / ~50% idle CPU + workload model)
Storage topology corrected: local-nonprod = S1 disk (not separate); per-host storage.cfg
cnode5 (706) deleted; cnode4 (601) stopped (delete pending)
tsys6 S1 typo + tsys9 missing S1 — both FIXED by operator (verified: 8/8 NFS stores active on both)
CPU ~50%-idle goal: fleet idles 0-5% (under-utilized; chase via workloads, not hardware)
tsys4: blocked on PCIe NIC + RAM install
tsys5: blocked on 2nd ethernet cable + NVMe install; also single-CPU (CPU2 socket empty)
tsys2: pending rebuild from Win10 to Proxmox

Kubernetes Control Plane (k3s HA — LIVE)

3-node k3s HA control plane deployed: cnode1/2/3 (v1.36.2+k3s1, embedded etcd)
All traffic over Tailscale IPs — no LAN IPs in node status or certs
All 3 cnodes tainted control-plane:NoSchedule (zero user workloads)
13/13 health checks pass (verify.sh): nodes Ready, etcd quorum, Tailscale IPs, CoreDNS, API server, workload isolation
Scripts in k8s/: wipe, install-cp, join-servers, post-setup, verify
Kubeconfig saved to ~/.kube/config.pfv-k8s (gitignored, embedded certs)
Workers (wnodes) not yet joined to this cluster
Distro decision: k3s chosen for regular R&D cluster. Talos docs preserved in k8s/docs/ for future ITAR/classified cluster.

PDU Management (powerman on pfv-tsys1 — LIVE)

Cyclades AlterPath PM10i (10 outlets) managed via powerman over serial
USB-DB9 adapter (Prolific pl2303) with stable udev symlink /dev/cyclades-pm10
powermand listening on 127.0.0.1:10101 + Tailscale 100.121.189.98:10101
All 10 outlets defined as outlet-1 through outlet-10
Validated: outlet 10 cycled off → on (8/8 test checks passed)
Rename outlets to match physical devices (Friday onsite)

Console Management (ser2net + conman on pfv-tsys4 — LIVE)

7 network switch/router consoles managed via ser2net + conman
USB enumeration problem SOLVED: udev rules pin each adapter by ID_PATH (physical USB port topology) to stable /dev/consoles/<name> symlinks that survive reboot regardless of enumeration order
ser2net exposes all 7 consoles on TCP ports (2001-2007) bound to Tailscale IP only (100.70.77.93)
conman connects to TCP ports for logging + multiplexing (7 log files active in /var/log/conman/)
Both ser2net + conmand enabled via systemd (survive reboot)
conmand systemd unit created (Debian package doesn't ship one)
Old /root/conmap + manual screen workflow replaced

Repo Merge

KNELServerBuild merged into PFVCluster (history preserved)
Directory structure reorganized (provisioning/, tests/, perf/, docs/)
All docs gardened: links fixed, stale refs removed, tailscale.md updated

Known Issues

Issue	Impact	Status
2 of 3 k3s cnodes on tsys4	tsys4 failure = etcd quorum lost (1 of 3)	Accepted — fundamental constraint with 3 cnodes + 2 NFS servers. Revisit after Friday (local-SSD boot for one cnode). Deferred storage concern.
tsys1/5/9 RAM-overcommitted (alloc)	115%/134%/129% allocation; actual usage 73%/37%/62% — all under 80% goal	Migrate NFS-backed VMs to tsys6/7 (117/171 GB free actual). See AUDIT-2026-07-29 §11
tsys5 is single-CPU	CPU2 socket unpopulated; half the documented CPU capacity	Consider seating a spare LGA771 Xeon (also activates CPU2 DIMM branch)
cnode4/5 stale VMs	cnode4 (601) stopped; cnode5 (706) deleted	cnode4 delete pending; cnode5 done ✓
Fleet CPU 0-5% idle	Misses ~50%-idle utilization goal	Consolidate baseline; solar-gate idle PowerEdge; grow via workloads
D2 IOPS hotspot	17 of 26 NFS-backed VMs on one spindle; S1/S4 underused	Phase 2 disk migrations: D2→S2/S3/S4/D5 (see audit §11)
Technitium AXFR uses port 53 (occupied by Pi-hole)	Zone transfer via rsync instead of native AXFR	Workaround in place
tsys4/5 hardware pending	Perf tuning incomplete on 2 hosts	Friday physical install

Pending (next session priorities)

Full phased action checklist with current/target inventory is in proxmox/docs/AUDIT-2026-07-29.md §11. Summary below. SSD/NVMe tiering is deferred to post-Friday.

Phase 0 — Storage config fixes — DONE ✓ (tsys6 S1 typo, tsys9 S1 missing)

Phase 1 — Cleanup:

Delete cnode4 (601) on tsys6 (stopped, stale)
cnode5 (706) already deleted ✓

Phase 2 — Disk migrations off D2 (IOPS rebalancing, 6 moves):

wnode-tsys5 (509): D2→S4 · tsys-awx (600): D2→D5 · hfnoc-uisp (702): D2→S4
rr-middleware (703): D2→S4 · TCTC (704): D2→S3 · cnode2 (705): D2→S2

Phase 3 — Host + disk migrations (RAM/CPU relief, 6 VMs):

tsys1→tsys7: librenms(104), pdm(105), tailscale-router(109), kali-tsys(114), workbench(117)
tsys9→tsys6: tsys-siem(901)

Phase 4 — Grow wnode-tsys9: 4c/4GB → 6c/8GB (after siem leaves tsys9)

Phase 5 — Join wnodes to k3s (after migrations free resources)

Other:

Friday maintenance: tsys4 (PCIe NIC + RAM), tsys5 (2nd CPU candidate, NVMe, 2nd ethernet)
PDU: Rename outlets in powerman.conf (Friday onsite)
k8s deferred: ETL tooling, HPC scheduler, vcluster policy, solar-aware scale-out

Infrastructure Summary

Component	Details
Proxmox hosts	7 standalone PVE, managed via PDM
DNS primary	pfv-netinfra-01 (192.168.3.252) — Technitium + Pi-hole
DNS secondary	pfv-netinfra-02 (192.168.3.253) — Technitium + Pi-hole
DNS production	tailscale-router (read-only source of truth)
NTP	pfv-netinfra-01/02 (redundant, LAN IPs, stratum 2/3)
Sandbox VM	sectestbed-sandbox (VMID 6000 on pfv-tsys5)
Backup	Proxmox Backup Server (PBS)
k8s control plane	3-node k3s HA (cnode1/2/3), all traffic over Tailscale
PDU	Cyclades PM10i via powerman on pfv-tsys1 (port 10101)
Console	7 switch consoles via ser2net+conman on pfv-tsys4 (TCP 2001-2007 on Tailscale)
Production	Cloudron VPS, Reston VA (this cluster is R&D only)

8.0 KiB Raw Blame History