KNEL/PFVCluster
5
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
575d79f6112d3a3311c9de032d1f3986cce78bf5
PFVCluster/STATUS.md
T
mrcharles 575d79f611 docs(audit): RAM/CPU placement audit + codify capacity goals 
Add CAPACITY-GOALS.md (80% RAM / ~50% idle CPU targets + workload placement
model) and AUDIT-2026-07-29.md (live end-to-end analysis of all 7 hosts). Key
findings: memory goal met fleet-wide (highest 72%); CPU idles 0-6% (chase via
workloads, not hardware); tsys5 is single-CPU not dual; cnode4/5 are stale
running VMs. Answers the wnode-sizing question (grow tsys9, not tsys1), confirms
tsys6/7 for RackRental, and gives a DDR3 compatibility decision tree (RDIMM-only
servers reject consumer UDIMM). Refreshed STATUS/docmap and the returned audit
logs, and fixed an empty-array abort in deploy-check.sh.

💘 Generated with Crush

Assisted-by: Crush:glm-5.2
2026-07-29 05:32:12 -05:00

7.6 KiB
Raw Blame History

Project Status

Human read-only. Agents maintain this file automatically after each work session. Do not edit by hand — the next agent run will overwrite it.

Last updated: 2026-07-29 by Crush (GLM-5.2)

Capacity targets are now codified in proxmox/docs/CAPACITY-GOALS.md: 80% RAM steady-state, ~50% idle CPU. The current placement analysis lives in proxmox/docs/AUDIT-2026-07-29.md.

Current State: STABLE

The merged PFVCluster repo is fully operational across provisioning, DNS infrastructure, Proxmox cluster ops, and k8s control plane.

Completed Work

Server Provisioning (validated on sectestbed-sandbox)

  • SetupNewSystem.sh deploys end-to-end to rc=0 (Debian 13 trixie)
  • Security hardening: SSH, SCAP-STIG, Wazuh, 2FA (SSH+Cockpit+Webmin)
  • NTP: redundant pfv-netinfra-01/02 (192.168.3.252/253), synced stratum 3
  • DNS resolv.conf: managed static file pointing at netinfra pair
  • Test suite: 5 tests (framework, safe-download, 2fa, https, system-req)
  • VM validation harness: git-based deploy + auto-rollback + guest-agent access

DNS Cluster (pfv-netinfra-01/02)

  • Production Technitium config replicated from tailscale-router (read-only)
  • 124 zones on both nodes (knel.net + reverse DNS)
  • pfv-netinfra-01 = PRIMARY, pfv-netinfra-02 = SECONDARY
  • Zone replication via rsync systemd timer (every 60s)
  • Credentials + 2FA replicated identically to production
  • Both LAN IPs resolve knel.net device names + recurse externally

Proxmox Cluster Ops

  • 5 of 7 hosts fully performance-tuned (tsys1/3/6/7/9)
  • Fleet audit refreshed 2026-07-29 (live data, all 7 hosts)
  • Memory goal (≤80% actual) met fleet-wide — highest host is tsys1 at 72%
  • Capacity goals codified (80% RAM / ~50% idle CPU + workload model)
  • CPU ~50%-idle goal: fleet idles 0-6% (under-utilized; chase via workloads, not hardware)
  • tsys4: blocked on PCIe NIC + RAM install
  • tsys5: blocked on 2nd ethernet cable + NVMe install; also single-CPU (CPU2 socket empty)
  • tsys2: pending rebuild from Win10 to Proxmox

Kubernetes Control Plane (k3s HA — LIVE)

  • 3-node k3s HA control plane deployed: cnode1/2/3 (v1.36.2+k3s1, embedded etcd)
  • All traffic over Tailscale IPs — no LAN IPs in node status or certs
  • All 3 cnodes tainted control-plane:NoSchedule (zero user workloads)
  • 13/13 health checks pass (verify.sh): nodes Ready, etcd quorum, Tailscale IPs, CoreDNS, API server, workload isolation
  • Scripts in k8s/: wipe, install-cp, join-servers, post-setup, verify
  • Kubeconfig saved to ~/.kube/config.pfv-k8s (gitignored, embedded certs)
  • Workers (wnodes) not yet joined to this cluster
  • Distro decision: k3s chosen for regular R&D cluster. Talos docs preserved in k8s/docs/ for future ITAR/classified cluster.

PDU Management (powerman on pfv-tsys1 — LIVE)

  • Cyclades AlterPath PM10i (10 outlets) managed via powerman over serial
  • USB-DB9 adapter (Prolific pl2303) with stable udev symlink /dev/cyclades-pm10
  • powermand listening on 127.0.0.1:10101 + Tailscale 100.121.189.98:10101
  • All 10 outlets defined as outlet-1 through outlet-10
  • Validated: outlet 10 cycled off → on (8/8 test checks passed)
  • Rename outlets to match physical devices (Friday onsite)

Console Management (ser2net + conman on pfv-tsys4 — LIVE)

  • 7 network switch/router consoles managed via ser2net + conman
  • USB enumeration problem SOLVED: udev rules pin each adapter by ID_PATH (physical USB port topology) to stable /dev/consoles/<name> symlinks that survive reboot regardless of enumeration order
  • ser2net exposes all 7 consoles on TCP ports (2001-2007) bound to Tailscale IP only (100.70.77.93)
  • conman connects to TCP ports for logging + multiplexing (7 log files active in /var/log/conman/)
  • Both ser2net + conmand enabled via systemd (survive reboot)
  • conmand systemd unit created (Debian package doesn't ship one)
  • Old /root/conmap + manual screen workflow replaced

Repo Merge

  • KNELServerBuild merged into PFVCluster (history preserved)
  • Directory structure reorganized (provisioning/, tests/, perf/, docs/)
  • All docs gardened: links fixed, stale refs removed, tailscale.md updated

Known Issues

Issue Impact Status
2 of 3 k3s cnodes on tsys4 tsys4 failure = etcd quorum lost (1 of 3) Accepted — fundamental constraint with 3 cnodes + 2 NFS servers. Revisit after Friday (local-SSD boot for one cnode). Deferred storage concern.
tsys1/5/9 RAM-overcommitted (alloc) 115%/134%/129% allocation; actual usage 72%/37%/60% — all under 80% goal Migrate NFS-backed VMs to tsys6/7 (117/171 GB free actual). See AUDIT-2026-07-29 §8
tsys5 is single-CPU CPU2 socket unpopulated; half the documented CPU capacity Consider seating a spare LGA771 Xeon (also activates CPU2 DIMM branch)
cnode4/5 stale VMs running Not joined to the 3-node cluster; waste 8 GB + 8 vCPU Stop + delete (AUDIT-2026-07-29 §10)
Fleet CPU 0-6% idle Misses ~50%-idle utilization goal Consolidate baseline; solar-gate idle PowerEdge; grow via workloads
Technitium AXFR uses port 53 (occupied by Pi-hole) Zone transfer via rsync instead of native AXFR Workaround in place
tsys4/5 hardware pending Perf tuning incomplete on 2 hosts Friday physical install

Pending (next session priorities)

RAM/CPU placement actions are itemized in proxmox/docs/AUDIT-2026-07-29.md §12. Storage/disk actions are deferred to post-Friday (see below).

  1. VM migrations (RAM/CPU relief, zero hardware):
    • tsys1 → tsys7: librenms(104), pdm(105), workbench(117), tailscale-router(109), kali-tsys(114)
    • tsys9 → tsys6: tsys-siem(901)
  2. Grow wnode-tsys9 to 6 vCPU / 16 GB (after siem move) — cross-compile node
  3. Stop + delete stale cnode4 (601) / cnode5 (706) — not in the 3-node cluster
  4. Join wnodes to the k3s cluster (after migrations free resources)
  5. Friday maintenance: physical hardware + storage work (see proxmox/docs/AUDIT-2026-07-28.md §5 and proxmox/docs/TODO.md)
  6. Wnode storage rebalancing (disk — deferred): wnode-tsys6 D5→S4, wnode-tsys7 D5→S1
  7. PDU: Rename outlets in powerman.conf to match physical devices (Friday)
  8. k8s deferred topics: ETL tooling, HPC scheduler, vcluster policy, solar-aware scale-out

See proxmox/docs/AUDIT-2026-07-28.md for the full fresh audit with VM inventory, capacity analysis, and Friday migration plan.

Infrastructure Summary

Component Details
Proxmox hosts 7 standalone PVE, managed via PDM
DNS primary pfv-netinfra-01 (192.168.3.252) — Technitium + Pi-hole
DNS secondary pfv-netinfra-02 (192.168.3.253) — Technitium + Pi-hole
DNS production tailscale-router (read-only source of truth)
NTP pfv-netinfra-01/02 (redundant, LAN IPs, stratum 2/3)
Sandbox VM sectestbed-sandbox (VMID 6000 on pfv-tsys5)
Backup Proxmox Backup Server (PBS)
k8s control plane 3-node k3s HA (cnode1/2/3), all traffic over Tailscale
PDU Cyclades PM10i via powerman on pfv-tsys1 (port 10101)
Console 7 switch consoles via ser2net+conman on pfv-tsys4 (TCP 2001-2007 on Tailscale)
Production Cloudron VPS, Reston VA (this cluster is R&D only)
Reference in New Issue View Git Blame Copy Permalink