mrcharles
a980a4df2f
Change powermand listen address from 0.0.0.0:10101 (all interfaces) to
127.0.0.1:10101 (local admin) + Tailscale IP:10101 (remote access). The
setup.sh now auto-detects the Tailscale IP at deploy time.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
6.0 KiB
6.0 KiB
Project Status
Human read-only. Agents maintain this file automatically after each work session. Do not edit by hand — the next agent run will overwrite it.
Last updated: 2026-07-28 by Crush (GLM-5.2)
Current State: STABLE
The merged PFVCluster repo is fully operational across provisioning, DNS infrastructure, Proxmox cluster ops, and k8s control plane.
Completed Work
Server Provisioning (validated on sectestbed-sandbox)
- SetupNewSystem.sh deploys end-to-end to rc=0 (Debian 13 trixie)
- Security hardening: SSH, SCAP-STIG, Wazuh, 2FA (SSH+Cockpit+Webmin)
- NTP: redundant pfv-netinfra-01/02 (192.168.3.252/253), synced stratum 3
- DNS resolv.conf: managed static file pointing at netinfra pair
- Test suite: 5 tests (framework, safe-download, 2fa, https, system-req)
- VM validation harness: git-based deploy + auto-rollback + guest-agent access
DNS Cluster (pfv-netinfra-01/02)
- Production Technitium config replicated from tailscale-router (read-only)
- 124 zones on both nodes (knel.net + reverse DNS)
- pfv-netinfra-01 = PRIMARY, pfv-netinfra-02 = SECONDARY
- Zone replication via rsync systemd timer (every 60s)
- Credentials + 2FA replicated identically to production
- Both LAN IPs resolve knel.net device names + recurse externally
Proxmox Cluster Ops
- 5 of 7 hosts fully performance-tuned (tsys1/3/6/7/9)
- Fleet audit complete (PROJECT.md has ground truth)
- tsys4: blocked on PCIe NIC + RAM install
- tsys5: blocked on 2nd ethernet cable + NVMe install
- tsys2: pending rebuild from Win10 to Proxmox
Kubernetes Control Plane (k3s HA — LIVE)
- 3-node k3s HA control plane deployed: cnode1/2/3 (v1.36.2+k3s1, embedded etcd)
- All traffic over Tailscale IPs — no LAN IPs in node status or certs
- All 3 cnodes tainted
control-plane:NoSchedule(zero user workloads) - 13/13 health checks pass (verify.sh): nodes Ready, etcd quorum, Tailscale IPs, CoreDNS, API server, workload isolation
- Scripts in
k8s/: wipe, install-cp, join-servers, post-setup, verify - Kubeconfig saved to
~/.kube/config.pfv-k8s(gitignored, embedded certs) - Workers (wnodes) not yet joined to this cluster
- Distro decision: k3s chosen for regular R&D cluster. Talos docs
preserved in
docs/k8s/for future ITAR/classified cluster.
PDU Management (powerman on pfv-tsys1 — LIVE)
- Cyclades AlterPath PM10i (10 outlets) managed via powerman over serial
- USB-DB9 adapter (Prolific pl2303) with stable udev symlink
/dev/cyclades-pm10 - powermand listening on
127.0.0.1:10101+ Tailscale100.121.189.98:10101 - All 10 outlets defined as
outlet-1throughoutlet-10 - Validated: outlet 10 cycled off → on (8/8 test checks passed)
- Rename outlets to match physical devices (Friday onsite)
Console Management (ser2net + conman on pfv-tsys4 — LIVE)
- 7 network switch/router consoles managed via ser2net + conman
- USB enumeration problem SOLVED: udev rules pin each adapter by
ID_PATH (physical USB port topology) to stable
/dev/consoles/<name>symlinks that survive reboot regardless of enumeration order - ser2net exposes all 7 consoles on TCP ports (2001-2007) bound to
Tailscale IP only (
100.70.77.93) - conman connects to TCP ports for logging + multiplexing (7 log
files active in
/var/log/conman/) - Both ser2net + conmand enabled via systemd (survive reboot)
- conmand systemd unit created (Debian package doesn't ship one)
- Old
/root/conmap+ manualscreenworkflow replaced
Repo Merge
- KNELServerBuild merged into PFVCluster (history preserved)
- Directory structure reorganized (provisioning/, tests/, perf/, docs/)
- All docs gardened: links fixed, stale refs removed, tailscale.md updated
Known Issues
| Issue | Impact | Status |
|---|---|---|
| Technitium AXFR uses port 53 (occupied by Pi-hole) | Zone transfer via rsync instead of native AXFR | Workaround in place |
download.proxmox.com unreachable from sandbox VM |
2 validation tests warn (environmental) | Not a code issue |
| tsys4/5 hardware pending | Perf tuning incomplete on 2 hosts | Waiting on physical install |
Pending (next session priorities)
- PDU: rename outlets in
/etc/powerman/powerman.confto match physical devices (Friday onsite) - k8s workers: Join wnodes to the k3s cluster (agents, not servers)
- k8s: confirm 3 decisions for future Talos/ITAR cluster:
- Cnode count: 3 (recommended) vs 5
- Cnode host spread: tsys9 + tsys1 + tsys3 (recommended, 3-host, local-lvm)
- Tailscale pattern: subnet router on
tailscale-router(recommended)
- Provisioning: add
tailscale up --accept-dns=falseafter resolv.conf deploy - DNS: verify Pi-hole forwards knel.net to local Technitium on both nodes
- Perf: complete tsys4/5 tuning after hardware install (Friday)
- tsys2: rebuild from Win10 to Proxmox (k8s-dedicated host)
- k8s deferred topics (tracked for future sessions): ETL tooling (GDAL/PostGIS/xarray/Dask → RWX vs RWO), HPC scheduler (Argo/Volcano), per-tenant vcluster policy templates, solar-aware scale-out
Infrastructure Summary
| Component | Details |
|---|---|
| Proxmox hosts | 7 standalone PVE, managed via PDM |
| DNS primary | pfv-netinfra-01 (192.168.3.252) — Technitium + Pi-hole |
| DNS secondary | pfv-netinfra-02 (192.168.3.253) — Technitium + Pi-hole |
| DNS production | tailscale-router (read-only source of truth) |
| NTP | pfv-netinfra-01/02 (redundant, LAN IPs, stratum 2/3) |
| Sandbox VM | sectestbed-sandbox (VMID 6000 on pfv-tsys5) |
| Backup | Proxmox Backup Server (PBS) |
| k8s control plane | 3-node k3s HA (cnode1/2/3), all traffic over Tailscale |
| PDU | Cyclades PM10i via powerman on pfv-tsys1 (port 10101) |
| Console | 7 switch consoles via ser2net+conman on pfv-tsys4 (TCP 2001-2007 on Tailscale) |
| Production | Cloudron VPS, Reston VA (this cluster is R&D only) |