KNEL/PFVCluster
5
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
28e0b0c7a664e4b1a45aa8094ac2d165dac43815
PFVCluster/STATUS.md
T
mrcharles 28e0b0c7a6 docs(audit): fresh fleet audit + fix stale paths across 13 perf scripts 
Fresh Proxmox fleet audit (2026-07-28) with current VM placements, RAM,
CPU, and storage for all 7 reachable hosts. Written to
docs/proxmox/AUDIT-2026-07-28.md — supersedes placement data in
PROJECT.md sections 4-8.

Key audit findings:
- CRITICAL: UCS01/02 and netinfra01/02 HA pairs both still on tsys4
  storage. tsys4 failure = DNS/DHCP/NTP + LDAP/AD fully dark. These
  migrations were the #1 recommendation from the previous audit and
  have not been done.
- CRITICAL: 2 of 3 active k3s cnodes (cnode1 + cnode2) on tsys4 NFS.
  tsys4 failure = etcd quorum lost.
- 59% of running VMs still on tsys4 storage (improved from 68%).
- cnode VMIDs have changed since PROJECT.md was written (cnode1 is now
  VMID 906 on tsys9, cnode2 is VMID 705 on tsys7, etc.)

Gardening fixes:
- Removed duplicate fleet-audit.sh (check.sh + deploy-check.sh already
  exist for this purpose)
- Fixed hardcoded path /home/reachableceo/projects/perfopt in 13 perf/
  scripts to use BASH_SOURCE-derived relative paths (per AGENTS.md
  self-locating scripts convention)
- Updated STATUS.md Known Issues with the two critical findings
- Updated STATUS.md Pending with prioritized pre-k8s action items
- Registered AUDIT-2026-07-28.md in docmap.md

💘 Generated with Crush

Assisted-by: Crush:glm-5.2
2026-07-28 20:07:09 -05:00

119 lines
6.1 KiB
Markdown
Raw Blame History

# Project Status
> **Human read-only. Agents maintain this file automatically after each work
> session.** Do not edit by hand — the next agent run will overwrite it.
> **Last updated:** 2026-07-28 by Crush (GLM-5.2)
## Current State: STABLE
The merged PFVCluster repo is fully operational across provisioning, DNS
infrastructure, Proxmox cluster ops, and k8s control plane.
## Completed Work
### Server Provisioning (validated on sectestbed-sandbox)
- [x] SetupNewSystem.sh deploys end-to-end to rc=0 (Debian 13 trixie)
- [x] Security hardening: SSH, SCAP-STIG, Wazuh, 2FA (SSH+Cockpit+Webmin)
- [x] NTP: redundant pfv-netinfra-01/02 (192.168.3.252/253), synced stratum 3
- [x] DNS resolv.conf: managed static file pointing at netinfra pair
- [x] Test suite: 5 tests (framework, safe-download, 2fa, https, system-req)
- [x] VM validation harness: git-based deploy + auto-rollback + guest-agent access
### DNS Cluster (pfv-netinfra-01/02)
- [x] Production Technitium config replicated from tailscale-router (read-only)
- [x] 124 zones on both nodes (knel.net + reverse DNS)
- [x] pfv-netinfra-01 = PRIMARY, pfv-netinfra-02 = SECONDARY
- [x] Zone replication via rsync systemd timer (every 60s)
- [x] Credentials + 2FA replicated identically to production
- [x] Both LAN IPs resolve knel.net device names + recurse externally
### Proxmox Cluster Ops
- [x] 5 of 7 hosts fully performance-tuned (tsys1/3/6/7/9)
- [x] Fleet audit complete (PROJECT.md has ground truth)
- [ ] tsys4: blocked on PCIe NIC + RAM install
- [ ] tsys5: blocked on 2nd ethernet cable + NVMe install
- [ ] tsys2: pending rebuild from Win10 to Proxmox
### Kubernetes Control Plane (k3s HA — LIVE)
- [x] 3-node k3s HA control plane deployed: cnode1/2/3 (v1.36.2+k3s1, embedded etcd)
- [x] **All traffic over Tailscale IPs** — no LAN IPs in node status or certs
- [x] All 3 cnodes tainted `control-plane:NoSchedule` (zero user workloads)
- [x] 13/13 health checks pass (verify.sh): nodes Ready, etcd quorum,
Tailscale IPs, CoreDNS, API server, workload isolation
- [x] Scripts in `k8s/`: wipe, install-cp, join-servers, post-setup, verify
- [x] Kubeconfig saved to `~/.kube/config.pfv-k8s` (gitignored, embedded certs)
- [ ] Workers (wnodes) not yet joined to this cluster
- [ ] Distro decision: **k3s chosen for regular R&D cluster**. Talos docs
preserved in `docs/k8s/` for future ITAR/classified cluster.
### PDU Management (powerman on pfv-tsys1 — LIVE)
- [x] Cyclades AlterPath PM10i (10 outlets) managed via powerman over serial
- [x] USB-DB9 adapter (Prolific pl2303) with stable udev symlink
`/dev/cyclades-pm10`
- [x] powermand listening on `127.0.0.1:10101` + Tailscale `100.121.189.98:10101`
- [x] All 10 outlets defined as `outlet-1` through `outlet-10`
- [x] Validated: outlet 10 cycled off → on (8/8 test checks passed)
- [ ] Rename outlets to match physical devices (Friday onsite)
### Console Management (ser2net + conman on pfv-tsys4 — LIVE)
- [x] 7 network switch/router consoles managed via ser2net + conman
- [x] **USB enumeration problem SOLVED:** udev rules pin each adapter by
ID_PATH (physical USB port topology) to stable `/dev/consoles/<name>`
symlinks that survive reboot regardless of enumeration order
- [x] ser2net exposes all 7 consoles on TCP ports (2001-2007) bound to
**Tailscale IP only** (`100.70.77.93`)
- [x] conman connects to TCP ports for logging + multiplexing (7 log
files active in `/var/log/conman/`)
- [x] Both ser2net + conmand enabled via systemd (survive reboot)
- [x] conmand systemd unit created (Debian package doesn't ship one)
- [x] Old `/root/conmap` + manual `screen` workflow replaced
### Repo Merge
- [x] KNELServerBuild merged into PFVCluster (history preserved)
- [x] Directory structure reorganized (provisioning/, tests/, perf/, docs/)
- [x] All docs gardened: links fixed, stale refs removed, tailscale.md updated
## Known Issues
| Issue | Impact | Status |
|-------|--------|--------|
| **UCS + netinfra HA pairs both on tsys4** | tsys4 failure = DNS/DHCP/NTP + LDAP/AD fully dark | **CRITICAL — needs PDM migration** |
| **2 of 3 k3s cnodes on tsys4** | tsys4 failure = etcd quorum lost | **CRITICAL — needs PDM migration** |
| Technitium AXFR uses port 53 (occupied by Pi-hole) | Zone transfer via rsync instead of native AXFR | Workaround in place |
| `download.proxmox.com` unreachable from sandbox VM | 2 validation tests warn (environmental) | Not a code issue |
| tsys4/5 hardware pending | Perf tuning incomplete on 2 hosts | Waiting on physical install |
## Pending (next session priorities)
1. **CRITICAL: Migrate HA pairs to separate storage** (PDM, 10 min):
- netinfra-02 (VMID 904): D2 → S3 (tsys4 → tsys5)
- ucs-02 (VMID 902): D5 → S2 (tsys4 → tsys5)
2. **CRITICAL: Fix k3s cnode quorum risk** (PDM, 5 min):
- Move cnode1 (VMID 906) or cnode2 (VMID 705) from tsys4 to tsys5 storage
3. **k8s workers:** Join wnodes to the k3s cluster (agents, not servers)
4. **PDU:** Rename outlets in powerman.conf to match physical devices (Friday)
5. Perf: complete tsys4/5 tuning after hardware install (Friday)
6. tsys2: rebuild from Win10 to Proxmox (k8s-dedicated host)
7. **k8s deferred topics:** ETL tooling, HPC scheduler, vcluster policy,
solar-aware scale-out
> **See [`docs/proxmox/AUDIT-2026-07-28.md`](docs/proxmox/AUDIT-2026-07-28.md)
> for the full fresh audit with VM inventory and action items.**
## Infrastructure Summary
| Component | Details |
|-----------|---------|
| Proxmox hosts | 7 standalone PVE, managed via PDM |
| DNS primary | pfv-netinfra-01 (192.168.3.252) — Technitium + Pi-hole |
| DNS secondary | pfv-netinfra-02 (192.168.3.253) — Technitium + Pi-hole |
| DNS production | tailscale-router (read-only source of truth) |
| NTP | pfv-netinfra-01/02 (redundant, LAN IPs, stratum 2/3) |
| Sandbox VM | sectestbed-sandbox (VMID 6000 on pfv-tsys5) |
| Backup | Proxmox Backup Server (PBS) |
| **k8s control plane** | **3-node k3s HA (cnode1/2/3), all traffic over Tailscale** |
| **PDU** | **Cyclades PM10i via powerman on pfv-tsys1 (port 10101)** |
| **Console** | **7 switch consoles via ser2net+conman on pfv-tsys4 (TCP 2001-2007 on Tailscale)** |
| Production | Cloudron VPS, Reston VA (this cluster is R&D only) |
Reference in New Issue View Git Blame Copy Permalink