KNEL/PFVCluster
5
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
5456c783c647488bcc1541171c6e687dc1cd2389
PFVCluster/STATUS.md
T
mrcharles 5456c783c6 feat(ups): NUT setup for APC Smart-UPS C 1500 on pfv-tsys1 
Deploy Network UPS Tools to monitor the server-room UPS units and feed data to
Home Assistant for power-usage tracking.

APC Smart-UPS C 1500 is fully operational: battery charge, runtime, load,
voltage, and status all reporting via usbhid-ups + APC HID 0.100 subdriver.
upsd listens on Tailscale (100.121.189.98:3493) for HA polling; local upsmon
provides graceful hypervisor shutdown on battery-low.

Key deployment fix: the udev rule must cover SUBSYSTEM=="usb" (raw
/dev/bus/usb) in addition to hidraw, because usbhid-ups opens the raw USB
device file after dropping to the nut user via setuid(). Matching only hidraw
causes EACCES on driver start.

Tripp Lite UPS (09ae:3016) is blocked — driver matches TrippLite HID 0.85
subdriver but fails reading the 878-byte HID Report Descriptor (EAGAIN via USB
hub). Needs physical reseat to direct motherboard USB port. Driver masked to
prevent restart-loop spam.

Files: ups/discover.sh, ups/setup.sh (idempotent), ups/status.sh, ups/README.md

💘 Generated with Crush

Assisted-by: Crush:glm-5.2
2026-07-30 09:34:05 -05:00

11 KiB
Raw Blame History

Project Status

Human read-only. Agents maintain this file automatically after each work session. Do not edit by hand — the next agent run will overwrite it.

Last updated: 2026-07-30 by Crush (GLM-5.2)

Capacity targets are now codified in proxmox/docs/CAPACITY-GOALS.md: 80% RAM steady-state, ~50% idle CPU. The current placement analysis lives in proxmox/docs/AUDIT-2026-07-29.md.

Current State: STABLE

The merged PFVCluster repo is fully operational across provisioning, DNS infrastructure, Proxmox cluster ops, and k8s control plane.

Completed Work

Server Provisioning (validated on sectestbed-sandbox)

  • SetupNewSystem.sh deploys end-to-end to rc=0 (Debian 13 trixie)
  • Security hardening: SSH, SCAP-STIG, Wazuh, 2FA (SSH+Cockpit+Webmin)
  • NTP: redundant pfv-netinfra-01/02 (192.168.3.252/253), synced stratum 3
  • DNS resolv.conf: managed static file pointing at netinfra pair
  • Test suite: 5 tests (framework, safe-download, 2fa, https, system-req)
  • VM validation harness: git-based deploy + auto-rollback + guest-agent access

DNS Cluster (pfv-netinfra-01/02)

  • Production Technitium config replicated from tailscale-router (read-only)
  • 124 zones on both nodes (knel.net + reverse DNS)
  • pfv-netinfra-01 = PRIMARY, pfv-netinfra-02 = SECONDARY
  • Zone replication via rsync systemd timer (every 60s)
  • Credentials + 2FA replicated identically to production
  • Both LAN IPs resolve knel.net device names + recurse externally
  • DHCP migrated from pfv-netboot → netinfra-01/02 (ISC failover-peer, primary/secondary). Both normal, 333 active leases load-balanced. DNS+NTP now handed out as 252/253. Netboot DHCP stopped + disabled. Webmin DHCP module on both nodes.

Proxmox Cluster Ops

  • 5 of 7 hosts fully performance-tuned (tsys1/3/6/7/9)
  • Fleet audit refreshed 2026-07-29 (consolidated, storage-corrected)
  • Memory goal (≤80% actual) met fleet-wide — highest host is tsys1 at 73%
  • Capacity goals codified (80% RAM / ~50% idle CPU + workload model)
  • Storage topology corrected: local-nonprod = S1 disk (not separate); per-host storage.cfg
  • cnode5 (706) deleted; cnode4 (601) stopped (delete pending)
  • tsys6 S1 typo + tsys9 missing S1 — both FIXED by operator (verified: 8/8 NFS stores active on both)
  • CPU ~50%-idle goal: fleet idles 0-5% (under-utilized; chase via workloads, not hardware)
  • tsys4: blocked on PCIe NIC + RAM install
  • tsys5: blocked on 2nd ethernet cable + NVMe install; also single-CPU (CPU2 socket empty)
  • tsys2: pending rebuild from Win10 to Proxmox

Kubernetes Control Plane (k3s HA — LIVE)

  • 3-node k3s HA control plane deployed: cnode1/2/3 (v1.36.2+k3s1, embedded etcd)
  • All traffic over Tailscale IPs — no LAN IPs in node status or certs
  • All 3 cnodes tainted control-plane:NoSchedule (zero user workloads)
  • 13/13 health checks pass (verify.sh): nodes Ready, etcd quorum, Tailscale IPs, CoreDNS, API server, workload isolation
  • Scripts in k8s/: wipe, install-cp, join-servers, post-setup, verify
  • Kubeconfig saved to ~/.kube/config.pfv-k8s (gitignored, embedded certs)
  • Workers (wnodes) not yet joined to this cluster
  • Distro decision: k3s chosen for regular R&D cluster. Talos docs preserved in k8s/docs/ for future ITAR/classified cluster.

PDU Management (powerman on pfv-tsys1 — LIVE)

  • Cyclades AlterPath PM10i (10 outlets) managed via powerman over serial
  • USB-DB9 adapter (Prolific pl2303) with stable udev symlink /dev/cyclades-pm10
  • powermand listening on 127.0.0.1:10101 + Tailscale 100.121.189.98:10101
  • All 10 outlets defined as outlet-1 through outlet-10
  • Validated: outlet 10 cycled off → on (8/8 test checks passed)
  • Rename outlets to match physical devices (Friday onsite)

UPS Management (NUT on pfv-tsys1 — LIVE)

  • APC Smart-UPS C 1500 fully reporting via NUT usbhid-ups + APC HID 0.100
  • upsd listening on Tailscale 100.121.189.98:3493 (port 3493)
  • Local upsmon (master) configured for graceful hypervisor shutdown
  • Home Assistant credentials generated (homeassistant user in upsd.users)
  • Scripts in ups/: discover, setup (idempotent), status
  • Key fix: udev rule covers BOTH usb and hidraw subsystems (driver opens raw /dev/bus/usb after setuid drop, not /dev/hidraw)
  • Tripp Lite UPS blocked: driver matches TrippLite HID 0.85 subdriver but fails reading 878-byte HID Report Descriptor (EAGAIN via USB hub). Needs physical reseat to direct motherboard USB port or cable replacement. Driver masked to prevent restart-loop spam.
  • Home Assistant NUT integration: not yet configured in HA UI (user action)

Console Management (ser2net telnet(rfc2217) + conman on pfv-tsys4 — LIVE)

  • 7 network switch/router consoles managed via ser2net + conman
  • USB enumeration problem SOLVED: udev rules pin each adapter by ID_PATH (physical USB port topology) to stable /dev/consoles/<name> symlinks that survive reboot regardless of enumeration order
  • ser2net exposes all 7 consoles on telnet(rfc2217) TCP ports (2001-2007) bound to Tailscale IP only (100.70.77.93)
  • conman connects to TCP ports for logging + multiplexing (7 log files active in /var/log/conman/)
  • Both ser2net + conmand enabled via systemd (survive reboot)
  • conmand systemd unit created (Debian package doesn't ship one)
  • conmand remote access enabled (loopback=off, port 7890)
  • Old /root/conmap + manual screen workflow replaced
  • Stair-stepping fix: ser2net accepter changed from raw tcp to telnet(rfc2217) so conman's telnet NVT negotiates binary mode properly
  • Validated 6/7 consoles working via conman from workstation (core-sw01, tor3-mgmt, tor3-stor, rrinfra-rtr, r2-tor-top, subodev-torsw)
  • pfv-r2-sw (port 2007): DEAD — zero bytes at all baud rates (1200-115200), adapter present and mapped correctly (/dev/ttyUSB3). Physical issue: device powered off, console cable disconnected, or console port disabled in device config. Requires onsite investigation.

Ansible AWX (tsys-awx — LIVE)

  • AWX 24.6.1 deployed via AWX Operator 2.19.1 on k3s single-node
  • Accessible at http://tsys-awx.knel.net (port 80, LoadBalancer via k3s ServiceLB)
  • Admin login: admin / REDACTED_PASSWORD
  • PostgreSQL 15 (bundled, 8 GiB PVC on local-path)
  • VM resized: 32→60 GB disk, swap converted to swapfile
  • Scripts in awx/: install-k3s, deploy-awx, verify-awx
  • HTTPS/TLS not yet configured (currently HTTP on port 80)
  • Execution environments not yet customized

Repo Merge

  • KNELServerBuild merged into PFVCluster (history preserved)
  • Directory structure reorganized (provisioning/, tests/, perf/, docs/)
  • All docs gardened: links fixed, stale refs removed, tailscale.md updated

Known Issues

Issue Impact Status
2 of 3 k3s cnodes on tsys4 tsys4 failure = etcd quorum lost (1 of 3) Accepted — fundamental constraint with 3 cnodes + 2 NFS servers. Revisit after Friday (local-SSD boot for one cnode). Deferred storage concern.
tsys1/5/9 RAM-overcommitted (alloc) 115%/134%/129% allocation; actual usage 73%/37%/62% — all under 80% goal Migrate NFS-backed VMs to tsys6/7 (117/171 GB free actual). See AUDIT-2026-07-29 §11
tsys5 is single-CPU CPU2 socket unpopulated; half the documented CPU capacity Consider seating a spare LGA771 Xeon (also activates CPU2 DIMM branch)
cnode4/5 stale VMs cnode4 (601) stopped; cnode5 (706) deleted cnode4 delete pending; cnode5 done ✓
Fleet CPU 0-5% idle Misses ~50%-idle utilization goal Consolidate baseline; solar-gate idle PowerEdge; grow via workloads
D2 IOPS hotspot 17 of 26 NFS-backed VMs on one spindle; S1/S4 underused Phase 2 disk migrations: D2→S2/S3/S4/D5 (see audit §11)
Technitium AXFR uses port 53 (occupied by Pi-hole) Zone transfer via rsync instead of native AXFR Workaround in place
tsys4/5 hardware pending Perf tuning incomplete on 2 hosts Friday physical install

Pending (next session priorities)

Full phased action checklist with current/target inventory is in proxmox/docs/AUDIT-2026-07-29.md §11. Summary below. SSD/NVMe tiering is deferred to post-Friday.

Phase 0 — Storage config fixes — DONE ✓ (tsys6 S1 typo, tsys9 S1 missing)

Phase 1 — Cleanup:

  • Delete cnode4 (601) on tsys6 (stopped, stale)
  • cnode5 (706) already deleted ✓

Phase 2 — Disk migrations off D2 (IOPS rebalancing, 6 moves):

  • wnode-tsys5 (509): D2→S4 · tsys-awx (600): D2→D5 · hfnoc-uisp (702): D2→S4
  • rr-middleware (703): D2→S4 · TCTC (704): D2→S3 · cnode2 (705): D2→S2

Phase 3 — Host + disk migrations (RAM/CPU relief, 6 VMs):

  • tsys1→tsys7: librenms(104), pdm(105), tailscale-router(109), kali-tsys(114), workbench(117)
  • tsys9→tsys6: tsys-siem(901)

Phase 4 — Grow wnode-tsys9: 4c/4GB → 6c/8GB (after siem leaves tsys9)

Phase 5 — Join wnodes to k3s (after migrations free resources)

Other:

  • Friday maintenance: tsys4 (PCIe NIC + RAM), tsys5 (2nd CPU candidate, NVMe, 2nd ethernet)
  • PDU: Rename outlets in powerman.conf (Friday onsite)
  • k8s deferred: ETL tooling, HPC scheduler, vcluster policy, solar-aware scale-out
  • UPS: APC Smart-UPS C 1500 operational via NUT on pfv-tsys1. Tripp Lite blocked on HID report descriptor (USB hub/cable issue — try direct motherboard port). HA NUT integration pending user config in HA UI.

Infrastructure Summary

Component Details
Proxmox hosts 7 standalone PVE, managed via PDM
DNS primary pfv-netinfra-01 (192.168.3.252) — Technitium + Pi-hole
DNS secondary pfv-netinfra-02 (192.168.3.253) — Technitium + Pi-hole
DNS production tailscale-router (read-only source of truth)
NTP pfv-netinfra-01/02 (redundant, LAN IPs, stratum 2/3)
Sandbox VM sectestbed-sandbox (VMID 6000 on pfv-tsys5)
Backup Proxmox Backup Server (PBS)
k8s control plane 3-node k3s HA (cnode1/2/3), all traffic over Tailscale
PDU Cyclades PM10i via powerman on pfv-tsys1 (port 10101)
AWX tsys-awx: AWX 24.6.1 on k3s (http://tsys-awx.knel.net:80)
Console 7 switch consoles via ser2net telnet(rfc2217)+conman on pfv-tsys4 (conmand:7890 on Tailscale)
UPS APC Smart-UPS C 1500 via NUT on pfv-tsys1 (upsd:3493 on Tailscale)
Production Cloudron VPS, Reston VA (this cluster is R&D only)
Reference in New Issue View Git Blame Copy Permalink