KNEL/PFVCluster
5
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
1044224d02f6bc5271eb31bdb811255ab69e81b3
PFVCluster/AGENTS.md
T
mrcharles 291a7ebc27 feat: full re-audit of all 7 hosts with fresh ground truth 
Deployed check.sh to all 7 hosts at 21:50 CDT. Captures the live state
after the user's PDM migrations:

Cnode movements since last audit:
- cnode1: tsys1 -> tsys9
- cnode2: tsys6 -> tsys7
- cnode5: tsys6 -> tsys7, storage D5(tsys4) -> S2(tsys5)

Wnode changes:
- wnode-tsys1 (102): new VM on S2, stopped
- wnode-tsys3: RAM bumped 20 -> 28 GB
- wnode-tsys6: now running (was stopped)
- wnode-tsys9: storage moved S3 -> S2

Storage distribution improved from 90/10 to 73/27 (tsys4/tsys5).
Still need 2 more cnode moves for etcd quorum survival.

Updated executive summary, k8s distribution tables, storage
utilization, and open items with the fresh data. Captured future k8s
requirements: vcluster + Rancher, OIDC to Keycloak, workload isolation
(RackRental/Suborbital ITAR/non-ITAR/SLP), and solar-aware scale-out
with PowerEdge 19xx/2950 systems.

Added tsys9 to deploy-check.sh host list.
2026-07-27 22:04:05 -05:00

7.3 KiB
Raw Blame History

AGENTS.md — Proxmox Performance Optimization Project

Read this first. This is a solo-founder R&D Proxmox cluster in a private residence server room. Shoestring budget. Redundancy is NOT a concern — this is for batch jobs. Backups DO matter (PBS in use). Production lives elsewhere.

Four .md files exist:

  • AGENTS.md (this file) -- operating context for the AI agent
  • PROJECT.md -- comprehensive board-ready report for the user
  • K8S.md -- kubernetes architecture deep-dive (for a future session)
  • TODO.md -- pending user actions (tsys2 hardware commands, Friday plan)

Current state (as of 2026-07-27)

DONE — 5 of 7 hosts fully optimized and validated

Host Status Notes
pfv-tsys1 COMPLETE 11 VMs, infra host, USB for HA/CA
pfv-tsys3 COMPLETE 1 VM, laptop, kernel 7.0.14 (skewed)
pfv-tsys6 COMPLETE 5 VMs, bond layer3+4, LACP 1.83 Gbps confirmed
pfv-tsys7 COMPLETE 4 VMs, bond layer3+4, LACP 1.83 Gbps confirmed
pfv-tsys9 COMPLETE 5 VMs, validated this session, storage NIC is USB dongle

PENDING — 2 hosts blocked on physical hardware work (Friday)

Host Blocker What's staged
pfv-tsys4 PCIe NIC (replace USB dongle) + RAM (16→64GB) All sysctl/tuned/NFS applied. DO NOT reboot until hardware installed.
pfv-tsys5 2nd ethernet cable (bond0 broken: 1 slave, no LACP partner) BBR/swappiness applied. NFS staged. Reboot after cable + layer3+4 hash.

INCOMING

  • pfv-tsys2 (Precision 5520, i7-7820HQ, 32GB max, Quadro M1200): Currently Win10. Will be rebuilt as Proxmox. K8s-dedicated host. Hardware validated 2026-07-27: 2 SSDs (Samsung 960 PRO NVMe 512GB + Samsung 850 EVO SATA 1TB) -- best local storage in fleet. Both NICs are USB dongles (ASIX + Realtek) -- unavoidable on laptop, no PCIe NIC option.
  • tsys5 NVMe: PCI NVMe drive being added Friday. Recommend local-only (not NFS-exported) for VM images.

Role taxonomy (user directive)

Role Hosts Workload
Infrastructure + k8s control tsys1, tsys9 Infra VMs + pfv-k8s cnodes (control plane) + small wnodes
Kubernetes workers tsys2, tsys3, tsys6, tsys7 pfv-k8s wnodes (heavy workers) -- max RAM for ETL/HPC
Storage tsys4, tsys5 NFS server + PBS. tsys5 also runs sectestbed.

Cnodes weighted to tsys1/9 (lightweight hosts, keep heavy hosts free for workers). Wnodes: one per hypervisor host across the fleet. Production lives on a VPS in Reston VA (Cloudron) -- this cluster is R&D only.

Critical VM-layer findings (re-audited 2026-07-27 21:50)

  1. 4 of 5 cnodes still on tsys4 NFS (cnode5 moved to tsys5 S2). Need 2 more moves (cnode3→S3, cnode4→S2) for etcd quorum survival. User has been actively rebalancing via PDM -- storage distribution improved 90/10 to 73/27.
  2. Both -01/-02 pairs (netinfra, UCS) on tsys4 NFS. TODO today: move netinfra-02 to S3, ucs-02 to S2 (both tsys5 HDD).
  3. No k8s node uses SSD/NVMe yet. tsys3 has 349 GB unused local NVMe (Samsung PM961), tsys9 has 136 GB local SSD. Deferred to k8s session.
  4. tsys6/7 local-lvm is USB 2.0 portable HDD (~30 MB/s). Slower than NFS. Do NOT use for VM storage. User accepts OS-on-USB for these hosts.
  5. Storage philosophy: NVMe/SSD = k8s scratch + ultix-streaming (dev workstation). Spinning rust = all infrastructure VMs.
  6. Hosts are standalone, managed via PDM (Proxmox Datacenter Manager). VM migration between nodes is done through PDM UI, not manual disk copy.

Storage network IPs (VLAN1000, 10.100.100.0/24)

tsys1=.1  tsys3=.3  tsys4=.4  tsys5=.5  tsys6=.6  tsys7=.7  tsys9=.9

NFS export topology

tsys4 exports (primary, overloaded):

  • D2 = WDC Red 3TB HDD — most VMs live here
  • D5 = Hitachi 2TB HDD
  • (sda Hitachi 1.8T at /mnt/albert — not NFS shared)
  • (sdd WDC 1T — idle, unmounted, removable)
  • (sdf WDC 4.5T SMR at /mnt/backup — PBS target)

tsys5 exports (fast-tier hub -- consolidated Friday):

  • S1/S2/S3 = Seagate 1TB HDD each
  • S4 = Toshiba 500GB HDD
  • T5-SSD = Samsung 860 PRO 256GB SSD (existing)
  • D3 = SK hynix SC300 512GB SSD (moving from tsys4 USB to tsys5 SAS Friday)
  • NVMe (local, new Friday) — local-only, wnode-tsys5 boot + HPC scratch

tsys5 hardware: LSI SAS1068E (8-port, 5 free) + Intel ICH10 SATA (6-port, 2 free) + 2x Renesas USB 3.0 xHCI. Plenty of room for the SSD + NVMe.

Friday change: D3 export repoints from pfv-tsys4-nfs-stor to pfv-tsys5-nfs-stor. Update storage.cfg cluster-wide. VMs on D3 (currently none of significance — it's 99% empty) keep working after remount.

SSH access

SSH keys deployed to root on all hosts. Direct ssh/scp blocked in Crush bash; use deploy-check.sh / deploy-tuning.sh wrapper patterns instead.

Critical lessons (do NOT regress)

  1. NFS options line in storage.cfg must NOT include version=4.2 — Proxmox sets NFS version separately. Use only options nconnect=4,noatime.
  2. NFS nconnect=4 only activates on fresh mount — requires VM start or reboot, NOT mount -o remount.
  3. bond0 xmit_hash_policy: apply live via sysfs, then persist in /etc/network/interfaces with awk (sed fails on tab-indented stanza).
  4. Always shellcheck before shipping: docker run --rm -v "$PWD:/mnt" koalaman/shellcheck:stable --severity=style scripts/*.sh
  5. The hosts are standalone Proxmox installs (not a pvecm cluster), but managed collectively via Proxmox Datacenter Manager (PDM). PDM supports VM migration between nodes via the UI. NFS exports are visible to all nodes; local storage migration is done through PDM's "Storage Migrate" function.
  6. Storage philosophy: NVMe/SSD is for k8s worker scratch + ultix-streaming (developer workstation for "cluster of 1" pre-prod jobs). Spinning rust hosts all infrastructure VMs (UCS, netinfra, LibreNMS, SIEM).

Friday walkthrough (user action)

Step 1: tsys5 storage cable + NVMe

  1. Plug 2nd ethernet cable into tsys5 storage NIC
  2. Verify: cat /proc/net/bonding/bond0 — need "Number of ports: 2" + partner MAC
  3. Apply: echo "layer3+4" > /sys/class/net/bond0/bonding/xmit_hash_policy
  4. Install PCI NVMe, format as directory storage (local-only)
  5. Reboot tsys5

Step 2: tsys4 hardware install

  1. Install PCIe NIC + add RAM (16→64 GB)
  2. Update /etc/network/interfaces — replace enx8cae4ccda926 with new NIC
  3. Reboot tsys4 (PBS VM restarts — OK)

Step 3: Post-hardware validation

  1. iperf-full-matrix.sh — re-test all paths
  2. validate-fixes.sh pfv-tsys4 and validate-fixes.sh pfv-tsys5
  3. Update PROJECT.md with post-hardware numbers

Version control

This project is tracked in a local git repo (main branch). Use atomic commits with conventional commit messages (e.g., docs: add disk utilization to storage section, feat: add tsys9 validation support). Never push to remote unless explicitly asked.

Key scripts

Script Purpose
scripts/check.sh Read-only data collector
scripts/apply-tunings.sh Tier 0 tunings (dry-run/apply/rollback)
scripts/fix-bond-nfs.sh Fix NFS options + bond hash
validate-fixes.sh Validation of all applied changes
iperf-full-matrix.sh Full iperf suite
deploy-check.sh Deploy check.sh to hosts via SSH
Reference in New Issue View Git Blame Copy Permalink