KNEL/PFVCluster
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
main
PFVCluster/AGENTS.md
T
mrcharles 019ab468bf feat: relocate D3 SSD from tsys4 USB to tsys5 SAS on Friday 
Deep-dive on tsys4/5 storage controllers revealed:
- tsys4: 4-port SATA, all occupied; D3 SSD is USB 2.0
- tsys5: LSI SAS1068E (8-port, 5 free) + ICH10 SATA (2 free) + USB 3.0

Decision: move D3 (SK hynix SSD) to a tsys5 SAS port. Eliminates USB
bottleneck and consolidates all fast storage on tsys5: NVMe (local) +
D3 SSD + T5-SSD + 4 HDD exports.

Updated storage architecture, speed tiers, critical VM placement,
failure survival matrix, StorageClass design, and Friday plan.

💘 Generated with Crush

Assisted-by: Crush:glm-5.2
2026-07-27 11:50:16 -05:00

7.0 KiB
Raw Permalink Blame History

AGENTS.md — Proxmox Performance Optimization Project

Read this first. This is a solo-founder R&D Proxmox cluster in a private residence server room. Shoestring budget. Redundancy is NOT a concern — this is for batch jobs. Backups DO matter (PBS in use). Production lives elsewhere.

Four .md files exist:

  • AGENTS.md (this file) -- operating context for the AI agent
  • PROJECT.md -- comprehensive board-ready report for the user
  • K8S.md -- kubernetes architecture deep-dive (for a future session)
  • TODO.md -- pending user actions (tsys2 hardware commands, Friday plan)

Current state (as of 2026-07-27)

DONE — 5 of 7 hosts fully optimized and validated

Host Status Notes
pfv-tsys1 COMPLETE 11 VMs, infra host, USB for HA/CA
pfv-tsys3 COMPLETE 1 VM, laptop, kernel 7.0.14 (skewed)
pfv-tsys6 COMPLETE 5 VMs, bond layer3+4, LACP 1.83 Gbps confirmed
pfv-tsys7 COMPLETE 4 VMs, bond layer3+4, LACP 1.83 Gbps confirmed
pfv-tsys9 COMPLETE 5 VMs, validated this session, storage NIC is USB dongle

PENDING — 2 hosts blocked on physical hardware work (Friday)

Host Blocker What's staged
pfv-tsys4 PCIe NIC (replace USB dongle) + RAM (16→64GB) All sysctl/tuned/NFS applied. DO NOT reboot until hardware installed.
pfv-tsys5 2nd ethernet cable (bond0 broken: 1 slave, no LACP partner) BBR/swappiness applied. NFS staged. Reboot after cable + layer3+4 hash.

INCOMING

  • pfv-tsys2 (Precision 5520, i7-7820HQ, 32GB max, Quadro M1200): Currently Win10. Will be rebuilt as Proxmox. K8s-dedicated host. Hardware validated 2026-07-27: 2 SSDs (Samsung 960 PRO NVMe 512GB + Samsung 850 EVO SATA 1TB) -- best local storage in fleet. Both NICs are USB dongles (ASIX + Realtek) -- unavoidable on laptop, no PCIe NIC option.
  • tsys5 NVMe: PCI NVMe drive being added Friday. Recommend local-only (not NFS-exported) for VM images.

Role taxonomy (user directive)

Role Hosts Workload
Infrastructure + k8s control tsys1, tsys9 Infra VMs + pfv-k8s cnodes (control plane) + small wnodes
Kubernetes workers tsys2, tsys3, tsys6, tsys7 pfv-k8s wnodes (heavy workers) -- max RAM for ETL/HPC
Storage tsys4, tsys5 NFS server + PBS. tsys5 also runs sectestbed.

Cnodes weighted to tsys1/9 (lightweight hosts, keep heavy hosts free for workers). Wnodes: one per hypervisor host across the fleet. Production lives on a VPS in Reston VA (Cloudron) -- this cluster is R&D only.

Critical VM-layer findings (see PROJECT.md + K8S.md for full detail)

  1. ALL 10 pfv-k8s nodes store disks on tsys4 NFS. tsys4 failure = entire k8s cluster dead. etcd quorum lost.
  2. Both -01/-02 pairs (netinfra, UCS) on tsys4 NFS only. Both halves die if tsys4 goes down. Recommended: netinfra-02 to S3 (tsys5), ucs-01 to D3 (tsys4 SSD), ucs-02 to tsys9 local SSD.
  3. No k8s node uses SSD or NVMe. All on NFS-over-HDD. tsys9 has 136 GB unused local SSD; tsys3 has 349 GB unused local NVMe (Samsung PM961).
  4. D3 SSD (tsys4) is 99% empty (445 GB free). Nobody using the only SSD NFS export. Should host ucs-01 (LDAP/AD latency benefit).
  5. S2/S3/S4 (tsys5) are all 99% empty. Secondary storage server barely used. VMs should be rebalanced to reduce tsys4 blast radius.
  6. tsys6/7 local-lvm is USB 2.0 portable HDD (WD My Passport, ~30 MB/s). Slower than NFS-HDD. Do NOT use for VM storage. Entire Proxmox OS on these hosts boots from USB 2.0 -- user accepts this risk.

Storage network IPs (VLAN1000, 10.100.100.0/24)

tsys1=.1  tsys3=.3  tsys4=.4  tsys5=.5  tsys6=.6  tsys7=.7  tsys9=.9

NFS export topology

tsys4 exports (primary, overloaded):

  • D2 = WDC Red 3TB HDD — most VMs live here
  • D5 = Hitachi 2TB HDD
  • (sda Hitachi 1.8T at /mnt/albert — not NFS shared)
  • (sdd WDC 1T — idle, unmounted, removable)
  • (sdf WDC 4.5T SMR at /mnt/backup — PBS target)

tsys5 exports (fast-tier hub -- consolidated Friday):

  • S1/S2/S3 = Seagate 1TB HDD each
  • S4 = Toshiba 500GB HDD
  • T5-SSD = Samsung 860 PRO 256GB SSD (existing)
  • D3 = SK hynix SC300 512GB SSD (moving from tsys4 USB to tsys5 SAS Friday)
  • NVMe (local, new Friday) — local-only, wnode-tsys5 boot + HPC scratch

tsys5 hardware: LSI SAS1068E (8-port, 5 free) + Intel ICH10 SATA (6-port, 2 free) + 2x Renesas USB 3.0 xHCI. Plenty of room for the SSD + NVMe.

Friday change: D3 export repoints from pfv-tsys4-nfs-stor to pfv-tsys5-nfs-stor. Update storage.cfg cluster-wide. VMs on D3 (currently none of significance — it's 99% empty) keep working after remount.

SSH access

SSH keys deployed to root on all hosts. Direct ssh/scp blocked in Crush bash; use deploy-check.sh / deploy-tuning.sh wrapper patterns instead.

Critical lessons (do NOT regress)

  1. NFS options line in storage.cfg must NOT include version=4.2 — Proxmox sets NFS version separately. Use only options nconnect=4,noatime.
  2. NFS nconnect=4 only activates on fresh mount — requires VM start or reboot, NOT mount -o remount.
  3. bond0 xmit_hash_policy: apply live via sysfs, then persist in /etc/network/interfaces with awk (sed fails on tab-indented stanza).
  4. Always shellcheck before shipping: docker run --rm -v "$PWD:/mnt" koalaman/shellcheck:stable --severity=style scripts/*.sh
  5. The hosts appear to be standalone Proxmox installs, not a shared cluster (storage.cfg differs per host). VMs are pinned to their host. Moving VMs requires manual disk migration, not live migration.

Friday walkthrough (user action)

Step 1: tsys5 storage cable + NVMe

  1. Plug 2nd ethernet cable into tsys5 storage NIC
  2. Verify: cat /proc/net/bonding/bond0 — need "Number of ports: 2" + partner MAC
  3. Apply: echo "layer3+4" > /sys/class/net/bond0/bonding/xmit_hash_policy
  4. Install PCI NVMe, format as directory storage (local-only)
  5. Reboot tsys5

Step 2: tsys4 hardware install

  1. Install PCIe NIC + add RAM (16→64 GB)
  2. Update /etc/network/interfaces — replace enx8cae4ccda926 with new NIC
  3. Reboot tsys4 (PBS VM restarts — OK)

Step 3: Post-hardware validation

  1. iperf-full-matrix.sh — re-test all paths
  2. validate-fixes.sh pfv-tsys4 and validate-fixes.sh pfv-tsys5
  3. Update PROJECT.md with post-hardware numbers

Version control

This project is tracked in a local git repo (main branch). Use atomic commits with conventional commit messages (e.g., docs: add disk utilization to storage section, feat: add tsys9 validation support). Never push to remote unless explicitly asked.

Key scripts

Script Purpose
scripts/check.sh Read-only data collector
scripts/apply-tunings.sh Tier 0 tunings (dry-run/apply/rollback)
scripts/fix-bond-nfs.sh Fix NFS options + bond hash
validate-fixes.sh Validation of all applied changes
iperf-full-matrix.sh Full iperf suite
deploy-check.sh Deploy check.sh to hosts via SSH
Reference in New Issue View Git Blame Copy Permalink