mrcharles
89469ff028
Deployed check.sh to all 7 hosts at 21:50 CDT. Captures the live state after the user's PDM migrations: Cnode movements since last audit: - cnode1: tsys1 -> tsys9 - cnode2: tsys6 -> tsys7 - cnode5: tsys6 -> tsys7, storage D5(tsys4) -> S2(tsys5) Wnode changes: - wnode-tsys1 (102): new VM on S2, stopped - wnode-tsys3: RAM bumped 20 -> 28 GB - wnode-tsys6: now running (was stopped) - wnode-tsys9: storage moved S3 -> S2 Storage distribution improved from 90/10 to 73/27 (tsys4/tsys5). Still need 2 more cnode moves for etcd quorum survival. Updated executive summary, k8s distribution tables, storage utilization, and open items with the fresh data. Captured future k8s requirements: vcluster + Rancher, OIDC to Keycloak, workload isolation (RackRental/Suborbital ITAR/non-ITAR/SLP), and solar-aware scale-out with PowerEdge 19xx/2950 systems. Added tsys9 to deploy-check.sh host list.
158 lines
7.3 KiB
Markdown
158 lines
7.3 KiB
Markdown
# AGENTS.md — Proxmox Performance Optimization Project
|
|
|
|
**Read this first.** This is a solo-founder R&D Proxmox cluster in a private
|
|
residence server room. Shoestring budget. Redundancy is NOT a concern — this
|
|
is for batch jobs. Backups DO matter (PBS in use). Production lives elsewhere.
|
|
|
|
**Four .md files exist:**
|
|
- `AGENTS.md` (this file) -- operating context for the AI agent
|
|
- `PROJECT.md` -- comprehensive board-ready report for the user
|
|
- `K8S.md` -- kubernetes architecture deep-dive (for a future session)
|
|
- `TODO.md` -- pending user actions (tsys2 hardware commands, Friday plan)
|
|
|
|
## Current state (as of 2026-07-27)
|
|
|
|
### DONE — 5 of 7 hosts fully optimized and validated
|
|
|
|
| Host | Status | Notes |
|
|
|------|--------|-------|
|
|
| pfv-tsys1 | COMPLETE | 11 VMs, infra host, USB for HA/CA |
|
|
| pfv-tsys3 | COMPLETE | 1 VM, laptop, kernel 7.0.14 (skewed) |
|
|
| pfv-tsys6 | COMPLETE | 5 VMs, bond layer3+4, LACP 1.83 Gbps confirmed |
|
|
| pfv-tsys7 | COMPLETE | 4 VMs, bond layer3+4, LACP 1.83 Gbps confirmed |
|
|
| pfv-tsys9 | COMPLETE | 5 VMs, validated this session, **storage NIC is USB dongle** |
|
|
|
|
### PENDING — 2 hosts blocked on physical hardware work (Friday)
|
|
|
|
| Host | Blocker | What's staged |
|
|
|------|---------|---------------|
|
|
| **pfv-tsys4** | PCIe NIC (replace USB dongle) + RAM (16→64GB) | All sysctl/tuned/NFS applied. DO NOT reboot until hardware installed. |
|
|
| **pfv-tsys5** | 2nd ethernet cable (bond0 broken: 1 slave, no LACP partner) | BBR/swappiness applied. NFS staged. Reboot after cable + layer3+4 hash. |
|
|
|
|
### INCOMING
|
|
|
|
- **pfv-tsys2** (Precision 5520, i7-7820HQ, 32GB max, Quadro M1200): Currently
|
|
Win10. Will be rebuilt as Proxmox. K8s-dedicated host. **Hardware validated
|
|
2026-07-27: 2 SSDs (Samsung 960 PRO NVMe 512GB + Samsung 850 EVO SATA 1TB)
|
|
-- best local storage in fleet. Both NICs are USB dongles (ASIX + Realtek)
|
|
-- unavoidable on laptop, no PCIe NIC option.**
|
|
- **tsys5 NVMe**: PCI NVMe drive being added Friday. Recommend local-only
|
|
(not NFS-exported) for VM images.
|
|
|
|
## Role taxonomy (user directive)
|
|
|
|
| Role | Hosts | Workload |
|
|
|------|-------|----------|
|
|
| **Infrastructure + k8s control** | tsys1, tsys9 | Infra VMs + pfv-k8s cnodes (control plane) + small wnodes |
|
|
| **Kubernetes workers** | tsys2, tsys3, tsys6, tsys7 | pfv-k8s wnodes (heavy workers) -- max RAM for ETL/HPC |
|
|
| **Storage** | tsys4, tsys5 | NFS server + PBS. tsys5 also runs sectestbed. |
|
|
|
|
Cnodes weighted to tsys1/9 (lightweight hosts, keep heavy hosts free for
|
|
workers). Wnodes: one per hypervisor host across the fleet. Production lives
|
|
on a VPS in Reston VA (Cloudron) -- this cluster is R&D only.
|
|
|
|
## Critical VM-layer findings (re-audited 2026-07-27 21:50)
|
|
|
|
1. **4 of 5 cnodes still on tsys4 NFS** (cnode5 moved to tsys5 S2). Need 2
|
|
more moves (cnode3→S3, cnode4→S2) for etcd quorum survival. User has been
|
|
actively rebalancing via PDM -- storage distribution improved 90/10 to 73/27.
|
|
2. **Both -01/-02 pairs (netinfra, UCS) on tsys4 NFS.** TODO today: move
|
|
netinfra-02 to S3, ucs-02 to S2 (both tsys5 HDD).
|
|
3. **No k8s node uses SSD/NVMe yet.** tsys3 has 349 GB unused local NVMe
|
|
(Samsung PM961), tsys9 has 136 GB local SSD. Deferred to k8s session.
|
|
4. **tsys6/7 local-lvm is USB 2.0 portable HDD** (~30 MB/s). Slower than NFS.
|
|
Do NOT use for VM storage. User accepts OS-on-USB for these hosts.
|
|
5. **Storage philosophy:** NVMe/SSD = k8s scratch + ultix-streaming (dev
|
|
workstation). Spinning rust = all infrastructure VMs.
|
|
6. **Hosts are standalone, managed via PDM** (Proxmox Datacenter Manager).
|
|
VM migration between nodes is done through PDM UI, not manual disk copy.
|
|
|
|
## Storage network IPs (VLAN1000, 10.100.100.0/24)
|
|
|
|
```
|
|
tsys1=.1 tsys3=.3 tsys4=.4 tsys5=.5 tsys6=.6 tsys7=.7 tsys9=.9
|
|
```
|
|
|
|
## NFS export topology
|
|
|
|
**tsys4 exports (primary, overloaded):**
|
|
- D2 = WDC Red 3TB HDD — most VMs live here
|
|
- D5 = Hitachi 2TB HDD
|
|
- (sda Hitachi 1.8T at /mnt/albert — not NFS shared)
|
|
- (sdd WDC 1T — idle, unmounted, removable)
|
|
- (sdf WDC 4.5T SMR at /mnt/backup — PBS target)
|
|
|
|
**tsys5 exports (fast-tier hub -- consolidated Friday):**
|
|
- S1/S2/S3 = Seagate 1TB HDD each
|
|
- S4 = Toshiba 500GB HDD
|
|
- T5-SSD = Samsung 860 PRO 256GB SSD (existing)
|
|
- **D3 = SK hynix SC300 512GB SSD** (moving from tsys4 USB to tsys5 SAS Friday)
|
|
- **NVMe (local, new Friday)** — local-only, wnode-tsys5 boot + HPC scratch
|
|
|
|
**tsys5 hardware:** LSI SAS1068E (8-port, 5 free) + Intel ICH10 SATA (6-port,
|
|
2 free) + 2x Renesas USB 3.0 xHCI. Plenty of room for the SSD + NVMe.
|
|
|
|
**Friday change:** D3 export repoints from pfv-tsys4-nfs-stor to
|
|
pfv-tsys5-nfs-stor. Update storage.cfg cluster-wide. VMs on D3 (currently
|
|
none of significance — it's 99% empty) keep working after remount.
|
|
|
|
## SSH access
|
|
|
|
SSH keys deployed to root on all hosts. Direct ssh/scp blocked in Crush bash;
|
|
use `deploy-check.sh` / `deploy-tuning.sh` wrapper patterns instead.
|
|
|
|
## Critical lessons (do NOT regress)
|
|
|
|
1. NFS `options` line in storage.cfg must NOT include `version=4.2` — Proxmox
|
|
sets NFS version separately. Use only `options nconnect=4,noatime`.
|
|
2. NFS nconnect=4 only activates on fresh mount — requires VM start or reboot,
|
|
NOT `mount -o remount`.
|
|
3. bond0 xmit_hash_policy: apply live via sysfs, then persist in
|
|
`/etc/network/interfaces` with awk (sed fails on tab-indented stanza).
|
|
4. Always shellcheck before shipping: `docker run --rm -v "$PWD:/mnt"
|
|
koalaman/shellcheck:stable --severity=style scripts/*.sh`
|
|
5. The hosts are **standalone Proxmox installs** (not a pvecm cluster), but
|
|
managed collectively via **Proxmox Datacenter Manager (PDM)**. PDM supports
|
|
VM migration between nodes via the UI. NFS exports are visible to all nodes;
|
|
local storage migration is done through PDM's "Storage Migrate" function.
|
|
6. **Storage philosophy:** NVMe/SSD is for k8s worker scratch + ultix-streaming
|
|
(developer workstation for "cluster of 1" pre-prod jobs). Spinning rust
|
|
hosts all infrastructure VMs (UCS, netinfra, LibreNMS, SIEM).
|
|
|
|
## Friday walkthrough (user action)
|
|
|
|
### Step 1: tsys5 storage cable + NVMe
|
|
1. Plug 2nd ethernet cable into tsys5 storage NIC
|
|
2. Verify: `cat /proc/net/bonding/bond0` — need "Number of ports: 2" + partner MAC
|
|
3. Apply: `echo "layer3+4" > /sys/class/net/bond0/bonding/xmit_hash_policy`
|
|
4. Install PCI NVMe, format as directory storage (local-only)
|
|
5. Reboot tsys5
|
|
|
|
### Step 2: tsys4 hardware install
|
|
1. Install PCIe NIC + add RAM (16→64 GB)
|
|
2. Update `/etc/network/interfaces` — replace `enx8cae4ccda926` with new NIC
|
|
3. Reboot tsys4 (PBS VM restarts — OK)
|
|
|
|
### Step 3: Post-hardware validation
|
|
1. `iperf-full-matrix.sh` — re-test all paths
|
|
2. `validate-fixes.sh pfv-tsys4` and `validate-fixes.sh pfv-tsys5`
|
|
3. Update PROJECT.md with post-hardware numbers
|
|
|
|
## Version control
|
|
|
|
This project is tracked in a local git repo (`main` branch). Use **atomic
|
|
commits with conventional commit messages** (e.g., `docs: add disk utilization
|
|
to storage section`, `feat: add tsys9 validation support`). Never push to
|
|
remote unless explicitly asked.
|
|
|
|
## Key scripts
|
|
|
|
| Script | Purpose |
|
|
|--------|---------|
|
|
| `scripts/check.sh` | Read-only data collector |
|
|
| `scripts/apply-tunings.sh` | Tier 0 tunings (dry-run/apply/rollback) |
|
|
| `scripts/fix-bond-nfs.sh` | Fix NFS options + bond hash |
|
|
| `validate-fixes.sh` | Validation of all applied changes |
|
|
| `iperf-full-matrix.sh` | Full iperf suite |
|
|
| `deploy-check.sh` | Deploy check.sh to hosts via SSH |
|