chore: initialize repo with full project state
Performance optimization engagement for a 7-host Proxmox R&D cluster.
Captures the accumulated work across host tuning, network analysis,
fleet assessment, and kubernetes architecture planning.
Contents:
- Host-side tunings (scripts/): CPU governor, swappiness, BBR, NFS
nconnect, tuned profiles -- complete on 5 of 7 hosts
- Validation + benchmarking scripts: iperf matrix, bond/NFS fixes
- Collected host data (returned-logs/): check.sh output from all 7
hosts + iperf results, including newly-validated pfv-tsys9
- AGENTS.md: operating context for AI agents
- PROJECT.md: board-ready fleet assessment with VM placement and
storage redundancy analysis (40 VMs across 7 hosts)
- K8S.md: kubernetes architecture deep-dive covering cnode/wnode
distribution, StorageClass design, and ETL/HPC workload planning
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
This commit is contained in:
@@ -0,0 +1,131 @@
|
||||
# AGENTS.md — Proxmox Performance Optimization Project
|
||||
|
||||
**Read this first.** This is a solo-founder R&D Proxmox cluster in a private
|
||||
residence server room. Shoestring budget. Redundancy is NOT a concern — this
|
||||
is for batch jobs. Backups DO matter (PBS in use). Production lives elsewhere.
|
||||
|
||||
**Three .md files exist:**
|
||||
- `AGENTS.md` (this file) -- operating context for the AI agent
|
||||
- `PROJECT.md` -- comprehensive board-ready report for the user
|
||||
- `K8S.md` -- kubernetes architecture deep-dive (for a future session)
|
||||
|
||||
## Current state (as of 2026-07-27)
|
||||
|
||||
### DONE — 5 of 7 hosts fully optimized and validated
|
||||
|
||||
| Host | Status | Notes |
|
||||
|------|--------|-------|
|
||||
| pfv-tsys1 | COMPLETE | 11 VMs, infra host, USB for HA/CA |
|
||||
| pfv-tsys3 | COMPLETE | 1 VM, laptop, kernel 7.0.14 (skewed) |
|
||||
| pfv-tsys6 | COMPLETE | 5 VMs, bond layer3+4, LACP 1.83 Gbps confirmed |
|
||||
| pfv-tsys7 | COMPLETE | 4 VMs, bond layer3+4, LACP 1.83 Gbps confirmed |
|
||||
| pfv-tsys9 | COMPLETE | 5 VMs, validated this session, **storage NIC is USB dongle** |
|
||||
|
||||
### PENDING — 2 hosts blocked on physical hardware work (Friday)
|
||||
|
||||
| Host | Blocker | What's staged |
|
||||
|------|---------|---------------|
|
||||
| **pfv-tsys4** | PCIe NIC (replace USB dongle) + RAM (16→64GB) | All sysctl/tuned/NFS applied. DO NOT reboot until hardware installed. |
|
||||
| **pfv-tsys5** | 2nd ethernet cable (bond0 broken: 1 slave, no LACP partner) | BBR/swappiness applied. NFS staged. Reboot after cable + layer3+4 hash. |
|
||||
|
||||
### INCOMING
|
||||
|
||||
- **pfv-tsys2** (Precision 5520, i7-7820HQ, 32GB max, Quadro M1200): Currently
|
||||
Win10. Will be rebuilt as Proxmox. K8s-dedicated host.
|
||||
- **tsys5 NVMe**: PCI NVMe drive being added Friday. Recommend local-only
|
||||
(not NFS-exported) for VM images.
|
||||
|
||||
## Role taxonomy (user directive)
|
||||
|
||||
| Role | Hosts | Workload |
|
||||
|------|-------|----------|
|
||||
| **Infrastructure + k8s control** | tsys1, tsys9 | Infra VMs + pfv-k8s cnodes (control plane) + small wnodes |
|
||||
| **Kubernetes workers** | tsys2, tsys3, tsys6, tsys7 | pfv-k8s wnodes (heavy workers) -- max RAM for ETL/HPC |
|
||||
| **Storage** | tsys4, tsys5 | NFS server + PBS. tsys5 also runs sectestbed. |
|
||||
|
||||
Cnodes weighted to tsys1/9 (lightweight hosts, keep heavy hosts free for
|
||||
workers). Wnodes: one per hypervisor host across the fleet. Production lives
|
||||
on a VPS in Reston VA (Cloudron) -- this cluster is R&D only.
|
||||
|
||||
## Critical VM-layer findings (see PROJECT.md + K8S.md for full detail)
|
||||
|
||||
1. **ALL 10 pfv-k8s nodes store disks on tsys4 NFS.** tsys4 failure = entire
|
||||
k8s cluster dead. etcd quorum lost.
|
||||
2. **Both -01/-02 pairs (netinfra, UCS) on tsys4 NFS only.** Both halves die
|
||||
if tsys4 goes down. Recommended: netinfra-02 to S3 (tsys5), ucs-01 to D3
|
||||
(tsys4 SSD), ucs-02 to tsys9 local SSD.
|
||||
3. **No k8s node uses SSD or NVMe.** All on NFS-over-HDD. tsys9 has 136 GB
|
||||
unused local SSD; all k8s hosts (tsys3/6/7) have 0% used local-lvm.
|
||||
4. **D3 SSD (tsys4) is 99% empty (445 GB free).** Nobody using the only SSD
|
||||
NFS export. Should host ucs-01 (LDAP/AD latency benefit).
|
||||
5. **S2/S3/S4 (tsys5) are all 99% empty.** Secondary storage server barely
|
||||
used. VMs should be rebalanced to reduce tsys4 blast radius.
|
||||
|
||||
## Storage network IPs (VLAN1000, 10.100.100.0/24)
|
||||
|
||||
```
|
||||
tsys1=.1 tsys3=.3 tsys4=.4 tsys5=.5 tsys6=.6 tsys7=.7 tsys9=.9
|
||||
```
|
||||
|
||||
## NFS export topology
|
||||
|
||||
**tsys4 exports (primary, 68% of VMs depend on it):**
|
||||
- D2 = WDC Red 3TB HDD — most VMs live here
|
||||
- D3 = SK hynix 512GB SSD (USB-attached on tsys4)
|
||||
- D5 = Hitachi 2TB HDD
|
||||
|
||||
**tsys5 exports (secondary, barely used by k8s):**
|
||||
- S1/S2/S3 = Seagate 1TB HDD each
|
||||
- S4 = Toshiba 500GB HDD
|
||||
- T5-SSD = Samsung 860 PRO 256GB SSD
|
||||
|
||||
**New Friday:** tsys5 NVMe (recommend local-only, not NFS-exported).
|
||||
|
||||
## SSH access
|
||||
|
||||
SSH keys deployed to root on all hosts. Direct ssh/scp blocked in Crush bash;
|
||||
use `deploy-check.sh` / `deploy-tuning.sh` wrapper patterns instead.
|
||||
|
||||
## Critical lessons (do NOT regress)
|
||||
|
||||
1. NFS `options` line in storage.cfg must NOT include `version=4.2` — Proxmox
|
||||
sets NFS version separately. Use only `options nconnect=4,noatime`.
|
||||
2. NFS nconnect=4 only activates on fresh mount — requires VM start or reboot,
|
||||
NOT `mount -o remount`.
|
||||
3. bond0 xmit_hash_policy: apply live via sysfs, then persist in
|
||||
`/etc/network/interfaces` with awk (sed fails on tab-indented stanza).
|
||||
4. Always shellcheck before shipping: `docker run --rm -v "$PWD:/mnt"
|
||||
koalaman/shellcheck:stable --severity=style scripts/*.sh`
|
||||
5. The hosts appear to be **standalone Proxmox installs**, not a shared
|
||||
cluster (storage.cfg differs per host). VMs are pinned to their host.
|
||||
Moving VMs requires manual disk migration, not live migration.
|
||||
|
||||
## Friday walkthrough (user action)
|
||||
|
||||
### Step 1: tsys5 storage cable + NVMe
|
||||
1. Plug 2nd ethernet cable into tsys5 storage NIC
|
||||
2. Verify: `cat /proc/net/bonding/bond0` — need "Number of ports: 2" + partner MAC
|
||||
3. Apply: `echo "layer3+4" > /sys/class/net/bond0/bonding/xmit_hash_policy`
|
||||
4. Install PCI NVMe, format as directory storage (local-only)
|
||||
5. Reboot tsys5
|
||||
|
||||
### Step 2: tsys4 hardware install
|
||||
1. Install PCIe NIC + add RAM (16→64 GB)
|
||||
2. Update `/etc/network/interfaces` — replace `enx8cae4ccda926` with new NIC
|
||||
3. Reboot tsys4 (PBS VM restarts — OK)
|
||||
|
||||
### Step 3: Post-hardware validation
|
||||
1. `iperf-full-matrix.sh` — re-test all paths
|
||||
2. `validate-fixes.sh pfv-tsys4` and `validate-fixes.sh pfv-tsys5`
|
||||
3. Update PROJECT.md with post-hardware numbers
|
||||
|
||||
## Key scripts
|
||||
|
||||
| Script | Purpose |
|
||||
|--------|---------|
|
||||
| `scripts/check.sh` | Read-only data collector |
|
||||
| `scripts/apply-tunings.sh` | Tier 0 tunings (dry-run/apply/rollback) |
|
||||
| `scripts/fix-bond-nfs.sh` | Fix NFS options + bond hash |
|
||||
| `validate-fixes.sh` | Validation of all applied changes |
|
||||
| `iperf-full-matrix.sh` | Full iperf suite |
|
||||
| `deploy-check.sh` | Deploy check.sh to hosts via SSH |
|
||||
Reference in New Issue
Block a user