KNEL/PFVCluster
5
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

merge: combine KNELServerBuild into PFVCluster

Browse Source 
Merge the KNELServerBuild repository (server provisioning, security
hardening, DNS cluster setup, test suite) into PFVCluster (Proxmox
cluster ops, performance tuning, fleet audit). Both histories are
preserved via --allow-unrelated-histories.

The two repos had no source-file collisions; only AGENTS.md and
.gitignore conflicted (both resolved by merging content from both).

Directory reorganization and doc gardening will follow in subsequent
commits.

🤖 Generated with [Crush](https://github.com/charmassociates/crush)

Assisted-by: GLM-5 via Crush <crush@charm.land>
This commit is contained in:
mrcharles
2026-07-28 11:23:22 -05:00
parent 1044224d02 b1088e8487
commit 132c0854d1
127 changed files with 15532 additions and 145 deletions
Download Patch File Download Diff File Expand all files Collapse all files
AGENTS.md
+51 -145
View File
@@ -1,157 +1,63 @@
# AGENTS.md — Proxmox Performance Optimization Project
# Agent Guidelines
**Read this first.** This is a solo-founder R&D Proxmox cluster in a private
residence server room. Shoestring budget. Redundancy is NOT a concern — this
is for batch jobs. Backups DO matter (PBS in use). Production lives elsewhere.
This repo combines two formerly-separate projects:
- **Server provisioning** (formerly KNELServerBuild): `provisioning/`, `tests/`,
`vendor/`, `dns-cluster-setup/`
- **Proxmox cluster ops** (formerly PFVCluster/perfopt): `perf/`, `netinfra/`,
`switches/`, `returned-logs/`
**Four .md files exist:**
- `AGENTS.md` (this file) -- operating context for the AI agent
- `PROJECT.md` -- comprehensive board-ready report for the user
- `K8S.md` -- kubernetes architecture deep-dive (for a future session)
- `TODO.md` -- pending user actions (tsys2 hardware commands, Friday plan)
## Repository Layout
## Current state (as of 2026-07-27)
- **Vendored framework**: `KNELShellFramework` lives at
`vendor/git@git.knownelement.com/29418/KNEL/KNELShellFramework/`. Its includes
are under `Framework-Includes/` there. Never assume `./Framework-Includes`
exists relative to the repo root.
- **Self-locating scripts**: All provisioning scripts derive their own location
via `BASH_SOURCE` and compute `PROJECT_ROOT_PATH` from it. They must never
depend on the current working directory. Run from anywhere.
- **Local config files are the source of truth**: Configs in
`provisioning/ConfigFiles/` are read with `cat`/`cp`. Do NOT re-introduce
`curl ${DL_ROOT}/...` downloads — that CDN is deprecated.
- **Non-bash agents**: Some files under `provisioning/Agents/` carry a `.sh`
extension but are PHP (e.g. `mysql.sh`, shebang `#!/usr/bin/php`). Syntax
checkers must skip these.
- **Proxmox hosts** are standalone installs managed via **PDM** (Proxmox
Datacenter Manager). SSH keys deployed to root on all hosts.
- **SSH in Crush**: Direct ssh/scp is blocked in the Crush bash environment.
Use the wrapper scripts: `tests/remote.sh`, `dns-cluster-setup/remote-dns.sh`,
or the `deploy-check.sh` / `deploy-tuning.sh` patterns.
### DONE — 5 of 7 hosts fully optimized and validated
## Git Commit Requirements
| Host | Status | Notes |
|------|--------|-------|
| pfv-tsys1 | COMPLETE | 11 VMs, infra host, USB for HA/CA |
| pfv-tsys3 | COMPLETE | 1 VM, laptop, kernel 7.0.14 (skewed) |
| pfv-tsys6 | COMPLETE | 5 VMs, bond layer3+4, LACP 1.83 Gbps confirmed |
| pfv-tsys7 | COMPLETE | 4 VMs, bond layer3+4, LACP 1.83 Gbps confirmed |
| pfv-tsys9 | COMPLETE | 5 VMs, validated this session, **storage NIC is USB dongle** |
1. **Commit atomically**: each logical change its own commit.
2. **Conventional commit format**: `feat(scope): desc`, `fix(scope): desc`,
`docs: desc`, `refactor(scope): desc`, `test(scope): desc`, `chore: desc`.
3. **Verbose messages**: title (50 chars max), blank line, body explaining
WHAT and WHY, footer with attribution.
### PENDING — 2 hosts blocked on physical hardware work (Friday)
## Autonomous Git Workflow
| Host | Blocker | What's staged |
|------|---------|---------------|
| **pfv-tsys4** | PCIe NIC (replace USB dongle) + RAM (16→64GB) | All sysctl/tuned/NFS applied. DO NOT reboot until hardware installed. |
| **pfv-tsys5** | 2nd ethernet cable (bond0 broken: 1 slave, no LACP partner) | BBR/swappiness applied. NFS staged. Reboot after cable + layer3+4 hash. |
### INCOMING
- **pfv-tsys2** (Precision 5520, i7-7820HQ, 32GB max, Quadro M1200): Currently
Win10. Will be rebuilt as Proxmox. K8s-dedicated host. **Hardware validated
2026-07-27: 2 SSDs (Samsung 960 PRO NVMe 512GB + Samsung 850 EVO SATA 1TB)
-- best local storage in fleet. Both NICs are USB dongles (ASIX + Realtek)
-- unavoidable on laptop, no PCIe NIC option.**
- **tsys5 NVMe**: PCI NVMe drive being added Friday. Recommend local-only
(not NFS-exported) for VM images.
## Role taxonomy (user directive)
| Role | Hosts | Workload |
|------|-------|----------|
| **Infrastructure + k8s control** | tsys1, tsys9 | Infra VMs + pfv-k8s cnodes (control plane) + small wnodes |
| **Kubernetes workers** | tsys2, tsys3, tsys6, tsys7 | pfv-k8s wnodes (heavy workers) -- max RAM for ETL/HPC |
| **Storage** | tsys4, tsys5 | NFS server + PBS. tsys5 also runs sectestbed. |
Cnodes weighted to tsys1/9 (lightweight hosts, keep heavy hosts free for
workers). Wnodes: one per hypervisor host across the fleet. Production lives
on a VPS in Reston VA (Cloudron) -- this cluster is R&D only.
## Critical VM-layer findings (re-audited 2026-07-27 21:50)
1. **4 of 5 cnodes still on tsys4 NFS** (cnode5 moved to tsys5 S2). Need 2
more moves (cnode3→S3, cnode4→S2) for etcd quorum survival. User has been
actively rebalancing via PDM -- storage distribution improved 90/10 to 73/27.
2. **Both -01/-02 pairs (netinfra, UCS) on tsys4 NFS.** TODO today: move
netinfra-02 to S3, ucs-02 to S2 (both tsys5 HDD).
3. **No k8s node uses SSD/NVMe yet.** tsys3 has 349 GB unused local NVMe
(Samsung PM961), tsys9 has 136 GB local SSD. Deferred to k8s session.
4. **tsys6/7 local-lvm is USB 2.0 portable HDD** (~30 MB/s). Slower than NFS.
Do NOT use for VM storage. User accepts OS-on-USB for these hosts.
5. **Storage philosophy:** NVMe/SSD = k8s scratch + ultix-streaming (dev
workstation). Spinning rust = all infrastructure VMs.
6. **Hosts are standalone, managed via PDM** (Proxmox Datacenter Manager).
VM migration between nodes is done through PDM UI, not manual disk copy.
## Storage network IPs (VLAN1000, 10.100.100.0/24)
```
tsys1=.1 tsys3=.3 tsys4=.4 tsys5=.5 tsys6=.6 tsys7=.7 tsys9=.9
```
## NFS export topology
**tsys4 exports (primary, overloaded):**
- D2 = WDC Red 3TB HDD — most VMs live here
- D5 = Hitachi 2TB HDD
- (sda Hitachi 1.8T at /mnt/albert — not NFS shared)
- (sdd WDC 1T — idle, unmounted, removable)
- (sdf WDC 4.5T SMR at /mnt/backup — PBS target)
**tsys5 exports (fast-tier hub -- consolidated Friday):**
- S1/S2/S3 = Seagate 1TB HDD each
- S4 = Toshiba 500GB HDD
- T5-SSD = Samsung 860 PRO 256GB SSD (existing)
- **D3 = SK hynix SC300 512GB SSD** (moving from tsys4 USB to tsys5 SAS Friday)
- **NVMe (local, new Friday)** — local-only, wnode-tsys5 boot + HPC scratch
**tsys5 hardware:** LSI SAS1068E (8-port, 5 free) + Intel ICH10 SATA (6-port,
2 free) + 2x Renesas USB 3.0 xHCI. Plenty of room for the SSD + NVMe.
**Friday change:** D3 export repoints from pfv-tsys4-nfs-stor to
pfv-tsys5-nfs-stor. Update storage.cfg cluster-wide. VMs on D3 (currently
none of significance — it's 99% empty) keep working after remount.
## SSH access
SSH keys deployed to root on all hosts. Direct ssh/scp blocked in Crush bash;
use `deploy-check.sh` / `deploy-tuning.sh` wrapper patterns instead.
## Critical lessons (do NOT regress)
1. NFS `options` line in storage.cfg must NOT include `version=4.2` — Proxmox
sets NFS version separately. Use only `options nconnect=4,noatime`.
2. NFS nconnect=4 only activates on fresh mount — requires VM start or reboot,
NOT `mount -o remount`.
3. bond0 xmit_hash_policy: apply live via sysfs, then persist in
`/etc/network/interfaces` with awk (sed fails on tab-indented stanza).
4. Always shellcheck before shipping: `docker run --rm -v "$PWD:/mnt"
koalaman/shellcheck:stable --severity=style scripts/*.sh`
5. The hosts are **standalone Proxmox installs** (not a pvecm cluster), but
managed collectively via **Proxmox Datacenter Manager (PDM)**. PDM supports
VM migration between nodes via the UI. NFS exports are visible to all nodes;
local storage migration is done through PDM's "Storage Migrate" function.
6. **Storage philosophy:** NVMe/SSD is for k8s worker scratch + ultix-streaming
(developer workstation for "cluster of 1" pre-prod jobs). Spinning rust
hosts all infrastructure VMs (UCS, netinfra, LibreNMS, SIEM).
## Friday walkthrough (user action)
### Step 1: tsys5 storage cable + NVMe
1. Plug 2nd ethernet cable into tsys5 storage NIC
2. Verify: `cat /proc/net/bonding/bond0` — need "Number of ports: 2" + partner MAC
3. Apply: `echo "layer3+4" > /sys/class/net/bond0/bonding/xmit_hash_policy`
4. Install PCI NVMe, format as directory storage (local-only)
5. Reboot tsys5
### Step 2: tsys4 hardware install
1. Install PCIe NIC + add RAM (16→64 GB)
2. Update `/etc/network/interfaces` — replace `enx8cae4ccda926` with new NIC
3. Reboot tsys4 (PBS VM restarts — OK)
### Step 3: Post-hardware validation
1. `iperf-full-matrix.sh` — re-test all paths
2. `validate-fixes.sh pfv-tsys4` and `validate-fixes.sh pfv-tsys5`
3. Update PROJECT.md with post-hardware numbers
## Version control
This project is tracked in a local git repo (`main` branch). Use **atomic
commits with conventional commit messages** (e.g., `docs: add disk utilization
to storage section`, `feat: add tsys9 validation support`). Never push to
remote unless explicitly asked.
Agents are authorized to commit AND push autonomously. After each logical unit
of work: stage, commit, push to `origin/main`. Group changes so each commit is
coherent on its own.
## Key scripts
| Script | Purpose |
|--------|---------|
| `scripts/check.sh` | Read-only data collector |
| `scripts/apply-tunings.sh` | Tier 0 tunings (dry-run/apply/rollback) |
| `scripts/fix-bond-nfs.sh` | Fix NFS options + bond hash |
| `validate-fixes.sh` | Validation of all applied changes |
| `iperf-full-matrix.sh` | Full iperf suite |
| `deploy-check.sh` | Deploy check.sh to hosts via SSH |
| `provisioning/SetupNewSystem.sh` | Full server provisioning (packages, hardening, 2FA) |
| `tests/vm-validation.sh` | End-to-end deploy + validate on sandbox VM |
| `tests/run-tests.sh` | Project test suite (unit/security/validation) |
| `dns-cluster-setup/setup.sh` | Technitium DNS cluster replication |
| `perf/deploy-check.sh` | Deploy read-only data collector to hosts |
| `perf/deploy-tuning.sh` | Deploy perf tunings to hosts |
| `perf/validate-fixes.sh` | Validate applied tuning changes |
| `perf/iperf-full-matrix.sh` | Full iperf throughput suite |
## Project context
This is a solo-founder R&D Proxmox cluster in a private residence. Shoestring
budget. Redundancy is not a concern for the R&D cluster. Backups DO matter
(PBS in use). Production lives on a VPS in Reston VA (Cloudron). See
`docs/PROJECT.md` for the comprehensive fleet report and `docs/TODO.md` for
pending hardware work.