Commit Graph
5 Commits
Author SHA1 Message Date
mrcharles c115ea8ea1 feat(proxmox): add e1000e offload fix script for Intel NIC hang bug
Script disables TSO/GRO/GSO/tx/rx offload on all e1000e NICs to prevent
the known "hardware unit hang" bug affecting Intel I217-LM, I219-LM, and
82571EB controllers under Proxmox. Applies live + installs systemd
service for persistence.

Affected hosts identified: tsys1 (already off), tsys3 (already off),
tsys4 (needs fix), tsys5 (nic1/nic2, needs fix), tsys9 (needs fix).
tsys6/7 use Broadcom (bnx2/tg3), not affected.

Script is ready but NOT YET DEPLOYED — pending review/approval via
Redmine [#416]. Initial attempt to deploy did not execute properly.

Based on archive/provisioning/Dell/Server/fixeth.sh.
Refs: https://forum.proxmox.com/threads/e1000-driver-hang.58284/

💘 Generated with Crush

Assisted-by: Crush:glm-5.2
2026-08-10 11:28:55 -05:00
mrcharles 96e01987c1 feat(perf): add reusable perf-matrix.sh test harness + iperf3 instrumentation
perf-matrix.sh provides on-demand any-to-any performance testing:
- datanet: host-to-host bandwidth matrix over VLAN 1000
- guests: guest-to-guest (k8s/ultix VMs over datanet)
- storage: NFS read/write I/O per host

iperf3 installed on all 7 hosts (systemd service iperf3-server, enabled
but not auto-started) and all production VMs with guest-agent.

Also: tsys3 datanet persistence fix (USB NIC late-enumerate systemd
service), 4 missing reverse DNS zones created, netbird/ultix-mini/
ultix-sidecar/ultix-highside DNS records added (DNS now fully
consistent across all Tailscale nodes). [#413]

💘 Generated with Crush

Assisted-by: Crush:glm-5.2
2026-08-10 11:03:36 -05:00
mrcharles 75b8cb5d9b docs: perf testing complete + tsys3 datanet fix + Kuma DNS aliases
Perf testing results (#413):
- Host-to-host datanet: all 7 hosts at ~900-940 Mbps (1Gbps line rate)
- Guest-to-guest: all k8s nodes at ~920-1040 Mbps over VLAN 1000
- Storage I/O: NFS writes 27-65 MB/s (spinning disk), reads cached

Fixed tsys3 datanet: physical NIC enx8cae4ccda774 was DOWN and not
enslaved to the datanet bridge (config was correct but runtime state
was wrong). Enslaved manually, verified connectivity.

Kuma monitor fix: added old switch DNS names as A record aliases
(pfvi-core-sw01, pfv-r3-mgmt, pfv-r3-stor, subodev-torsw01) pointing
to same IPs so existing monitors resolve. Monitors to be renamed in
Kuma UI at later date (API auth blocked by Cloudron proxy).

💘 Generated with Crush

Assisted-by: Crush:glm-5.2
2026-08-10 10:45:10 -05:00
mrcharles f9954fd9b4 refactor(switches): standardize all switch names to <site>-<rack>-<function>-<NN>
Rename all 5 switches across all layers (DNS, DHCP, device hostname,
conman console, ser2net, udev symlinks, repo files) to follow the
<site>-<rack>-<function>-<count> convention:
  pfv-core-sw01    → pfv-r5-core-01     (rack 5 core)
  pfv-tor3-mgmt    → pfv-r3-tor-mgmt-01 (rack 3 mgmt TOR)
  pfv-tor3-stor    → pfv-r3-tor-stor-01 (rack 3 storage TOR)
  pfv-r2-tor-top   → pfv-r2-tor-01      (rack 2 TOR)
  subodev-torsw01  → pfv-r6-mgmt-01     (rack 6 mgmt)

Dead switch (pfv-r2-sw, port 2007) removed from mapping.txt.
pfv-rrinfra-rtr unchanged (router, not a switch).

Device hostnames changed via conman. Configs regenerated on pfv-tsys4.
All .cmds files, validate-conman.sh, and AGENTS.md updated. [#369]

💘 Generated with Crush

Assisted-by: Crush:glm-5.2
2026-08-10 10:29:13 -05:00
mrcharles ec6e228b05 chore: adopt TSYSGroupAIOS framework (git hooks, rules engine, SoR policy)
Brings in the enforcement layer from ~/daytoday/meta:
- Makefile, scripts/ (check-rules.sh, setup-hooks.sh, pre-commit/pre-push,
  docker-run.sh, garden.sh, lib/common.sh)
- WORKING.md, questions-v1.md, .env.example
- Git hooks installed (pre-commit: fast audit, pre-push: full audit)

Fixes to pass rule audit:
- Pin Pi-hole/autoheal Docker images (no :latest tags)
- Fix shellcheck SC2001 in probe-vm-dns.sh
- Prune vendor/ and archive/ from shellcheck + Discourse pointer checks
- Add Quick Start, Enforcement Model, Task Tracking, Working Style
  sections to AGENTS.md from template

💘 Generated with Crush

Assisted-by: Crush:glm-5.2
2026-08-07 12:29:36 -05:00