Commit Graph
3 Commits
Author SHA1 Message Date
reachableceo 88faf35ec3 docs: migrate all documentation to Discourse wiki topics
All knowledge docs (architecture, runbooks, references, audits, policies)
have been migrated to community.turnsys.com as wiki topics in the VP
TechOps category. Discourse is now the canonical source of truth for
documentation; git edit history no longer serves as the doc changelog.

37 .md files converted to short pointers linking to their Discourse
topics. AGENTS.md updated with new documentation workflow policy.
Code (scripts, configs, playbooks) remains authoritative in git.
2026-08-06 14:08:34 -05:00
mrcharles 24b08aa592 docs(audit): document observability stack + Saturday OAM day plan
Operator-defined monitoring architecture (no tool duplication):
- Beszel (Cloudron VPS): CPU/RAM/disk metrics on all Proxmox hosts via Tailscale
- Uptime Kuma (Cloudron VPS): up/down alerting via Pushover
- LibreNMS (VMID 104, on-prem): SNMP polling for switches (non-Tailsale gear)
- Home Assistant (VMID 100, on-prem): power/temp instrumentation + alerting

Saturday 2026-08-01 is designated OAM Day: goal is 100% instrumentation
coverage across all four tools + centralized syslog.

💘 Generated with Crush

Assisted-by: Crush:glm-5.2
2026-07-30 23:02:11 -05:00
mrcharles 26de312c42 docs(audit): full ground-truth audit — network, storage, compute, switches
Comprehensive end-to-end audit of the Proxmox cluster covering all three
layers (network, storage, compute) with live data gathered from all 7 hosts
+ 4 switches via SNMP, conman consoles, and LLDP.

Key findings documented in AUDIT-2026-07-30.md:

Network (cross-rack NFS bottleneck root-caused):
- core-sw01 LAG hash = layer-2-3 (no L4 ports) — NFS caps at 1 Gbps per
  host pair regardless of nconnect. tor3-stor already uses layer-2-3-4.
- core-sw01 ch1 member mismatch: g16 is UP but not in LAG; g17 is DOWN.
  Only 3 of 4 cross-rack links active.
- Both switches use static mode (no LACP) for the cross-rack trunk.
- tsys5 bond0 degraded (1 of 2 slaves, nic2 down).

Storage (major topology changes):
- tsys5 new NVMe (CT500P1 500GB) + relocated SK hynix SSD (512GB) — both
  installed but NOT mounted/configured yet.
- tsys4 D3 changed (SK hynix SSD → Seagate HDD, still USB tier-3).
- tsys4 /mnt/albert (Hitachi 2TB internal) + /mnt/backup (WD 5TB USB) added.
- tsys5 S3 unmounted (stale export), S4 gone (broken storage.cfg entry).
- Samsung 860 PRO SSD at 93% full.

Fleet consistency:
- tsys4 behind on PVE 9.1.5/kernel 6.17 (fleet: 9.2.5/7.0.14).
- tsys4/5 bond hash + nconnect + rmem/wmem inconsistent with tsys6/7.
- 10+ VMs using cache=writethrough on NFS (slow synchronous writes).

Gardening: updated STATUS.md, docmap.md, AGENTS.md key scripts table.
Superseded AUDIT-2026-07-29 as the authoritative audit.

💘 Generated with Crush

Assisted-by: Crush:glm-5.2
2026-07-30 23:01:43 -05:00