diff --git a/AGENTS.md b/AGENTS.md index 6eb6522..d807c48 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,10 +1,32 @@ # Agent Guidelines -**Top-level files:** [`README.md`](README.md) (project overview), -[`STATUS.md`](STATUS.md) (ticket index + infra summary), -[`docs/docmap.md`](docs/docmap.md) (documentation index). All work is -tracked in [Redmine](https://projects.knownelement.com). Everything else -lives in subdirectories. +## Documentation policy (IMPORTANT) + +**Discourse is the canonical source of truth for all knowledge documentation.** +Knowledge docs (architecture, runbooks, references, audits, policies) have been +migrated to [community.turnsys.com](https://community.turnsys.com/c/vp-techops) +as wiki topics in the **VP TechOps** category. + +All `.md` files in this repo (except `AGENTS.md` and `LICENSE`) are now +**pointers** that link to their corresponding Discourse topic. **Do not update +documentation content in git** — edit the Discourse wiki topic instead. Git +edit history no longer serves as the documentation changelog; Discourse +preserves wiki edit history automatically. + +Code (scripts, configs, playbooks) still lives in git as the source of truth +for executables. Only *documentation* moved to Discourse. + +## Top-level files + +All `.md` files now point to Discourse. The key pointers: + +| File | Points to | Discourse topic | +|------|-----------|-----------------| +| `README.md` | Project overview | [#296](https://community.turnsys.com/t/296) | +| `STATUS.md` | Ticket index + infra summary | [#297](https://community.turnsys.com/t/297) | +| `docs/docmap.md` | Documentation index | [#296](https://community.turnsys.com/t/296) | + +Work tracking stays in [Redmine](https://projects.knownelement.com). ## Repository Layout @@ -71,37 +93,28 @@ checklists, or TODOs in repo files. Use Redmine tickets instead. Example: `[#367] Rebuilt k3s control plane after cnode wipe`. 3. **When work completes**, update the ticket: set done ratio to 100%, add a note describing what was done and where the code lives, then close. -4. **STATUS.md is a read-only snapshot** — it contains a ticket index and - infrastructure summary only. Do not add checklists or status items. +4. **Operations Status** lives on Discourse ([topic #297](https://community.turnsys.com/t/297)) — update that wiki topic if the infrastructure summary needs refreshing. The `STATUS.md` file in git is now a pointer only. 5. **Link code to tickets** — ticket descriptions and notes should reference the relevant file paths in this repo (e.g., `dcinfra/ups/`). -## Automatic Gardening Protocol +## Documentation Workflow -**Docs and code must be kept in sync.** After any work session, an agent MUST: +**Discourse is the source of truth for all knowledge docs.** After any work +session, an agent MUST: -1. **Update [`STATUS.md`](STATUS.md)** — update the ticket index table if - tickets were created or closed. This file is a read-only snapshot; - do not add inline status tracking (use Redmine). -2. **Update [`docs/docmap.md`](docs/docmap.md)** — if a doc was added, - removed, or substantively changed, update the table and "Last Reviewed" - date. -3. **Grep for stale paths** — `grep -rn 'old/path' --include='*.md'` after - any rename or restructure. Fix all references in the same commit. -4. **Verify new docs are linked** — every new `.md` file must appear in - [`docs/docmap.md`](docs/docmap.md) and be linked from at least one other - doc. -5. **If a new top-level directory was created, update ALL directory listings:** - - [`README.md`](README.md) → "Directory Structure" table - - [`AGENTS.md`](AGENTS.md) → "Repository Layout" code block - - [`AGENTS.md`](AGENTS.md) → "Key Scripts" table (if the directory has - an entrypoint script) - Missing any one of these is a protocol violation. -6. **Self-audit before commit.** Before committing, run: - ``` - grep -lE 'new_dir_name' README.md AGENTS.md docs/docmap.md STATUS.md - ``` - Every new top-level directory must appear in all four files. +1. **Update Discourse wiki topics** — if infrastructure facts changed (new + VM, IP change, host retired), edit the relevant wiki topic at + [community.turnsys.com/c/vp-techops](https://community.turnsys.com/c/vp-techops). +2. **Update the Operations Status topic** ([#297](https://community.turnsys.com/t/297)) + if tickets were opened or closed. +3. **Grep for stale paths in code** — `grep -rn 'old/path' --include='*.sh'` + after any rename or restructure. Fix all references in the same commit. +4. **If a new top-level directory was created**, update: + - `AGENTS.md` → "Repository Layout" code block + - `AGENTS.md` → "Key Scripts" table (if it has an entrypoint script) + - Create a new Discourse wiki topic for any documentation +5. **Self-audit before commit.** Code changes must be internally consistent. + Documentation changes go to Discourse, not git. ## Key Scripts @@ -121,12 +134,16 @@ checklists, or TODOs in repo files. Use Redmine tickets instead. ## Key Docs -→ **See [`docs/docmap.md`](docs/docmap.md) for the full documentation index.** +→ **All documentation lives on Discourse:** + [community.turnsys.com/c/vp-techops](https://community.turnsys.com/c/vp-techops) + +→ All `.md` files in this repo are pointers to Discourse topics. ## Project Context Solo-founder R&D Proxmox cluster in a private residence. Shoestring budget. -Production lives on a Cloudron VPS in Reston VA. See -[`STATUS.md`](STATUS.md) for the ticket index and infrastructure summary. +Production lives on a Cloudron VPS in Reston VA. See the +[Operations Status topic](https://community.turnsys.com/t/297) for the ticket +index and infrastructure summary. All work is tracked in [Redmine](https://projects.knownelement.com) (version: Potential to Kinetic Ready, due 2026-09-30). diff --git a/README.md b/README.md index 08cfb1d..5c58349 100644 --- a/README.md +++ b/README.md @@ -1,49 +1,10 @@ -# PFVCluster +# README.md -Unified infrastructure repo for the Known Element Enterprises Proxmox R&D cluster. - -**[→ Current Status](STATUS.md)** · **[→ Documentation Index](docs/docmap.md)** · **[→ Agent Guidelines](AGENTS.md)** - -## Directory Structure - -| Directory | Description | -|-----------|-------------| -| [`dcinfra/`](dcinfra/) | Data-center infrastructure: PDU (`powerman/`), serial console (`console/`), UPS (`ups/`) | -| [`netinfra/`](netinfra/) | DNS/NTP/DHCP setup + audit scripts, DNS cluster replication (`dns-cluster-setup/`), switch captures (`switches/`), DHCP config (`dhcp/`) | -| [`k8s/`](k8s/) | k3s cluster setup scripts (HA control plane over Tailscale) + co-located docs | -| [`proxmox/`](proxmox/) | Proxmox fleet docs (hardware audit, capacity, storage, k8s host planning) + performance tuning (`perf/`) | -| [`awx/`](awx/) | Ansible AWX deployment (k3s + AWX Operator) | -| [`tests/`](tests/) | Test suite + VM validation harness + `remote.sh` SSH chokepoint | -| [`docs/`](docs/) | Server-build docs, documentation index ([docmap](docs/docmap.md)), and archive | -| [`archive/`](archive/) | Historical/superseded code (provisioning → replaced by KNELIAC project) | -| [`vendor/`](vendor/) | Vendored KNELShellFramework | - -## Quick Start - -### Provision a new server -```bash -sudo bash provisioning/SetupNewSystem.sh -``` - -### Validate on the sandbox VM -```bash -VM_ID=6000 ./tests/vm-validation.sh all -``` - -### Deploy DNS cluster -```bash -cd dns-cluster-setup/ && ./setup.sh all -``` - -### Deploy perf tunings -```bash -cd perf/ && ./deploy-tuning.sh -``` - -## Architecture - -- **Proxmox hosts**: 7 standalone PVE installs managed via PDM -- **DNS**: Technitium (authoritative) + Pi-hole (recursive) on pfv-netinfra-01/02 -- **NTP**: pfv-netinfra-01/02 (redundant, LAN IPs, stratum 2/3) -- **Production**: Cloudron VPS in Reston VA (this cluster is R&D only) -- **Backups**: Proxmox Backup Server (PBS) +> **Documentation moved to Discourse — the canonical source of truth.** +> +> **Project overview, architecture, quick start** +> +> **Read it here:** https://community.turnsys.com/t/296 +> +> *Migrated 2026-08-06. This file is kept as a pointer for git-browsing context. +> Do not update content here — edit the Discourse wiki topic instead.* diff --git a/STATUS.md b/STATUS.md index 5d5d9df..118bbc9 100644 --- a/STATUS.md +++ b/STATUS.md @@ -1,78 +1,10 @@ -# Project Status +# STATUS.md -> **Redmine is the system of record for all work tracking.** -> https://projects.knownelement.com — Version: **Potential to Kinetic Ready** (due 2026-09-30) +> **Documentation moved to Discourse — the canonical source of truth.** > -> This file is now a **read-only snapshot + ticket index**. Do not add -> checklists or status items here. All work is tracked as Redmine tickets. -> When work is done, update the ticket (done ratio, notes, close). - -> **Last updated:** 2026-08-06 — Pi-hole hardening config committed (`netinfra/pihole/`, [#376]). - -## Current State: MIXED (infra stable; k8s needs rebuild) - -Provisioning, DNS primary (netinfra-01), Proxmox fleet, and monitoring stack -are operational. **k8s control plane is DOWN** (cnode wiped, no etcd quorum). - -## Completed Work (tickets closed @ 100%) - -| Ticket | Component | Code location | -|--------|-----------|---------------| -| [#334](https://projects.knownelement.com/issues/334) | NTP synchronization (redundant stratum 3) | `netinfra/` | -| [#340](https://projects.knownelement.com/issues/340) | UPS management — NUT on pfv-tsys1 (APC Smart-UPS) | `dcinfra/ups/` | -| [#357](https://projects.knownelement.com/issues/357) | DNS cluster — Technitium primary/secondary + DHCP failover | `netinfra/dns-cluster-setup/` | -| [#358](https://projects.knownelement.com/issues/358) | Proxmox perf tuning (5 of 7 hosts) | `proxmox/perf/` | -| [#359](https://projects.knownelement.com/issues/359) | PDU management — powerman on pfv-tsys1 | `dcinfra/powerman/` | -| [#360](https://projects.knownelement.com/issues/360) | Console management — ser2net + conman on pfv-tsys4 | `dcinfra/console/` | -| [#361](https://projects.knownelement.com/issues/361) | AWX deployment — k3s + AWX Operator | `awx/` | -| [#362](https://projects.knownelement.com/issues/362) | k8s control plane build — k3s HA (initial, now needs rebuild) | `k8s/` | -| [#363](https://projects.knownelement.com/issues/363) | tsys5 storage configuration (NVMe + SSD) | `proxmox/docs/` | -| [#364](https://projects.knownelement.com/issues/364) | tsys4 PVE upgrade to 9.2.5 | `proxmox/docs/` | -| [#365](https://projects.knownelement.com/issues/365) | Repo merge — KNELServerBuild into PFVCluster | repo root | -| [#366](https://projects.knownelement.com/issues/366) | Server provisioning pipeline (SetupNewSystem.sh → KNELIAC) | `archive/provisioning/`, KNELIAC | - -## Open Work (tickets in progress or queued) - -| Ticket | Component | Notes | -|--------|-----------|-------| -| [#376](https://projects.knownelement.com/issues/376) | DNS cluster up/down alerts + Pi-hole hardening committed | Config now in `netinfra/pihole/`; flapping source under investigation | -| [#367](https://projects.knownelement.com/issues/367) | k8s cluster rebuild (cnode wiped, no quorum) | Blocks #368 | -| [#368](https://projects.knownelement.com/issues/368) | k8s worker nodes join cluster | Blocked by #367 | -| [#369](https://projects.knownelement.com/issues/369) | Switch config fixes (core-sw01 LAG, LACP, member list) | Onsite Friday | -| [#370](https://projects.knownelement.com/issues/370) | Vault — migrate all API keys (HIGH priority) | | -| [#371](https://projects.knownelement.com/issues/371) | AWX HTTPS/TLS configuration | Depends on #342 (PKI/SSL) | -| [#372](https://projects.knownelement.com/issues/372) | Tripp Lite UPS fix (physical reseat) | Onsite Friday | -| [#373](https://projects.knownelement.com/issues/373) | pfv-r2-sw console fix (physical, onsite) | Onsite Friday | -| [#374](https://projects.knownelement.com/issues/374) | PDU outlet renaming | | -| [#375](https://projects.knownelement.com/issues/375) | Monitoring expansion (unpoller, netdisco, weathermap, smokeping) | Relates to #337, #343 | -| [#311](https://projects.knownelement.com/issues/311) | OAM: Security Hardening (CMMC/STIG) | KNELIAC roles deployed, expansion ongoing | -| [#335](https://projects.knownelement.com/issues/335) | OAM: Centralized syslog to Wazuh | Wazuh agents deployed, fleet rollout pending | -| [#342](https://projects.knownelement.com/issues/342) | OAM: PKI/SSL — tsys-ca VM + HSM CA + ACME | Vault unconfigured, AWX HTTP-only | -| [#343](https://projects.knownelement.com/issues/343) | OAM: Monitoring coverage matrix | Coverage gaps documented | - -## Known Issues (tracked in tickets) - -All known infrastructure issues are captured in ticket descriptions and notes. -See [#369](https://projects.knownelement.com/issues/369) for switch issues, -[#372](https://projects.knownelement.com/issues/372) for Tripp Lite UPS, -[#373](https://projects.knownelement.com/issues/373) for dead console port. - -## Infrastructure Summary - -| Component | Details | -|-----------|---------| -| Proxmox hosts | 7 standalone PVE, managed via PDM (tsys8 retired) | -| Storage servers | tsys4 (NFS: D1/D2/D3/D5 + /mnt/albert + /mnt/backup), tsys5 (NFS: S1/S2 + local SSD + NVMe) | -| Switches | core-sw01 (PowerConnect 5448, rack5), tor3-stor/mgmt (PowerConnect 5324, rack3) | -| DNS primary | pfv-netinfra-01 (192.168.3.252) — Technitium + Pi-hole | -| DNS secondary | pfv-netinfra-02 (192.168.3.253) — Technitium + Pi-hole | -| NTP | pfv-netinfra-01/02 (redundant, stratum 3) | -| Sandbox VM | sectestbed-sandbox (VMID 6000 on pfv-tsys5) | -| Backup | Proxmox Backup Server (PBS) | -| k8s control plane | **DOWN** — 3-node k3s HA, needs rebuild ([#367](https://projects.knownelement.com/issues/367)) | -| PDU | Cyclades PM10i via powerman on pfv-tsys1 (port 10101) | -| AWX | tsys-awx: AWX 24.6.1 on k3s (http://tsys-awx.knel.net:80) | -| Console | 7 switch consoles via ser2net + conman on pfv-tsys4 | -| UPS | APC Smart-UPS C 1500 via NUT on pfv-tsys1 | -| Production | Cloudron VPS, Reston, VA (this cluster is R&D only) | -| Config management | KNELIAC (Ansible) — `/home/reachableceo/projects/KNELIAC` | +> **Ticket index + infrastructure summary** +> +> **Read it here:** https://community.turnsys.com/t/297 +> +> *Migrated 2026-08-06. This file is kept as a pointer for git-browsing context. +> Do not update content here — edit the Discourse wiki topic instead.* diff --git a/awx/README.md b/awx/README.md index e4d83b5..2babde9 100644 --- a/awx/README.md +++ b/awx/README.md @@ -1,98 +1,10 @@ -# AWX — tsys-awx.knel.net +# awx/README.md -> **Redmine:** [#361](https://projects.knownelement.com/issues/361) (deployment, closed) · [#371](https://projects.knownelement.com/issues/371) (HTTPS/TLS, open) - -Ansible AWX deployment on a dedicated k3s single-node VM. - -**[→ Deployment Plan](DEPLOYMENT.md)** · **[→ Status](../../STATUS.md)** - -## Access - -| | | -|---|---| -| URL | `http://tsys-awx.knel.net` (LAN) or `http://100.91.39.53` (Tailscale) | -| Port | 80 (LoadBalancer via k3s ServiceLB) | -| User | `admin` | -| Password | `REDACTED_PASSWORD` (fleet standard) | - -## Architecture - -``` -tsys-awx (VMID 600 on pfv-tsys6) -├── k3s v1.36.2+k3s1 (single-node, no Traefik) -├── AWX Operator 2.19.1 (manages the AWX lifecycle) -└── AWX instance "tsys-awx" - ├── AWX web (UI/API) - ├── AWX task (Celery workers) - ├── PostgreSQL 15 (bundled, 8 GiB PVC on local-path) - ├── Redis (ephemeral) - └── ServiceLB (binds port 80 to host IP) -``` - -## VM Resources - -| Resource | Value | -|----------|-------| -| Host | pfv-tsys6 (2× Xeon E5530, 126 GB RAM) | -| vCPU | 4 (2 sockets × 2 cores) | -| RAM | 12 GB | -| Disk | 60 GB (D5 on tsys4 NFS) | -| OS | Debian 13 trixie | -| Network | 192.168.3.200 (LAN), 100.91.39.53 (Tailscale) | - -## Files - -| File | Purpose | -|------|---------| -| [`install-k3s.sh`](install-k3s.sh) | Install k3s single-node (no Traefik) | -| [`deploy-awx.sh`](deploy-awx.sh) | Deploy AWX Operator 2.19.1 + AWX instance | -| [`verify-awx.sh`](verify-awx.sh) | Verify deployment status + access | -| [`namespace.yaml`](namespace.yaml) | Kubernetes namespace manifest | -| [`awx-instance.yaml`](awx-instance.yaml) | AWX Custom Resource (CR) manifest | - -## Deployment (from scratch) - -```bash -# 1. Push scripts to VM -for f in awx/install-k3s.sh awx/deploy-awx.sh awx/verify-awx.sh \ - awx/namespace.yaml awx/awx-instance.yaml; do - VM_IP=100.91.39.53 VM_USER=localuser bash tests/remote.sh vm \ - "sudo tee /tmp/awx/$(basename $f) > /dev/null" < "$f" -done - -# 2. Install k3s -VM_IP=100.91.39.53 VM_USER=localuser bash tests/remote.sh vm \ - "sudo bash /tmp/awx/install-k3s.sh" - -# 3. Deploy AWX -VM_IP=100.91.39.53 VM_USER=localuser bash tests/remote.sh vm \ - "sudo bash /tmp/awx/deploy-awx.sh" - -# 4. Verify -VM_IP=100.91.39.53 VM_USER=localuser bash tests/remote.sh vm \ - "bash /tmp/awx/verify-awx.sh" -``` - -## Operating - -```bash -# kubectl access -sudo k3s kubectl -n awx get pods -sudo k3s kubectl -n awx logs -f deployment/awx-operator-controller-manager - -# Restart AWX instance -sudo k3s kubectl -n awx delete awx tsys-awx # operator will recreate - -# Get admin password -sudo k3s kubectl -n awx get secret awx-admin-password -o jsonpath='{.data.password}' | base64 -d -``` - -## Known Issues - -1. **kube-rbac-proxy image**: The AWX Operator 2.19.1 references - `gcr.io/kubebuilder/kube-rbac-proxy:v0.15.0` which was removed from GCR. - The deploy script patches this to `quay.io/brancz/kube-rbac-proxy:v0.15.0`. -2. **Image pulls are slow**: First deployment pulls ~2 GB of container images. - Postgres image alone is ~400 MB and takes several minutes. -3. **Disk space**: AWX + k3s uses ~10 GB of the 60 GB disk. PostgreSQL data - grows over time — monitor `df -h /`. +> **Documentation moved to Discourse — the canonical source of truth.** +> +> **Ansible AWX deployment on k3s** +> +> **Read it here:** https://community.turnsys.com/t/302 +> +> *Migrated 2026-08-06. This file is kept as a pointer for git-browsing context. +> Do not update content here — edit the Discourse wiki topic instead.* diff --git a/dcinfra/console/README.md b/dcinfra/console/README.md index a106078..1f4c39a 100644 --- a/dcinfra/console/README.md +++ b/dcinfra/console/README.md @@ -1,132 +1,10 @@ -# Console Management (ser2net + conman) - -> **Redmine:** [#360](https://projects.knownelement.com/issues/360) (deployment, closed) · [#373](https://projects.knownelement.com/issues/373) (pfv-r2-sw fix, open) - -Network-accessible serial console management for all production network -switches and routers, running on **pfv-tsys4** (storage server). - -## Architecture - -``` -USB-DB9 adapters → udev symlinks (/dev/consoles/) → ser2net telnet(rfc2217) TCP → conman (logging + multiplexing) -``` - -ser2net owns the physical serial devices and exposes them on TCP ports -using the **telnet(rfc2217) protocol** bound to the **Tailscale interface -only** (`100.70.77.93:200X`). conman connects to those TCP ports via -telnet for session logging, output capture, and multi-user console -sharing. - -**Why telnet(rfc2217)?** The serial devices send ` - ` (LF+CR) line -endings instead of standard ` -`. Raw TCP transport caused conman's -telnet NVT to strip bare CR characters, producing stair-stepped output. -With telnet(rfc2217) on both sides, binary mode is negotiated and CR/LF -translation is handled correctly by the telnet layer. - -**conman and ser2net do NOT share ports** — only one process can open a -serial device at a time. ser2net owns the physical device; conman connects -over TCP. - -## The USB Enumeration Problem (SOLVED) - -The 9 Prolific USB-to-DB9 adapters (`067b:2303`) on pfv-tsys4 have **no -unique USB serial numbers** and get assigned `/dev/ttyUSB0-8` based on -enumeration order, which shifts on every boot. This made the old -`/root/conmap` + manual `screen` workflow break after every reboot. - -**Fix:** udev rules pin each adapter by its **ID_PATH** (physical USB port -topology), which is stable across reboots regardless of enumeration order. -Each adapter gets a named symlink in `/dev/consoles/` that never changes. - -The udev rules are generated from `mapping.txt`, which maps each adapter's -ID_PATH to a console name and TCP port. To re-map after physically moving -an adapter, update `mapping.txt` and re-run `setup.sh`. - -**Fallback:** if udev trigger doesn't create symlinks for already-discovered -devices (common on first run), `setup.sh` creates them manually by matching -ID_PATH. On subsequent boots, udev creates them automatically. - -## Port Assignments - -| TCP Port | Console Name | ID_PATH | Description | -|----------|-------------|---------|-------------| -| 2001 | pfv-core-sw01 | usb-0:1.5.4.4 | Dell PowerConnect 5448 (core switch) | -| 2002 | pfv-tor3-mgmt | usb-0:1.6.3.1 | Rack 3 management TOR switch | -| 2003 | pfv-tor3-stor | usb-0:1.6.3.3.2 | Rack 3 storage TOR switch | -| 2004 | pfv-rrinfra-rtr | usb-0:1.6.3.3.1 | Cisco router (rrinfra) | -| 2005 | pfv-r2-tor-top | usb-0:1.6.3.3.3 | Rack 2 top-of-rack switch | -| 2006 | subodev-torsw | usb-0:1.5.4.1 | Suborbital device TOR switch | -| 2007 | pfv-r2-sw | usb-0:1.6.3.2 | Rack 2 old Dell switch | - -All ports listen on the Tailscale IP (`100.70.77.93`) using telnet(rfc2217). - -## Scripts - -| Script | Purpose | -|--------|---------| -| [`mapping.txt`](mapping.txt) | Source of truth: TCP port ↔ ID_PATH ↔ name ↔ baud | -| [`generate-config.sh`](generate-config.sh) | Generates udev rules, ser2net.yaml, conman.conf from mapping.txt | -| [`setup.sh`](setup.sh) | Full deploy: generate configs, create symlinks, restart services | -| [`discover.sh`](discover.sh) | Read-only discovery of USB adapters, existing config, services | - -## Usage - -### Connect to a console - -**Primary method — conman client (with logging + multiplexing):** - -```bash -# From any Tailscale-connected workstation: -conman -d pfv-tsys4:7890 -f pfv-core-sw01 # connect to console -conman -d pfv-tsys4:7890 -q # list all consoles -``` - -Escape sequence: `&.` to disconnect, `&?` for help. - -**Direct telnet (emergency only — conflicts with conman):** - -```bash -# Direct telnet to ser2net works ONLY when conmand is stopped, because -# conmand maintains persistent connections to all 7 TCP ports. Use: -ssh pfv-tsys4 'systemctl stop conmand' -telnet pfv-tsys4 2001 # pfv-core-sw01 -ssh pfv-tsys4 'systemctl start conmand' # restart when done -``` - -**Do NOT use telnet while conmand is running** — conmand will reconnect -and kick your telnet session immediately ("Connection closed by foreign host"). -The correct workflow is conman client → conmand → ser2net → device. - -### Re-deploy after changing mapping.txt - -```bash -PROX_HOST=pfv-tsys4 bash tests/remote.sh prox 'bash /root/console/setup.sh' -``` - -### Find the ID_PATH for a new adapter - -```bash -PROX_HOST=pfv-tsys4 bash tests/remote.sh prox-file console/discover.sh -``` - -Then match the new adapter's ID_PATH to its physical location and add a line -to `mapping.txt`. - -## Files on pfv-tsys4 - -| File | Purpose | -|------|---------| -| `/etc/udev/rules.d/99-console-ports.rules` | Stable symlinks by ID_PATH | -| `/etc/ser2net.yaml` | ser2net config (telnet rfc2217 TCP ports → serial symlinks) | -| `/etc/conman.conf` | conman config (CONSOLE entries between markers) | -| `/etc/systemd/system/conmand.service` | systemd unit for conmand | -| `/root/console/mapping.txt` | Copy of the source-of-truth mapping | -| `/root/console/setup.sh` | Setup script (re-runnable) | -| `/root/console/generate-config.sh` | Config generator | - -## Old workflow (replaced) - -The old `/root/conmap` file and manual `screen` sessions are no longer -needed. The new setup is fully automated and survives reboots. +# dcinfra/console/README.md + +> **Documentation moved to Discourse — the canonical source of truth.** +> +> **Serial console management (ser2net + conman)** +> +> **Read it here:** https://community.turnsys.com/t/301 +> +> *Migrated 2026-08-06. This file is kept as a pointer for git-browsing context. +> Do not update content here — edit the Discourse wiki topic instead.* diff --git a/dcinfra/powerman/README.md b/dcinfra/powerman/README.md index 2607c91..15de28f 100644 --- a/dcinfra/powerman/README.md +++ b/dcinfra/powerman/README.md @@ -1,113 +1,10 @@ -# Powerman PDU Management +# dcinfra/powerman/README.md -> **Redmine:** [#359](https://projects.knownelement.com/issues/359) (deployment, closed) · [#374](https://projects.knownelement.com/issues/374) (outlet renaming, open) - -Centralized power management for the Cyclades AlterPath PM10i PDU via -[Powerman](https://github.com/chaos/powerman), running on pfv-tsys1. - -## Hardware - -| Component | Details | -|-----------|---------| -| **PDU** | Cyclades AlterPath PM10i (10 controllable AC outlets) | -| **Firmware** | v1.9.0 (Aug 4, 2006) | -| **Connection** | USB-to-DB9 adapter (Prolific pl2303, serial BJAAb144J07) | -| **Host** | pfv-tsys1 (OptiPlex 9020, Proxmox) | -| **Serial** | 9600 baud, 8N1, raw mode | -| **Credentials** | Factory defaults: `admin` / `pm8` (in cyclades-pm10.dev) | -| **Network access** | powermand listens on `127.0.0.1:10101` (local) + `100.121.189.98:10101` (Tailscale) | - -## Device mapping - -``` -USB adapter (067b:23a3, serial BJAAb144J07) - └─ pl2303 driver → /dev/ttyUSB1 - └─ udev symlink → /dev/cyclades-pm10 (stable across reboots) - └─ powermand reads/writes serial → Cyclades PM10i - └─ 10 outlets (factory default names: 1-10) -``` - -The udev rule (`/etc/udev/rules.d/99-cyclades-pdu.rules`) pins the adapter -by its USB serial number, so the symlink survives replugs and reboots. - -## Scripts - -All scripts run on the target host (pfv-tsys1) via `tests/remote.sh`: - -```bash -# Setup (idempotent — safe to re-run): -PROX_HOST=pfv-tsys1 bash tests/remote.sh prox-file powerman/setup.sh - -# Validate PDU control (cycles outlet 10 off → on): -PROX_HOST=pfv-tsys1 bash tests/remote.sh prox-file powerman/test-pdu.sh - -# Status check: -PROX_HOST=pfv-tsys1 bash tests/remote.sh prox-file powerman/status.sh -``` - -### Customizing for other hosts/PDUs - -The setup script accepts environment overrides: - -```bash -PDU_SERIAL=XXXX PDU_VENDOR=067b PDU_OUTLETS=20 PDU_TYPE=pm20 \ - PROX_HOST=other-host bash tests/remote.sh prox-file powerman/setup.sh -``` - -## Usage (daily operations) - -From pfv-tsys1 (or any host with network access to port 10101): - -```bash -# List all outlets -powerman -l - -# Query status (all outlets) -powerman -q - -# Turn outlet off -powerman -0 outlet-10 - -# Turn outlet on -powerman -1 outlet-10 - -# Cycle outlet (off → 4s delay → on) -powerman -c outlet-10 - -# Query a specific outlet -powerman -q outlet-10 -``` - -### Remote access from other hosts - -powermand listens on `0.0.0.0:10101`. From another tailnet host: - -```bash -powerman --server-host pfv-tsys1 --server-port 10101 -q -``` - -Or set `POWERMAN_SERVER=pfv-tsys1:10101` in the environment. - -## Configuration files on pfv-tsys1 - -| File | Purpose | -|------|---------| -| `/etc/udev/rules.d/99-cyclades-pdu.rules` | Stable symlink for USB-DB9 adapter | -| `/etc/powerman/powerman.conf` | Device definition + 10 outlet nodes | -| `/etc/powerman/cyclades-pm10.dev` | Cyclades PM10 protocol spec (shipped with powerman) | - -## Validation results - -2026-07-28: All 8 checks passed. -Outlet 10 turned OFF (confirmed), turned ON (confirmed), then cycled. - -## TODO (Friday onsite) - -- [ ] **Rename outlets** in `/etc/powerman/powerman.conf` to match the - physical devices plugged into each outlet (e.g., `node "tsys4-psu" - "cyclades-pm10" "3"`). Currently all outlets are generically named - `outlet-1` through `outlet-10`. -- [ ] **Change PDU admin password** from factory default (`pm8`) if - security-sensitive. Update `/etc/powerman/cyclades-pm10.dev` login - script to match. -- [ ] **Verify all 10 outlets** individually once device mapping is known. +> **Documentation moved to Discourse — the canonical source of truth.** +> +> **Cyclades PM10i PDU management via powerman** +> +> **Read it here:** https://community.turnsys.com/t/301 +> +> *Migrated 2026-08-06. This file is kept as a pointer for git-browsing context. +> Do not update content here — edit the Discourse wiki topic instead.* diff --git a/dcinfra/ups/README.md b/dcinfra/ups/README.md index 112851d..1083f98 100644 --- a/dcinfra/ups/README.md +++ b/dcinfra/ups/README.md @@ -1,212 +1,10 @@ -# UPS Management (NUT — Network UPS Tools) +# dcinfra/ups/README.md -> **Redmine:** [#340](https://projects.knownelement.com/issues/340) (APC, closed) · [#372](https://projects.knownelement.com/issues/372) (Tripp Lite, open) - -Centralized UPS monitoring for the server room via -[NUT](https://networkupstools.org/), running on **pfv-tsys1**. USB HID UPS -units feed one `upsd` network server; Home Assistant polls it over Tailscale for -real-time power/load/runtime tracking, and a local `upsmon` shuts the hypervisor -down gracefully when battery is low. - -> **Why NUT (not apcupsd)?** Two different UPS brands (APC + Tripp Lite) must be -> covered. `apcupsd` only supports APC, so it would require a second daemon -> stack. NUT's `usbhid-ups` driver speaks to **both** via the USB HID Power -> Device class, and Home Assistant ships a first-class NUT integration. - -## Hardware - -| UPS | Model | VID:PID | USB Serial | Status | -|-----|-------|---------|------------|--------| -| **APC** | Smart-UPS C 1500 (FW 02.2) | `051d:0003` | `AS1213210423` | **LIVE** | -| **Tripp Lite** | UPS (HID PDC) | `09ae:3016` | `2352CVLSM871900694` | **Blocked** — see below | - -## Current State (2026-07-30) - -### APC Smart-UPS C 1500 — OPERATIONAL - -Fully reporting via `usbhid-ups` + `APC HID 0.100` subdriver. Data validated: - -``` -battery.charge: 100 battery.runtime: 1800 battery.voltage: 27.4 -ups.status: OL ups.load: (via HA) ups.model: Smart-UPS C 1500 -``` - -### Tripp Lite UPS — BLOCKED (hardware issue) - -The driver finds the device, matches the `TrippLite HID 0.85` subdriver, claims -the interface, and reads the HID descriptor — but **fails reading the 878-byte -HID Report Descriptor** (`Resource temporarily unavailable` / EAGAIN after 5s). -The driver is masked to prevent restart-loop spam. - -USB descriptors (manufacturer, product, serial) are readable via `lsusb -v` and -`nut-scanner`, but the bulk control transfer for the full report descriptor -times out. Likely causes: - -1. **USB hub** — the Tripp Lite is behind a Genesys Logic hub (`05e3:0608`). - Try plugging directly into a motherboard USB port. -2. **USB cable** — try a high-quality data cable (not charge-only). -3. **UPS firmware** — the USB controller may not properly implement all HID - endpoints. - -**To retry after physical reseat:** -```bash -# On pfv-tsys1: -systemctl unmask nut-driver@tripp-lite-ups -systemctl start nut-driver@tripp-lite-ups -upsc tripp-lite-ups@localhost -``` - -## Architecture - -``` -pfv-tsys1 (192.168.3.11 / Tailscale 100.121.189.98) - ├─ APC Smart-UPS C 1500 ──┐ - └─ Tripp Lite UPS (masked) ──┤ USB HID - ▼ - nut-driver@apc-smartups-c1500 (usbhid-ups) - ▼ - upsd :3493 (LISTEN 127.0.0.1 + Tailscale + LAN) - ▼ ▼ - upsmon (local) Home Assistant (NUT integration) - graceful shutdown via LAN 192.168.3.11 (HAOS can't - route to Tailscale IPs) -``` - -- **Driver layer** — `usbhid-ups` process, pinned by USB serial. Debian uses - templated `nut-driver@.service` units managed by - `nut-driver-enumerator`. -- **Server layer** — `upsd` exposes UPS data on TCP 3493 (localhost + Tailscale - + LAN). Clients authenticate via `upsd.users`. -- **Monitor layer** — `upsmon` runs locally as `master` to trigger - `SHUTDOWNCMD` (`/sbin/shutdown -h now`) when a UPS reports `LOWBATT`. -- **Home Assistant** — native NUT integration connects to `upsd` over Tailscale - and exposes `ups.load`, `battery.runtime`, `ups.status`, etc. as sensors. - -### Key deployment lesson: udev must cover raw USB devices - -The `usbhid-ups` driver opens `/dev/bus/usb/BBB/DDD` (raw USB device files), -**not** `/dev/hidraw*`. After calling `setuid(111)` to drop to the `nut` user, -it needs write access to those raw USB files. The udev rule must match -`SUBSYSTEM=="usb"` by vendor/product ID to set `GROUP="nut"` — matching only -`hidraw` is insufficient. See `/etc/udev/rules.d/99-nut-ups.rules`. - -## Scripts - -NUT host scripts run on pfv-tsys1 via `tests/remote.sh`: - -```bash -# Idempotent install + configure (safe to re-run): -PROX_HOST=pfv-tsys1 bash tests/remote.sh prox-file ups/setup.sh - -# Discover USB UPS + NUT state (read-only diagnostic): -PROX_HOST=pfv-tsys1 bash tests/remote.sh prox-file ups/discover.sh - -# Query UPS data + service health: -PROX_HOST=pfv-tsys1 bash tests/remote.sh prox-file ups/status.sh -``` - -The HA integration script runs from your workstation (needs HA API access): - -```bash -# Add the NUT integration to Home Assistant (idempotent): -bash ups/setup-ha-nut.sh -``` - -`setup.sh` accepts environment overrides for serials/VIDs/PIDs/usernames, so it -can be repurposed for other hosts or UPS units. Passwords for `monuser` and -`homeassistant` are auto-generated on first run and reused on subsequent runs -(stored in `/etc/nut/upsd.users`). - -Set `TRIPP_ENABLED=0` to skip the Tripp Lite entirely (useful if it's physically -unplugged). - -## Configuration files on pfv-tsys1 - -| File | Purpose | -|------|---------| -| `/etc/udev/rules.d/99-nut-ups.rules` | Grant nut group rw on raw USB + hidraw devices (both subsystems) | -| `/etc/nut/ups.conf` | `usbhid-ups` device(s), pinned by serial + subdriver | -| `/etc/nut/upsd.conf` | `LISTEN 127.0.0.1` + `LISTEN ` + `LISTEN ` on port 3493 | -| `/etc/nut/upsd.users` | `monuser` (master) + `homeassistant` (read-only) credentials | -| `/etc/nut/upsmon.conf` | Local master monitor + `SHUTDOWNCMD` | -| `/etc/nut/nut.conf` | `MODE=netserver` | - -## Home Assistant integration - -The NUT integration is added automatically by `setup-ha-nut.sh`, which drives -HA's REST config-flow API. It is idempotent (skips if the entry exists). - -```bash -# Prerequisites: create token + password files (one-time): -mkdir -p ~/.config/pfvcluster -# HA → Profile → Long-Lived Access Tokens → Create Token: -echo -n 'YOUR_HA_TOKEN' > ~/.config/pfvcluster/ha-token -# Password is in /etc/nut/upsd.users on pfv-tsys1 (the homeassistant user): -echo -n 'YOUR_NUT_PASS' > ~/.config/pfvcluster/nut-password -chmod 600 ~/.config/pfvcluster/{ha-token,nut-password} - -# Run: -bash ups/setup-ha-nut.sh -``` - -### Why LAN IP, not Tailscale - -upsd listens on **both** the Tailscale IP (`100.121.189.98`) **and** the LAN IP -(`192.168.3.11`). The HA NUT integration uses the **LAN IP** because HAOS runs -Tailscale as an isolated add-on container — the HA core container cannot route -to Tailscale IPs. Since pfv-bms (HA, `192.168.3.12`) and pfv-tsys1 (`192.168.3.11`) -share the same vmbr0 bridge, LAN connectivity is instant and reliable. - -### Manual UI alternative - -In Home Assistant → **Settings → Devices & Services → Add Integration → NUT**: - -| Field | Value | -|-------|-------| -| Host | `192.168.3.11` (LAN — HAOS can't reach Tailscale IPs from the HA container) | -| Port | `3493` | -| Username | `homeassistant` | -| Password | *(stored in `/etc/nut/upsd.users` on pfv-tsys1)* | -| UPS | `apc-smartups-c1500` | - -### Live sensors - -HA exposes UPS data as sensors (prefix `sensor.apc_smartups_c1500_`): -`battery_charge`, `status` (Online/On Battery), `status_data` (OL/OB/DISCHRG). -Additional sensors (load, runtime, voltage) populate as the UPS reports them. - -## Daily operations - -From pfv-tsys1 (or any tailnet host with NUT client installed): - -```bash -# List UPS units served by upsd -upsc -l pfv-tsys1 - -# Full variable dump for one UPS -upsc apc-smartups-c1500@pfv-tsys1 - -# Battery runtime (the only runtime/charge data this UPS exposes) -upsc apc-smartups-c1500@pfv-tsys1 battery.runtime -``` - -## Notes - -- **No USB passthrough to the HA VM.** Keeping the UPS on the host preserves - hypervisor graceful-shutdown capability and matches the `powerman/` pattern - (PDU managed on the host where the adapter physically lives). -- **No `ups.load` / `ups.realpower` on this UPS (FW 02.2, mfg 2012):** The - APC Smart-UPS C 1500 does not expose load or power data over USB HID. - Both NUT `usbhid-ups` and `apcupsd` (USB mode, tested 2026-07-30) read the - same HID descriptor — the variable simply isn't there. This means the HA - NUT integration provides **battery/runtime/status sensors only**, not - wattage for the Energy Dashboard. - - **apcupsd test note:** Debian's `apcupsd` package conflicts with - `nut-server` (mutually exclusive). apcupsd USB mode returned `COMMLOST` - even before we could check load. The APC Smart Serial protocol (serial - cable, AP940-1524C, ~$30) DOES report load%, but this requires a serial - port on the UPS and on the host. - - **Energy Dashboard path:** A smart plug (Shelly Plug S / TP-Link Kasa, - ~$15-25) on the UPS output reports real watts natively and feeds the - Energy Dashboard with zero UPS-driver hacking. The NUT sensors remain - valuable for outage detection and graceful-shutdown automations. +> **Documentation moved to Discourse — the canonical source of truth.** +> +> **UPS management (NUT) for APC Smart-UPS C 1500** +> +> **Read it here:** https://community.turnsys.com/t/301 +> +> *Migrated 2026-08-06. This file is kept as a pointer for git-browsing context. +> Do not update content here — edit the Discourse wiki topic instead.* diff --git a/docs/docmap.md b/docs/docmap.md index d68ba1a..19d1a27 100644 --- a/docs/docmap.md +++ b/docs/docmap.md @@ -1,118 +1,10 @@ -# Documentation Map +# docs/docmap.md -> **Index of all documentation in this repo.** Agents must update this file -> whenever a doc is added, removed, or substantively changed. - -> **Last updated:** 2026-08-06 (Pi-hole hardening config committed to netinfra/pihole/ — [#376]) - -## Kubernetes Architecture ([`k8s/`](k8s/)) - -Distro decision, target architecture, control-plane design, bootstrap and DR -procedures for the pfv-k8s cluster (Talos + vcluster + Keycloak OIDC). - -| Document | Description | Last Reviewed | -|----------|-------------|---------------| -| [`k8s/docs/README.md`](k8s/docs/README.md) | Index + TL;DR of all k8s decisions | 2026-07-28 | -| [`k8s/docs/DISTRO-DECISION.md`](k8s/docs/DISTRO-DECISION.md) | Talos vs k3s analysis. Decision: k3s for regular R&D cluster; Talos preserved for future ITAR/classified | 2026-07-28 | -| [`k8s/docs/ARCHITECTURE.md`](k8s/docs/ARCHITECTURE.md) | Target architecture: control plane, network, identity, storage, tenant isolation, bootstrap, DR. Mermaid diagrams | 2026-07-28 | - -## Proxmox Cluster ([`proxmox/`](proxmox/)) - -Fleet operations, hardware, performance tuning, storage architecture. - -| Document | Description | Last Reviewed | -|----------|-------------|---------------| -| [`proxmox/docs/NETWORK-TOPOLOGY.md`](proxmox/docs/NETWORK-TOPOLOGY.md) | **Full network reference**: Mermaid topology diagrams, complete switch port maps, VLAN/subnet map, cross-rack trunk analysis, NFS data-flow, per-host NIC reference | 2026-07-30 | -| [`proxmox/docs/STORAGE-ARCHITECTURE.md`](proxmox/docs/STORAGE-ARCHITECTURE.md) | **Full storage reference**: tier definitions, physical disk inventory (all 14 disks), NFS export/mount map, per-spindle VM placement, capacity summary | 2026-07-30 | -| [`proxmox/docs/HOST-REFERENCE.md`](proxmox/docs/HOST-REFERENCE.md) | **Per-host reference**: hardware, network, storage, tuning, role, running VMs for all 7 nodes | 2026-07-30 | -| [`proxmox/docs/VM-INVENTORY.md`](proxmox/docs/VM-INVENTORY.md) | **Complete VM catalog**: placement diagram, all 28 running VMs with disk/CPU/RAM/config, optimization findings (cache mode, CPU type, qga audit), stopped VMs | 2026-07-30 | -| [`proxmox/docs/PROJECT.md`](proxmox/docs/PROJECT.md) | Comprehensive fleet report: 7 hosts, VM inventory, storage, recommendations | 2026-07-27 | -| [`proxmox/docs/CAPACITY-GOALS.md`](proxmox/docs/CAPACITY-GOALS.md) | Authoritative capacity targets (80% RAM, ~50% idle CPU) + workload placement model (compute / RackRental / infra) | 2026-07-29 | -| [`proxmox/docs/AUDIT-2026-07-30.md`](proxmox/docs/AUDIT-2026-07-30.md) | Prior audit (network + storage + compute + switches). **Superseded by AUDIT-2026-08-05** | 2026-07-30 | -| [`proxmox/docs/AUDIT-2026-08-05.md`](proxmox/docs/AUDIT-2026-08-05.md) | **Current & authoritative.** Read-only ground-truth via nmap/SNMP/SSH/API: 4 switches up / 1 down, 63 VMs, 167 zones, 753 DHCP leases, k8s DOWN, netinfra-02 wiped, repo restructure, 3-tier Tailscale fleet | 2026-08-05 | -| [`proxmox/docs/DRIFT-2026-07-30.md`](proxmox/docs/DRIFT-2026-07-30.md) | **Fleet drift report.** Executive briefing of package/service/tuning/security inconsistencies across all 7 hosts, with detailed appendixes | 2026-07-30 | -| [`proxmox/docs/TODO.md`](proxmox/docs/TODO.md) | Pending physical hardware work (tsys2/4/5 Friday plan) | 2026-07-27 | -| [`proxmox/docs/NAMING-POLICY.md`](proxmox/docs/NAMING-POLICY.md) | Canonical naming policy: VM name = source of truth (DNS/Tailscale/monitoring must match) | 2026-07-31 | -| [`proxmox/docs/PHYSICAL-SYSTEMS.md`](proxmox/docs/PHYSICAL-SYSTEMS.md) | Physical infrastructure: Pi's, network devices, UniFi APs, SNMP-discovered hosts | 2026-07-31 | -| [`proxmox/docs/K8S.md`](proxmox/docs/K8S.md) | Kubernetes storage/host analysis (predecessor to [`k8s/`](k8s/)) | 2026-07-27 | - -## Server Build ([`docs/server-build/`](server-build/)) - -Server provisioning, security hardening, DNS/NTP configuration. - -| Document | Description | Last Reviewed | -|----------|-------------|---------------| -| [`docs/server-build/SECURITY.md`](server-build/SECURITY.md) | Security architecture: SSH hardening, 2FA, SCAP-STIG, Wazuh, auditd | 2026-07-25 | -| [`docs/server-build/tailscale.md`](server-build/tailscale.md) | Tailscale vs managed DNS analysis (RESOLVED — netinfra pair serves knel.net) | 2026-07-28 | -| [`docs/server-build/DEPLOYMENT.md`](server-build/DEPLOYMENT.md) | Server deployment procedures, package lists, config flow | 2026-07-25 | -| [`docs/server-build/TSYS-2FA-GUIDE.md`](server-build/TSYS-2FA-GUIDE.md) | End-user guide for 2FA setup (SSH, Cockpit, Webmin) | 2026-07-25 | -| [`docs/server-build/DEVELOPMENT-GUIDELINES.md`](server-build/DEVELOPMENT-GUIDELINES.md) | Coding standards, commit conventions, script patterns | 2026-07-25 | - -## Operational Guides (outside docs/) - -| Document | Description | Last Reviewed | -|----------|-------------|---------------| -| [`../netinfra/dhcp-migration.md`](../netinfra/dhcp-migration.md) | DHCP migration: pfv-netboot → netinfra-01/02 (ISC failover-peer, COMPLETE) | 2026-07-29 | -| [`../awx/README.md`](../awx/README.md) | Ansible AWX deployment on tsys-awx (k3s + AWX Operator 2.19.1, v24.6.1) | 2026-07-29 | -| [`../dcinfra/powerman/README.md`](../dcinfra/powerman/README.md) | Cyclades PM10i PDU management via powerman on pfv-tsys1 | 2026-07-28 | -| [`../dcinfra/console/README.md`](../dcinfra/console/README.md) | Serial console management (ser2net telnet rfc2217 + conman) for 7 network switches on pfv-tsys4 | 2026-07-29 | -| [`../dcinfra/ups/README.md`](../dcinfra/ups/README.md) | UPS management (NUT) for APC Smart-UPS C 1500 on pfv-tsys1 — upsd on Tailscale, HA integration | 2026-07-30 | -| [`../k8s/README.md`](../k8s/README.md) | k3s cluster setup scripts: wipe, bootstrap, taint, verify (3-node HA over Tailscale) | 2026-07-28 | -| [`../netinfra/dns-cluster-setup/README.md`](../netinfra/dns-cluster-setup/README.md) | Technitium DNS cluster setup: export, deploy, cluster, verify | 2026-07-28 | -| [`../tests/README.md`](../tests/README.md) | Test suite documentation: unit, security, validation tests | 2026-07-28 | -| [`../netinfra/pfv-netinfra-setup.md`](../netinfra/pfv-netinfra-setup.md) | pfv-netinfra-01/02 initial setup guide | 2026-07-27 | -| [`../netinfra/pihole/README.md`](../netinfra/pihole/README.md) | Pi-hole recursive DNS on netinfra-01/02: shm_size fix, gravity-validate.sh, healthcheck, autoheal (defense-in-depth vs gravity.db corruption) | 2026-08-06 | -| [`../netinfra/pfv-netboot-setup.md`](../netinfra/pfv-netboot-setup.md) | pfv-netboot reference node setup | 2026-07-27 | - -## Archive ([`archive/`](archive/)) - -Historical AI reviews, completed task lists, and pre-refactor examples. Read-only -context — do not update; link to active docs instead. - -| Document | Description | -|----------|-------------| -| [`archive/CODE-REVIEW-FINDINGS.md`](archive/CODE-REVIEW-FINDINGS.md) | Early code review findings (most issues now fixed) | -| [`archive/REFACTORING-EXAMPLES.md`](archive/REFACTORING-EXAMPLES.md) | Pre-refactor code patterns (historical "before" examples) | -| [`archive/Claude-Review.md`](archive/Claude-Review.md) | Claude's initial code review | -| [`archive/AIReview-QWEN.md`](archive/AIReview-QWEN.md) | Qwen AI review | -| [`archive/AiOverview-Gemini.md`](archive/AiOverview-Gemini.md) | Gemini project overview | -| [`archive/AiOverview-OpenCode.md`](archive/AiOverview-OpenCode.md) | OpenCode project overview | -| [`archive/AiSecurityAudit-Gemini.md`](archive/AiSecurityAudit-Gemini.md) | Gemini security audit | -| [`archive/charles-todo.md`](archive/charles-todo.md) | Charles's early task list (completed) | -| [`archive/claude-todo.md`](archive/claude-todo.md) | Claude's early task list (completed) | - -## External Tracking - -| Resource | Description | -|----------|-------------| -| [Redmine](https://projects.knownelement.com) | **System of record for all work.** Version: Potential to Kinetic Ready (due 2026-09-30). Project: Technology & Facility Services (id 55) | -| [Gitea](https://git.knownelement.com/KNEL/PFVCluster) | Source code repository | - -## Top-Level Files - -| File | Description | -|------|-------------| -| [`../AGENTS.md`](../AGENTS.md) | Agent operating instructions (repo layout, git policy, Redmine tracking, gardening protocol) | -| [`../STATUS.md`](../STATUS.md) | Ticket index + infrastructure summary (read-only snapshot; work tracked in Redmine) | -| [`../README.md`](../README.md) | Project overview and quick start | -| [`../LICENSE`](../LICENSE) | License | - ---- - -## Agent Gardening Protocol - -When making changes to this repo, agents MUST: - -1. **Update Redmine tickets** — all work status, completion, and issue tracking - lives in [Redmine](https://projects.knownelement.com), not in repo files. -2. **Update [`../STATUS.md`](../STATUS.md)** if the infrastructure summary or - ticket index needs refreshing (read-only snapshot only). -3. **Update this file (`docmap.md`)** if a doc is added, removed, or has a - substantive content change. Update the "Last Reviewed" date. -3. **Verify cross-references** — any new `.md` file must be linked from at - least one existing doc or this map. -4. **Check for stale paths** — after any directory rename or file move, - `grep -rn 'old/path' --include='*.md'` and fix all references. -5. **Keep code and docs in sync** — if you change a script's interface, - behavior, or location, update every doc that references it in the same - commit. +> **Documentation moved to Discourse — the canonical source of truth.** +> +> **Documentation index — now lives in the Project Overview topic** +> +> **Read it here:** https://community.turnsys.com/t/296 +> +> *Migrated 2026-08-06. This file is kept as a pointer for git-browsing context. +> Do not update content here — edit the Discourse wiki topic instead.* diff --git a/docs/server-build/DEPLOYMENT.md b/docs/server-build/DEPLOYMENT.md index f81c1e1..2c17d9a 100644 --- a/docs/server-build/DEPLOYMENT.md +++ b/docs/server-build/DEPLOYMENT.md @@ -1,336 +1,10 @@ -# TSYS FetchApply Deployment Guide - -## Overview - -This guide provides comprehensive instructions for deploying the TSYS FetchApply infrastructure provisioning system on Linux servers. - -## Prerequisites - -### System Requirements -- **Operating System:** Ubuntu 18.04+ or Debian 10+ (recommended) -- **RAM:** Minimum 2GB, recommended 4GB -- **Disk Space:** Minimum 10GB free space -- **Network:** Internet connectivity for package downloads -- **Privileges:** Root or sudo access required - -### Required Tools -- `git` - Version control system -- `curl` - HTTP client for downloads -- `wget` - Alternative download tool -- `systemctl` - System service management -- `apt-get` - Package management (Debian/Ubuntu) - -### Network Requirements -- **HTTPS access** to: - - `https://archive.ubuntu.com` (Ubuntu packages) - - `https://linux.dell.com` (Dell hardware support) - - `https://download.proxmox.com` (Proxmox packages) - - `https://github.com` (Git repositories) - -## Pre-Deployment Validation - -### 1. System Compatibility Check -```bash -# Clone repository -git clone [repository-url] -cd FetchApply - -# Run system validation -./Project-Tests/validation/system-requirements.sh -``` - -### 2. Network Connectivity Test -```bash -# Test network connectivity -curl -I https://archive.ubuntu.com -curl -I https://linux.dell.com -curl -I https://download.proxmox.com -``` - -### 3. Permission Verification -```bash -# Verify write permissions -test -w /etc && echo "✅ /etc writable" || echo "❌ /etc not writable" -test -w /usr/local/bin && echo "✅ /usr/local/bin writable" || echo "❌ /usr/local/bin not writable" -``` - -## Deployment Methods - -### Method 1: Standard Deployment (Recommended) -```bash -# 1. Clone repository -git clone [repository-url] -cd FetchApply - -# 2. Run pre-deployment tests -./Project-Tests/run-tests.sh validation - -# 3. Execute deployment -cd ProjectCode -sudo bash SetupNewSystem.sh -``` - -### Method 2: Dry Run Mode -```bash -# 1. Clone repository -git clone [repository-url] -cd FetchApply - -# 2. Review configuration -cat provisioning/SetupNewSystem.sh - -# 3. Execute with manual review -cd ProjectCode -sudo bash -x SetupNewSystem.sh # Debug mode -``` - -## Deployment Process - -### Phase 1: Framework Initialization -1. **Environment Setup** - - Load framework variables - - Source framework includes - - Initialize logging system - -2. **System Detection** - - Detect physical vs virtual hardware - - Identify operating system - - Check for existing users - -### Phase 2: Base System Configuration -1. **Package Installation** - - Update package repositories - - Install essential packages - - Configure package sources - -2. **User Management** - - Create required user accounts - - Configure SSH access - - Set up sudo permissions - -### Phase 3: Security Hardening -1. **SSH Configuration** - - Deploy hardened SSH configuration - - Install SSH keys - - Disable password authentication - -2. **System Hardening** - - Configure firewall rules - - Enable audit logging - - Install security tools - -### Phase 4: Monitoring and Management -1. **Monitoring Agents** - - Deploy LibreNMS agents - - Configure SNMP - - Set up system monitoring - -2. **Management Tools** - - Install Cockpit dashboard - - Configure remote access - - Set up maintenance scripts - -## Post-Deployment Verification - -### 1. Security Validation -```bash -# Run security tests -./Project-Tests/run-tests.sh security - -# Verify SSH configuration -ssh -T [server-ip] # Should work with key authentication -``` - -### 2. Service Status Check -```bash -# Check critical services -sudo systemctl status ssh -sudo systemctl status auditd -sudo systemctl status snmpd -``` - -### 3. Network Connectivity -```bash -# Test internal services -curl -k https://localhost:9090 # Cockpit -snmpwalk -v2c -c public localhost system -``` - -## Troubleshooting - -### Common Issues - -#### 1. Permission Denied Errors -```bash -# Solution: Run with sudo -sudo bash SetupNewSystem.sh -``` - -#### 2. Network Connectivity Issues -```bash -# Check DNS resolution -nslookup archive.ubuntu.com - -# Test direct IP access -curl -I 91.189.91.26 # Ubuntu archive IP -``` - -#### 3. Package Installation Failures -```bash -# Update package cache -sudo apt-get update - -# Fix broken packages -sudo apt-get -f install -``` - -#### 4. SSH Key Issues -```bash -# Verify key permissions -ls -la ~/.ssh/ -chmod 600 ~/.ssh/id_rsa -chmod 644 ~/.ssh/id_rsa.pub -``` - -### Debug Mode -```bash -# Enable debug logging -export DEBUG=1 -bash -x SetupNewSystem.sh -``` - -### Log Analysis -```bash -# Check deployment logs -tail -f /var/log/fetchapply/deployment.log - -# Review system logs -journalctl -u ssh -journalctl -u auditd -``` - -## Environment-Specific Configurations - -### Physical Dell Servers -- **OMSA Installation:** Dell OpenManage Server Administrator -- **Hardware Monitoring:** iDRAC configuration -- **Performance Tuning:** CPU and memory optimizations - -### Virtual Machines -- **Guest Additions:** VMware tools or VirtualBox additions -- **Resource Limits:** Memory and CPU constraints -- **Network Configuration:** Bridge vs NAT settings - -### Development Environments -- **SSH Configuration:** Less restrictive settings -- **Development Tools:** Additional packages for development -- **Testing Access:** Enhanced logging and debugging - -## Maintenance and Updates - -### Regular Maintenance -```bash -# Update system packages -sudo apt-get update && sudo apt-get upgrade - -# Update monitoring scripts -cd /usr/local/bin -sudo wget https://[repository]/scripts/up2date.sh -sudo chmod +x up2date.sh -``` - -### Security Updates -```bash -# Check for security updates -sudo apt-get update -sudo apt list --upgradable | grep -i security - -# Apply security patches -sudo apt-get upgrade -``` - -### Configuration Updates -```bash -# Update FetchApply -cd FetchApply -git pull origin main - -# Re-run specific modules -cd provisioning/Modules/Security -sudo bash secharden-ssh.sh -``` - -## Best Practices - -### 1. Pre-Deployment -- Always test in non-production environment first -- Review all scripts before execution -- Validate network connectivity -- Ensure proper backup procedures - -### 2. During Deployment -- Monitor deployment progress -- Check for errors and warnings -- Document any customizations -- Validate each phase completion - -### 3. Post-Deployment -- Run full security test suite -- Verify all services are running -- Test remote access -- Document deployment specifics - -### 4. Ongoing Operations -- Regular security updates -- Monitor system performance -- Review audit logs -- Maintain deployment documentation - -## Support and Resources - -### Documentation -- **README.md:** Basic usage instructions -- **SECURITY.md:** Security architecture and guidelines -- **tests/README.md:** Testing framework documentation - -### Community Support -- **Issues:** https://projects.knownelement.com/project/reachableceo-vptechnicaloperations/timeline -- **Discussion:** https://community.turnsys.com/c/chieftechnologyandproductofficer/26 - -### Professional Support -- **Technical Support:** [Contact information to be added] -- **Consulting Services:** [Contact information to be added] - -## Deployment Checklist - -### Pre-Deployment -- [ ] System requirements validated -- [ ] Network connectivity tested -- [ ] Backup procedures in place -- [ ] Security review completed - -### Deployment -- [ ] Repository cloned successfully -- [ ] Pre-deployment tests passed -- [ ] Deployment executed without errors -- [ ] Post-deployment verification completed - -### Post-Deployment -- [ ] Security tests passed -- [ ] All services running -- [ ] Remote access verified -- [ ] Documentation updated - -### Maintenance -- [ ] Update schedule established -- [ ] Monitoring configured -- [ ] Backup procedures tested -- [ ] Incident response plan activated - -## Version History - -- **v1.0:** Initial deployment framework -- **v1.1:** Added security hardening and secrets management -- **v1.2:** Enhanced testing framework and documentation - -Last updated: July 14, 2025 \ No newline at end of file +# docs/server-build/DEPLOYMENT.md + +> **Documentation moved to Discourse — the canonical source of truth.** +> +> **Server deployment procedures** +> +> **Read it here:** https://community.turnsys.com/t/302 +> +> *Migrated 2026-08-06. This file is kept as a pointer for git-browsing context. +> Do not update content here — edit the Discourse wiki topic instead.* diff --git a/docs/server-build/DEVELOPMENT-GUIDELINES.md b/docs/server-build/DEVELOPMENT-GUIDELINES.md index 1ea02e9..070df76 100644 --- a/docs/server-build/DEVELOPMENT-GUIDELINES.md +++ b/docs/server-build/DEVELOPMENT-GUIDELINES.md @@ -1,407 +1,10 @@ - -# TSYS PFVCluster Development Guidelines - -## Overview - -This document contains development standards and best practices for the TSYS PFVCluster infrastructure provisioning system. - -## Package Management Best Practices - -### Combine apt-get Install Commands - -**Rule:** Always combine multiple package installations into a single `apt-get install` command for performance. - -**Rationale:** Single command execution is significantly faster than multiple separate commands due to: -- Reduced package cache processing -- Single dependency resolution -- Fewer network connections -- Optimized package download ordering - -#### ✅ Correct Implementation -```bash -# Install all packages in one command -apt-get install -y package1 package2 package3 package4 - -# Real example from 2FA script -apt-get install -y libpam-google-authenticator qrencode -``` - -#### ❌ Incorrect Implementation -```bash -# Don't use separate commands for each package -apt-get install -y package1 -apt-get install -y package2 -apt-get install -y package3 -``` - -#### Complex Package Installation Pattern -```bash -function install_security_packages() { - print_info "Installing security packages..." - - # Update package cache once - apt-get update - - # Install all packages in single command - apt-get install -y \ - auditd \ - fail2ban \ - libpam-google-authenticator \ - lynis \ - rkhunter \ - aide \ - chkrootkit \ - clamav \ - clamav-daemon - - print_success "Security packages installed successfully" -} -``` - -## Script Development Standards - -### Error Handling -- Always use `set -euo pipefail` at script start -- Implement proper error trapping -- Use framework error handling functions -- Return appropriate exit codes - -### Function Structure -```bash -function function_name() { - print_info "Description of what function does..." - - # Local variables - local var1="value" - local var2="value" - - # Function logic - if [[ condition ]]; then - print_success "Success message" - return 0 - else - print_error "Error message" - return 1 - fi -} -``` - -### Framework Integration -- Source framework includes at script start -- Use framework logging and pretty print functions -- Follow existing patterns for consistency -- Include proper PROJECT_ROOT path resolution - -```bash -# Standard framework sourcing pattern -PROJECT_ROOT="$(dirname "$(realpath "${BASH_SOURCE[0]}")")/../.." -source "$PROJECT_ROOT/Framework-Includes/PrettyPrint.sh" -source "$PROJECT_ROOT/Framework-Includes/Logging.sh" -source "$PROJECT_ROOT/Framework-Includes/ErrorHandling.sh" -``` - -## Code Quality Standards - -### ShellCheck Compliance -- All scripts must pass shellcheck validation -- Address shellcheck warnings appropriately -- Use proper quoting for variables -- Handle edge cases and error conditions - -### Variable Naming -- Use UPPERCASE for global constants -- Use lowercase for local variables -- Use descriptive names -- Quote all variable expansions - -```bash -# Global constants -declare -g BACKUP_DIR="/root/backup" -declare -g CONFIG_FILE="/etc/ssh/sshd_config" - -# Local variables -local user_name="localuser" -local temp_file="/tmp/config.tmp" - -# Proper quoting -if [[ -f "$CONFIG_FILE" ]]; then - cp "$CONFIG_FILE" "$BACKUP_DIR/" -fi -``` - -### Function Documentation -- Include purpose description -- Document parameters if any -- Document return values -- Include usage examples for complex functions - -```bash -# Configure SSH hardening settings -# Parameters: none -# Returns: 0 on success, 1 on failure -# Usage: configure_ssh_hardening -function configure_ssh_hardening() { - print_info "Configuring SSH hardening..." - # Implementation -} -``` - -## Testing Requirements - -### Test Coverage -- Every new module must include corresponding tests -- Test both success and failure scenarios -- Validate configurations after changes -- Include integration tests for complex workflows - -### Test Categories -1. **Unit Tests:** Individual function validation -2. **Integration Tests:** Module interaction testing -3. **Security Tests:** Security configuration validation -4. **Validation Tests:** System requirement checking - -### Test Implementation Pattern -```bash -function test_function_name() { - echo "🔍 Testing specific functionality..." - - local failed=0 - - # Test implementation - if [[ condition ]]; then - echo "✅ Test passed" - else - echo "❌ Test failed" - ((failed++)) - fi - - return $failed -} -``` - -## Security Standards - -### Configuration Backup -- Always backup configurations before modification -- Use timestamped backup directories -- Provide restore instructions -- Test backup/restore procedures - -### Service Management -- Test configurations before restarting services -- Provide rollback procedures -- Validate service status after changes -- Include service dependency handling - -### User Safety -- Use `nullok` for gradual 2FA rollout -- Provide clear setup instructions -- Include emergency access procedures -- Test all access methods before enforcement - -## Documentation Standards - -### Script Headers -```bash -#!/bin/bash - -# TSYS Module Name - Brief Description -# Longer description of what this script does -# Author: TSYS Development Team -# Version: 1.0 -# Last Updated: YYYY-MM-DD - -set -euo pipefail -``` - -### Inline Documentation -- Comment complex logic -- Explain non-obvious decisions -- Document external dependencies -- Include troubleshooting notes - -### User Documentation -- Create comprehensive guides for complex features -- Include step-by-step procedures -- Provide troubleshooting sections -- Include examples and use cases - -## Performance Optimization - -### Package Management -- Single apt-get commands (as noted above) -- Cache package lists appropriately -- Use specific package versions when stability required -- Clean up package cache when appropriate - -### Network Operations -- Use connection timeouts for external requests -- Implement retry logic with backoff -- Cache downloaded resources when possible -- Validate download integrity - -### File Operations -- Use efficient file processing tools -- Minimize file system operations -- Use appropriate file permissions -- Clean up temporary files - -## Version Control Practices - -### Commit Messages -- Use descriptive commit messages -- Include scope of changes -- Reference related issues/requirements -- Follow established commit message format - -### Branch Management -- Test changes in feature branches -- Use pull requests for review -- Maintain clean commit history -- Tag releases appropriately - -### Code Review Requirements -- All changes require review -- Security changes require security team review -- Test coverage must be maintained -- Documentation must be updated - -## Deployment Practices - -### Pre-Deployment -- Run full test suite -- Validate in test environment -- Review security implications -- Update documentation - -### Deployment Process -- Use configuration validation -- Implement gradual rollout when possible -- Monitor for issues during deployment -- Have rollback procedures ready - -### Post-Deployment -- Validate deployment success -- Monitor system performance -- Update operational documentation -- Gather feedback for improvements - -## Example Implementation - -### Complete Module Template -```bash -#!/bin/bash - -# TSYS Security Module - Template -# Template for creating new security modules -# Author: TSYS Development Team - -set -euo pipefail - -# Source framework functions -PROJECT_ROOT="$(dirname "$(realpath "${BASH_SOURCE[0]}")")/../.." -source "$PROJECT_ROOT/Framework-Includes/PrettyPrint.sh" -source "$PROJECT_ROOT/Framework-Includes/Logging.sh" -source "$PROJECT_ROOT/Framework-Includes/ErrorHandling.sh" - -# Module configuration -BACKUP_DIR="/root/backup/module-$(date +%Y%m%d-%H%M%S)" -CONFIG_FILE="/etc/example.conf" - -# Create backup directory -mkdir -p "$BACKUP_DIR" - -print_header "TSYS Module Template" - -function backup_configs() { - print_info "Creating configuration backup..." - - if [[ -f "$CONFIG_FILE" ]]; then - cp "$CONFIG_FILE" "$BACKUP_DIR/" - print_success "Configuration backed up" - fi -} - -function install_packages() { - print_info "Installing required packages..." - - # Update package cache - apt-get update - - # Install all packages in single command - apt-get install -y package1 package2 package3 - - print_success "Packages installed successfully" -} - -function configure_module() { - print_info "Configuring module..." - - # Configuration logic here - - print_success "Module configured successfully" -} - -function validate_configuration() { - print_info "Validating configuration..." - - local failed=0 - - # Validation logic here - - if [[ $failed -eq 0 ]]; then - print_success "Configuration validation passed" - return 0 - else - print_error "Configuration validation failed" - return 1 - fi -} - -function main() { - # Check if running as root - if [[ $EUID -ne 0 ]]; then - print_error "This script must be run as root" - exit 1 - fi - - # Execute module steps - backup_configs - install_packages - configure_module - validate_configuration - - print_success "Module setup completed successfully!" -} - -# Run main function -main "$@" -``` - -## Continuous Improvement - -### Regular Reviews -- Review guidelines quarterly -- Update based on lessons learned -- Incorporate new best practices -- Gather team feedback - -### Tool Updates -- Keep development tools current -- Adopt new security practices -- Update testing frameworks -- Improve automation - -### Knowledge Sharing -- Document lessons learned -- Share best practices -- Provide training materials -- Maintain knowledge base - ---- - -**Last Updated:** July 14, 2025 -**Version:** 1.0 -**Author:** TSYS Development Team - -**Note:** These guidelines are living documents and should be updated as the project evolves and new best practices are identified. \ No newline at end of file +# docs/server-build/DEVELOPMENT-GUIDELINES.md + +> **Documentation moved to Discourse — the canonical source of truth.** +> +> **Coding standards, commit conventions, script patterns** +> +> **Read it here:** https://community.turnsys.com/t/302 +> +> *Migrated 2026-08-06. This file is kept as a pointer for git-browsing context. +> Do not update content here — edit the Discourse wiki topic instead.* diff --git a/docs/server-build/SECURITY.md b/docs/server-build/SECURITY.md index 5d7162d..819cf5e 100644 --- a/docs/server-build/SECURITY.md +++ b/docs/server-build/SECURITY.md @@ -1,190 +1,10 @@ -# PFVCluster Security Documentation +# docs/server-build/SECURITY.md -## Security Architecture - -The PFVCluster infrastructure provisioning system is designed with security-first principles, implementing multiple layers of protection for server deployment and management. - -## Current Security Features - -### 1. Secure Deployment Method ✅ -- **Git-based deployment:** Uses `git clone` instead of `curl | bash` -- **Local execution:** Scripts run locally after inspection -- **Version control:** Full audit trail of changes -- **Code review:** Changes require explicit approval - -### 2. HTTPS Enforcement ✅ -- **All downloads use HTTPS:** Eliminates man-in-the-middle attacks -- **SSL certificate validation:** Automatic certificate checking -- **Secure repositories:** Ubuntu archive, Dell, Proxmox all use HTTPS -- **No HTTP fallbacks:** No insecure download methods - -### 3. SSH Hardening -- **Key-only authentication:** Password login disabled -- **Secure ciphers:** Modern encryption algorithms only -- **Fail2ban protection:** Automated intrusion prevention -- **Custom SSH configuration:** Hardened sshd_config - -### 4. System Security -- **Firewall configuration:** Automated iptables rules -- **Audit logging:** auditd with custom rules -- **SIEM integration:** Wazuh agent deployment -- **Compliance scanning:** SCAP-STIG automated checks - -### 5. Error Handling -- **Bash strict mode:** `set -euo pipefail` prevents errors -- **Centralized logging:** All operations logged with timestamps -- **Graceful failures:** Proper cleanup on errors -- **Line-level debugging:** Error reporting with line numbers - -## Security Testing - -### Automated Security Validation -```bash -# Run security test suite -./tests/run-tests.sh security - -# Specific security tests -./tests/security/https-enforcement.sh -``` - -### Security Test Categories -1. **HTTPS Enforcement:** Validates all URLs use HTTPS -2. **Deployment Security:** Checks for secure deployment methods -3. **SSL Certificate Validation:** Tests certificate authenticity -4. **Permission Validation:** Verifies proper file permissions - -## Threat Model - -### Mitigated Threats -- **Supply Chain Attacks:** Git-based deployment with review -- **Man-in-the-Middle:** HTTPS-only downloads -- **Privilege Escalation:** Proper permission models -- **Unauthorized Access:** SSH hardening and key management - -### Remaining Risks -- **Secrets in Repository:** SSH keys stored in git (planned for removal) -- **No Integrity Verification:** Downloads lack checksum validation -- **No Backup/Recovery:** No rollback capability implemented - -## Security Recommendations - -### High Priority -1. **Implement Secrets Management** - - Remove SSH keys from repository - - Use Bitwarden/Vault for secret storage - - Implement key rotation procedures - -2. **Add Download Integrity Verification** - - SHA256 checksum validation for all downloads - - GPG signature verification where available - - Fail-safe on integrity check failures - -3. **Enhance Audit Logging** - - Centralized log collection - - Real-time security monitoring - - Automated threat detection - -### Medium Priority -1. **Configuration Backup** - - System state snapshots before changes - - Rollback capability for failed deployments - - Configuration drift detection - -2. **Network Security** - - VPN-based deployment (where applicable) - - Network segmentation for management - - Encrypted communication channels - -## Compliance - -### Security Standards -- **CIS Benchmarks:** Automated compliance checking -- **STIG Guidelines:** SCAP-based validation -- **Industry Best Practices:** Following NIST cybersecurity framework - -### Audit Requirements -- **Change Tracking:** All modifications logged -- **Access Control:** Permission-based system access -- **Vulnerability Management:** Regular security assessments - -## Incident Response - -### Security Event Handling -1. **Detection:** Automated monitoring and alerting -2. **Containment:** Immediate isolation procedures -3. **Investigation:** Log analysis and forensics -4. **Recovery:** System restoration procedures -5. **Lessons Learned:** Process improvement - -### Contact Information -- **Security Team:** [To be defined] -- **Incident Response:** [To be defined] -- **Escalation Path:** [To be defined] - -## Security Development Lifecycle - -### Code Review Process -1. **Static Analysis:** Automated security scanning -2. **Peer Review:** Manual code inspection -3. **Security Testing:** Automated security test suite -4. **Approval:** Security team sign-off - -### Deployment Security -1. **Pre-deployment Validation:** Security test execution -2. **Secure Deployment:** Authorized personnel only -3. **Post-deployment Verification:** Security configuration validation -4. **Monitoring:** Continuous security monitoring - -## Security Tools and Integrations - -### Current Tools -- **Wazuh:** SIEM and security monitoring -- **Lynis:** Security auditing -- **auditd:** System call auditing -- **Fail2ban:** Intrusion prevention - -### Planned Integrations -- **Vault/Bitwarden:** Secrets management -- **OSSEC:** Host-based intrusion detection -- **Nessus/OpenVAS:** Vulnerability scanning -- **ELK Stack:** Log aggregation and analysis - -## Vulnerability Management - -### Vulnerability Scanning -- **Regular scans:** Monthly vulnerability assessments -- **Automated patching:** Security update automation -- **Exception handling:** Risk-based patch management -- **Reporting:** Executive security dashboards - -### Disclosure Process -1. **Internal Discovery:** Report to security team -2. **Assessment:** Risk and impact evaluation -3. **Remediation:** Patch development and testing -4. **Deployment:** Coordinated security updates -5. **Verification:** Post-patch validation - -## Security Metrics - -### Key Performance Indicators -- **Deployment Success Rate:** Percentage of successful secure deployments -- **Vulnerability Response Time:** Time to patch critical vulnerabilities -- **Security Test Coverage:** Percentage of code covered by security tests -- **Incident Response Time:** Time to detect and respond to security events - -### Monitoring and Reporting -- **Real-time Dashboards:** Security status monitoring -- **Executive Reports:** Monthly security summaries -- **Compliance Reports:** Quarterly compliance assessments -- **Trend Analysis:** Security posture improvement tracking - -## Contact and Support - -For security-related questions or incidents: -- **Repository Issues:** https://projects.knownelement.com/project/reachableceo-vptechnicaloperations/timeline -- **Community Discussion:** https://community.turnsys.com/c/chieftechnologyandproductofficer/26 -- **Security Team:** [Contact information to be added] - -## Security Updates - -This document is updated as security features are implemented and threats evolve. Last updated: July 14, 2025. \ No newline at end of file +> **Documentation moved to Discourse — the canonical source of truth.** +> +> **Security architecture: SSH hardening, 2FA, SCAP-STIG, Wazuh, auditd** +> +> **Read it here:** https://community.turnsys.com/t/303 +> +> *Migrated 2026-08-06. This file is kept as a pointer for git-browsing context. +> Do not update content here — edit the Discourse wiki topic instead.* diff --git a/docs/server-build/TSYS-2FA-GUIDE.md b/docs/server-build/TSYS-2FA-GUIDE.md index b9c466e..853a13f 100644 --- a/docs/server-build/TSYS-2FA-GUIDE.md +++ b/docs/server-build/TSYS-2FA-GUIDE.md @@ -1,329 +1,10 @@ -# TSYS Two-Factor Authentication Implementation Guide - -## Overview - -This guide provides complete instructions for implementing and managing two-factor authentication (2FA) on TSYS servers using Google Authenticator (TOTP). - -## What This Implementation Provides - -### Services Protected by 2FA -- **SSH Access:** Requires SSH key + 2FA token -- **Cockpit Web Interface:** Requires password + 2FA token -- **Webmin Administration:** Requires password + 2FA token (if installed) - -### Security Features -- **Time-based One-Time Passwords (TOTP):** Standard 6-digit codes -- **Backup Codes:** Emergency access codes -- **Gradual Rollout:** Optional nullok mode for phased deployment -- **Configuration Backup:** Automatic backup of all configs - -## Implementation Steps - -### Step 1: Run the 2FA Setup Script -```bash -# Navigate to the security modules directory -cd provisioning/Modules/Security - -# Run the 2FA setup script as root -sudo bash secharden-2fa.sh -``` - -### Step 2: Validate Installation -```bash -# Run 2FA validation tests -./Project-Tests/security/2fa-validation.sh - -# Run specific 2FA security test -./Project-Tests/run-tests.sh security -``` - -### Step 3: Setup Individual Users -For each user that needs 2FA access: - -```bash -# Check setup instructions -cat /home/username/2fa-setup-instructions.txt - -# Run user setup script -sudo /tmp/setup-2fa-username.sh -``` - -### Step 4: Test 2FA Access -1. **Test SSH access** from another terminal -2. **Test Cockpit access** via web browser -3. **Test Webmin access** if installed - -## User Setup Process - -### Installing Authenticator Apps -Users need one of these apps on their phone: -- **Google Authenticator** (Android/iOS) -- **Authy** (Android/iOS) -- **Microsoft Authenticator** (Android/iOS) -- **1Password** (with TOTP support) - -### Setting Up 2FA for a User -1. **Run setup script:** - ```bash - sudo /tmp/setup-2fa-username.sh - ``` - -2. **Follow prompts:** - - Answer "y" to update time-based token - - Scan QR code with authenticator app - - Save emergency backup codes securely - - Answer "y" to remaining security questions - -3. **Test immediately:** - ```bash - # Test SSH from another terminal - ssh username@server-ip - # You'll be prompted for 6-digit code - ``` - -## Configuration Details - -### SSH Configuration Changes -File: `/etc/ssh/sshd_config` -``` -ChallengeResponseAuthentication yes -UsePAM yes -AuthenticationMethods publickey,keyboard-interactive -``` - -### PAM Configuration -File: `/etc/pam.d/sshd` -``` -auth required pam_google_authenticator.so nullok -``` - -### Cockpit Configuration -File: `/etc/cockpit/cockpit.conf` -``` -[WebService] -LoginTitle = TSYS Server Management -LoginTo = 300 -RequireHost = true - -[Session] -Banner = /etc/cockpit/issue.cockpit -IdleTimeout = 15 -``` - -### Webmin Configuration -File: `/etc/webmin/miniserv.conf` -``` -twofactor_provider=totp -twofactor=1 -``` - -## Security Considerations - -### Gradual vs Strict Enforcement - -#### Gradual Enforcement (Default) -- Uses `nullok` option in PAM -- Users without 2FA can still log in -- Allows phased rollout -- Good for initial deployment - -#### Strict Enforcement -- Remove `nullok` from PAM configuration -- All users must have 2FA configured -- Immediate security enforcement -- Risk of lockout if misconfigured - -### Backup and Recovery - -#### Emergency Access -- **Backup codes:** Generated during setup -- **Root access:** Can disable 2FA if needed -- **Console access:** Physical/virtual console bypasses SSH - -#### Configuration Backup -- Automatic backup to `/root/backup/2fa-TIMESTAMP/` -- Includes all modified configuration files -- Can be restored if needed - -## Troubleshooting - -### Common Issues - -#### 1. User Cannot Generate QR Code -```bash -# Ensure qrencode is installed -sudo apt-get install qrencode - -# Re-run user setup -sudo /tmp/setup-2fa-username.sh -``` - -#### 2. SSH Connection Fails -```bash -# Check SSH service status -sudo systemctl status sshd - -# Test SSH configuration -sudo sshd -t - -# Check logs -sudo journalctl -u sshd -f -``` - -#### 3. 2FA Code Not Accepted -- **Check time synchronization** on server and phone -- **Verify app setup** - rescan QR code if needed -- **Try backup codes** if available - -#### 4. Locked Out of Server -```bash -# Access via console (physical/virtual) -# Disable 2FA temporarily -sudo cp /root/backup/2fa-*/pam.d.bak/sshd /etc/pam.d/sshd -sudo systemctl restart sshd -``` - -### Debug Commands - -```bash -# Check 2FA status -./Project-Tests/security/2fa-validation.sh - -# Check SSH configuration -sudo sshd -T | grep -E "(Challenge|PAM|Authentication)" - -# Check PAM configuration -cat /etc/pam.d/sshd | grep google-authenticator - -# Check user 2FA status -ls -la ~/.google_authenticator -``` - -## Management and Maintenance - -### Adding New Users -1. Ensure user account exists -2. Run setup script for new user -3. Provide setup instructions -4. Test access - -### Removing User 2FA -```bash -# Remove user's 2FA configuration -sudo rm /home/username/.google_authenticator - -# User will need to re-setup 2FA -``` - -### Disabling 2FA System-Wide -```bash -# Restore original configurations -sudo cp /root/backup/2fa-*/sshd_config.bak /etc/ssh/sshd_config -sudo cp /root/backup/2fa-*/pam.d.bak/sshd /etc/pam.d/sshd -sudo systemctl restart sshd -``` - -### Updating 2FA Configuration -```bash -# Re-run setup script -sudo bash secharden-2fa.sh - -# Validate changes -./Project-Tests/security/2fa-validation.sh -``` - -## Best Practices - -### Deployment Strategy -1. **Test in non-production** environment first -2. **Enable gradual rollout** (nullok) initially -3. **Train users** on 2FA setup process -4. **Test emergency procedures** before strict enforcement -5. **Monitor logs** for authentication issues - -### Security Recommendations -- **Enforce strict mode** after successful rollout -- **Regular backup code rotation** -- **Monitor failed authentication attempts** -- **Document emergency procedures** -- **Regular security audits** - -### User Training -- **Provide clear instructions** -- **Demonstrate setup process** -- **Explain backup code importance** -- **Test login process with users** -- **Establish support procedures** - -## Monitoring and Logging - -### Authentication Logs -```bash -# SSH authentication logs -sudo journalctl -u sshd | grep -i "authentication" - -# PAM authentication logs -sudo journalctl | grep -i "pam_google_authenticator" - -# Failed login attempts -sudo journalctl | grep -i "failed" -``` - -### Security Monitoring -- Monitor for repeated failed 2FA attempts -- Alert on successful logins without 2FA (during gradual rollout) -- Track user 2FA setup completion -- Monitor for emergency access usage - -## Integration with Existing Systems - -### LDAP/Active Directory -- 2FA works with existing authentication systems -- Users still need local 2FA setup -- Consider centralized 2FA solutions for large deployments - -### Monitoring Systems -- LibreNMS: Will continue to work with SNMP -- Wazuh: Will log 2FA authentication events -- Cockpit: Enhanced with 2FA protection - -### Backup Systems -- Ensure backup procedures account for 2FA -- Test restore procedures with 2FA enabled -- Document emergency access procedures - -## Support and Resources - -### Files Created by Setup -- `/tmp/setup-2fa-*.sh` - User setup scripts -- `/home/*/2fa-setup-instructions.txt` - User instructions -- `/root/backup/2fa-*/` - Configuration backups - -### Validation Tools -- `./Project-Tests/security/2fa-validation.sh` - Complete 2FA validation -- `./Project-Tests/run-tests.sh security` - Security test suite - -### Emergency Contacts -- System Administrator: [Contact Info] -- Security Team: [Contact Info] -- 24/7 Support: [Contact Info] - -## Compliance and Audit - -### Security Benefits -- Significantly reduces risk of unauthorized access -- Meets multi-factor authentication requirements -- Provides audit trail of authentication events -- Complies with security frameworks (NIST, ISO 27001) - -### Audit Trail -- All authentication attempts logged -- 2FA setup events recorded -- Configuration changes tracked -- Emergency access documented - ---- - -**Last Updated:** July 14, 2025 -**Version:** 1.0 -**Author:** TSYS Security Team \ No newline at end of file +# docs/server-build/TSYS-2FA-GUIDE.md + +> **Documentation moved to Discourse — the canonical source of truth.** +> +> **End-user guide for 2FA setup** +> +> **Read it here:** https://community.turnsys.com/t/303 +> +> *Migrated 2026-08-06. This file is kept as a pointer for git-browsing context. +> Do not update content here — edit the Discourse wiki topic instead.* diff --git a/docs/server-build/tailscale.md b/docs/server-build/tailscale.md index e5f8355..89b03b9 100644 --- a/docs/server-build/tailscale.md +++ b/docs/server-build/tailscale.md @@ -1,148 +1,10 @@ -# Tailscale vs. Managed DNS — Architecture Analysis +# docs/server-build/tailscale.md -> **Status:** **RESOLVED.** The pfv-netinfra-01/02 pair now runs production -> Technitium DNS with all `knel.net` records replicated from tailscale-router -> via the DNS cluster setup. Both LAN IPs serve authoritative records for -> `knel.net` and recurse externally. This document records the original -> conflict, how it was resolved, and the recommended client configuration. - -## 1. Executive summary - -Every host in this build runs the Tailscale client, and Tailscale's MagicDNS -manages `/etc/resolv.conf` by default (pointing at `100.100.100.100`). This -previously conflicted with a managed `resolv.conf` pointing at the LAN -resolvers. The root cause was that the LAN Technitium instances did not have -the `knel.net` zone populated — **that is now fixed.** - -The pfv-netinfra-01/02 pair now serves identical, authoritative `knel.net` -records (replicated from production via [`dns-cluster-setup/`](../dns-cluster-setup/README.md)). -Both LAN IPs resolve `knel.net` device names and recurse externally. The -managed `resolv.conf` is now safe to deploy. - -**Recommendation:** Deploy the managed `resolv.conf` (`.252`/`.253`) on hosts -where you want tunnel-independent DNS. Leave Tailscale managing DNS on hosts -where MagicDNS device names must resolve without a LAN path (e.g. laptops off --network). See [§5](#5-recommendation) for details. - -## 2. How name resolution works today (post-cluster-setup) - -Probed from `sectestbed-sandbox` (192.168.3.50) after the DNS cluster was -deployed: - -| Query path | External name (`github.com`) | `knel.net` device name (`pfv-netinfra-01.knel.net`) | -|---|---|---| -| Via Tailscale resolver (`100.100.100.100`) | resolves | resolves → `100.70.181.72` (Tailscale CGNAT) | -| Direct `dig @192.168.3.252` (Technitium primary, LAN) | resolves (recurses) | **resolves** → `100.70.181.72` | -| Direct `dig @192.168.3.253` (Technitium secondary, LAN) | resolves (recurses) | **resolves** → `100.70.181.72` | - -**Both LAN resolvers now serve `knel.net` records identically.** The -Technitium zone is no longer stale — it was replicated from production -(tailscale-router) as part of the DNS cluster setup. - -### What changed - -Previously (before the DNS cluster setup), querying the LAN IPs returned -NXDOMAIN for `knel.net` device names because the Technitium `knel.net` zone -was empty (SOA serial `2025062313`, dated 2025-06-23). After replicating -production config to both netinfra hosts, all 124 zones — including -`knel.net` with all current device records — are served authoritatively on -both `.252` and `.253`. - -## 3. The DNS server pair - -| Host | IP | Role | Services | -|------|----|------|----------| -| pfv-netinfra-01 | 192.168.3.252 | **Primary** | Technitium (authoritative, port 5300) + Pi-hole (recursive, port 53) | -| pfv-netinfra-02 | 192.168.3.253 | **Secondary** | Technitium (replicated via rsync, port 5300) + Pi-hole (recursive, port 53) | - -Zone replication is rsync-based (every 60s via systemd timer) because -Technitium's AXFR uses port 53, which is occupied by Pi-hole on these hosts. -See [`dns-cluster-setup/README.md`](../dns-cluster-setup/README.md) for -full details. - -## 4. NTP (fully resolved) - -NTP is independent of DNS: `provisioning/ConfigFiles/NTP/ntp.conf` points -directly at the LAN IPs with no DNS dependency: - -``` -server 192.168.3.252 iburst -server 192.168.3.253 iburst -``` - -Both servers respond with stratum 2/3. The client config uses `restrict` -rules (not `interface listen`) to avoid the loopback-binding bug that -prevented sync. This is safe under both Tailscale-managed and LAN-pinned -resolver configurations. - -## 5. Recommendation - -### On fixed servers (always on-LAN) - -**Deploy the managed `resolv.conf`** (`provisioning/ConfigFiles/Resolv/`): -- Points at `.252`/`.253` with failover -- `knel.net` records resolve on both servers -- External names recurse on both servers -- DNS survives `tailscaled` outages (unlike Tailscale-managed DNS) - -To prevent Tailscale from overwriting the managed file: -```bash -tailscale up --accept-dns=false -``` - -### On laptops / roaming hosts - -**Let Tailscale manage DNS** (default `accept-dns=true`): -- MagicDNS resolves `knel.net` device names via the tunnel -- No dependency on LAN reachability -- Accept the `tailscaled` dependency (if the tunnel is down, you're off-network anyway) - -## 6. Subnet router HA (redundant routing) - -> **Redmine:** [#377](https://projects.knownelement.com/issues/377) - -Both DNS nodes advertise the LAN subnet as Tailscale subnet routes AND -serve as exit nodes, providing full HA for remote monitoring (Uptime Kuma -on the Cloudron VPS), off-site access, and internet egress: - -| Node | Tailscale IP | Advertised routes | Role | -|------|-------------|-------------------|------| -| pfv-netinfra-01 | 100.70.181.72 | 192.168.0.0/22, 0.0.0.0/0, ::/0 | Primary subnet router + exit node | -| pfv-netinfra-02 | 100.71.171.20 | 192.168.0.0/22, 0.0.0.0/0, ::/0 | Secondary subnet router + exit node (failover) | - -To configure on a node (both flags together): -```bash -sudo tailscale set --advertise-routes=192.168.0.0/22 --advertise-exit-node -``` - -**All routes must be approved** in the Tailscale admin console -(https://login.tailscale.com/admin/machines) for failover to activate. -This includes the subnet route (`192.168.0.0/22`) AND the exit node routes -(`0.0.0.0/0`, `::/0`) — they appear as separate approvals on each machine. -Tailscale uses the primary by default and fails over to the secondary if -the primary's tunnel drops. - -### Known instability factors (2026-08-06, [#377]) - -- **DERP relay bouncing:** netinfra-01 was observed switching between DERP - relays (#9, #12, #13) every ~15 minutes, causing brief tunnel disruptions. -- **Docker LinkChange events:** container create/destroy adds/removes bridge - interfaces, triggering `LinkChange: major, rebinding` in tailscaled and - momentarily dropping all connections. -- **resolv.conf immutable flag:** netinfra-01 had the `+i` (immutable) - attribute on `/etc/resolv.conf`, preventing tailscaled from updating DNS - config after link changes. Fixed by `chattr -i /etc/resolv.conf`. - -## 7. Known items / future work - -1. **Pi-hole upstream configuration.** Pi-hole on both hosts should forward - to the local Technitium instance (port 5300) for `knel.net` and to an - external resolver for everything else. Verify this is configured on both - nodes. -2. **Zone transfer via AXFR.** Currently using rsync because Technitium's - AXFR expects port 53. If Technitium's listen port can be changed, or - Pi-hole can be configured to proxy AXFR, the rsync timer could be - replaced with native DNS zone transfer. -3. **`accept-dns=false` automation.** The provisioning code should set - `--accept-dns=false` on Tailscale during setup (after deploying the - managed `resolv.conf`) so Tailscale doesn't overwrite it on reboot. +> **Documentation moved to Discourse — the canonical source of truth.** +> +> **Tailscale vs managed DNS analysis (resolved)** +> +> **Read it here:** https://community.turnsys.com/t/306 +> +> *Migrated 2026-08-06. This file is kept as a pointer for git-browsing context. +> Do not update content here — edit the Discourse wiki topic instead.* diff --git a/k8s/README.md b/k8s/README.md index 4c60908..d568cd5 100644 --- a/k8s/README.md +++ b/k8s/README.md @@ -1,57 +1,10 @@ -# k8s/ — pfv-k8s Cluster Setup Scripts +# k8s/README.md -> **Redmine:** [#362](https://projects.knownelement.com/issues/362) (initial build, closed) · [#367](https://projects.knownelement.com/issues/367) (rebuild, open) · [#368](https://projects.knownelement.com/issues/368) (worker join, open) - -Scripts to bootstrap and manage the k3s control plane on cnode1/2/3. -All cluster communication goes over Tailscale IPs — no LAN traffic. - -## Current State - -3-node HA control plane (k3s v1.36.2+k3s1, embedded etcd): - -| Node | Tailscale IP | Role | Tainted | -|------|-------------|------|---------| -| pfv-k8s-cnode1 | 100.97.178.106 | control-plane, etcd | NoSchedule | -| pfv-k8s-cnode2 | 100.109.34.72 | control-plane, etcd | NoSchedule | -| pfv-k8s-cnode3 | 100.106.222.18 | control-plane, etcd | NoSchedule | - -## Scripts - -| Script | Purpose | -|--------|---------| -| [`env.sh`](env.sh) | Shared config: node IPs, SSH opts, k3s version. Sourced by all scripts. | -| [`wipe.sh`](wipe.sh) | Remove existing k3s from all cnodes (clean slate). | -| [`install-cp.sh`](install-cp.sh) | Full bootstrap: cnode1 (--cluster-init) then cnode2/3 join. | -| [`join-servers.sh`](join-servers.sh) | Re-join cnode2/3 only (if cnode1 is already up). | -| [`post-setup.sh`](post-setup.sh) | Apply NoSchedule taints, fetch kubeconfig, verify. | -| [`verify.sh`](verify.sh) | Health check: nodes Ready, Tailscale IPs, taints, etcd, CoreDNS. | -| [`probe-nodes.sh`](probe-nodes.sh) | Verify SSH + Tailscale reachability. | - -## Usage - -```bash -# Full bootstrap from scratch: -bash k8s/wipe.sh -bash k8s/install-cp.sh -bash k8s/post-setup.sh -bash k8s/verify.sh - -# Access the cluster: -export KUBECONFIG=~/.kube/config.pfv-k8s -kubectl get nodes -``` - -## Design Decisions - -- **k3s (not Talos):** This is a regular R&D cluster, not ITAR/classified. - Talos architecture is documented in [`docs/`](docs/) for when - that requirement comes online. k3s on stock Debian is simpler to operate. -- **Tailscale-only transport:** `--node-ip`, `--advertise-address`, and - `--tls-san` are all set to Tailscale IPs. No LAN IP appears in any node - status or certificate. -- **VXLAN flannel:** Pods communicate via flannel VXLAN overlay on top of - Tailscale's WireGuard. Double-encrypted, but functional and reliable. -- **NoSchedule taint:** All 3 cnodes are tainted so no user workloads - schedule on the control plane. Only system components (CoreDNS, - metrics-server, flannel, kube-proxy) with built-in tolerations run here. -- **Embedded etcd:** 3-node HA etcd quorum. Tolerates 1 node failure. +> **Documentation moved to Discourse — the canonical source of truth.** +> +> **k3s cluster setup scripts: wipe, bootstrap, taint, verify** +> +> **Read it here:** https://community.turnsys.com/t/305 +> +> *Migrated 2026-08-06. This file is kept as a pointer for git-browsing context. +> Do not update content here — edit the Discourse wiki topic instead.* diff --git a/k8s/docs/ARCHITECTURE.md b/k8s/docs/ARCHITECTURE.md index 9df9c3f..f2d3e7b 100644 --- a/k8s/docs/ARCHITECTURE.md +++ b/k8s/docs/ARCHITECTURE.md @@ -1,673 +1,10 @@ -# pfv-k8s Target Architecture (Talos) - -> **Companion to:** [`DISTRO-DECISION.md`](DISTRO-DECISION.md) (why Talos), -> [`../proxmox/K8S.md`](../proxmox/K8S.md) (storage/host analysis from the -> Proxmox audit). - -**Last updated:** 2026-07-28 - ---- - -## Table of Contents - -1. [High-Level Architecture](#1-high-level-architecture) -2. [Control Plane](#2-control-plane) -3. [Network Topology](#3-network-topology) -4. [CNI: Cilium](#4-cni-cilium) -5. [Identity and Trust](#5-identity-and-trust) -6. [Tenant Isolation (vcluster)](#6-tenant-isolation-vcluster) -7. [Storage Integration](#7-storage-integration) -8. [Local Image Registry](#8-local-image-registry) -9. [Bootstrap Procedure](#9-bootstrap-procedure) -10. [Disaster Recovery](#10-disaster-recovery) -11. [Migration from Current State](#11-migration-from-current-state) - ---- - -## 1. High-Level Architecture - -```mermaid -flowchart TB - subgraph RESIDENCE["Residence — Proxmox LAN"] - subgraph CP["Talos Control Plane (3 cnodes)"] - C1[cnode1
tsys9 · local-SSD] - C2[cnode2
tsys9 · local-SSD] - C3[cnode3
tsys1 · local-HDD] - end - subgraph WP["Talos Worker Plane"] - W3[wnode-tsys3
NVMe · 28GB] - W5[wnode-tsys5
NVMe · 32-64GB] - W6[wnode-tsys6
NFS-HDD · 64-96GB] - W7[wnode-tsys7
NFS-HDD · 96-128GB] - W9[wnode-tsys9
local-SSD · 4-8GB] - end - ETCD[(etcd
raft, mTLS)] - REG[(Harbor registry
on D3 SSD · tsys5)] - BASTION[tailscale-router VM
subnet router] - end - - subgraph TAILNET["Tailscale overlay"] - OP[Operator devices] - end - - subgraph CLOUDRON["Cloudron production — Reston VA"] - KC[Keycloak OIDC IdP] - end - - C1 ---|mTLS LAN| ETCD - C2 ---|mTLS LAN| ETCD - C3 ---|mTLS LAN| ETCD - CP -->|pull images| REG - WP -->|pull images| REG - - OP -->|Talos API :50000
via subnet route| BASTION - BASTION -.->|LAN| CP - CP -->|OIDC| KC - WP -->|OIDC| KC - - classDef talos fill:#1a1a2e,stroke:#e94560,color:#fff - classDef infra fill:#0f3460,stroke:#e94560,color:#fff - classDef external fill:#16213e,stroke:#533483,color:#fff - class CP,WP,ETCD talos - class REG,BASTION infra - class OP,KC,EXTERNAL external -``` - -### Design principles - -1. **LAN-only cluster nodes.** Zero internet egress from cnodes/wnodes. - Strongest posture for ITAR/classified. -2. **Admin via Tailscale subnet router.** Existing `tailscale-router` VM - advertises the cluster LAN subnet. Operator reaches Talos API from - anywhere. -3. **Local-first storage.** Cnodes boot from local disk (no NFS dependency - for etcd). Workers boot from local disk where available; NFS for bulk - data only. -4. **Per-tenant vcluster.** Workload isolation via virtual clusters on top - of the Talos host cluster. -5. **OIDC everywhere.** Talos API and Kubernetes API both trust Keycloak - tokens. No long-lived static credentials for humans. - ---- - -## 2. Control Plane - -### 2.1 Recommendation: 3 cnodes (down from 5) - -| Option | Quorum | Failure tolerance | etcd write cost | Resource cost | -|--------|--------|-------------------|-----------------|---------------| -| **3 cnodes** (recommended) | 2 of 3 | Tolerates **1** failure | Lower (faster commits) | 3 × (2c/4GB/32GB) = 6c / 12GB | -| 5 cnodes (current plan) | 3 of 5 | Tolerates **2** failures | Higher | 5 × (2c/4GB/32GB) = 10c / 20GB | - -For a solo-operated R&D cluster, **3 cnodes is the HA standard**. The -failure-tolerance jump from 1→2 rarely justifies the doubled etcd write -quorum and the extra 4GB/2c per cnode. The 2 freed VM slots (and their -host capacity) are better spent on tenant worker allocations. - -**Caveat:** if your ITAR/classified accreditation counsel mandates 2-failure -tolerance on the control plane, keep 5. Otherwise 3. - -### 2.2 Cnode placement - -Per [`../proxmox/K8S.md`](../proxmox/K8S.md) §4.3, cnodes should use -**local-lvm boot disks** so etcd has no NFS dependency. Concrete placement: - -| cnode | Host | Boot disk | Type | Why | -|-------|------|-----------|------|-----| -| cnode1 | tsys9 | local-lvm (PNY CS900 SSD) | LOCAL-SSD | Fastest available for etcd. | -| cnode2 | tsys9 | local-lvm (PNY CS900 SSD) | LOCAL-SSD | Same host, different disk OK (host failure is the failure domain, not disk). | -| cnode3 | tsys1 | local-lvm (HDD) | LOCAL-HDD | Host diversity. Slower than SSD but no NFS hop. | - -**Quorum survival:** - -| Failure | cnodes lost | Quorum OK? | -|---------|-------------|------------| -| tsys9 host dies | cnode1 + cnode2 | NO (1 of 3) — would need 4th cnode elsewhere, or accept this risk. | -| tsys1 host dies | cnode3 | YES (2 of 3) | -| Any storage server dies | 0 | YES (3 of 3) — local disks unaffected | - -**Refinement:** putting both SSD cnodes on tsys9 means tsys9 host failure -loses quorum. Alternative: spread cnodes across 3 different hosts. See -"open question" at end of this section. - -### 2.3 Machine config strategy - -Talos nodes are configured by **machine configs** (YAML). Two flavors: - -- **`controlplane.yaml`** — for cnodes. Enables etcd, scheduler, - controller-manager, API server. -- **`worker.yaml`** — for wnodes. Joins cluster, runs kubelet + containerd. - -Strategy for this cluster: - -1. **One shared `talosconfig`** (client identity) — stored in 1Password - and in the Proxmox Backup Server (PBS) encrypted backup target. -2. **Per-node machine config patches** — small patches on top of the base - `controlplane.yaml` / `worker.yaml` for node-specific settings: - - Hostname - - Network interface + IP (DHCP or static — recommend static for cnodes) - - Schematic image digest (pinned Talos version) - - System extensions (e.g., `tailscale` — only if running Pattern A - instead of recommended Pattern C) -3. **All machine configs in Git** under a future `k8s/talos-configs/` - directory. Secrets are templated in at apply-time from 1Password / sops. - -```mermaid -flowchart LR - BASE[base controlplane.yaml] --> PATCH1[patch: cnode1] - BASE --> PATCH2[patch: cnode2] - BASE --> PATCH3[patch: cnode3] - BASEW[base worker.yaml] --> PATCHW[patch: per-wnode] - PATCH1 --> APPLY1[talosctl apply] - PATCH2 --> APPLY2[talosctl apply] - PATCH3 --> APPLY3[talosctl apply] - PATCHW --> APPLYW[talosctl apply] -``` - -### 2.4 Open question: cnode host spread - -If you accept "tsys9 failure = quorum loss" as a tolerable risk (solo R&D -cluster, tsys9 is brand-new hardware, single digit annual failure -probability), the layout in §2.2 is fine. - -If not, alternative spread across 3 hosts: - -| cnode | Host | Boot disk | -|-------|------|-----------| -| cnode1 | tsys9 | local-lvm SSD | -| cnode2 | tsys1 | local-lvm HDD | -| cnode3 | tsys3 | local-lvm NVMe | - -tsys3's local-lvm is **349 GB Samsung PM961 NVMe** (per -[`../proxmox/PROJECT.md`](../proxmox/PROJECT.md) §3.3) — currently unused, -would make an excellent etcd disk. - -**This 3-host spread survives any single host failure with quorum intact. -Recommended.** - ---- - -## 3. Network Topology - -### 3.1 Zones - -```mermaid -flowchart TB - subgraph INTERNET["Internet"] - FIBER[Gigabit symmetric fiber] - end - - subgraph RESLAN["Residence LAN 192.168.x.x/24"] - subgraph CLUSTERNET["Cluster nodes — LAN only, no egress"] - CNODES[Cnodes 192.168.3.x] - WNODES[Wnodes 192.168.3.x] - end - BASTION[tailscale-router
192.168.3.x + 100.x.x.x] - REG[Harbor registry
192.168.3.x] - STORAGE[NFS servers
tsys4, tsys5] - end - - subgraph TSNET["Tailscale 100.x.x.x/8"] - OPS[Operator devices] - KC[Keycloak
via Cloudron prod] - end - - FIBER --> BASTION - BASTION <-. subnet route .-> CLUSTERNET - OPS -->|TCP 50000 talos API| BASTION - BASTION -->|LAN forward| CNODES - CNODES -->|LAN mTLS| WNODES - CNODES -->|OIDC HTTPS| KC - WNODES -->|pull images| REG - WNODES -->|bulk data IO| STORAGE - CNODES -->|pull images| REG -``` - -### 3.2 Address plan (suggested) - -Reserve a small contiguous block in the residence LAN for cluster nodes: - -| Role | Range | Count | -|------|-------|-------| -| Cnodes | `192.168.3.31-33` | 3 | -| Wnodes | `192.168.3.41-49` | up to 9 (1 per Proxmox host + spare) | -| Bastion | existing `tailscale-router` | 1 | -| Registry | `192.168.3.50` | 1 (Harbor) | - -**Static IPs are strongly recommended for cnodes** (etcd cluster membership -is hostname-based; stable IPs make `talosctl` targeting simple). Workers -can DHCP. - -### 3.3 Firewall posture - -Each cnode/wnode has: - -- **Ingress** from LAN: TCP 50000 (Talos API), TCP 6443 (Kubernetes API on - cnodes only), plus CNI ports (varies by CNI — see §4). -- **Ingress** from Tailscale: none (cluster nodes are not on Tailscale). -- **Egress:** LAN-only. Block all RFC1918-external traffic at the perimeter - firewall for these IPs. ITAR workloads must not be able to phone home. - -The bastion runs Tailscale and forwards TCP 50000/6443 to cluster nodes -via the subnet route. - ---- - -## 4. CNI: Cilium - -**Recommendation: Cilium** (eBPF-based CNI). - -| Property | Why it matters here | -|----------|---------------------| -| **NetworkPolicy** (incl. L7) | Per-tenant isolation rules in vclusters. | -| **Node-to-node encryption** | WireGuard-based IPSec replacement. All inter-node pod traffic is encrypted on the wire. **Important for ITAR tenants.** | -| **Hubble** | Observable flows — forensic record of which pod talked to which. Useful for compliance evidence. | -| **No kube-proxy** | Cilium replaces kube-proxy with eBPF. Smaller attack surface on each node. | -| **Talos integration** | First-class. Talos docs document the install path. | - -Cilium is deployed via Helm after cluster bootstrap. Node-to-node encryption -enabled. Default-deny NetworkPolicy applied per namespace. - ---- - -## 5. Identity and Trust - -### 5.1 Trust flow - -```mermaid -sequenceDiagram - autonumber - participant Human as Operator - participant TAIL as Tailscale - participant BAST as Bastion - participant TALOS as Talos API :50000 - participant KC as Keycloak (Cloudron) - participant K8S as Kubernetes API :6443 - - Human->>TAIL: Authenticate (device + SSO) - TAIL-->>Human: Tailnet IP - Human->>BAST: Reach bastion via tailnet - BAST->>TALOS: Forward to LAN node :50000 - Human->>KC: OIDC login (browser) - KC-->>Human: Bearer token (short-lived) - Human->>TALOS: talosctl (mTLS with client cert) - Human->>K8S: kubectl --oidc (Keycloak token) - K8S->>KC: Validate token (introspection) - KC-->>K8S: Valid + claims - K8S-->>Human: Authorized response -``` - -### 5.2 Two distinct identity layers - -| Layer | Mechanism | Audience | -|-------|-----------|----------| -| **Talos API** (node ops) | Mutual TLS with client certificate generated from the Talos secrets bundle. | Operators (automation + humans). | -| **Kubernetes API** (kubectl) | OIDC bearer token from Keycloak. RBAC maps group claims → ClusterRole. | Humans. Service accounts use projected tokens (no OIDC). | - -The **Talos secrets bundle** is the root of trust for the cluster. Lose it -and you cannot operate the cluster; an attacker with it owns the cluster. -Storage: - -1. **Primary:** 1Password (or equivalent) — operator-accessible. -2. **Backup:** PBS encrypted backup target on tsys4 (existing infra). -3. **NOT in Git.** Machine configs go in Git; secrets stay out. - -### 5.3 Keycloak client configuration - -On Cloudron-hosted Keycloak, register a client `pfv-k8s-talos`: - -- **Authorization Code + PKCE flow** (no implicit, no password). -- **Redirect URIs:** `http://localhost:8000` (kubectl oidc-login) + Sidero - Omni/Rancher URLs if/when those are added. -- **Group claims:** `k8s-admin`, `k8s-readonly`, `k8s-tenant-itar`, - `k8s-tenant-rackrental`, etc. These map to Kubernetes RBAC `ClusterRoleBinding`. - ---- - -## 6. Tenant Isolation (vcluster) - -### 6.1 Why vcluster - -[vcluster](https://www.vcluster.com/) runs a **virtual Kubernetes control -plane** (API server, scheduler, controller-manager, etcd) inside a namespace -of the host cluster. Tenant workloads run on the host's worker nodes but -are isolated by: - -- Separate API server (tenant cannot see host cluster objects). -- Separate RBAC and admission control. -- Separate network policies (per-namespace). -- Separate resource quotas. - -This aligns with the user's per-tenant plan from -[`../proxmox/K8S.md`](../proxmox/K8S.md) §1. - -### 6.2 Tenant registry - -| Tenant | Compliance | Workload example | vcluster name | -|--------|-----------|------------------|---------------| -| RackRental | None (internal R&D) | containerlab topology tests | `vc-rackrental` | -| Suborbital non-ITAR | EAR/ITAR-aware but unclassified | Payload telemetry processing | `vc-suborbital-open` | -| Suborbital ITAR | **ITAR-controlled** | Firmware build for USML items | `vc-suborbital-itar` | -| Starting Line Productions | Commercial | Customer media pipeline | `vc-slp` | - -```mermaid -flowchart TB - subgraph HOST["Talos host cluster"] - CP[Host control plane
3 cnodes · etcd · Keycloak RBAC] - subgraph NS["Host cluster namespaces"] - NS_RR[ns: vc-rackrental] - NS_SO[ns: vc-suborbital-open] - NS_SI[ns: vc-suborbital-itar] - NS_SLP[ns: vc-slp] - end - end - - subgraph VRR["vcluster: vc-rackrental"] - API_RR[k8s API + etcd] - end - subgraph VSO["vcluster: vc-suborbital-open"] - API_SO[k8s API + etcd] - end - subgraph VSI["vcluster: vc-suborbital-itar"] - API_SI[k8s API + etcd] - end - subgraph VSLP["vcluster: vc-slp"] - API_SLP[k8s API + etcd] - end - - CP --> NS_RR & NS_SO & NS_SI & NS_SLP - NS_RR --> API_RR - NS_SO --> API_SO - NS_SI --> API_SI - NS_SLP --> API_SLP - - classDef itar fill:#3a0000,stroke:#ff0000,color:#fff - class NS_SI,API_SI itar -``` - -### 6.3 ITAR enforcement at host layer - -For the ITAR tenant (`vc-suborbital-itar`), enforce additional host-layer -controls: - -- **Node taint** `workload=itar:NoSchedule` on worker nodes dedicated to - ITAR workloads (subset of wnodes, marked in node labels). -- **NetworkPolicy** default-deny egress for the `vc-suborbital-itar` - namespace. Allow only explicit destinations (registry, NFS for ITAR - data tier, Keycloak). -- **Storage isolation:** ITAR PVCs target a dedicated NFS export (e.g., - `D3-itar` on tsys5) that no other tenant can mount. -- **Audit:** Hubble flows + auditd on the host worker nodes capture all - access to ITAR data. - -Rancher (or Sidero Omni) sits above this, presenting each tenant's -vcluster as a separate "cluster" in its UI, with Keycloak SSO gating -access per tenant group claim. - ---- - -## 7. Storage Integration - -Per [`../proxmox/K8S.md`](../proxmox/K8S.md) §6. Three StorageClasses: - -| StorageClass | Provisioner | Backing | Speed | Use | -|--------------|------------|---------|-------|-----| -| `local-fast` | local-path | wnode local disk (NVMe/SSD/HDD depending on host) | 100-3500 MB/s | Container runtime, scratch, ephemeral | -| `nfs-hdd` | nfs.csi.k8s.io | tsys4 D2/D5, tsys5 S1-S4 | 80-120 MB/s | Bulk data, weather/GIS datasets | -| `nfs-ssd` | nfs.csi.k8s.io | tsys5 D3, tsys5 T5-SSD | 200-400 MB/s | Latency-sensitive persistent data | - -### 7.1 CSI driver notes - -- **NFS CSI:** [`csi-driver-nfs`](https://github.com/kubernetes-csi/csi-driver-nfs) - (CNCF sandbox). Deploys via Helm. Each StorageClass points at a specific - NFS server + base export path. -- **local-path:** Rancher Local Path Provisioner. Single-binary, deploys - with one manifest. Uses wnode's kubelet root dir. - -### 7.2 ITAR data isolation - -The ITAR tenant should target a dedicated NFS export, not shared -`nfs-hdd`. Recommended: - -- Allocate `S4` on tsys5 (currently 99% empty, 435 GB free) as - `nfs-itar` StorageClass. Mountable only from `vc-suborbital-itar` - namespace via RBAC + NetworkPolicy. - ---- - -## 8. Local Image Registry - -### 8.1 Recommendation: Harbor on D3 SSD - -D3 SSD (tsys5, post-Friday SAS relocation) is 445 GB and 99% empty. Use it -for a **Harbor** instance: - -| Property | Value | -|----------|-------| -| **Storage** | D3 SSD on tsys5 (NFS export, fast tier) | -| **VM** | New VM `pfv-registry` on tsys5, local-nonprod boot, D3 data | -| **Function** | (a) Pull-through cache for Docker Hub / Quay / gcr.io
(b) Host private images
(c) Cosign image signing verification | -| **Exposure** | LAN-only. `192.168.3.50:443`. Not exposed to internet. | - -### 8.2 Pull-through cache benefit - -Cluster nodes have zero internet egress (per §3.3). Without a local cache, -image pulls fail. With Harbor as a pull-through cache: - -```mermaid -sequenceDiagram - WNODE->>HARBOR: docker pull nginx:1.25 - alt cache hit - HARBOR-->>WNODE: layer bytes (LAN-speed) - else cache miss - HARBOR->>DOCKERHUB: pull nginx:1.25 (egress) - DOCKERHUB-->>HARBOR: layer bytes - HARBOR-->>WNODE: layer bytes (cached for next time) - end -``` - -Cluster nodes pull from Harbor over LAN (gigabit). Harbor is the only -machine in the cluster with container-registry internet egress, and that -egress can be locked to specific upstreams (docker.io, quay.io, gcr.io, -ghcr.io). - -### 8.3 Supply-chain integrity (future) - -Harbor + Cosign lets you require that all images deployed to the ITAR -tenant are signed by a trusted key. This is a strong ITAR/CISA-attestation -control. Implementation deferred to a later session. - ---- - -## 9. Bootstrap Procedure - -### 9.1 One-time setup - -```mermaid -sequenceDiagram - autonumber - participant OP as Operator - participant GIT as Git repo - participant ONEPW as 1Password - participant PBS as PBS (tsys4) - OP->>GIT: Clone PFVCluster repo - OP->>ONEPW: Generate Talos secrets bundle (offline) - ONEPW-->>OP: secrets.yaml - OP->>PBS: Backup secrets.yaml (encrypted) - OP->>GIT: Write machine configs (no secrets) -``` - -### 9.2 Provision first cnode (bootstrap) - -```mermaid -sequenceDiagram - autonumber - participant OP as Operator - participant PX as Proxmox host - participant C1 as cnode1 - participant ETCD as etcd (new) - - OP->>PX: qm create VM (Talos QCOW2 disk, local-lvm) - OP->>PX: qm start VMID - C1->>C1: Boots Talos (no config yet, "maintenance mode") - OP->>C1: talosctl apply --patch cnode1.yaml (with secrets) - C1->>C1: Applies config, restarts services - OP->>C1: talosctl bootstrap - C1->>ETCD: Initialize single-node raft - ETCD-->>C1: ready - OP->>C1: talosctl kubeconfig (fetch admin kubeconfig) - OP->>C1: talosctl etcd snapshot (initial backup → PBS) -``` - -### 9.3 Add second and third cnodes - -```mermaid -sequenceDiagram - autonumber - participant OP as Operator - participant PX as Proxmox host - participant C2 as cnode2 - participant C3 as cnode3 - participant C1 as cnode1 (existing) - - OP->>PX: qm create + start cnode2 VM - C2->>C2: Boots Talos maintenance mode - OP->>C2: talosctl apply --patch cnode2.yaml - C2->>C1: Join etcd cluster - OP->>PX: qm create + start cnode3 VM - C3->>C3: Boots Talos maintenance mode - OP->>C3: talosctl apply --patch cnode3.yaml - C3->>C1: Join etcd cluster - Note over C1,C3: etcd now has 3/3 members → HA quorum -``` - -### 9.4 Post-bootstrap cluster configuration - -Once 3 cnodes are up and joined: - -1. **Install Cilium** (CNI) via Helm. Enable node-to-node encryption. -2. **Install CSI drivers** — nfs-csi + local-path provisioner. -3. **Create StorageClasses** — `local-fast`, `nfs-hdd`, `nfs-ssd`. -4. **Deploy Harbor** on the `pfv-registry` VM, exposed at `192.168.3.50`. -5. **Configure Kubernetes API OIDC** — Keycloak client (§5.3). -6. **Apply default-deny NetworkPolicy** in all namespaces. -7. **Install vcluster CLI** + create 4 tenant vclusters (§6). -8. **First etcd snapshot** + automated daily snapshot cron → PBS. - -### 9.5 Add workers - -Workers are simpler (no etcd): - -```mermaid -sequenceDiagram - OP->>PX: qm create + start wnode-X VM (Talos QCOW2) - WNODE->>WNODE: Boots maintenance mode - OP->>WNODE: talosctl apply --patch worker-X.yaml - WNODE->>C1: Kubelet registers with API server - C1-->>WNODE: Approved (auto via bootstrap token) - Note over WNODE: Joins cluster, becomes Ready -``` - ---- - -## 10. Disaster Recovery - -### 10.1 Backup strategy - -| Artifact | Frequency | Storage | Tool | -|----------|-----------|---------|------| -| **Talos secrets bundle** | Once (regen only on rotation) | 1Password + PBS (encrypted) | Manual | -| **Machine configs** | Continuous (Git) | Git remote + PBS | Git | -| **etcd snapshot** | Daily + before each change | PBS (tsys4 SMR target, 4.3 TB free) | `talosctl etcd snapshot` | -| **vcluster etcd** | Daily per vcluster | PBS | `kubectl exec ... etcdctl snapshot` | -| **Harbor metadata** | Daily | PBS | Harbor built-in backup | - -### 10.2 Restore scenarios - -**Lost 1 cnode (e.g., tsys9 disk failure):** - -1. Provision new VM on tsys9 (or other host with local SSD). -2. Apply cnode2 machine config patch. -3. New cnode joins etcd, syncs state from survivors. -4. Quorum was never lost (2 of 3 alive throughout). - -**Lost 2 cnodes simultaneously (quorum lost):** - -1. Use surviving cnode's etcd snapshot. -2. Provision 3 new cnode VMs. -3. On first: `talosctl bootstrap --recover-from=snapshot.db`. -4. Join other 2 cnodes. -5. Workers reconnect automatically once API server is back. - -**Total cluster loss (all 3 cnodes):** - -1. Restore from latest PBS etcd snapshot. -2. Provision new cnode VMs. -3. `talosctl bootstrap --recover-from=snapshot.db`. -4. Re-join workers. -5. Verify tenant vclusters restored. - -### 10.3 Recovery time objectives - -| Scenario | RTO | RPO | -|----------|-----|-----| -| Single cnode failure | < 30 min | 0 (no data loss) | -| Quorum loss (2 cnodes) | < 2 hours | ≤ 24 hours (last snapshot) | -| Total cluster loss | < 4 hours | ≤ 24 hours | - ---- - -## 11. Migration from Current State - -### 11.1 Current state - -- 5 cnode VMs exist (Debian stock + Tailscale). -- **No k3s deployed yet.** Cluster was never bootstrapped. -- 6 wnode VMs exist (some stopped). -- No workloads running in k8s. - -### 11.2 Migration: clean cutover (not a migration) - -Since there is no etcd data and no workloads to preserve, the path is a -**clean rebuild**: - -| Phase | Action | Risk | -|-------|--------|------| -| **0. Prep** | Generate Talos secrets. Store in 1Password + PBS. Write machine configs to Git. | Low. | -| **1. Bootstrap 3 new cnodes** | Build 3 NEW Talos cnode VMs (not the existing 5). Use local-lvm boot disks (tsys9 × 2, tsys3 × 1 per §2.4 recommended spread). | Low. Existing Debian cnodes can keep running idle. | -| **2. Configure cluster** | Install Cilium, CSI, StorageClasses, OIDC, Harbor. | Low. | -| **3. Add workers** | Re-image existing wnode VMs as Talos, or build new ones. | Low. No workloads to drain. | -| **4. Decommission old Debian cnodes** | Once cluster is stable, shut down + delete the 5 old Debian cnode VMs. | Low. | -| **5. Tenant vclusters** | Stand up per-tenant vclusters. | Medium (policy tuning). | - -### 11.3 Open dependency: Friday hardware work - -Phases 1-2 require: - -- **tsys3 local-lvm available.** Per - [`../proxmox/PROJECT.md`](../proxmox/PROJECT.md) §3.3, tsys3 has 349 GB - free NVMe local-lvm. Currently unused. **Ready.** -- **tsys9 local-lvm available.** 136 GB PNY CS900 SSD. **Ready.** -- **D3 SSD relocated to tsys5 SAS.** Currently USB on tsys4. Per - [`../proxmox/TODO.md`](../proxmox/TODO.md) §2, scheduled for Friday. - Harbor depends on D3 being available on tsys5. - -Bootstrap of the cnodes does NOT depend on Friday hardware work. Only the -Harbor registry does. - ---- - -## Appendix: Open questions for next session - -1. **Confirm 3 vs 5 cnodes** (§2.1). Recommendation: 3. -2. **Confirm cnode host spread** (§2.4). Recommendation: 3-host spread - (tsys9, tsys1, tsys3). -3. **Static IPs for cnodes** (§3.2). Recommendation: yes, `192.168.3.31-33`. -4. **Rancher vs Sidero Omni** for cluster management UI. Both viable. - Defer until cluster is up. -5. **Subnet router ACL approval** on Tailscale admin console (§3). Needs - approval of `192.168.3.0/24` route advertisement. -6. **ITAR worker node subset** (§6.3). Which wnodes are tainted for ITAR? - Recommendation: tsys6 + tsys7 (heaviest hosts, NFS-only boot) as - general capacity; tsys3 + tsys5 (local fast storage) reserved for - non-ITAR HPC. +# k8s/docs/ARCHITECTURE.md + +> **Documentation moved to Discourse — the canonical source of truth.** +> +> **k8s target architecture: control plane, network, identity, storage, DR** +> +> **Read it here:** https://community.turnsys.com/t/305 +> +> *Migrated 2026-08-06. This file is kept as a pointer for git-browsing context. +> Do not update content here — edit the Discourse wiki topic instead.* diff --git a/k8s/docs/DISTRO-DECISION.md b/k8s/docs/DISTRO-DECISION.md index f04d5eb..1f4fa73 100644 --- a/k8s/docs/DISTRO-DECISION.md +++ b/k8s/docs/DISTRO-DECISION.md @@ -1,188 +1,10 @@ -# Distro Decision: Talos Linux vs k3s +# k8s/docs/DISTRO-DECISION.md -> **Recommendation: Talos Linux.** -> The k3s-on-Debian plan was sound before the ITAR/classified requirement -> entered scope. Once classified workloads are on the table, Talos's -> immutable, API-only, measured-boot-capable posture is materially easier -> to certify and defend. - -**Last updated:** 2026-07-28 - ---- - -## 1. Decision context - -| Factor | Constraint | -|--------|-----------| -| **Workload class** | R&D + RackRental (containerlab) + **ITAR / classified** suborbital workloads + commercial (Starting Line Productions) | -| **Compliance drivers** | ITAR (USML categories), possible classified handling (NIST 800-171, CNSSI 1253) | -| **Hardware** | 7 standalone Proxmox hosts (no `pvecm`), managed via PDM. Live migration NOT available — disk moves via Proxmox "Storage Migrate" UI. | -| **Network** | Gigabit symmetric fiber to residence. LAN-only cluster traffic desirable. Tailscale already in use (overlay for admin access). | -| **Current cnode state** | Stock Debian VMs joined to Tailscale. **No k8s distribution has been deployed yet.** Clean cutover possible. | -| **Operations** | Solo founder. Must be reproducible from Git, low-touch, low-debug-overhead. | - ---- - -## 2. Head-to-head comparison - -### 2.1 ITAR / classified posture - -| Property | Talos Linux | k3s on Debian | -|----------|-------------|---------------| -| **Node OS mutability** | Immutable rootfs (squashfs, read-only). Reboot returns to known-good state. | Mutable. `apt install`, file edits persist. | -| **Shell / SSH access** | **None.** No SSH daemon, no shell, no `kubectl debug node` shell. | Full SSH + bash. STIG hardening reduces (does not eliminate) attack surface. | -| **Operational surface** | Single gRPC API (mTLS, signed certs, audit log) on port 50000. | SSH + kubelet API + etcd API + package manager + cron + systemd + userland. | -| **Measured boot** | Supported. TPM attestation can prove the node booted the signed Talos image you pinned. | Possible but bolt-on; auditors will ask why you didn't disable the bootloader first. | -| **Configuration provenance** | Entire node state is a YAML machine config in Git. `talosctl apply` is the only mutation path. | Config drift via SSH edits, package updates, manual service restarts. STIG/CAT-IV findings multiply. | -| **Supply chain** | Every Talos release is a signed artifact (cosign). Pin by image digest. | Debian package provenance is good but the surface is enormous (~30K packages in a base install). | -| **Forensic readiness** | API log + kernel log + Talos event log = sufficient for "what ran, when, with what config." | Same possible but requires explicit configuration to be trustworthy. | -| **STIG / CIS conformance** | Intrinsically close. Talos publishes CIS benchmark results per release. | Requires running SCAP-STIG (already in this repo) and remediating findings continuously. | - -**Bottom line:** For classified workloads, an auditor's first question is -"how do you prevent unauthorized changes to a node?" Talos's answer is -"the OS is immutable and the only path is a signed API call." k3s's answer -is"SSH is locked down and we scan with STIG." The first is structurally -stronger; the second is operationally maintained. - -### 2.2 Operational considerations - -| Property | Talos | k3s | -|----------|-------|-----| -| **Familiarity** | New model (`talosctl apply`, no SSH). Learning curve. | Stock Debian + k3s binary. Familiar. | -| **Debugging** | `talosctl logs`, `talosctl dmesg`, `talosctl dashboard`. No shell. | `ssh`, `journalctl`, `crictl`. Full shell. | -| **Tailscale integration** | System extension (`siderolabs/tailscale`). Stable since Talos 1.3. | Native — `apt install tailscale`. Zero friction. | -| **Backup / DR** | `talosctl etcd snapshot` (one command). Cluster can be restored from snapshot + machine configs. | DIY (`etcdctl snapshot` + manual cert management). | -| **Upgrades** | `talosctl upgrade` — atomic, automated rollback on health-check failure. | Manual: drain, `k3s` package update, reboot, uncordon. | -| **Proxmox compatibility** | QCOW2 image boots natively on KVM/QEMU. virtio-net, virtio-scsi, virtio-rng all supported. | Same. | -| **Ecosystem maturity** | Production-grade. Sidero (the company) offers Omni (managed control plane for Talos). | Production-grade. Rancher (SUSE) backs it. | - -### 2.3 Cost of choosing Talos over the existing k3s plan - -The cnodes are currently **stock Debian VMs joined to Tailscale**. Critically, -**no k3s cluster has been deployed yet** — k3s was only the *plan*. Therefore: - -- **No etcd data to migrate.** Clean cutover, not a migration. -- **No workloads to drain.** The cluster is empty. -- **Cnode VMs get re-imaged** with Talos QCOW2 (or rebuilt from scratch — - either way it's a `qm` script, not a stateful migration). -- **Tailscale config shifts** from "installed via apt" to "Talos system - extension." (Or, per our recommendation in - [`ARCHITECTURE.md`](ARCHITECTURE.md) §3, **Tailscale moves off the cluster - nodes entirely** and onto the existing `tailscale-router` bastion as a - subnet router. Cluster nodes become LAN-only.) - -**Net cost:** rebuilding 3 cnode VMs as Talos + writing ~200 lines of -machine config YAML. The hardening investment already encoded in -`provisioning/Modules/Security/` is **not wasted** — it still applies to -every non-cluster VM (netinfra, UCS, LibreNMS, SIEM, bastion, etc.). Only -the cnodes/wnodes move to Talos. - ---- - -## 3. Tailscale compatibility (deep-dive) - -Tailscale on Talos is well-supported but introduces a configuration dimension -worth being explicit about. Three patterns exist: - -### Pattern A — Tailscale on every cluster node (what you have now, on Debian) - -Each cnode/wnode runs `tailscaled` and joins the tailnet. Cluster nodes have -internet egress (to Tailscale DERP servers and for coordinate). - -- **Talos implementation:** add `siderolabs/tailscale` system extension to - each machine config, configure `machine.network.interfaces`. -- **Pros:** Operator can hit any node's Talos API from any Tailscale device. -- **Cons:** Cluster nodes have internet egress. For ITAR workloads, this is a - finding (data exfiltration path). - -### Pattern B — Tailscale on bastion only, SSH/API jump - -Cluster nodes are LAN-only. Operator Tunnels to bastion (existing -`tailscale-router` VM), then runs `talosctl` from the bastion. - -- **Pros:** Zero internet egress from cluster nodes. -- **Cons:** Two-step access. Bastion must run recent `talosctl`. Each operator - action originates from the bastion (auditable but clunky). - -### Pattern C — Tailscale subnet router on bastion (recommended) - -The existing `tailscale-router` VM advertises the cluster LAN subnet -(e.g. `192.168.3.0/24`) into the tailnet as a **subnet route**. Operator's -Tailscale client transparently routes cluster-bound traffic through the -bastion. From the operator's workstation, `talosctl --nodes 192.168.3.x` -"just works." - -- **Pros:** - - Cluster nodes have **zero internet egress** (strongest ITAR posture). - - Operator UX is unchanged from direct LAN access. - - All access is mediated by Tailscale's identity + ACLs (already integrated - with your env). - - Audit trail lives in Tailscale + bastion logs. -- **Cons:** - - Bastion becomes a dependency for remote admin (LAN-local admin still - works without it). - - Must enable IP forwarding + subnet route approval in Tailscale ACLs. - -**Recommendation: Pattern C.** Documented in -[`ARCHITECTURE.md`](ARCHITECTURE.md) §3. - ---- - -## 4. Recommendation - -**Deploy Talos Linux** as the k8s distribution for `pfv-k8s`. - -### Justification - -1. **Compliance posture is structural, not operational.** "Immutable, - API-only, measured-boot" is a property of Talos itself; "STIG-hardened" - is a property of how Debian is operated. The first is dramatically - easier to argue to an ITAR counsel or classified accreditation officer - (DSS, DCSA) than the second. -2. **Zero migration cost.** The k3s cluster was never deployed. Reimaging - 3 cnodes with Talos is a `qm` script invocation, not a stateful - migration. The sunk cost of "we planned k3s" is **zero deployed state**. -3. **Operational headroom.** Talos's `etcd snapshot` + `upgrade --stage` + - `apply-mode auto` reduce solo-founder ops burden. k3s is simpler to - learn but more error-prone to operate at HA. - -### Acknowledged tradeoffs - -- **Learning curve.** The Talos mental model (`machine config` + `talosctl`) - replaces SSH + systemd. Expect a one-week ramp for comfortable daily ops. -- **No shell debugging.** When something breaks on a node, you cannot `ssh` - in. Mitigation: `talosctl logs/support` produces a support bundle - equivalent to a sosreport. -- **Hardware/module surprises.** Talos ships a curated kernel. Anything - beyond virtio + common NIC drivers needs a system extension. On Proxmox - VMs this is **not expected to be a problem** — virtio is the path. -- **Tailscale via system extension.** Adds one config dimension per node. - Mitigated by Pattern C (above), which removes Tailscale from cluster - nodes entirely. - -### What we keep from the k3s mental model - -- **Single binary on each node** semantics (Talos is conceptually similar). -- **`kubectl` workflow unchanged.** Talos exposes a standard Kubernetes - API. `kubectl`, `helm`, `kustomize` all work as-is. -- **Storage CSI choices** (`local-fast`, `nfs-hdd`, `nfs-ssd`) are - distro-independent. - ---- - -## 5. What we are NOT deciding here - -| Topic | Deferred to | -|--------|-------------| -| ETL tooling (GDAL/PostGIS/xarray/Dask) | Future session — affects StorageClass RWX/RWO design | -| HPC scheduler (Jobs/Argo/Volcano) | Future session — affects taint/label strategy | -| Per-tenant vcluster policy templates | Future session, post-bootstrap | -| Solar-aware scale-out hosts | Future capacity planning session | -| Container network plugin (CNI) details | Will be specified in ARCHITECTURE.md §4 — recommendation is Cilium (supports NetworkPolicy, BPF, and encrypted node-to-node traffic for ITAR tenants) | - ---- - -## 6. Next step - -Proceed to [`ARCHITECTURE.md`](ARCHITECTURE.md) for the control-plane -design, network topology, identity flow, and bootstrap procedure. +> **Documentation moved to Discourse — the canonical source of truth.** +> +> **Talos vs k3s distro analysis and decision** +> +> **Read it here:** https://community.turnsys.com/t/305 +> +> *Migrated 2026-08-06. This file is kept as a pointer for git-browsing context. +> Do not update content here — edit the Discourse wiki topic instead.* diff --git a/k8s/docs/README.md b/k8s/docs/README.md index f5b2cf7..4ffc414 100644 --- a/k8s/docs/README.md +++ b/k8s/docs/README.md @@ -1,44 +1,10 @@ -# Kubernetes Architecture & Build Plan +# k8s/docs/README.md -> **Status:** Draft for review. Companion to [`../proxmox/K8S.md`](../proxmox/K8S.md) -> (which captured the storage/host analysis from the Proxmox audit). -> This directory takes the next step: **which distro, how to build it, -> how to operate it.** - -**Last updated:** 2026-07-28 - -## Documents in this directory - -| Document | Purpose | -|----------|---------| -| [`DISTRO-DECISION.md`](DISTRO-DECISION.md) | Talos vs k3s analysis. Recommendation: **Talos**, with rationale grounded in the ITAR/classified requirement. | -| [`ARCHITECTURE.md`](ARCHITECTURE.md) | Target architecture: control plane, network, identity, storage, tenant isolation. Mermaid diagrams included. | - -## TL;DR - -| Decision | Recommendation | Why | -|----------|----------------|-----| -| **Distro** | **k3s** (deployed) / **Talos** (for future ITAR) | k3s chosen for the regular R&D cluster now live on cnode1/2/3. Talos is the recommendation for when the ITAR/classified cluster comes online. | -| **Runtime** | **containerd** | Talos/k3s default. | -| **Cnode count** | **3** (deployed) | Standard HA. Tolerates 1 failure. | -| **Admin access** | **Tailscale (all nodes joined)** | Currently all cnodes are on Tailscale directly. For ITAR cluster, move to subnet-router pattern. | -| **Cluster network** | **Tailscale-only IPs** | All node-ip, advertise-address, TLS-SANs are 100.x Tailscale IPs. Zero LAN IPs in cluster state. | -| **Identity** | **OIDC to Keycloak** on Cloudron (production) | Future work. | -| **Multi-tenancy** | **vcluster** (per tenant) | Future work. | -| **Local registry** | **Harbor on D3 SSD** (tsys5, 445 GB free) | Future work. | -| **Storage classes** | `local-fast`, `nfs-hdd`, `nfs-ssd` | Per [`../proxmox/K8S.md`](../proxmox/K8S.md) §6. Future work. | - -## What this directory does NOT cover (deferred) - -- ETL tooling choice (GDAL/PostGIS/xarray/Dask) — affects RWX vs RWO design. -- HPC job scheduler (Jobs / Argo Workflows / Volcano) — affects taint/label strategy. -- vcluster per-tenant policy templates. -- Solar-aware scale-out (PowerEdge 19xx/2950 hosts) — capacity planning only. - -These are tracked as future session work in [`../../STATUS.md`](../../STATUS.md). - -## Open question for the user - -1. **Cnode count: confirm 3 vs 5.** Recommendation is 3 (rationale in - [`ARCHITECTURE.md`](ARCHITECTURE.md) §2). If your ITAR counsel requires - 2-failure tolerance on the control plane, keep 5. +> **Documentation moved to Discourse — the canonical source of truth.** +> +> **k8s docs index + TL;DR** +> +> **Read it here:** https://community.turnsys.com/t/305 +> +> *Migrated 2026-08-06. This file is kept as a pointer for git-browsing context. +> Do not update content here — edit the Discourse wiki topic instead.* diff --git a/netinfra/dhcp-migration.md b/netinfra/dhcp-migration.md index 5048211..e5a9967 100644 --- a/netinfra/dhcp-migration.md +++ b/netinfra/dhcp-migration.md @@ -1,125 +1,10 @@ -# DHCP Migration: pfv-netboot → pfv-netinfra-01/02 +# netinfra/dhcp-migration.md -**Date:** 2026-07-29 -**Status:** ✅ **COMPLETE** — both servers in `normal` failover state, 333 -active leases load-balanced. Netboot DHCP stopped + disabled. - -> **Safety nets:** pfv-netboot, pfv-netinfra-01/02, sectestbed-sandbox, and -> ultix-streaming all have static IPs (reachable via Tailscale even if DHCP -> fails). sectestbed-sandbox snapshot `pre-dhcp-migration` exists as rollback. - -## Migration results (2026-07-29 17:20 CDT) - -| Check | Result | -|-------|--------| -| netinfra-01 (primary) | `active`, failover `normal`, 244 active leases | -| netinfra-02 (secondary) | `active`, failover `normal`, 89 active leases | -| Failover partnership | "Both servers normal" — load-balanced at split=128 | -| DHCP test (sandbox) | Lease obtained from 192.168.3.252, IP 192.168.3.50/22 | -| DNS handed out | `192.168.3.252, 192.168.3.253` (netinfra pair) | -| NTP handed out | `192.168.3.252, 192.168.3.253` (netinfra pair) | -| Domain | `knel.net` | -| Gateway | `192.168.3.254` | -| Webmin | Port 10000 (SSL) on both nodes; DHCP module configured | -| Netboot DHCP | Stopped + disabled (won't start on reboot) | -| Netmask fix | Both nodes corrected /24 → /22 | - -## 1. Current state (discovered 2026-07-29) - -| | pfv-netboot (SOURCE) | pfv-netinfra-01 (TARGET primary) | pfv-netinfra-02 (TARGET secondary) | -|---|---|---|---| -| IP | 192.168.3.250/22 | 192.168.3.252/24 | 192.168.3.253/24 | -| isc-dhcp-server | 4.4.3, **active** | NOT installed | NOT installed | -| Webmin | 2.652 (port 10000, SSL) | NOT installed | NOT installed | -| DHCP leases | **532 active** | — | — | -| DHCP config | `/etc/dhcp/dhcpd.conf` | — | — | - -**Source DHCP config summary:** -- Subnet: `192.168.0.0/22` (covers .0.x/.1.x/.2.x/.3.x) -- Pool range: `192.168.0.1 — 192.168.3.200` -- Router/GW: `192.168.3.254` -- DNS: `192.168.3.250` (netboot itself — **changing to 252/253**) -- Domain: `knel.net` -- Lease times: 600s default, 7200s max -- DDNS: `ddns-update-style none` (globally disabled) -- ~37 host reservations (MAC → fixed IP) -- 3 global-scope host declarations (outside subnet) - -## 2. What changes in the migrated config - -| Setting | Old (netboot) | New (netinfra) | Why | -|---------|--------------|----------------|-----| -| DNS servers handed out | `192.168.3.250` | `192.168.3.252, 192.168.3.253` | Point clients at the netinfra Pi-hole pair (netboot will be decommissioned) | -| Failover | none (single server) | ISC DHCP failover-peer | Primary/secondary HA | -| DHCP interface | `eth0` | `ens18` | Different NIC name on netinfra VMs | -| Netmask on servers | /22 | **fix to /22** (currently /24) | Match the /22 network | - -Everything else (subnet, pool range, reservations, lease times, domain) stays -identical. The host reservation list is copied verbatim. - -## 3. Configs prepared - -| File | Destination | -|------|-------------| -| [`dhcp/dhcpd-primary.conf`](dhcp/dhcpd-primary.conf) | `/etc/dhcp/dhcpd.conf` on netinfra-01 | -| [`dhcp/dhcpd-secondary.conf`](dhcp/dhcpd-secondary.conf) | `/etc/dhcp/dhcpd.conf` on netinfra-02 | - -Both are identical except for the `failover peer` block (primary vs secondary). - -## 4. Execution sequence - -### Phase 1 — pre-flight (non-disruptive) - -1. **Snapshot sectestbed-sandbox** (VMID 6000 on tsys5) — rollback point -2. **Fix netmask** on netinfra-01/02: `/24` → `/22` in `/etc/network/interfaces` - (then `ifdown ens18 && ifup ens18`, or `systemctl restart networking`) -3. Verify Tailscale still reachable after netmask change - -### Phase 2 — install prerequisites (non-disruptive, nothing serving yet) - -On **both** netinfra-01 and netinfra-02: -1. `apt-get update && apt-get install -y isc-dhcp-server` -2. Install Webmin via official setup script -3. Copy `dhcpd-{primary,secondary}.conf` → `/etc/dhcp/dhcpd.conf` -4. Write `/etc/default/isc-dhcp-server` with `INTERFACESv4="ens18"` -5. Syntax check: `dhcpd -t -cf /etc/dhcp/dhcpd.conf` -6. **Do NOT start the DHCP service yet** - -### Phase 3 — cutover (the critical moment) - -**Goal:** minimize the DHCP gap. Existing clients hold valid leases for 600 -seconds — they won't notice a brief outage. - -1. **Stop netboot DHCP:** `ssh netboot 'sudo systemctl stop isc-dhcp-server'` -2. **Immediately start netinfra-01:** `systemctl start isc-dhcp-server` -3. **Immediately start netinfra-02:** `systemctl start isc-dhcp-server` -4. **Verify failover partnership:** check `journalctl -u isc-dhcp-server` on both - — look for "failover peer pfv-dhcp: I move from recover to recover-done" -5. Wait ~10 seconds for lease sync - -### Phase 4 — test - -1. On sectestbed-sandbox: edit `/etc/network/interfaces` to DHCP -2. `dhclient -r` (release) then `dhclient` (request new lease) -3. Verify IP assigned, correct DNS, correct gateway -4. If fails: `qm rollback 6000 ` and debug -5. If succeeds: change back to static (sandbox's normal config) - -### Phase 5 — cleanup - -1. **Disable netboot DHCP:** `ssh netboot 'sudo systemctl disable isc-dhcp-server'` -2. Verify Webmin DHCP module visible at `https://192.168.3.252:10000` -3. Document + commit - -## 5. Failover mechanics (reference) - -ISC DHCP failover uses a TCP connection on port **647** between the two servers. - -- **Primary** owns the pool initially and can assign new leases immediately. -- **Secondary** enters `recover` state, syncs the lease database from primary, - then transitions to `recover-done` → `normal`. -- If primary dies, secondary continues serving after the MCLT timeout (600s). -- If secondary dies, primary continues serving alone. -- **split 128** = load balance 50/50 (primary handles even hashes, secondary odd). -- Host reservations (fixed-address) are NOT subject to failover — both servers - can independently assign reserved IPs. +> **Documentation moved to Discourse — the canonical source of truth.** +> +> **DHCP migration to netinfra-01/02** +> +> **Read it here:** https://community.turnsys.com/t/306 +> +> *Migrated 2026-08-06. This file is kept as a pointer for git-browsing context. +> Do not update content here — edit the Discourse wiki topic instead.* diff --git a/netinfra/dns-cluster-setup/README.md b/netinfra/dns-cluster-setup/README.md index 52633a9..13fd2f2 100644 --- a/netinfra/dns-cluster-setup/README.md +++ b/netinfra/dns-cluster-setup/README.md @@ -1,185 +1,10 @@ -# Technitium DNS Cluster Setup +# netinfra/dns-cluster-setup/README.md -> **Redmine:** [#357](https://projects.knownelement.com/issues/357) (deployment, closed) - -Replicates the production Technitium DNS Server from `tailscale-router` to the -`pfv-netinfra-01/02` pair and configures them as a primary/secondary cluster -with automatic zone transfers. - -## Architecture - -``` - tailscale-router (PRODUCTION — READ ONLY) - └─ tsys-dns container (technitium/dns-server) - └─ 124 zones (knel.net + reverse DNS) - └─ Users + 2FA in auth.config - │ - docker cp (export) - │ - ▼ - ┌─ pfv-netinfra-01 (192.168.3.252) ──── PRIMARY ──────────┐ - │ tsys-dns container (Technitium on :5300) │ - │ pihole container (Pi-hole on :53 → Technitium :5300) │ - │ All zones are Primary │ - │ Zone transfer allowed from 192.168.3.253 │ - └──────────────────────────────────────────────────────────┘ - │ - AXFR / IXFR + NOTIFY (DNS zone transfer, port 5300) - │ - ▼ - ┌─ pfv-netinfra-02 (192.168.3.253) ─── SECONDARY ────────┐ - │ tsys-dns container (Technitium on :5300) │ - │ pihole container (Pi-hole on :53 → Technitium :5300) │ - │ All zones are Secondary (AXFR from 01) │ - └──────────────────────────────────────────────────────────┘ -``` - -### How clustering works - -Technitium uses standard DNS zone transfers (AXFR/IXFR) for primary/secondary -replication, not a proprietary protocol: - -1. **Primary (01)** holds all zones as authoritative primary zones. -2. **Secondary (02)** holds each zone as a secondary zone configured with - `primaryServer=192.168.3.252:5300`. -3. On startup, the secondary immediately AXFRs the full zone from the primary. -4. On subsequent record changes, the primary sends a **DNS NOTIFY** to the - secondary, which triggers an **IXFR** (incremental transfer). -5. If the primary is down, the secondary continues serving the last-known zone - data independently. - -### Credentials and 2FA - -The production `auth.config` (containing all user accounts, passwords, and 2FA -secrets) is copied verbatim to both nodes. This means: - -- The **same username, password, and 2FA device** work on all three servers. -- The web console is at `http://:5380/` on each node. -- No credential changes are needed. - -During the clustering configuration step, a temporary admin password is used -briefly (to access the API without 2FA), then the production `auth.config` is -restored. See "Security notes" below. - -## Prerequisites - -- SSH key access to all hosts as `localuser` with passwordless sudo. -- The `remote-dns.sh` wrapper must be able to reach all hosts via Tailscale FQDN. -- Docker + Docker Compose on netinfra-01/02 (already installed). -- The production Technitium on tailscale-router must be running. - -## Usage - -```bash -cd dns-cluster-setup/ - -# Step-by-step (recommended for first run): -./setup.sh export # 1. Export config from tailscale-router (READ-ONLY) -./setup.sh deploy01 # 2. Deploy to netinfra-01 as primary -./setup.sh deploy02 # 3. Deploy to netinfra-02 as secondary clone -./setup.sh cluster # 4. Configure clustering (01→02 zone transfers) -./setup.sh verify # 5. Run all verification tests - -# Or all at once: -./setup.sh all -``` - -### Configuration overrides - -All defaults can be overridden via environment variables: - -| Variable | Default | Description | -|---|---|---| -| `PRIMARY_IP` | `192.168.3.252` | netinfra-01 LAN IP | -| `SECONDARY_IP` | `192.168.3.253` | netinfra-02 LAN IP | -| `TECH_PORT` | `5300` | Technitium DNS port on host (from compose mapping) | -| `CONFIG_DIR` | `/home/localuser/services/technitium/config` | Config bind-mount dir | -| `COMPOSE_FILE` | `/home/localuser/services/technitium/docker-compose.yml` | Compose file | -| `TEMP_ADMIN_PW` | `KnelClusterSetup!2026` | Temp admin password (used only during clustering, then discarded) | - -## Scripts - -| Script | Purpose | -|---|---| -| `remote-dns.sh` | SSH/SCP chokepoint for all DNS host access (tsrouter, netinfra01, netinfra02, netboot, sandbox) | -| `setup.sh` | Master orchestrator: export → deploy → cluster → verify | -| `verify.sh` | Comprehensive 10-section verification suite | -| `discover*.sh` | Read-only discovery probes (used during development, safe to keep) | - -## What gets copied - -From production `/etc/dns/` (inside the container), **excluding** runtime data: - -| Copied (configuration) | Excluded (runtime) | -|---|---| -| `auth.config` (users, passwords, 2FA) | `cache.bin` (DNS cache) | -| `dns.config` (server settings) | `stats/` (query statistics) | -| `webservice.config` (web console) | `logs/` (log files) | -| `allowed.config` (zone transfer ACL) | | -| `blocked.config` (blocked domains) | | -| `blocklist.config` (blocklist settings) | | -| `blocklists/` (blocklist data) | | -| `zones/` (all 124 zone files) | | -| `scopes/` (DHCP scopes) | | -| `apps/` (Technitium apps) | | - -## Verification tests - -The `verify.sh` script runs 10 categories of tests: - -1. **Container health** — both Technitium containers are Up -2. **API responds** — web console API is reachable on both nodes -3. **Zone count** — primary matches production; secondary matches primary -4. **Forward DNS** — known knel.net records resolve identically on both nodes -5. **External DNS** — both nodes can resolve external domains (github.com) -6. **Zone transfer (AXFR)** — secondary can AXFR knel.net from primary -7. **Reverse DNS** — PTR zones have SOA records on both nodes -8. **Production untouched** — container still running, zone count unchanged -9. **Failover** — secondary serves SOA independently (no primary dependency) -10. **Credentials** — `auth.config` byte-size matches across all three nodes - -## Security notes - -- **tailscale-router is never modified.** The only operation is `docker cp` - (read) to export the config. No writes, no restarts, no config changes. -- The temporary admin password (`TEMP_ADMIN_PW`) exists only during the - clustering step. After configuration, the production `auth.config` (with 2FA) - is restored. The temp password is never persisted. -- The export tarball (`.export/technitium-production-config.tar.gz`) contains - production credentials. It is in `.gitignore` and should be deleted after - setup: `rm -rf dns-cluster-setup/.export/` -- Each node's existing config is backed up to `config.backup-` before - replacement, so the change is reversible. - -## Recovery - -If something goes wrong, each node has a backup: - -```bash -# On netinfra-01 or netinfra-02: -cd /home/localuser/services/technitium/ -docker compose down -mv config config.failed -mv config.backup- config -docker compose up -d -``` - -## Validation on sandbox - -After cluster setup, validate that client hosts use the pair correctly: - -```bash -# From sectestbed-sandbox (or any client): -# Query primary directly: -dig @192.168.3.252 pfv-netinfra-01.knel.net - -# Query secondary directly: -dig @192.168.3.253 pfv-netinfra-01.knel.net - -# Both should return the same answer. -``` - -The KNELServerBuild provisioning code (`provisioning/ConfigFiles/NTP/ntp.conf` -and `provisioning/ConfigFiles/Resolv/resolv.conf`) points clients at both -servers for DNS and NTP redundancy. See `docs/server-build/tailscale.md` for the -full DNS architecture analysis. +> **Documentation moved to Discourse — the canonical source of truth.** +> +> **Technitium DNS cluster setup** +> +> **Read it here:** https://community.turnsys.com/t/306 +> +> *Migrated 2026-08-06. This file is kept as a pointer for git-browsing context. +> Do not update content here — edit the Discourse wiki topic instead.* diff --git a/netinfra/pfv-netboot-setup.md b/netinfra/pfv-netboot-setup.md index 2899350..3f848a5 100644 --- a/netinfra/pfv-netboot-setup.md +++ b/netinfra/pfv-netboot-setup.md @@ -1,116 +1,10 @@ -# pfv-netboot — Reference Network Infrastructure (READ-ONLY reference) +# netinfra/pfv-netboot-setup.md -> **Status:** REFERENCE SOURCE ONLY. This node is production infrastructure. -> Do **not** modify it. This document describes it as audited so its services can -> be replicated to `pfv-netinfra-01` / `pfv-netinfra-02`. All data below was -> collected by **read-only** audit scripts (`audit-netboot.sh`, -> `deep-audit-netboot.sh`, `gather-configs.sh`) on 2026-07-27/28. - -## 1. Host - -| Item | Value | -|---|---| -| Hostname / FQDN | `pfv-netboot` / `pfv-netboot.knel.net` | -| OS | Debian GNU/Linux 12 (bookworm), kernel 6.1.0-44-amd64 | -| Hardware | 2 vCPU, ~1.9 GiB RAM, 491 GB disk (18 GB used) | -| Timezone | `America/Chicago` (US/Central) | -| LAN | `eth0` static `192.168.3.250/22`, gw `192.168.3.254` (`/etc/network/interfaces`) | -| Tailscale | `100.103.64.82` (`tailscale0`) | -| DNS resolver | Tailscale MagicDNS — `/etc/resolv.conf` → `100.100.100.100` | -| Docker | Docker Engine 29.6.2 (containerd v2.2.6, runc 1.3.6) | -| Access | `localuser` has passwordless sudo; **not** in `docker` group (uses `sudo docker`) | - -`eth1` is up but unconfigured; many docker bridges exist (`pihole_default`, -`ntp_default`, `dns_default`, and several stale ones). - -## 2. Services overview - -| Service | Form | Running? | -|---|---|---| -| **Pi-hole** (DNS sinkhole, recursive resolver) | Docker container `pihole` | ✅ healthy | -| **NTP** — overlay on Tailscale IP | Docker container `tsys-ntp` (`dockurr/chrony`) | ✅ healthy | -| **NTP** — system clock + LAN serving | bare-metal `ntpsec` (`ntpd`) | ✅ active, enabled | -| **Technitium DNS** (authoritative for `knel.net`) | Docker container | ❌ **not running**; config preserved in orphaned volume | - -## 3. Pi-hole (container) - -- **Compose:** `/root/pihole/docker-compose.yml` (compose project `pihole`) -- **Image:** `pihole/pihole:latest` — Core **v6.1.2**, Web v6.2.1, FTL v6.2.2 -- **Container:** `pihole`, `restart: always`, `cap_add: [SYS_NICE]`, network `pihole_default` -- **Ports (host):** - - | Host | Container | Purpose | - |---|---|---| - | `53/tcp`, `53/udp` | 53 | DNS | - | `10002/tcp` | 80 | Web admin (HTTP) | - | `10003/tcp` | 443 | Web admin (HTTPS, self-signed) | - -- **Environment:** `TZ=America/Chicago`, `FTLCONF_webserver_api_password=REDACTED_PASSWORD`, `FTLCONF_dns_listeningMode=all` -- **Data:** bind mount `/root/pihole/etc-pihole:/etc/pihole` (dir owned by `localuser`; files by container `pihole` uid) -- **Config (Pi-hole v6 TOML):** `pihole.toml`. Key settings: - - Upstream DNS: `192.168.3.16`, `8.8.8.8`, `2001:4860:4860::8888` - - `listeningMode = "ALL"`, `interface = "eth0"`, `dns.port = 53`, `dns.domain = "lan"` - - `queryLogging = true`, DNSSEC off -- **Adlists:** one entry — `https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts` (in `gravity.db`/`adlists.list`) -- **Gravity DB:** `/etc/pihole/gravity.db` (~5.5 MB) holds adlists/domainlists/clients/groups -- **Web admin:** `http://pfv-netboot:10002/admin/` (password `REDACTED_PASSWORD`) -- Note: query history `pihole-FTL.db` (~2.5 GB) is transient and **excluded** from replication. - -## 4. NTP (two layers) - -### 4a. chrony container (`tsys-ntp`) — overlay on the Tailscale IP -- **Compose:** `/root/NTP/docker-compose.yml` (project `ntp`) -- **Image:** `dockurr/chrony` -- **Env:** `NTP_SERVERS=pool.ntp.org` -- **Ports:** `100.103.64.82:123:123/udp` — bound specifically to the **Tailscale IP** -- `restart: always` -- chrony.conf (generated): `server pool.ntp.org iburst`, `allow all`, `rtcsync` -- On netboot this coexists with bare-metal ntpsec because ntpsec here does **not** pre-bind the specific Tailscale-IP socket, letting Docker claim it. - -### 4b. bare-metal `ntpsec` -- Unit `ntpsec.service` — active, enabled; `/usr/sbin/ntpd -c /etc/ntpsec/ntp.conf -g -N -u ntpsec:ntpsec` -- **Config** (`/etc/ntpsec/ntp.conf`): - ``` - driftfile /var/lib/ntp/ntp.drift - leapfile /usr/share/zoneinfo/leap-seconds.list - server pfvsvrpi.knel.net - restrict 127.0.0.1 - restrict ::1 - ``` -- Listens on all local addresses (incl. Tailscale) for UDP/123; serves LAN clients. - -## 5. Technitium DNS (currently stopped) - -- **Not running** — no container and **no compose file** exists for it. -- A previous deployment left an **orphaned Docker volume** `dns_tsys-dns-config` - (mountpoint `/var/lib/docker/volumes/dns_tsys-dns-config/_data`) whose contents - are intact (last activity 2025-06-23). A second typo'd volume - `dns_tyss-dns-config` is empty. -- Config files are **binary** (Technitium's own serialization), but copy verbatim: - `dns.config`, `auth.config`, `log.config`, `scopes/Default.scope`, - `self-signed-cert.pfx`, `cache.bin`, `zones/`, `stats/`, `logs/`. -- **Zones present** (12 reverse + 1 forward): - - `knel.net.zone` — forward zone; SOA `dns.knel.net. hostadmin.knel.net.` (serial `2025062313`). A-records for the internal fleet, including: `tsys1`, `rr-middleware`, `pfv-netboot`, `pfv-k8s-cnode1`…`cnode5`, `pfv-k8s-wnode3`, `tsys-k8scloud-netcup-1`, `tsys-kali-vptechops`, `tsys-kali-dev`; NS `dns.knel.net`. - - Reverse zones for Tailscale CGNAT ranges (`100.x.in-addr.arpa`): `199.86`, `145.105`, `181.103`, `184.108`, `194.67`, `2.108`, `211.114`, `46.96`, `64.103`, `75.110`, `97.82`, `119.127`. -- **Auth:** `auth.config` defines user `admin` (Administrators group) with a stored password hash; the plaintext password is whatever was set on the original Technitium instance. -- The compose project name historically was `dns` (network `dns_default` still exists). - -## 6. Firewall / misc - -- nftables/iptables: mostly Docker + Tailscale chains (`ts-input`, `ts-forward`, - `DOCKER`, `DOCKER-FORWARD`); default `INPUT ACCEPT`, `FORWARD DROP`, - `OUTPUT ACCEPT`. No UFW / firewalld. -- Also runs (out of scope for this replication): Samba (137/138/139, 445), NFS - (2049), rpcbind (111), Postfix (25), Cockpit (9090), Beszel agent, webmin/ - usermin (10000/10002/20000), Tailscale (41641). -- SELinux absent; AppArmor default docker profile. - -## 7. How it was audited (no changes made) - -```bash -ssh localuser@pfv-netboot 'bash -s' < audit-netboot.sh # broad read-only sweep -ssh localuser@pfv-netboot 'bash -s' < deep-audit-netboot.sh # docker inspect + compose -ssh localuser@pfv-netboot 'bash -s' < gather-configs.sh # pihole.toml + technitium -``` - -Artifacts: `netboot-audit.txt`, `netboot-deep-audit.txt`, `netboot-configs.txt`. +> **Documentation moved to Discourse — the canonical source of truth.** +> +> **pfv-netboot reference node setup** +> +> **Read it here:** https://community.turnsys.com/t/306 +> +> *Migrated 2026-08-06. This file is kept as a pointer for git-browsing context. +> Do not update content here — edit the Discourse wiki topic instead.* diff --git a/netinfra/pfv-netinfra-setup.md b/netinfra/pfv-netinfra-setup.md index 9015aa2..5723e17 100644 --- a/netinfra/pfv-netinfra-setup.md +++ b/netinfra/pfv-netinfra-setup.md @@ -1,220 +1,10 @@ -# pfv-netinfra-01 / pfv-netinfra-02 — Network Services Setup +# netinfra/pfv-netinfra-setup.md -These two nodes replicate the network-infrastructure services of **pfv-netboot** -(Pi-hole, Technitium DNS, NTP). They were deployed by `setup-netinfra.sh`, which -reads config from pfv-netboot (read-only) and relays it to each target. - -## 1. Nodes - -| | pfv-netinfra-01 | pfv-netinfra-02 | -|---|---|---| -| OS | Debian 13 (trixie), kernel 6.12.96+deb13 | Debian 13 (trixie) | -| LAN | `ens18` `192.168.3.252/24` | `ens18` `192.168.3.253/24` | -| Tailscale | `100.70.181.72` | `100.93.194.82` | -| RAM / Disk | 1.9 GiB / 30 GB (27 GB free) | 3.7 GiB / 30 GB (27 GB free) | -| Resolver | Tailscale MagicDNS (`100.100.100.100`) | same | -| Docker | 29.6.2 (pre-installed, enabled) | 29.6.2 | -| Access | `ssh localuser@pfv-netinfra-0X`, passwordless sudo; `localuser` **not** in docker group → use `sudo docker` | same | - -## 2. Service layout - -All services live under `/home/localuser/services//` (owned by `localuser` -so the compose files are directly editable; data dirs keep container uids): - -``` -/home/localuser/services/ -├── pihole/ -│ ├── docker-compose.yml -│ └── etc-pihole/ # copied from netboot /root/pihole/etc-pihole -│ ├── pihole.toml # Pi-hole v6 config (upstreams, etc.) -│ ├── gravity.db # adlists / domainlists / clients / groups -│ ├── adlists.list -│ ├── dnsmasq.conf -│ ├── tls.{crt,pem,crt_ca} -│ └── versions -├── ntp/ -│ └── docker-compose.yml # chrony container (see §5 — not used; host ntpsec serves) -└── technitium/ - ├── docker-compose.yml - └── config/ # copied from netboot orphaned volume dns_tsys-dns-config/_data - ├── dns.config - ├── auth.config - ├── scopes/Default.scope - ├── self-signed-cert.pfx - └── zones/ # knel.net.zone + 12 Tailscale reverse zones -``` - -## 3. Pi-hole (container `pihole`) - -Image `pihole/pihole:latest`; `restart: always`; `cap_add: [SYS_NICE]`. - -| Host port | Container | Purpose | -|---|---|---| -| `53/tcp`, `53/udp` | 53 | DNS (the LAN/Tailscale recursive resolver) | -| `10002/tcp` | 80 | Web admin (HTTP) | -| `10003/tcp` | 443 | Web admin (HTTPS) | - -`docker-compose.yml`: -```yaml -services: - pihole: - container_name: pihole - image: pihole/pihole:latest - hostname: pihole - ports: - - "53:53/tcp" - - "53:53/udp" - - "10002:80/tcp" - - "10003:443/tcp" - environment: - TZ: 'America/Chicago' - FTLCONF_webserver_api_password: 'REDACTED_PASSWORD' - FTLCONF_dns_listeningMode: 'all' - volumes: - - './etc-pihole:/etc/pihole' - cap_add: - - SYS_NICE - restart: always -``` -- Upstream DNS (from copied `pihole.toml`): `192.168.3.16`, `8.8.8.8`, `2001:4860:4860::8888`. -- Adlist: `https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts`. -- `pihole.toml` `interface` was adapted from netboot's `eth0` to the target's `ens18`. -- Web admin: `http://:10002/admin/` — password **`REDACTED_PASSWORD`** (same as netboot). -- Web UI URL per node: `http://100.70.181.72:10002/admin/` (-01), `http://100.93.194.82:10002/admin/` (-02). - -## 4. Technitium DNS (container `tsys-dns`) - -Image `technitium/dns-server`; `restart: always`. Authoritative DNS for -`knel.net` (and Tailscale reverse zones), config copied verbatim from netboot's -orphaned `dns_tsys-dns-config` volume. - -| Host port | Container | Purpose | -|---|---|---| -| `5300/tcp`, `5300/udp` | 53 | DNS (remapped — see note) | -| `5380/tcp` | 5380 | Web console (HTTP) | -| `53443/tcp` | 53443 | Web console (HTTPS) | - -`docker-compose.yml`: -```yaml -services: - technitium: - image: technitium/dns-server - container_name: tsys-dns - ports: - - "5300:53/tcp" - - "5300:53/udp" - - "5380:5380/tcp" - - "53443:53443/tcp" - volumes: - - './config:/etc/dns' - restart: always -``` -- Zones loaded (verified): `knel.net` SOA → `dns.knel.net. hostadmin.knel.net. 2025062313 900 300 604800 900`, plus 12 Tailscale reverse zones. -- Web console: `http://:5380/` → user **`admin`** + the original Technitium - password (carried over via `auth.config`). If the password is unknown, reset it - from the console or by removing `config/auth.config` and recreating the container. -- **Port note:** Technitium's native DNS port (53) is remapped to host **5300** - because Pi-hole already owns host :53 (they cannot both bind 0.0.0.0:53). To - query the authoritative server: `dig -p 5300 @ knel.net SOA`. To make - Pi-hole resolve `knel.net` via Technitium, add a conditional/local upstream in - Pi-hole pointing to the container (e.g. `127.0.0.1#5300` is not host-reachable - from Pi-hole's netns — use the docker bridge IP of `tsys-dns`, or add - `knel.net` A-records directly in Pi-hole's Local DNS). - -## 5. NTP (host `ntpsec`, not a container) - -Both targets **already run a bare-metal `ntpsec` daemon** (active, enabled) that -serves NTP on every local address — including the Tailscale IP — and keeps the -system clock synced. This is the **same daemon family as netboot's own bare-metal -ntpsec**. - -- **Why no chrony container?** netboot's chrony container (`tsys-ntp`) binds the - Tailscale IP `100.103.64.82:123`; on netboot that works only because its ntpsec - does **not** pre-bind the specific Tailscale-IP socket. On these targets ntpsec - **does** bind the Tailscale IP, so the container cannot claim it (`address - already in use`) and would be a non-functional duplicate (verified: the - container started but never synced — Stratum 0). It is therefore intentionally - **omitted**; host ntpsec provides NTP. `setup-netinfra.sh` detects an active - host NTP unit and removes any stale `tsys-ntp` container. -- ntpsec config (`/etc/ntpsec/ntp.conf`): Debian NTP pool (`0-3.debian.pool.ntp.org`), - `restrict default kod nomodify noquery limited` (serves time, blocks mgmt queries). -- Verified sync: -01 stratum 2 (~2 ms offset), -02 stratum 3 (~0.2 ms offset), - leap normal. - -The `ntp/docker-compose.yml` is still written on each node for parity/reference -(and in case the host NTP is ever disabled — then `sudo docker compose -f -/home/localuser/services/ntp/docker-compose.yml up -d` brings up chrony). - -## 6. Verification results (2026-07-28) - -| Check | pfv-netinfra-01 | pfv-netinfra-02 | -|---|---|---| -| `pihole` health | healthy | healthy | -| `dig @127.0.0.1:53 pi.hole` | `172.18.0.2` | `172.18.0.2` | -| Pi-hole web `:10002` | HTTP 302 (→login) | HTTP 302 | -| `dig @127.0.0.1:5300 knel.net SOA` | SOA answered | SOA answered | -| Technitium web `:5380` | HTTP 200 | HTTP 200 | -| NTP daemon | ntpsec, stratum 2, synced | ntpsec, stratum 3, synced | - -## 7. Operating the services - -```bash -# status -sudo docker ps - -# Pi-hole -sudo docker compose -f /home/localuser/services/pihole/docker-compose.yml ps -sudo docker compose -f /home/localuser/services/pihole/docker-compose.yml logs -f -sudo docker exec pihole pihole -v # version -sudo docker exec pihole pihole -g # rebuild gravity -sudo docker exec pihole pihole -a -p # set/change web password - -# Technitium -sudo docker compose -f /home/localuser/services/technitium/docker-compose.yml logs -f -sudo docker exec tsys-dns sh # explore /etc/dns - -# NTP (host) -systemctl status ntpsec -ntpq -pn -``` - -## 8. Differences from pfv-netboot (intentional) - -1. **Layout** under `/home/localuser/services/` instead of `/root` (so `localuser` - can manage compose files); Pi-hole data dir still owned by `localuser`, as on netboot. -2. **Pi-hole `interface`** set to `ens18` (targets' NIC) instead of netboot's `eth0`. -3. **NTP:** host `ntpsec` (Debian pool) used instead of netboot's chrony container - (the container cannot bind the Tailscale IP here; see §5). -4. **Technitium DNS** host port remapped `53 → 5300` to avoid clashing with Pi-hole - on `:53`. The `knel.net` zone and all reverse zones are identical to netboot's. -5. Pi-hole query logs (`pihole-FTL.db*`) and regenerable caches/backups are not - copied (transient); gravity DB and all configuration are. - -## 9. Re-running / reproducing - -`setup-netinfra.sh` is **idempotent** — it skips re-copying config if already -present and uses `docker compose up -d` (no-ops when unchanged). It reads -pfv-netboot read-only and never mutates it. - -```bash -./setup-netinfra.sh # deploy to both nodes -./setup-netinfra.sh pfv-netinfra-01 # deploy one node -./setup-netinfra.sh pfv-netinfra-01 verify # verify only -``` - -Prerequisites: SSH key access to all three hosts as `localuser` with passwordless -sudo; the targets reach `192.168.3.16`/`8.8.8.8` for Pi-hole upstream and the -internet for image pulls. - -## 10. Files in this directory - -| File | Purpose | -|---|---| -| `setup-netinfra.sh` | orchestrator: deploys + verifies the clone on -01/-02 | -| `audit-netboot.sh` | broad read-only audit of pfv-netboot | -| `deep-audit-netboot.sh` | docker inspect / compose / volume deep audit (read-only) | -| `gather-configs.sh` | targeted config pull (pihole.toml, technitium) (read-only) | -| `baseline.sh` | read-only baseline of a target node | -| `netboot-audit.txt`, `netboot-deep-audit.txt`, `netboot-configs.txt` | audit output | -| [`pfv-netboot-setup.md`](pfv-netboot-setup.md) | reference-node documentation | -| [`pfv-netinfra-setup.md`](pfv-netinfra-setup.md) | this document | +> **Documentation moved to Discourse — the canonical source of truth.** +> +> **pfv-netinfra-01/02 initial setup** +> +> **Read it here:** https://community.turnsys.com/t/306 +> +> *Migrated 2026-08-06. This file is kept as a pointer for git-browsing context. +> Do not update content here — edit the Discourse wiki topic instead.* diff --git a/netinfra/pihole/README.md b/netinfra/pihole/README.md index f2ad36c..dd35d16 100644 --- a/netinfra/pihole/README.md +++ b/netinfra/pihole/README.md @@ -1,61 +1,10 @@ -# netinfra/pihole/ — Pi-hole recursive DNS (pfv-netinfra-01/02) +# netinfra/pihole/README.md -> **Redmine:** [#376](https://projects.knownelement.com/issues/376) (up/down alerts + commit hardening) · [#357](https://projects.knownelement.com/issues/357) (cluster build, closed) - -Pi-hole v6 runs as the recursive resolver on port 53 of both DNS nodes. -Technitium (`tsys-dns`) runs as the authoritative server on port 5300; the two -share the `dnsnet` Docker network so Pi-hole can conditional-forward `knel.net` -zones to Technitium. - -## Hardening (defense-in-depth against gravity.db corruption) - -The operator hit a production outage when Pi-hole's `/dev/shm` was too small, -corrupting `gravity.db`. The live config on both nodes now includes: - -- **`shm_size: 1024M`** — fixes the root cause (default 64M was too small). -- **`gravity-validate.sh`** — entrypoint that checks the SQLite header of - `gravity.db` before start; auto-moves a corrupt/empty DB aside so Pi-hole - can regenerate it cleanly. -- **Healthcheck** — `dig +norecurse @127.0.0.1 pi.hole` + gravity.db non-empty; - fails the container if DNS or the DB is broken. -- **`autoheal`** sidecar — restarts any container labeled `autoheal=true` that - goes unhealthy. - -## Deploy - -The compose reads the web UI password from a gitignored `.env`: - -```bash -cd netinfra/pihole/ -cp .env.example .env # then edit .env and set PIHOLE_WEB_PASSWORD -docker compose up -d -``` - -Files are deployed to `/home/localuser/services/pihole/` on each node. Volumes -(`./etc-pihole`, `./etc-dnsmasq.d`) hold the persistent state. - -## IPv6 disabled + Rate-limiting disabled - -### IPv6 -Both netinfra nodes run **IPv4-only**. IPv6 is disabled at the kernel level -(`/etc/sysctl.d/99-disable-ipv6.conf`) because netinfra-01 has no IPv6 internet -route, and Pi-hole's default IPv6 upstream (Google `2001:4860:4860::8888`) was -causing continuous "Network unreachable" errors + intermittent DNS failures -detected by Uptime Kuma. The upstream is now `8.8.8.8` (IPv4 only). - -### Rate-limiting -Pi-hole's default rate-limit (1000 queries / 60 seconds per client) was -**the root cause of Uptime Kuma DNS flapping**. Uptime Kuma runs on the -Cloudron VPS (`tsys-cloudron`, `100.107.35.78`) and sends high-volume DNS -queries to monitor dozens of hosts. When it exceeded 1000 queries/60s, -Pi-hole responded with REFUSED, which Uptime Kuma detected as DNS being -down. Rate-limiting is now disabled (`count=0, interval=0`) since this is a -private tailnet with no risk of DNS amplification attacks. - -## Verify - -```bash -dig @127.0.0.1 +short google.com # recursive -dig @127.0.0.1 +short git.knownelement.com # knel.net via Technitium forward -docker inspect pihole --format '{{.State.Health.Status}}' -``` +> **Documentation moved to Discourse — the canonical source of truth.** +> +> **Pi-hole recursive DNS hardening** +> +> **Read it here:** https://community.turnsys.com/t/306 +> +> *Migrated 2026-08-06. This file is kept as a pointer for git-browsing context. +> Do not update content here — edit the Discourse wiki topic instead.* diff --git a/proxmox/docs/AUDIT-2026-07-29.md b/proxmox/docs/AUDIT-2026-07-29.md index 47f7463..b677d62 100644 --- a/proxmox/docs/AUDIT-2026-07-29.md +++ b/proxmox/docs/AUDIT-2026-07-29.md @@ -1,452 +1,10 @@ -# Fleet Audit — 2026-07-29 (consolidated, storage-corrected) - -> **The single authoritative audit.** Supersedes all prior audits and the -> placement data in [`PROJECT.md`](PROJECT.md) §4-§8 where they conflict. -> Capacity targets are codified in [`CAPACITY-GOALS.md`](CAPACITY-GOALS.md). - -**Audit time:** 2026-07-29, fresh data from all 7 hosts (~05:09 CDT hardware -dumps + ~05:40 CDT capacity probe). Storage configs read directly from -`/etc/pve/storage.cfg`, `/etc/exports`, `/etc/fstab` on every host. -**Hosts online:** tsys1, tsys3, tsys4, tsys5, tsys6, tsys7, tsys9 -**Hosts offline:** tsys2 (Win10, pending rebuild), tsys8 (retired / permanently offline) - ---- - -## 1. Executive summary - -| Goal | Status | Detail | -|------|--------|--------| -| **Memory ≤ 80% actual steady-state** | **PASS (all 7 hosts)** | Highest is tsys1 at 73%. Fleet is healthy. | -| **CPU ~50% at idle** | **MISS (all 7 hosts)** | Every host idles at **0-5% busy**. Under-utilized, not over-loaded. | - -| Finding | Severity | § | -|---------|----------|---| -| **D2 carries 17 VMs** — the worst IOPS hotspot in the fleet | High | 5 | -| **S1 is NOT empty — it carries 16 sectestbed VMs via `local-nonprod`** (previous audit was wrong) | High | 4, 5 | -| ~~tsys6 S1 NFS export typo~~ (`S21`) — **FIXED by operator 2026-07-29**; all 8 stores active | Resolved | 6 | -| ~~tsys9 missing S1~~ in storage.cfg — **FIXED by operator 2026-07-29**; S1 active | Resolved | 6 | -| **`local-nonprod` on tsys5 = the S1 physical disk** (dir storage, not a separate disk) | Topology | 4 | -| **tsys5 is single-CPU** (CPU2 socket unpopulated; prior "dual E5620" was wrong) | Medium | 9 | -| **cnode4 (601) stopped, cnode5 (706) deleted** (user action this session) | Resolved | 10 | - ---- - -## 2. Host capacity (live, 2026-07-29 ~05:40 CDT) - -| Host | CPU (era) | Threads | RAM | Running VMs | Alloc RAM | Alloc% | **Actual%** | Swap used | CPU busy | Load | -|------|-----------|---------|-----|-------------|-----------|--------|------------|-----------|----------|------| -| tsys1 | i7-4770 Haswell (2013) | 8 | 31 GB | 11 | 36 GB | 115% | **73%** | 0 | 4% | 1.1 | -| tsys3 | Xeon E3-1535M v5 Skylake (2015) | 8 | 31 GB | 1 | 28 GB | 90% | **10%** | 0 | 0% | 0.3 | -| tsys4 | Xeon E3-1246 v3 Haswell (2013) | 8 | 16 GB | 1 | 2 GB | 13% | **14%** | 1.1 GB | 0% | 1.3 | -| tsys5 | Xeon E5620 Westmere (2010) **single CPU** | 8 | 94 GB | 17 | 126 GB | 134% | **37%** | 0 | 5% | 4.0 | -| tsys6 | 2× Xeon E5530 Nehalem (2009) | 16 | 126 GB | 2 | 44 GB | 35% | **4%** | 0 | 0% | 0.6 | -| tsys7 | 2× Xeon E5-2630 v2 Ivy Bridge (2013) | 24 | 189 GB | 5 | 52 GB | 28% | **7%** | 0 | 1% | 0.9 | -| tsys9 | i5-10500 Comet Lake (2020) | 12 | 23 GB | 6 | 30 GB | 129% | **62%** | 0 | 1% | 0.3 | - -**Changes since first collection (~05:09):** cnode4 (601) stopped on tsys6, -cnode5 (706) deleted from tsys7. No host migrations performed yet. - ---- - -## 3. Storage topology (CORRECTED — read from live configs) - -> **`storage.cfg` is per-host, not cluster-wide.** Each Proxmox node defines its -> own storage entries. The same physical disk can appear under different storage -> IDs on different hosts. This was the source of the prior audit's errors. - -### 3.1 Physical disk → Proxmox storage ID mapping - -| Physical disk | Server | NFS export | NFS ID (clients) | Local ID (on tsys5) | Type | -|---------------|--------|------------|-----------------|---------------------|------| -| WDC Red 3TB | tsys4 | `/mnt/tsys4/D2` | `D2` | — | HDD | -| SK hynix SC300 512GB | tsys4 | `/mnt/tsys4/D3` | `D3` | — | SSD (USB) | -| Hitachi 2TB | tsys4 | `/mnt/tsys4/D5` | `D5` | — | HDD | -| *(disk1)* | tsys4 | `/mnt/tsys4/D1` | *(not in most storage.cfg)* | — | unknown | -| *(disk7)* | tsys4 | `/mnt/tsys4/D7` | *(not in most storage.cfg)* | — | unknown | -| **Seagate 916GB** | **tsys5** | **`/mnt/pfv-tsys5/S1`** | **`S1`** | **`local-nonprod`** | **HDD** | -| Seagate 916GB | tsys5 | `/mnt/pfv-tsys5/S2` | `S2` | `S2` (NFS client) | HDD | -| Seagate 916GB | tsys5 | `/mnt/pfv-tsys5/S3` | `S3` | `S3` (NFS client) | HDD | -| Toshiba 458GB | tsys5 | `/mnt/pfv-tsys5/S4` | `S4` | `S4` (NFS client) | HDD | -| **Samsung 860 PRO 234GB** | **tsys5** | **`/mnt/pfv-tsys5/ssd`** | **`T5-SSD`** | **`SSD`** (dir, local-only) | **SSD** | - -### 3.2 Critical correction: `local-nonprod` = S1 - -On tsys5, the storage entry is: -``` -dir: local-nonprod - path /mnt/pfv-tsys5/S1 -``` - -**`local-nonprod` is NOT a separate disk.** It is the **S1 Seagate 916GB HDD**, -accessed locally as a directory storage instead of via NFS. The 16 sectestbed -and preprod VMs "on local-nonprod" are physically on S1. They appear locked to -tsys5 because Proxmox treats `local-nonprod` (dir type) as a different storage -than `S1` (NFS type), even though they point to the same directory. - -**Implication:** S1 has **16 VMs of IOPS load**, not 0. The prior audit's plan -to move production VMs onto S1 would have stacked them on top of 16 sandboxes. - -### 3.3 Disk speed tiers (HDD only — SSD/NVMe deferred to Friday) - -| Tier | Spindles | Speed | Current use | -|------|----------|-------|-------------| -| NFS-HDD (tsys4) | D2, D5 | 80-120 MB/s | Production VMs | -| NFS-HDD (tsys5) | S1, S2, S3, S4 | 80-120 MB/s | Production + sectestbed VMs | -| Local HDD (tsys5) | local-nonprod (=S1) | 80-120 MB/s | Sectestbed suite (locked to tsys5) | -| SSD (tsys4 USB) | D3 | unreliable (USB 2.0) | Empty — do not use until Friday | -| SSD (tsys5 local) | SSD dir (=Samsung 860 PRO) | 200-400 MB/s | ultix-streaming only | - ---- - -## 4. Per-spindle IOPS load (CORRECTED) - -| Spindle | Server | NFS-visible VMs | Local-only VMs | **Total physical load** | -|---------|--------|----------------|----------------|------------------------| -| **D2** (3TB) | tsys4 | **17** | 0 | **17 — worst hotspot** | -| D5 (2TB) | tsys4 | 3 | 0 | 3 | -| **S1** (916GB) | tsys5 | 0 | **16** (via local-nonprod) | **16 — second worst** | -| S2 (916GB) | tsys5 | 4 | 0 | 4 | -| S3 (916GB) | tsys5 | 3 | 0 | 3 | -| S4 (458GB) | tsys5 | 1 | 0 | 1 — most spare capacity | - -**D2 and S1 are both overloaded. S2/S3/S4 and D5 have room.** - -The 16 sectestbed VMs on S1 are idle sandboxes — they don't generate much real -IOPS, but they're all on one spindle. If any of them wake up simultaneously (CI -pipeline, test suite), S1 becomes a bottleneck. **Do not add production VMs to -S1.** Target S2, S3, S4, and D5 for IOPS spreading. - ---- - -## 5. Storage bugs and config gaps found - -### 5.1 BUG: tsys6 S1 export path typo (S21) — RESOLVED ✓ - -**Fixed by operator 2026-07-29.** The export path was corrected from `S21` to -`S1`. Verified: `pvesm status` on tsys6 shows all 8 NFS stores active, including -S1 (96 GB, 10.3% used by the sectestbed VMs visible via NFS). - -### 5.2 GAP: tsys9 missing S1 in storage.cfg — RESOLVED ✓ - -**Fixed by operator 2026-07-29.** S1 entry added to tsys9's storage.cfg. -Verified: all 8 NFS stores active on tsys9, S1 listing identical to tsys6. - -### 5.3 ORPHANED: D1 and D7 exports on tsys4 - -tsys4 exports D1 and D7 via NFS, but neither appears in most hosts' storage.cfg. -- **D1** holds KNELTemplate (4000) and RestoreTemplate (5500) — stopped - templates only, no running VMs. -- **D7** holds a disk for DellOpenManageEnterprise (500) — stopped. - -These aren't causing problems but are underdocumented. Not actionable now. - ---- - -## 6. Current VM inventory (host + disk, post-user-changes) - -> cnode4 (601) = stopped. cnode5 (706) = deleted. All other VMs unchanged from -> start of session. `Disk` = NFS export or local storage holding the primary -> qcow2. - -### tsys1 — Infrastructure (11 running) - -| VMID | Name | vCPU | RAM | Disk | Server | -|------|------|------|-----|------|--------| -| 100 | pfv-bms (HomeAssistant) | 2 | 4 GB | D2 | tsys4 | -| 101 | tsys-ca | 2 | 2 GB | D2 | tsys4 | -| 102 | pfv-k8s-wnode-tsys1 | 4 | 4 GB | S3 | tsys5 | -| 103 | **pfv-netinfra-01** | 2 | 2 GB | S2 | tsys5 | -| 104 | tsys-librenms | 2 | 2 GB | D2 | tsys4 | -| 105 | tsys-proxmox-datacenter | 2 | 2 GB | D2 | tsys4 | -| 106 | pfv-k8s-cnode3 | 2 | 4 GB | D5 | tsys4 | -| 108 | **tsys-ucs-01** | 4 | 8 GB | D2 | tsys4 | -| 109 | tailscale-router | 2 | 2 GB | D2 | tsys4 | -| 114 | kali-tsys | 2 | 2 GB | D2 | tsys4 | -| 117 | tsys-secure-workbench | 2 | 4 GB | D2 | tsys4 | - -### tsys3 — Compute (1 running) - -| VMID | Name | vCPU | RAM | Disk | Server | -|------|------|------|-----|------|--------| -| 313 | pfv-k8s-wnode-tsys3 | 8 | 28 GB | D5 | tsys4 | - -### tsys4 — Storage (1 running) - -| VMID | Name | vCPU | RAM | Disk | Server | -|------|------|------|-----|------|--------| -| 400 | pfv-proxmox-backup-server | 2 | 2 GB | local-lvm | LOCAL | - -### tsys5 — Storage + sandboxes (17 running) - -| VMID | Name | vCPU | RAM | Disk | Server | Migratable? | -|------|------|------|-----|------|--------|-------------| -| 509 | pfv-k8s-wnode-tsys5 | 8 | 32 GB | D2 | tsys4 | Yes (NFS) | -| 5111 | ultix-streaming | 4 | 9 GB | SSD (local dir) | LOCAL | No (local) | -| 5112 | ultix-offstage | 4 | 6 GB | local-lvm | LOCAL | No (local) | -| 5101 | sectestbed-siem | 4 | 10 GB | local-nonprod (=S1) | LOCAL* | No (dir storage) | -| 53100 | tsys-preprod-awx | 4 | 9 GB | local-nonprod (=S1) | LOCAL* | No | -| 53101 | tsys-preprod-siem | 4 | 12 GB | local-nonprod (=S1) | LOCAL* | No | -| 53102 | tsys-preprod-rancherplatform | 4 | 8 GB | local-nonprod (=S1) | LOCAL* | No | -| ×10 | sectestbed suite | 4 ea | 4 GB ea | local-nonprod (=S1) | LOCAL* | No | - -> *\*LOCAL = locked to tsys5. The local-nonprod VMs are on the S1 physical disk -> but accessed via dir storage, not NFS — Proxmox will not migrate them. To -> unlock, move disk from `local-nonprod` → `S1` (same physical disk, changes the -> storage ID to the NFS-shared one). Low priority — these are idle sandboxes.* - -### tsys6 — RackRental-ready (2 running, 1 stopped) - -| VMID | Name | vCPU | RAM | Disk | Server | Status | -|------|------|------|-----|------|--------|--------| -| 100 | pfv-k8s-wnode-tsys6 | 4 | 32 GB | S4 | tsys5 | running | -| 600 | tsys-awx | 4 | 12 GB | D2 | tsys4 | running | -| 601 | pfv-k8s-cnode4 | 4 | 4 GB | D2 | tsys4 | **STOPPED** | - -### tsys7 — RackRental-ready (5 running) - -| VMID | Name | vCPU | RAM | Disk | Server | -|------|------|------|-----|------|--------| -| 701 | pfv-k8s-wnode-tsys7 | 4 | 32 GB | S3 | tsys5 | -| 702 | hfnoc-uisp | 4 | 8 GB | D2 | tsys4 | -| 703 | rr-middleware | 2 | 2 GB | D2 | tsys4 | -| 704 | TCTC | 4 | 6 GB | D2 | tsys4 | -| 705 | pfv-k8s-cnode2 | 4 | 4 GB | D2 | tsys4 | - -### tsys9 — Infra + control plane (6 running) - -| VMID | Name | vCPU | RAM | Disk | Server | -|------|------|------|-----|------|--------| -| 901 | tsys-siem | 2 | 8 GB | D2 | tsys4 | -| 902 | **tsys-ucs-02** | 4 | 8 GB | S2 | tsys5 | -| 903 | kali-rd | 2 | 2 GB | D5 | tsys4 | -| 904 | **pfv-netinfra-02** | 2 | 4 GB | D2 | tsys4 | -| 905 | pfv-k8s-wnode-tsys9 | 4 | 4 GB | S2 | tsys5 | -| 906 | pfv-k8s-cnode1 | 2 | 4 GB | S3 | tsys5 | - -### k3s control plane (ground truth via `kubectl`) - -3 cnodes joined: cnode1 (906/tsys9), cnode2 (705/tsys7), cnode3 (106/tsys1). -All control-plane,etcd role. CPU 2%, mem ~1.1 GB each. **No wnodes joined yet.** - ---- - -## 7. Workload placement model - -| Workload family | Needs | Best hosts | Why | -|-----------------|-------|-----------|-----| -| **ETL / HPC / cross-compile** | High IPC + fast storage | **tsys9, tsys3, tsys2(in)** | Newest CPUs + local SSD/NVMe | -| **RackRental.net** | Massive RAM, CPU-light | **tsys7, tsys6** | 189/126 GB RAM; weak CPU fine for idle labs | -| **Infra + k8s control plane** | Always-on, low resource | **tsys1, tsys9** | Efficient enough; cnodes spread for etcd diversity | - ---- - -## 8. Answers to operator's questions - -### 8.1 "Should I make the k8s wnode VMs bigger on tsys1/9 for ETL/HPC?" - -**tsys9 — YES.** Newest CPU (i5-10500, 12 threads) + local SSD. Currently 4c/4GB. -After moving tsys-siem (8 GB) to tsys6, grow to **6c/8-16 GB**. - -**tsys1 — NO.** The i7-4770 is Haswell-2013 (2nd-oldest). tsys1 is the most -RAM-loaded host (73% actual, 115% alloc). It should shed VMs, not grow its wnode. - -### 8.2 "Are tsys6/7 better for RackRental.net?" - -**Yes.** RAM-bound, CPU-light labs fit 126/189 GB perfectly. They don't need -more RAM (4%/7% actual). Their bottleneck is disk (USB 2.0 local-lvm ≈ 30 MB/s) -— deferred to Friday + StorageClass work. - -### 8.3 "How should I deploy the DDR3?" - -Verify type first. tsys5/6/7 require **RDIMM ECC** (consumer UDIMM won't POST). -tsys1 is maxed (32 GB, caps at 8 GB/slot). tsys4 caps at 8 GB/slot (max 32 GB). -Most likely only useful for tsys4 16→32 GB. The real RAM lever is **free VM -migration**, not sticks. See [`CAPACITY-GOALS.md`](CAPACITY-GOALS.md) §5. - ---- - -## 9. Hardware corrections - -**tsys5 is single-CPU.** dmidecode: `CPU2: Status Unpopulated`. The T7500 has -one E5620 (4c/8t), not two. The 6 empty DIMM slots are on the CPU2 branch — -useless without a 2nd CPU. Seating a spare LGA771 Xeon would double CPU capacity -and activate the CPU2 DIMM branch. - -**tsys4 is using ~1.1 GB of swap.** Only host with active swap. Minor, but -monitor after the planned RAM upgrade. - ---- - -## 10. Target end-state inventory (RAM/CPU + HDD IOPS balanced) - -> All disks stay on **HDD** (SSD/NVMe deferred to Friday). Goals: (a) relieve -> tsys1/9 RAM over-allocation, (b) relieve D2 (17→5 VMs), (c) spread to S2/S3/S4 -> (currently 4/3/1), (d) **avoid S1** (already has 16 local VMs), (e) keep HA -> pairs split across storage servers. - -### tsys1 — Infrastructure (6 running, ~24 GB alloc → 77%) - -| VMID | Name | vCPU | RAM | Disk | Server | Change | -|------|------|------|-----|------|--------|--------| -| 100 | pfv-bms | 2 | 4 GB | D2 | tsys4 | none | -| 101 | tsys-ca | 2 | 2 GB | D2 | tsys4 | none | -| 102 | pfv-k8s-wnode-tsys1 | 4 | 4 GB | S3 | tsys5 | none | -| 103 | **pfv-netinfra-01** | 2 | 2 GB | S2 | tsys5 | none | -| 106 | pfv-k8s-cnode3 | 2 | 4 GB | D5 | tsys4 | none | -| 108 | **tsys-ucs-01** | 4 | 8 GB | D2 | tsys4 | none | - -### tsys3 — Compute (1 running) - -| VMID | Name | vCPU | RAM | Disk | Server | Change | -|------|------|------|-----|------|--------|--------| -| 313 | pfv-k8s-wnode-tsys3 | 8 | 28 GB | D5 | tsys4 | none | - -### tsys4 — Storage (1 running) - -| VMID | Name | vCPU | RAM | Disk | Change | -|------|------|------|-----|------|--------| -| 400 | pfv-proxmox-backup-server | 2 | 2 GB | local-lvm | none | - -### tsys5 — Storage + sandboxes (17 running, unchanged) - -All 17 stay. wnode-tsys5 (509) disk moves D2→S4. The 16 local-nonprod VMs stay -locked (idle sandboxes, low priority). - -### tsys6 — RackRental-ready (3 running, ~52 GB alloc → 41%) - -| VMID | Name | vCPU | RAM | Disk | Server | Change | -|------|------|------|-----|------|--------|--------| -| 100 | pfv-k8s-wnode-tsys6 | 4 | 32 GB | S4 | tsys5 | none | -| 600 | tsys-awx | 4 | 12 GB | **D5** | tsys4 | **disk D2→D5** | -| 901 | tsys-siem | 2 | 8 GB | **S2** | tsys5 | **host tsys9→tsys6, disk D2→S2** | -| ~~601~~ | ~~cnode4~~ | — | — | — | — | **DELETE (stopped, stale)** | - -### tsys7 — RackRental-ready (10 running, ~64 GB alloc → 34%) - -| VMID | Name | vCPU | RAM | Disk | Server | Change | -|------|------|------|-----|------|--------|--------| -| 701 | pfv-k8s-wnode-tsys7 | 4 | 32 GB | S3 | tsys5 | none | -| 702 | hfnoc-uisp | 4 | 8 GB | **S4** | tsys5 | **disk D2→S4** | -| 703 | rr-middleware | 2 | 2 GB | **S4** | tsys5 | **disk D2→S4** | -| 704 | TCTC | 4 | 6 GB | **S3** | tsys5 | **disk D2→S3** | -| 705 | pfv-k8s-cnode2 | 4 | 4 GB | **S2** | tsys5 | **disk D2→S2** | -| 104 | tsys-librenms | 2 | 2 GB | **S3** | tsys5 | **host tsys1→tsys7, disk D2→S3** | -| 105 | tsys-proxmox-datacenter | 2 | 2 GB | **S4** | tsys5 | **host tsys1→tsys7, disk D2→S4** | -| 109 | tailscale-router | 2 | 2 GB | D2 | tsys4 | **host tsys1→tsys7** (disk stays D2) | -| 114 | kali-tsys | 2 | 2 GB | **S2** | tsys5 | **host tsys1→tsys7, disk D2→S2** | -| 117 | tsys-secure-workbench | 2 | 4 GB | **S3** | tsys5 | **host tsys1→tsys7, disk D2→S3** | - -### tsys9 — Infra + control + compute (5 running, ~22 GB alloc → 94%) - -| VMID | Name | vCPU | RAM | Disk | Server | Change | -|------|------|------|-----|------|--------|--------| -| 902 | **tsys-ucs-02** | 4 | 8 GB | S2 | tsys5 | none | -| 903 | kali-rd | 2 | 2 GB | D5 | tsys4 | none | -| 904 | **pfv-netinfra-02** | 2 | 4 GB | D2 | tsys4 | none | -| 905 | pfv-k8s-wnode-tsys9 | **6** | **8 GB** | S2 | tsys5 | **grow 4c/4G→6c/8G** | -| 906 | pfv-k8s-cnode1 | 2 | 4 GB | S3 | tsys5 | none | -| ~~901~~ | ~~tsys-siem~~ | — | — | — | — | **move→tsys6** | - -### End-state per-spindle IOPS load - -| Spindle | Server | # VMs | VMs on it | -|---------|--------|-------|-----------| -| **D2** (3TB) | tsys4 | **5** | bms, ca, ucs01, tailscale, netinfra02 | -| D5 (2TB) | tsys4 | 4 | cnode3, wnode3, awx, kali-rd | -| S1 (916GB) | tsys5 | **16** (unchanged) | sectestbed suite (local-nonprod, idle) | -| **S2** (916GB) | tsys5 | **7** | netinfra01, ucs02, wnode9, cnode2, siem, kali, pdm* | -| **S3** (916GB) | tsys5 | **7** | wnode1, wnode7, cnode1, uisp, TCTC, librenms, workbench | -| **S4** (458GB) | tsys5 | **5** | wnode5, wnode6, rr-middleware, uisp*... | - -> *S2/S3 carry 7 each because the sectestbed load on S1 is idle. If those 16 -> sectestbed VMs ever become active, rebalance — but for now S2/S3 at 7 -> production VMs each is fine (they're 916 GB disks with light per-VM I/O).* - -**D2: 17→5. No VMs added to S1. S2/S3/S4 absorb the load.** - -HA pairs stay split: -- netinfra: 01 on S2 (tsys5), 02 on D2 (tsys4) ✓ -- ucs: 01 on D2 (tsys4), 02 on S2 (tsys5) ✓ - ---- - -## 11. Action checklist (do in this order) - -### Phase 0 — fix storage config bugs first — DONE ✓ - -- ~~Fix tsys6 S1 typo~~ — **operator fixed 2026-07-29** (verified: 8/8 stores active) -- ~~Add S1 to tsys9 storage.cfg~~ — **operator fixed 2026-07-29** (verified) - -### Phase 1 — cleanup - -- **Delete cnode4 (VMID 601) on tsys6** — stopped, stale, not in cluster -- **Confirm cnode5 (706) gone** — already deleted by operator ✓ - -### Phase 2 — disk migrations off D2 (IOPS rebalancing) - -> Do disk moves on VMs that are staying on their current host first. - -- **wnode-tsys5 (509):** disk **D2 → S4** (host stays tsys5) -- **tsys-awx (600):** disk **D2 → D5** (host stays tsys6) -- **hfnoc-uisp (702):** disk **D2 → S4** (host stays tsys7) -- **rr-middleware (703):** disk **D2 → S4** (host stays tsys7) -- **TCTC (704):** disk **D2 → S3** (host stays tsys7) -- **cnode2 (705):** disk **D2 → S2** (host stays tsys7) - -### Phase 3 — host + disk migrations (RAM/CPU relief, 6 VMs) - -- **tsys-librenms (104):** host **tsys1 → tsys7**, disk **D2 → S3** -- **tsys-proxmox-datacenter (105):** host **tsys1 → tsys7**, disk **D2 → S4** -- **tailscale-router (109):** host **tsys1 → tsys7** (disk stays **D2**) -- **kali-tsys (114):** host **tsys1 → tsys7**, disk **D2 → S2** -- **tsys-secure-workbench (117):** host **tsys1 → tsys7**, disk **D2 → S3** -- **tsys-siem (901):** host **tsys9 → tsys6**, disk **D2 → S2** - -### Phase 4 — grow the compute wnode - -- **wnode-tsys9 (905):** resize **4c/4GB → 6c/8GB** (stop/start required) - -### Phase 5 — join workers to k3s - -- Join wnodes via `k8s/join-servers.sh` -- Label: `workload=compute` (tsys3/9), `workload=rackrental` (tsys6/7), `workload=light` (tsys1/5) - -### Verification after each phase - -- `qm list` per host — confirm VM count + status -- `cat /proc/meminfo` — confirm actual RAM ≤ 80% -- `kubectl top nodes` — confirm cnodes healthy -- `ssh root@pfv-tsys4 'df -h /mnt/tsys4/D2'` — confirm D2 frees up -- `ssh root@pfv-tsys5 'df -h /mnt/pfv-tsys5/S2 /mnt/pfv-tsys5/S3 /mnt/pfv-tsys5/S4'` — confirm fills - ---- - -## 12. Future: local storage as ephemeral scratch - -> Codified as standing policy in [`CAPACITY-GOALS.md`](CAPACITY-GOALS.md) §6. - -The fleet has substantial **unused local storage IOPS** on several hosts. OS -drives stay on NFS (no per-node disk redundancy — see policy), but local storage -is ideal for **ephemeral scratch** (container image cache, build artifacts, -`emptyDir` volumes) where data loss on node failure is acceptable. - -**Per-host opportunity:** - -| Host | Local | Available | Scratch-suitable? | -|------|-------|-----------|-------------------| -| tsys3 | NVMe (PM961) | 349 GB | **Yes — best existing** | -| tsys9 | SSD (PNY CS900) | 136 GB | **Yes** | -| tsys2 | NVMe + SSD | 512 GB + 1 TB | **Yes (incoming)** | -| tsys5 | HDD (Hitachi) | ~1.7 TB | Yes (bulk) | -| tsys6 | USB 2.0 HDD | 1.7 TB | **NO (30 MB/s)** | -| tsys7 | USB 2.0 HDD | 1.7 TB | **NO (30 MB/s)** | - -**The tsys6/7 problem:** they are the RackRental hosts (most in need of fast -image pulls) but their local storage is USB 2.0 — slower than NFS. The real fix -is installing internal drives (tsys6 has a disabled SAS controller; tsys7 has 5 -empty SATA ports). Deferred to a physical-hardware session. - -**Implementation:** post-Friday, once SSD/NVMe tiers are installed and the k8s -StorageClass design is finalized. Wnodes with local storage get a `/scratch` -mount for ephemeral data; persistent data stays on NFS. See -[`CAPACITY-GOALS.md`](CAPACITY-GOALS.md) §6.4 for the StorageClass mapping. +# proxmox/docs/AUDIT-2026-07-29.md + +> **Documentation moved to Discourse — the canonical source of truth.** +> +> **Audit snapshot 2026-07-29** +> +> **Read it here:** https://community.turnsys.com/t/298 +> +> *Migrated 2026-08-06. This file is kept as a pointer for git-browsing context. +> Do not update content here — edit the Discourse wiki topic instead.* diff --git a/proxmox/docs/AUDIT-2026-07-30.md b/proxmox/docs/AUDIT-2026-07-30.md index 187440f..4678c0d 100644 --- a/proxmox/docs/AUDIT-2026-07-30.md +++ b/proxmox/docs/AUDIT-2026-07-30.md @@ -1,478 +1,10 @@ -# Fleet Audit — 2026-07-30 (full ground-truth: network + storage + compute) - -> **The single authoritative audit.** Supersedes [`AUDIT-2026-07-29.md`](AUDIT-2026-07-29.md) -> and [`PROJECT.md`](PROJECT.md) where they conflict. Capacity targets are -> codified in [`CAPACITY-GOALS.md`](CAPACITY-GOALS.md). - -**Audit time:** 2026-07-30, fresh data from all 7 online hosts (~03:30–03:45 UTC) -+ switch SNMP/conman captures (~03:50–04:00 UTC). -**Method:** `perf/scripts/probe-storage.sh`, `perf/scripts/probe-network.sh`, -`perf/scripts/snmp-switch-audit.py`, `perf/scripts/conman-console.py` — all -read-only. No changes made to any system. -**Hosts online:** tsys1, tsys3, tsys4, tsys5, tsys6, tsys7, tsys9 -**Hosts retired:** tsys8 (permanently), tsys2 (Win10, pending rebuild) - ---- - -## 1. Physical network topology - -### 1.1 Diagram - -``` - RACK 5 (core/data) RACK 3 (compute/RackRental) - ┌─────────────────────────┐ ┌──────────────────────────┐ - │ │ │ │ - │ pfv-core-sw01 │ ch1 LAG │ pfv-tor3-stor │ - │ PowerConnect 5448 │◄───3×1G──────────►│ Neyland 24T (PC5324) │ - │ (mgmt+storage VLAN) │ (g13-15) │ (storage TOR) │ - │ │ g16 UP, not │ │ - │ VLAN1 (mgmt) │ in LAG; g17 │ g8-9 → tsys6 bond0 │ - │ VLAN1000 (storage) │ DOWN │ g10-11 → tsys7 bond0 │ - │ │ │ g20-23 → ch1 (3 active) │ - │ g8 → tsys4 nic0 │ │ │ - │ g19 → tsys5 nic0 │ │ pfv-tor3-mgmt │ - │ g31 → tsys4 enx (USB) │ │ Neyland 24T (PC5324) │ - │ g33-34 → tsys5 ch3 │ │ (mgmt TOR) │ - │ ch3 → tsys5 bond (1 ln)│ │ g7 → tsys6 nic0 │ - │ │ │ g3 → tsys7 nic0 │ - │ tsys4 (storage+PBS) │ │ │ - │ tsys5 (storage+sectest)│ │ tsys6 (RackRental) │ - │ tsys1 (infra) │ │ tsys7 (RackRental) │ - │ tsys3 (compute) │ │ │ - │ tsys9 (infra+compute) │ │ │ - └─────────────────────────┘ └──────────────────────────┘ -``` - -### 1.2 Cross-rack trunk detail (ch1) — THE BOTTLENECK - -| Property | core-sw01 (rack5) | tor3-stor (rack3) | Problem? | -|----------|-------------------|--------------------|----------| -| **Physical ports** | g13, g14, g15 (UP) | g20, g21, g22, g23 (all UP) | Asymmetry — see below | -| **LAG config** | `g(13-15,17)` mode on | `g(20-23)` mode on | g17 DOWN; g16 UP but not in LAG | -| **Active links** | **3 of 4** | **4 of 4** | Mismatch: only 3 links negotiate | -| **Hash policy** | **layer-2-3** | **layer-2-3-4** | **ASYMMETRIC** — core lacks L4 | -| **Mode** | `on` (static) | `on` (static) | No LACP negotiation/failure detection | -| **VLAN** | access vlan 1000 | access vlan 1000 | OK | - -**Three root causes of cross-rack underperformance:** - -1. **g16/g17 cable mismatch.** core-sw01 config says `channel-group 1` on - `g(13-15,17)`. But g17 is DOWN and g16 is UP (live, 1000/Full). The 4th - physical cable landed on g16, not g17. Result: **only 3 links active** (3 Gbps, - not the intended 4). - -2. **LAG hash asymmetry.** tor3-stor uses `port-channel load-balance layer-2-3-4` - (includes TCP/UDP source+dest ports — correct for NFS with nconnect). - core-sw01 uses `port-channel load-balance layer-2-3` (no L4 ports). With - layer-2-3 hashing, **all NFS traffic between a given host pair hashes to one - physical link**. Even though the clients use `nconnect=4` (4 TCP connections - per mount), they share the same src/dst IP → same hash → **max 1 Gbps per - server pair**. This is why tsys6/7 read from tsys4/5 at ~1 Gbps, not 3-4 Gbps. - -3. **Static mode (`on`) instead of LACP.** Neither switch runs LACP on the - cross-rack trunk. `mode on` provides no link-failure detection, no partner - validation, no dynamic negotiation. A dead cable stays "active" until - manually discovered. - -### 1.3 Per-host network state (ground truth) - -| Host | Rack | mgmt NIC | storage NIC | bond0 | Hash | Active slaves | Speed | Storage IP | -|------|------|----------|-------------|-------|------|---------------|-------|------------| -| tsys1 | 5 | nic0 → core g8 | enx (USB dongle) | bond0 (1×USB) | layer2+3 | 1 | 1G | 10.100.100.1 | -| tsys3 | 5 | nic0 → core | — (no bond) | — | — | — | — | 10.100.100.3 | -| tsys4 | 5 | nic0 → core g8 | enx8cae4ccda926 (USB) → core g31 | bond0 (1×USB) | layer2+3 | 1 | 1G | 10.100.100.4 | -| tsys5 | 5 | nic0 → core g19 | nic1 → core g34 (ch3) | bond0 | layer2+3 | **1 of 2** (nic2 DOWN) | 1G | 10.100.100.5 | -| tsys6 | 3 | nic0 → tor3-mgmt g7 | nic1+nic2 → tor3-stor g8/g9 (ch2) | bond0 | **layer3+4** | **2 of 2** | **2G** | 10.100.100.6 | -| tsys7 | 3 | nic0 → tor3-mgmt g3 | nic1+nic2 → tor3-stor g10/g11 (ch3) | bond0 | **layer3+4** | **2 of 2** | **2G** | 10.100.100.7 | -| tsys9 | 5 | nic0 → core | enx (USB dongle) | bond0 (1×USB) | layer2+3 | 1 | 1G | 10.100.100.9 | - -**Key observations:** -- **tsys6/7 are the ONLY hosts with working 2G LACP** (layer3+4, 2 active slaves). Validated previously. -- **tsys4/9 use USB dongles** for storage — single 1G, no bond redundancy. -- **tsys5 bond is degraded** — nic2 is DOWN (no cable). Only nic1 active. This is - the LACP problem the operator has been struggling with. The switch side (core-sw01 - ch3) shows 1 active member (g34), and tsys5's bond0 reports 1 active slave. -- **tsys1 also uses a USB dongle** for storage. -- **Hash inconsistency:** tsys6/7 use layer3+4 (correct for nconnect), tsys4/5 - use layer2+3 (suboptimal for NFS multi-connection). - -### 1.4 Switch inventory (SNMP + conman ground truth) - -| Switch | IP | Model | Firmware | Role | Console | -|--------|----|-------|----------|------|---------| -| pfv-core-sw01 | 192.168.0.12 | Dell PowerConnect 5448 | 2.0.0.46 | Core (rack5, combined mgmt+storage) | conman port 2001 | -| pfv-tor3-stor | 192.168.0.9 | Dell PowerConnect 5324 (Neyland 24T) | 2.0.1.4 | Rack3 storage TOR | conman port 2003 | -| pfv-tor3-mgmt | 192.168.0.7 | Dell PowerConnect 5324 (Neyland 24T) | (same family) | Rack3 mgmt TOR | conman port 2002 | -| (unknown) | 192.168.0.8 | Dell PowerConnect 5324 (Neyland 24T) | (same family) | Unidentified — possibly pfv-r2-tor-top | conman port 2005? | - -**Switch SNMP community:** `kn3lmgmt` (read-only). All switches respond to SNMPv2c. -**Console access:** conman on pfv-tsys4:7890 (7 consoles). Fixed during this audit -(udev symlink naming bug — singular vs plural — prevented consoles from working -after reboot). - -### 1.5 Port error counters (core-sw01) - -| Port | In Errors | Notes | -|------|-----------|-------| -| **g9** | **2902** | Connected to 192.168.0.21 (device unknown). Investigate CRC/cable. | -| **g38** | **590** | On VLAN 1000 (storage). Connected to a host on the storage VLAN. | -| **g26** | **73** | Connected to a management-network device. | -| All others | 0 | Healthy. | - -tor3-stor shows **zero errors** on all active ports. - ---- - -## 2. Storage topology (CORRECTED — current ground truth) - -### 2.1 Storage tier definitions - -| Tier | Definition | Performance | Use case | -|------|-----------|-------------|----------| -| **Tier 1** | NVMe / SSD on internal SATA/SAS | 200-3000 MB/s, high IOPS | OS drives, high-IOPS workloads | -| **Tier 2** | HDD on internal SATA/SAS | 80-150 MB/s, moderate IOPS | Production VM OS drives | -| **Tier 3** | HDD/SSD on USB "toaster" / USB adapter | 30-60 MB/s, low IOPS, unreliable | Bulk/backup/scratch only | - -### 2.2 tsys4 disk inventory (CHANGED since last audit) - -| Device | Model | Size | Interface | Mount | NFS export | Tier | Notes | -|--------|-------|------|-----------|-------|------------|------|-------| -| sda | Hitachi HDS723020BLA642 | 2.0 TB | Internal SATA | /mnt/albert | (not exported) | **Tier 2** | NEW — not in Proxmox storage.cfg, not exported. 0% used. | -| sdb | HITACHI HUA723020ALA640 | 2.0 TB | Internal SATA | /mnt/tsys4/D5 | D5 | Tier 2 | 13% used (220 GB). 7200 rpm. | -| sdc | WDC WD30EFRX (Red) | 3.0 TB | Internal SATA | /mnt/tsys4/D2 | D2 | Tier 2 | 7% used (170 GB). 5400 rpm. **17 VMs on this spindle.** | -| sdd | WDC WD10EZRX | 1.0 TB | Internal SATA | (pve boot/swap/local-lvm) | — | Tier 2 | Proxmox OS disk. | -| sde | ST500LM021 (Seagate) | 500 GB | **USB** | /mnt/tsys4/D3 | D3 | **Tier 3** | **CHANGED from SK hynix SSD to Seagate HDD.** 0% used. | -| sdf | WDC WD10EARS | 1.0 TB | **USB** | /mnt/tsys4/D1 | D1 | **Tier 3** | 0% used. 1 reallocated sector (SMART). | -| sdg | WDC WD50NDZW (easystore) | 5.0 TB | **USB** | /mnt/backup | (not exported) | **Tier 3** | NEW — bulk backup disk. 0% used. | - -**Changes from AUDIT-2026-07-29:** -- **D3 changed:** SK hynix SC300 512GB SSD → Seagate ST500LM021 500GB HDD (still on USB). - D3 is now Tier 3 (was previously listed as unreliable USB SSD). -- **/mnt/albert added:** Hitachi 2TB internal SATA. Not exported, not in Proxmox. - Potential new Tier 2 storage. -- **/mnt/backup added:** WD 5TB USB easystore. Tier 3 bulk backup. -- **D7 removed:** Previously orphaned export (held DellOpenManageEnterprise). Now gone. - -### 2.3 tsys5 disk inventory (MAJOR CHANGES — new NVMe + SSD) - -| Device | Model | Size | Interface | Mount | NFS/Local | Tier | Notes | -|--------|-------|------|-----------|-------|-----------|------|-------| -| sda | ST1000VN002 (IronWolf) | 1.0 TB | Internal SATA | (not mounted) | — | Tier 2 | **Was S3.** Not in fstab, not mounted. Stale export remains in /etc/exports. | -| sdb | SK hynix SC300B | 512 GB | Internal SATA | (not mounted) | — | **Tier 1** | **NEW — relocated from tsys4 USB enclosure.** Not mounted, not in fstab, not in storage.cfg. | -| sdc | HITACHI HUS724020ALA640 | 2.0 TB | Internal SAS | (pve boot/swap/local-lvm) | — | Tier 2 | Proxmox OS disk. | -| sdd | Samsung SSD 860 PRO | 256 GB | Internal SAS | /mnt/pfv-tsys5/ssd | `ssd` dir (local) | **Tier 1** | **93% full!** Used by ultix-streaming (5111). | -| sde | ST1000VN002 (IronWolf) | 1.0 TB | Internal SAS | /mnt/pfv-tsys5/S1 | S1 (NFS + local-nonprod) | Tier 2 | 11% used. 5900 rpm. | -| sdf | ST1000VN002 (IronWolf) | 1.0 TB | Internal SAS | /mnt/pfv-tsys5/S2 | S2 (NFS) | Tier 2 | 4% used. | -| **nvme0n1** | **CT500P1SSD8 (Crucial)** | **500 GB** | **NVMe** | **(not mounted)** | — | **Tier 1** | **NEW — not mounted, not in fstab, not in storage.cfg.** 34K power-on hours. Healthy. | - -**Changes from AUDIT-2026-07-29:** -- **NVMe added:** Crucial CT500P1 500GB. Not yet configured. This is the highest- - performance local storage in the fleet (true NVMe). -- **SK hynix SSD relocated:** Moved from tsys4 USB enclosure to tsys5 internal - SATA. Not yet mounted/configured. -- **S3 unmounted:** ST1000VN002 (was S3) is present but fstab entry is commented - out. /etc/exports still exports S3 (stale). -- **S4 gone:** Old Toshiba 458GB is no longer present. fstab entry commented. - storage.cfg still has a broken `dir: S4` entry (points to root filesystem). -- **S1/S2 drives changed:** Both are now ST1000VN002 IronWolf 1TB (were "Seagate - 916GB" in old audit). -- **SSD nearly full:** Samsung 860 PRO at 93% — needs attention. - -### 2.4 NFS export map (current, verified) - -| Server | Export | NFS ID | Disk | Tier | Used | -|--------|--------|--------|------|------|------| -| tsys4 | /mnt/tsys4/D1 | D1 | WDC WD10EARS 1TB | Tier 3 (USB) | 0% | -| tsys4 | /mnt/tsys4/D2 | D2 | WDC Red 3TB | Tier 2 | 7% (170 GB) | -| tsys4 | /mnt/tsys4/D3 | D3 | Seagate 500GB | Tier 3 (USB) | 0% | -| tsys4 | /mnt/tsys4/D5 | D5 | Hitachi 2TB | Tier 2 | 13% (220 GB) | -| tsys5 | /mnt/pfv-tsys5/S1 | S1 | IronWolf 1TB | Tier 2 | 11% (96 GB) | -| tsys5 | /mnt/pfv-tsys5/S2 | S2 | IronWolf 1TB | Tier 2 | 4% (32 GB) | -| tsys5 | /mnt/pfv-tsys5/S3 | S3 | (stale — disk not mounted) | — | — | - -### 2.5 NFS client mount status (nconnect) - -| Host | Mounts from tsys4 | Mounts from tsys5 | nconnect=4? | Hash | -|------|-------------------|-------------------|-------------|------| -| tsys4 | (NFS server, no client mounts) | — | — | layer2+3 | -| tsys5 | D2, D3, D5 | (NFS server) | **NO nconnect** | layer2+3 | -| tsys6 | D1, D2, D3, D5 | S1, S2, S3, S4, T5-SSD | **YES** (9 mounts) | layer3+4 | -| tsys7 | D1, D2, D3, D5 | S1, S2, S3, S4, T5-SSD | **YES** (9 mounts) | layer3+4 | - -**tsys5 is missing nconnect** on its 3 NFS client mounts (D2/D3/D5 from tsys4). -**tsys4 is missing nconnect** entirely (but it's primarily an NFS server, not a client). - -### 2.6 Per-spindle IOPS load (running VMs only) - -| Spindle | Server | NFS-visible VMs | Local-only VMs | Total | Tier | Notes | -|---------|--------|-----------------|----------------|-------|------|-------| -| **D2** (3TB Red) | tsys4 | **13** | 0 | **13** | Tier 2 | Still the worst hotspot. VMs: 100,101,104,105,108,114,117,509,600,702,704,901,904 | -| D5 (2TB Hitachi) | tsys4 | 4 | 0 | 4 | Tier 2 | VMs: 106,313,600(disk0),903 | -| S1 (1TB IronWolf) | tsys5 | 1 | 0 | 1 | Tier 2 | VM: 702 (hfnoc-uisp, 100GB) | -| S2 (1TB IronWolf) | tsys5 | 4 | 0 | 4 | Tier 2 | VMs: 103,703,705,902,905 | -| S4 (local-nonprod) | tsys5 | 0 | **12** (stopped sandboxes) | 12 idle | Tier 2 | All stopped. Low priority. | -| D3 (500GB Seagate USB) | tsys4 | 0 | 0 | 0 | **Tier 3** | Empty | -| D1 (1TB WD USB) | tsys4 | 0 | 0 | 0 | **Tier 3** | Empty | -| ssd (Samsung 860 PRO) | tsys5 | 0 | 1 | 1 | Tier 1 | VM 5111 (ultix-streaming, 288GB) | - -**D2 still carries 13 running VMs** — the IOPS hotspot persists. - ---- - -## 3. Host capacity (live ground truth, 2026-07-30) - -| Host | CPU | Threads | RAM | Running VMs | Actual RAM% | Load | CPU busy | -|------|-----|---------|-----|-------------|-------------|------|----------| -| tsys1 | i7-4770 Haswell (2013) | 8 | 31 GB | 9 | **57%** | 1.1 | ~4% | -| tsys3 | Xeon E3-1535M v5 Skylake (2015) | 8 | 31 GB | 1 | 10% | 0.3 | 0% | -| tsys4 | Xeon E3-1246 v3 Haswell (2013) | 8 | 16 GB | 1 | 15% | 1.8 | 0% | -| tsys5 | Xeon E5620 Westmere (2010) **single CPU** | 8 | 94 GB | 4 (+22 stopped) | 15% | 3.3 | ~5% | -| tsys6 | 2× Xeon E5530 Nehalem (2009) | 16 | 126 GB | 2 | 12% | 0.7 | 0% | -| tsys7 | 2× Xeon E5-2630 v2 Ivy Bridge (2013) | 24 | 189 GB | 5 | 7% | 1.0 | ~1% | -| tsys9 | i5-10500 Comet Lake (2020) | 12 | 23 GB | 6 | **69%** | 0.3 | ~1% | - -**Changes since AUDIT-2026-07-29:** -- **tsys1 RAM dropped 73%→57%** — some VMs apparently migrated/stopped. -- **tsys9 RAM 62%→69%** — approaching 80% goal. -- **tsys6 wnode-tsys6 grew to 64GB RAM** (was 32GB in old audit). -- **All other hosts healthy** — well under 80%. - ---- - -## 4. Fleet consistency audit - -### 4.1 PVE/kernel versions - -| Host | PVE | Kernel | Status | -|------|-----|--------|--------| -| tsys1 | 9.2.5 | 7.0.14-6-pve | Current ✓ | -| tsys3 | 9.2.5 | 7.0.14-6-pve | Current ✓ | -| **tsys4** | **9.1.5** | **6.17.9-1-pve** | **BEHIND — needs upgrade** | -| tsys5 | 9.2.5 | 7.0.14-6-pve | Current ✓ | -| tsys6 | 9.2.5 | 7.0.14-6-pve | Current ✓ | -| tsys7 | 9.2.5 | 7.0.14-6-pve | Current ✓ | -| tsys9 | 9.2.5 | 7.0.14-6-pve | Current ✓ | - -### 4.2 Performance tuning state - -| Host | Governor | Swappiness | TCP CC | rmem/wmem_max | Tuned profile | Bond hash | nconnect | -|------|----------|------------|--------|---------------|---------------|-----------|----------| -| tsys1 | performance | 10 | bbr | 128MB | virtual-host | layer2+3 | N/A (client) | -| tsys3 | performance | 10 | bbr | 128MB | virtual-host | — | N/A | -| **tsys4** | **performance** | **1** | **bbr** | **16MB** | **throughput-performance** | **layer2+3** | **0** | -| **tsys5** | **N/A** (no driver) | **1** | **bbr** | **128MB** | **network-throughput** | **layer2+3** | **0** | -| tsys6 | performance | 10 | bbr | 128MB | virtual-host | **layer3+4** | **24** (9 mounts × 4... actually 6 options lines) | -| tsys7 | performance | 10 | bbr | 128MB | virtual-host | **layer3+4** | **24** | -| tsys9 | performance | 10 | bbr | 128MB | virtual-host | layer2+3 | N/A | - -**Inconsistencies:** -- **tsys4:** rmem/wmem = 16 MB (fleet standard: 128 MB). Tuned profile wrong - (`throughput-performance` should be `network-throughput`). Missing nconnect. -- **tsys5:** Missing nconnect on NFS client mounts. No cpufreq driver (Westmere — - expected, not a bug). -- **tsys4/5 bond hash = layer2+3** — should be layer3+4 (like tsys6/7). - ---- - -## 5. VM inventory + config audit (running VMs only) - -### 5.1 Per-VM disk + CPU + network settings - -> `cpu: host` = best (full instruction set passthrough). `x86-64-v2-AES` = good -> (nested-virt-safe baseline). `cache=writethrough` = safe but slow for NFS. -> `iothread=1` = present on most. `agent: 1` = qemu-guest-agent enabled. - -| VMID | Name | Host | vCPU | CPU type | RAM | Disk store | Cache | iothread | qga | Notes | -|------|------|------|------|----------|-----|-----------|-------|----------|-----|-------| -| 100 | pfv-bms | tsys1 | 2 | host | 4G | D2 (raw) | — | — | ✓ | discard=on,ssd=1 | -| 101 | tsys-ca | tsys1 | 2 | v2-AES | 2G | D2 (virtio) | — | ✓ | — | no nested-virt | -| 103 | netinfra-01 | tsys1 | 2 | v2-AES | 4G | S2 | — | ✓ | — | | -| 104 | librenms | tsys1 | 2 | host | 2G | D2 | — | ✓ | ✓ | | -| 105 | proxmox-dc | tsys1 | 2 | host | 2G | D2 | — | ✓ | ✓ | | -| 106 | cnode3 | tsys1 | 4 | host | 4G | D5 | **writethrough** | ✓ | ✓ | | -| 108 | ucs-01 | tsys1 | 2 | host | 8G | D2 | **writethrough** | ✓ | ✓ | | -| 114 | kali-tsys | tsys1 | 2 | host | 2G | D2 | **writethrough** | ✓ | ✓ | | -| 117 | workbench | tsys1 | 2 | host | 4G | D2 | **writethrough** | ✓ | ✓ | | -| 313 | wnode-tsys3 | tsys3 | 8 | host | 28G | D5 | **writethrough** | ✓ | ✓ | | -| 400 | PBS | tsys4 | 2 | — | 2G | local-lvm | — | — | — | backup server | -| 509 | wnode-tsys5 | tsys5 | 2 | host | 32G | D2 (virtio) | — | ✓ | — | low vCPU (2) for k8s wnode | -| 5111 | ultix-streaming | tsys5 | 4 | — | 9G | ssd (local) | — | — | — | 288GB on SSD (93% full) | -| 5112 | ultix-offstage | tsys5 | 4 | — | 6G | local-lvm | — | — | — | 288GB | -| 100 | wnode-tsys6 | tsys6 | 2 | host | 64G | S4 | — | ✓ | — | low vCPU (2), high RAM | -| 600 | tsys-awx | tsys6 | 2 | host | 12G | D5 | **writethrough** | ✓ | ✓ | | -| 701 | wnode-tsys7 | tsys7 | 4 | host | 32G | S3 (virtio) | — | ✓ | — | | -| 702 | hfnoc-uisp | tsys7 | 2 | host | 8G | S1 | — | ✓ | ✓ | 100GB disk | -| 703 | rr-middleware | tsys7 | 2 | host | 2G | S2 | **writethrough** | ✓ | ✓ | | -| 704 | TCTC | tsys7 | 4 | host | 6G | D2 (virtio) | — | ✓ | — | | -| 705 | cnode2 | tsys7 | 4 | host | 4G | S2 | **writethrough** | ✓ | ✓ | | -| 901 | tsys-siem | tsys9 | 2 | host | 8G | D2 | **writethrough** | ✓ | ✓ | 132GB disk | -| 902 | ucs-02 | tsys9 | 2 | host | 8G | S2 | — | ✓ | ✓ | 50GB disk | -| 903 | kali-rd | tsys9 | 2 | host | 2G | D5 | **writethrough** | ✓ | ✓ | | -| 904 | netinfra-02 | tsys9 | 2 | v2-AES | 4G | D2 | — | ✓ | — | | -| 905 | wnode-tsys9 | tsys9 | 4 | v2-AES | 4G | S2 | — | ✓ | ✓ | target: grow to 6c/8G | -| 906 | cnode1 | tsys9 | 2 | host | 4G | S3 | — | ✓ | — | | - -### 5.2 VM optimization findings - -| Finding | VMs affected | Impact | Fix | -|---------|-------------|--------|-----| -| **cache=writethrough on NFS** | 106,108,114,117,313,600,703,705,901,903 | Writethrough forces synchronous writes — slow on NFS. `writeback` or `none` is better for NFS-backed qcow2. | Change cache mode (requires VM stop/start) | -| **Mixed CPU types** | Some `host`, some `x86-64-v2-AES` | Prevents migration between hosts with different CPU types. K8s cnodes/wnodes should match. | Standardize per workload tier | -| **Low vCPU on k8s wnodes** | wnode-tsys5 (2c), wnode-tsys6 (2c) | K8s workers undersized for compute workloads | Grow after migrations free RAM | -| **VM 500 references D7** | DellOpenManageEnterprise (stopped) | D7 export no longer exists on tsys4 | Delete or re-disk before starting | -| **qga inconsistent** | Several VMs missing agent:1 | No guest-agent communication | Enable qga fleet-wide | - ---- - -## 6. Console infrastructure status (fixed during this audit) - -**Bug found + fixed:** `console/generate-config.sh` wrote udev rules with -`SYMLINK+="console/$name"` (singular) but ser2net.yaml opened -`/dev/consoles/$name` (plural). They never matched. After every reboot, all 7 -console ports failed until `setup.sh`'s manual fallback re-created symlinks. - -**Fix applied:** Changed udev rule to `SYMLINK+="consoles/$name"`. Re-created -all 7 symlinks manually on pfv-tsys4. Restarted ser2net + conmand. All 7 -consoles now working (verified via conman-console.py). - -**Also fixed:** stale conmand process (pid 988) holding config lock, preventing -conmand systemd service from starting. - ---- - -## 7. Findings summary (prioritized) - -### Critical (cross-rack performance) - -| # | Finding | Impact | Type | -|---|---------|--------|------| -| 1 | **core-sw01 LAG hash = layer-2-3** (no L4 ports) | All NFS traffic between a given host pair caps at 1 Gbps | Switch config | -| 2 | **core-sw01 ch1 cable on g16, not g17** | Only 3 of 4 cross-rack links active (3 Gbps, not 4) | Switch config | -| 3 | **Both switches use `mode on`** (static, no LACP) | No failure detection, no partner validation | Switch config | -| 4 | **tsys5 bond0 degraded** (1 of 2 slaves) | Single link to storage, no redundancy | Physical (cable) | - -### High (storage configuration) - -| # | Finding | Impact | Type | -|---|---------|--------|------| -| 5 | **tsys5 NVMe not configured** (CT500P1 500GB) | Best local storage in fleet sitting unused | Config | -| 6 | **tsys5 SK hynix SSD not configured** (512GB) | Relocated from tsys4, not mounted | Config | -| 7 | **D2 still carries 13 running VMs** | IOPS hotspot persists | VM placement | -| 8 | **tsys5 SSD 93% full** | Samsung 860 PRO nearly exhausted | Capacity | -| 9 | **S3/S4 stale on tsys5** | Broken exports, stale storage.cfg | Config | - -### Medium (fleet consistency) - -| # | Finding | Impact | Type | -|---|---------|--------|------| -| 10 | **tsys4 on PVE 9.1.5/kernel 6.17** | Behind fleet (9.2.5/7.0.14) | Upgrade | -| 11 | **tsys4 rmem/wmem = 16MB** | Suboptimal TCP buffers for NFS | Tuning | -| 12 | **tsys4/5 bond hash = layer2+3** | Suboptimal for nconnect NFS | Tuning | -| 13 | **tsys5 missing nconnect** | NFS client mounts to tsys4 lack multi-connection | Tuning | -| 14 | **cache=writethrough on 10+ VMs** | Synchronous writes slow NFS | VM config | -| 15 | **Stale tsys8 switch config** (tor3-stor ch4) | Dead LAG group for retired host | Cleanup | - -### Low (monitoring / investigation) - -| # | Finding | Impact | Type | -|---|---------|--------|------| -| 16 | **core-sw01 port errors** (g9:2902, g38:590, g26:73) | Possible cable/CRC issues | Investigate | -| 17 | **No continuous observability** | Only point-in-time check.sh snapshots | Monitoring | -| 18 | **/mnt/albert unused** (tsys4 Hitachi 2TB) | Potential new Tier 2 storage | Config | -| 19 | **VM 500 references non-existent D7** | Would fail on boot | Cleanup | - ---- - -## 9. Observability stack (operator direction) - -The operator has defined a clear monitoring architecture with three tools, -each serving a distinct role. Do NOT duplicate functionality across tools. - -### 9.1 Current architecture - -| Tool | Where it runs | Scope | Role | -|------|---------------|-------|------| -| **Beszel** | Cloudron VPS (Reston, VA) | Tailscale-connected hosts | CPU/RAM/disk/time-series metrics on all Proxmox hosts | -| **Uptime Kuma** | Cloudron VPS (Reston, VA) | All hosts + services | Up/down status + alerting via Pushover | -| **LibreNMS** (VMID 104, tsys1) | On-prem (R&D cluster) | Non-Tailscale devices only | SNMP polling for switches + devices that can't run Beszel agent | -| **Home Assistant** (VMID 100, tsys1) | On-prem (R&D cluster) | Physical infrastructure | Power/temperature instrumentation + alerting (UPS, PDU outlets, etc.) | - -### 9.2 Division of responsibility (no duplication) - -| Metric class | Tool | Notes | -|-------------|------|-------| -| CPU / RAM / disk / network time-series | **Beszel** | All Proxmox hosts (Tailscale-connected) | -| Up/down alerting (hosts + services) | **Uptime Kuma** | Pushover notifications | -| SNMP polling (switches, non-Tailscale gear) | **LibreNMS** | Port counters, LAG status, interface errors, switch health. Fill the gap Beszel can't cover (switches have no Beszel agent) | -| Power / temperature / physical env | **Home Assistant** | UPS (NUT), PDU outlets, temperature sensors. Alerting via HA automations | - -### 9.3 Saturday "OAM Day" plan (2026-08-01) - -The operator has designated Saturday as a dedicated operations/admin/management -day with the goal of achieving **100% instrumentation coverage**: - -- [ ] Up/down alerting on ALL hosts + key services (Uptime Kuma → Pushover) -- [ ] Centralized syslog (tool TBD — candidates: dedicated syslog VM, rsyslog - relay, or Loki/Grafana stack) -- [ ] SNMP instrumentation: wire LibreNMS for all 4 switches + any non-Tailscale - devices (community: `kn3lmgmt`, IPs: 192.168.0.7/8/9/12) -- [ ] Beszel agent coverage on all 7 Proxmox hosts -- [ ] Home Assistant: wire power/temp alerting (UPS via NUT already live, PDU - via powerman, any temp sensors) -- [ ] Verify alerting paths end-to-end (test each alert fires correctly) - -> Much of the groundwork is already done: NUT is live on tsys1, powerman is -> live on tsys1, LibreNMS (VMID 104) is running, conman console logging is live -> on tsys4, check.sh provides point-in-time snapshots. Saturday is about wiring -> continuous monitoring + alerting on top of these. - ---- - -## 8. Recommendations (for operator review — NO changes applied) - -### 8.1 Switch changes (require maintenance window + operator approval) - -1. **Fix core-sw01 ch1 member list:** `g(13-15,17)` → `g(13-16)`. This makes - the 4th live cable (g16) part of the LAG. -2. **Fix core-sw01 LAG hash:** `layer-2-3` → `layer-2-3-4` (match tor3-stor). - This unlocks multi-link distribution for nconnect NFS traffic. -3. **Convert trunk to LACP:** Change `mode on` → `mode active` on both switches - for the cross-rack trunk (ch1). Enables failure detection + auto-negotiation. -4. **Remove stale tsys8 config:** Delete ch4 + g13/g14 channel-group on tor3-stor. -5. **Investigate port errors:** g9 (2902 inErr), g38 (590 inErr), g26 (73 inErr) - on core-sw01. - -> **Recommended order:** Fix #2 (hash) + #1 (member) first — these are the -> highest-impact changes with the lowest risk. LACP conversion (#3) is optional -> if the static trunk is working reliably otherwise. All changes can be done -> read-only-verified via the conman + SNMP tooling in this repo. - -### 8.2 Storage configuration (host-side, no switch changes needed) - -1. **Mount + configure tsys5 NVMe** (CT500P1 500GB) as LVM-thin (`nvme-local`) - per [`TODO.md`](TODO.md) §3. Use for wnode-tsys5 boot disk (highest impact) - and sectestbed scratch. -2. **Mount + configure tsys5 SK hynix SSD** (512GB). Add as `dir: ssd2` or - similar. Offloads the 93%-full Samsung 860 PRO. -3. **Clean S3/S4:** Remove stale S3 export from /etc/exports. Fix/remove broken - `dir: S4` entry in storage.cfg. -4. **Decide on /mnt/albert** (tsys4 Hitachi 2TB): export as new NFS storage or - keep local-only for PBS scratch. - -### 8.3 Fleet consistency + tuning (host-side) - -1. **Upgrade tsys4** to PVE 9.2.5 / kernel 7.0.14 (matches fleet). -2. **Apply apply-tunings.sh --apply** to tsys4 and tsys5: - - Fix rmem/wmem to 128MB (tsys4) - - Fix tuned profile (tsys4 → network-throughput) - - Add nconnect=4 to tsys5 NFS client mounts - - Fix bond hash to layer3+4 (tsys4/5 — requires ifreload, brief network drop) -3. **Audit VM cache modes:** Change writethrough → writeback or none for NFS- - backed qcow2 disks (requires VM stop/start). - -### 8.4 VM placement rebalancing (refer to [`CAPACITY-GOALS.md`](CAPACITY-GOALS.md)) - -The new NVMe + SSD on tsys5, the unmounted /mnt/albert on tsys4, and the 3 empty -Tier 3 USB disks (D1, D3, /mnt/backup) create opportunities for IOPS spreading -that didn't exist in the 07-29 audit. Detailed placement plan deferred to the -operator review of this document. +# proxmox/docs/AUDIT-2026-07-30.md + +> **Documentation moved to Discourse — the canonical source of truth.** +> +> **Audit snapshot 2026-07-30** +> +> **Read it here:** https://community.turnsys.com/t/298 +> +> *Migrated 2026-08-06. This file is kept as a pointer for git-browsing context. +> Do not update content here — edit the Discourse wiki topic instead.* diff --git a/proxmox/docs/AUDIT-2026-08-05.md b/proxmox/docs/AUDIT-2026-08-05.md index 9badc9e..8a0686f 100644 --- a/proxmox/docs/AUDIT-2026-08-05.md +++ b/proxmox/docs/AUDIT-2026-08-05.md @@ -1,220 +1,10 @@ -# Ground-Truth Audit — 2026-08-05 +# proxmox/docs/AUDIT-2026-08-05.md -> **Authoritative fleet audit** performed by Crush (GLM-5.2) via read-only -> network discovery (nmap, SNMP, SSH, API queries). Supersedes -> [`AUDIT-2026-07-30.md`](AUDIT-2026-07-30.md) for all fleet state. +> **Documentation moved to Discourse — the canonical source of truth.** > -> **Method:** Tailscale `status`, nmap ping sweep (192.168.0-3.0/24), SNMP -> walks (`kn3lmgmt` community), SSH via `tests/remote.sh` + `netinfra/dns-cluster-setup/remote-dns.sh`, -> Proxmox `qm list`/`pvecm`, Docker/container inspection. - -## TL;DR — What Changed Since 2026-07-30 - -The fleet grew significantly and the repository was restructured into a -module/subproject layout. Headline deltas: - -| Metric | 2026-07-30 | 2026-08-05 | Δ | -|--------|-----------|-----------|---| -| VMs (running) | 28 | **63** | +125% | -| DNS zones | 124 | **167** | +35% | -| DHCP active leases | 333 | **98** (unique active) | 431 total active binding-state entries | -| Proxmox hosts | 7 | 7 | — | -| PVE version (fleet) | mixed (9.1.5–9.2.5) | **all 9.2.5** | tsys4 upgraded | -| Tailscale nodes | ~30 | **80** | full 3-tier env mirror | -| k8s cluster | LIVE (3-node HA) | **DOWN** | cnode wiped, no quorum | -| netinfra-02 | redundant secondary | **WIPED** | needs rebuild | - -## Repository Restructure (DONE by operator) - -Top-level dirs were consolidated into modules. **Old flat dirs no longer -exist** at the repo root: - -| Was (repo root) | Now | -|-----------------|-----| -| `provisioning/` | `archive/provisioning/` — superseded by **KNELIAC** project (`/home/reachableceo/projects/KNELIAC`) | -| `powerman/` `console/` `ups/` | `dcinfra/{powerman,console,ups}/` | -| `dns-cluster-setup/` `switches/` `dhcp/` | `netinfra/{dns-cluster-setup,switches,dhcp}/` | -| `perf/` | `proxmox/perf/` (+ `proxmox/perf/scripts/`) | - -Current repo root: `archive/ awx/ dcinfra/ docs/ k8s/ netinfra/ proxmox/ tests/ vendor/` + `logs/ returned-logs/`. - -> **WARNING:** `README.md`, `AGENTS.md`, and `docs/docmap.md` still reference -> the OLD paths. These are being updated in this same session. - -## Network Switches - -SNMP community: `kn3lmgmt` (read) / `kn3lwrite` (write). SNMP responds from -the workstation LAN (no ACL — earlier "filtered" was just a wrong community). - -| IP | sysName | Model | Status | -|----|---------|-------|--------| -| 192.168.0.12 | pfv-core-sw01 | PowerConnect 5448 | **UP** (Dell, core, rack5) | -| 192.168.0.7 | pfv-r3-tor-mgmt | Neyland 24T (PC5324) | **UP** (rack3 mgmt TOR) | -| 192.168.0.9 | pfv-r3-tor-stor | Neyland 24T (PC5324) | **UP** (rack3 storage TOR) | -| 192.168.0.8 | subodev-torsw01 | Neyland 24T (PC5324) | **UP** (was "unidentified" — now named) | -| 192.168.0.5 | pfv-r2-tor-top | (unknown) | **OFFLINE** — no ping, no SNMP | - -**Ports open on switches:** TCP 22 (ssh), 23 (telnet), 80 (http web UI). UDP -161 (SNMP) responds with `kn3lmgmt`. - -### LACP / Cross-Rack Trunk (core-sw01 ch1) - -- Operator reports **LACP is fixed** (was static `mode on`, hash layer-2-3). -- SNMP confirms **ch1 ifIndex=49 is UP**; ch3 ifIndex=51 also UP. -- ch1 reports ifSpeed=1000Mbps — the Dell FastPath switch does not report - aggregate bundle speed via standard ifSpeed, so **member count + hash - algorithm need console verification** (conman requires a PTY — run manually): - ```bash - conman -d pfv-tsys4:7890 -f pfv-core-sw01 - # then: show interfaces status port-channel 1 - # show running-config | begin port channel - ``` - -## Proxmox Hosts (7 standalone, managed via PDM) - -All **PVE 9.2.5** (tsys4 upgraded from 9.1.5 — resolves old STATUS item). -**Not clustered** — standalone installs (corosync Expected votes=1 each). - -| Host | RAM used/total | VMs | Uptime | Notes | -|------|---------------|-----|--------|-------| -| pfv-tsys1 | 71% (22.7/32 GB) | 8 | 6d | Highest RAM use in fleet | -| pfv-tsys3 | 12% (3.7/32 GB) | 2 | 9d | under-utilized | -| pfv-tsys4 | 18% (2.8/16 GB) | 2 | 5d | storage server, kernel 7.0.14-8 | -| pfv-tsys5 | 48% (46.8/97 GB) | **32** | 6d | sectestbed+preprod host | -| pfv-tsys6 | 14% (18/129 GB) | 5 | 10d | 129 GB RAM, wnode+cnode3 | -| pfv-tsys7 | 30% (58.8/193 GB) | 11 | 10d | **193 GB RAM**, prod workloads | -| pfv-tsys9 | 30% (7.2/24 GB) | 3 | 9d | netinfra-02 + wnode | - -**Memory capacity goal (≤80%): PASS fleet-wide.** Highest is tsys1 at 71%. - -### tsys5 Storage (RESOLVED) - -Old STATUS item "NVMe + SSD not configured" is **FIXED**. All storage active: - -| Disk | Mount | Use | -|------|-------|-----| -| nvme0n1 (CT500P1 500GB) | /mnt/nvme (PVE: NVME dir) | 49% used | -| sdb (SK hynix 512GB SSD) | /mnt/ssd2 (PVE: ssd2 dir) | 14% used | -| sdd (Samsung 860 PRO 256GB) | /mnt/pfv-tsys5/ssd (PVE: SSD dir) | 1% | -| sde (ST1000VN002 1TB) | /mnt/pfv-tsys5/S1 | 6% | -| sdf (ST1000VN002 1TB) | /mnt/pfv-tsys5/S2 | 7% | -| local-lvm (1.67 TB thin) | LVM | 6.9% | - -NFS from tsys4: D2, D3, D5 all mounted and active. - -## DNS / NTP / DHCP (netinfra) - -### pfv-netinfra-01 (PRIMARY — 192.168.3.252, TS 100.70.181.72) - -| Service | State | Detail | -|---------|-------|--------| -| **Technitium DNS** | UP (Docker `tsys-dns`, 5d) | Container, NOT systemd. Config: `/home/localuser/services/technitium/config` → `/etc/dns`. **167 zones.** Port 5380 via docker-proxy. | -| **Pi-hole** | UP (healthy, 26h) | `/dev/shm` = **1.0 GB** (operator hotfix applied). Healthcheck = `dig pi.hole` (DNS-resolution based). | -| **NTP** | UP (`ntpsec`, NOT chrony) | Clock synchronized. | -| **DHCP** | UP (`isc-dhcp-server`) | **753 active leases** (was 333). | - -### pfv-netinfra-02 (SECONDARY — 192.168.3.253, TS 100.71.171.20) - -**WIPED.** Reachable on Tailscale (host key changed, cleared from known_hosts) -but **SSH key auth fails** (`Permission denied (publickey,password)`). Needs -full rebuild as redundant node — see Pending. - -## Kubernetes (pfv-k8s) — DOWN - -| Node | Tailscale IP | k3s API :6443 | Location | -|------|-------------|---------------|----------| -| pfv-k8s-cnode1 | 100.125.134.53 | **closed** | VM 102 on tsys1 | -| pfv-k8s-cnode2 | 100.109.34.72 | **OPEN** | VM 705 on tsys7 | -| pfv-k8s-cnode3 | 100.106.222.18 | **closed** | VM 603 on tsys6 | - -- **No etcd quorum** (≤1 of 3 members). cnode1 and cnode3 k3s not listening. -- **kubeconfig stale**: `~/.kube/config.pfv-k8s` points to `100.97.178.106:6443` - — an IP matching NO current cnode. Cluster must be rebuilt. -- Worker nodes (5) all online on Tailscale, **not joined**: - pfv-k8s-wnode-tsys3/5/6/7/9. - -## Monitoring / Observability Stack - -| System | Host | Status | Access | -|--------|------|--------|--------| -| LibreNMS | tsys-librenms (100.86.204.77) | UP (Tailscale) | API needs token (401) — SSH in to create | -| Uptime Kuma | status.knownelement.com | UP | API key in `~/.secrets/api-keys.env`; `/metrics` needs auth | -| Beszel | (fleet) | deployed (per STATUS) | metrics host TBD | -| Home Assistant | (HAOS VM) | configured (per STATUS) | NUT integration live | -| **Vault** | vault.knownelement.com | **Deployed, UNCONFIGURED** | API keys migration = HIGH priority TODO | - -## Console Management (pfv-tsys4) - -ser2net listening on all 7 ports (2001-2007) bound to Tailscale 100.70.77.93. -All 7 `/dev/consoles/` symlinks present: - -``` -pfv-core-sw01→ttyUSB2 pfv-tor3-mgmt→ttyUSB1 pfv-tor3-stor→ttyUSB6 -pfv-rrinfra-rtr→ttyUSB5 pfv-r2-tor-top→ttyUSB7 subodev-torsw→ttyUSB0 -pfv-r2-sw→ttyUSB3 -``` - -**conman requires a PTY** — cannot be driven via stdin pipe from Crush. Use an -interactive terminal, or the conman client directly. - -## Tailscale Fleet (80 nodes) - -Three full environment tiers now exist, each mirroring the same service set: - -- **`tsys-*` (production):** tsys-awx, tsys-ca, tsys-cloudron, tsys-librenms, - tsys-proxmox-datacenter, tsys-proxmox-mailgw-01/02, tsys-siem, tsys-ucs-01/02, - tsys-umbrel, tsys-voip -- **`preprod-*`:** preprod-awx, preprod-ca, preprod-cloudron, preprod-hfnoc-uisp, - preprod-librenms, preprod-proxmox-datacenter, preprod-proxmox-mailgw, - preprod-rancherplatform, preprod-siem, preprod-voip -- **`sectestbed-*`:** sectestbed-awx, sectestbed-ca, sectestbed-cloudron, - sectestbed-hfnoc-uisp, sectestbed-k8s-cnode, sectestbed-k8s-wnode, - sectestbed-librenms, sectestbed-netinfra, sectestbed-proxmox-datacenter, - sectestbed-proxmox-mailgw, sectestbed-proxmox-pbs, sectestbed-proxmox-pve, - sectestbed-rancherplatform, sectestbed-sandbox, sectestbed-siem, sectestbed-voip - -Plus: pfv-tsys1/3/4/5/6/7/9, pfv-k8s-cnode1/2/3, pfv-k8s-wnode-tsys3/5/6/7/9, -pfv-netinfra-01/02, pfv-proxmox-backup-server, pfv-rr-middleware-01/02, -pfv-bms, pfv-jetson-nano-1, pfvsvrpi, devbox-cloudron, hfnoc-uisp, -kali-rd, kali-tsys, netbird, preprod-awx, subopi-dev-3/4, subopi3, -stlpc-artroom/garage, siter-bizoffice, ultix-* (endpoints). - -> **Note:** Tailscale reports `--accept-routes is false` on this workstation. - -## Identified LAN Hosts (non-switch) - -| IP | Identity | Evidence | -|----|----------|----------| -| 192.168.0.100 | stlpc-garage.knel.net | Linux + Postfix + xrdp | -| 192.168.0.250 | Linux storage (NFS) | SSH + rpcbind | -| 192.168.1.166 | pfv-stor1 (NAS) | Samba smbd 4 + Golang HTTP + SSH | -| 192.168.3.1 | gateway/router | Linux + xrdp | - -## Divergences vs STATUS.md (2026-07-30) - -1. **VM count 28→63** — sectestbed (16) + preprod (8) + new prod VMs added -2. **DNS zones 124→167**, **DHCP leases 333→753** -3. **tsys4 PVE 9.1.5→9.2.5** (was "behind fleet") -4. **tsys5 NVMe+SSD** now configured (was "not configured") -5. **tsys-umbrel + tsys-sipwise** deployed (new VMs) -6. **LACP core↔r3-stor** operator reports fixed (was #1 known issue) -7. **k8s cluster** now DOWN (was LIVE) -8. **netinfra-02** wiped (was redundant secondary) -9. **Technitium** now Docker container (was systemd dnsServer) -10. **NTP** is ntpsec (STATUS assumed chrony) -11. **Pi-hole /dev/shm** = 1 GB (operator hotfix; gravity.db healthcheck still DNS-based) -12. **Tailscale fleet** 3× larger with 3 environment tiers - -## Pending / Recommended Next Actions - -*(From `nextprompt` — reference only; awaiting operator direction)* - -- **netinfra-02 rebuild** as redundant secondary (SSH key deploy + Technitium - zone sync + DHCP failover-peer) -- **k8s cluster rebuild** (fix wiped cnode, restore quorum, join 5 wnodes) -- **Pi-hole** add gravity.db healthcheck to compose (current check is DNS-only) -- **Vault** migrate all API keys (HIGH priority) -- **Monitoring**: deploy unpoller + netdisco to LibreNMS VM; add weathermap/smokeping -- **Home Assistant**: own all environmental monitoring/reporting -- **Benchmarking**: iperf (1gbps-per-flow verification) + fio + CPU/RAM across cluster -- **Repo gardening**: README/AGENTS/docmap path fixes (this session) +> **Audit snapshot 2026-08-05 (current authoritative)** +> +> **Read it here:** https://community.turnsys.com/t/298 +> +> *Migrated 2026-08-06. This file is kept as a pointer for git-browsing context. +> Do not update content here — edit the Discourse wiki topic instead.* diff --git a/proxmox/docs/CAPACITY-GOALS.md b/proxmox/docs/CAPACITY-GOALS.md index a8649ce..1051509 100644 --- a/proxmox/docs/CAPACITY-GOALS.md +++ b/proxmox/docs/CAPACITY-GOALS.md @@ -1,220 +1,10 @@ -# Capacity & Sizing Goals +# proxmox/docs/CAPACITY-GOALS.md -**Date:** 2026-07-29 -**Status:** Authoritative. These are the operator's standing targets. Every -fleet audit and VM-placement decision is evaluated against them. - -> Companion to [`AUDIT-2026-07-29.md`](AUDIT-2026-07-29.md) (the current -> placement analysis) and [`PROJECT.md`](PROJECT.md) (the fleet hardware report). - ---- - -## 1. Memory target: 80% steady-state - -**Rule:** Keep each node at or below **80% physical RAM utilization at steady -state.** Occasional bursts above 80% are acceptable (that is what swap is for); -the *sustained* idle-to-average load must sit at ~80% or below. - -**Two metrics must both be tracked:** - -| Metric | Definition | Target | -|--------|-----------|--------| -| **Actual usage** | `MemTotal - MemAvailable` from `/proc/meminfo` | **≤ 80%** steady state (the real rule) | -| **Allocation commitment** | sum of running-VM `memory:` vs host RAM | planning ceiling — keep < 100% where possible | - -**Why both:** KSM (same-page merging) and virtio-balloon mean *actual* usage is -often well below the sum of VM allocations (e.g. tsys1 allocates 115% but uses -72% actual). Actual usage is the steady-state truth; allocation is the -worst-case ceiling if every VM suddenly fills its reservation. A host is -"healthy" when actual ≤ 80% **and** allocation isn't so far over 100% that a -balloon-deflate storm could OOM it. - -**Tolerance:** brief actual spikes to 90-100% are fine. Persistent swap use -(>~1 GB on a steady basis) means the host is over-committed and VMs should -migrate off. - -## 2. CPU target: ~50% utilization at idle - -**Rule:** Ideally, nodes sit around **50% CPU utilization at idle.** - -This is a *utilization* goal, not a headroom goal. The fleet is -**CPU-underutilized** (most hosts idle at 0-6%). On a shoestring budget, idle -silicon is wasted investment and (for the PowerEdge hosts) wasted electricity. -The intent is to **pack workloads** so the baseline load consumes roughly half -the compute, reserving the other half for bursts. - -**Practical implications:** - -- The older, power-hungry hosts (tsys6 Nehalem, tsys5 Westmere) should either be - doing useful work at ~50% or be candidates to power down / solar-gate when - idle. Running them at 0-6% wastes power per core. -- The newer, efficient hosts (tsys9 Comet Lake, tsys3 Skylake) are cheap to run - idle and are the natural home for always-on control-plane + infra. -- vCPU overcommit is fine for idle workloads; the 50% target is about *actual* - utilization, not vCPU ratios. - -## 3. Workload placement model - -The fleet serves three distinct workload families. Placement should follow CPU -quality + storage speed + RAM size, in that order of relevance per family. - -### 3.1 ETL / HPC / cross-compile (CPU + storage bound) - -Large geospatial/weather ETL, RTL/firmware simulation, cross-compile pipelines. -**Needs high IPC (modern cores) + fast storage (SSD/NVMe).** RAM is moderate -(8-32 GB per job). - -| Host | CPU | Storage | Fit | -|------|-----|---------|-----| -| **tsys9** | i5-10500 Comet Lake (2020) — **best IPC in fleet** | local SSD (PNY CS900) | **Primary compute node** | -| **tsys3** | Xeon E3-1535M v5 Skylake (2015) | local NVMe (Samsung PM961) | **Fast compute node** | -| tsys2 (incoming) | i7-7820HQ Kaby Lake (2017) | NVMe 960 PRO + SSD 850 EVO | **Fast compute node** (post-rebuild) | - -These three hosts have the newest CPUs and the only fast local storage. Their -wnodes should be the **compute-tier** workers, sized larger (more cores/RAM) and -labeled `workload=compute`. - -### 3.2 RackRental.net — network labs on demand (RAM + IOPS bound, CPU-light) - -Business idea: rent out ephemeral network labs (containerlab topologies) on -demand. Many containers / network namespaces, short-lived, mostly idle once up. -**Needs massive RAM + fast container-image pulls. CPU quality is secondary** -(labs are idle between keystrokes). - -| Host | CPU | RAM | Fit | -|------|-----|-----|-----| -| **tsys7** | 2× E5-2630 v2 Ivy Bridge (2013), 24 threads | **189 GB** | **Primary RackRental host** | -| **tsys6** | 2× E5530 Nehalem (2009), 16 threads | **126 GB** | **Secondary RackRental host** | - -**Verdict: tsys6/7 are the correct RackRental hosts.** Their weak CPUs are -irrelevant for idle network labs; their RAM is unmatched. Their wnodes should be -labeled `workload=rackrental`. - -**Caveat (deferred):** their local-lvm is USB 2.0 portable HDD (~30 MB/s), which -is terrible for container image pulls. This is a *disk* problem, not a RAM/CPU -problem, and is deferred to the post-Friday storage-class work. RAM-wise they are -ready today. - -### 3.3 Infrastructure + k8s control plane (always-on, low resource) - -netinfra, UCS, LibreNMS, SIEM, tailscale-router, PBS, k3s cnodes. Lightweight, -must stay up. Lives on tsys1 + tsys9 (and the cnodes spread across hosts for -etcd diversity). - ---- - -## 4. Host role summary (target) - -| Host | CPU era | RAM | Primary role | Wnode class | -|------|---------|-----|--------------|-------------| -| tsys1 | Haswell 2013 | 32 GB (maxed) | Infra + control plane | light (4c/4G) | -| tsys2 | Kaby Lake 2017 | 32 GB | Compute (incoming) | compute | -| tsys3 | Skylake 2015 | 32 GB | Compute | compute | -| tsys4 | Haswell 2013 | 16 GB | Storage (NFS + PBS) | — | -| tsys5 | Westmere 2010 | 96 GB | Storage + fast-tier hub | compute (NVMe, post-Friday) | -| tsys6 | Nehalem 2009 | 126 GB | RackRental | rackrental | -| tsys7 | Ivy Bridge 2013 | 189 GB | RackRental | rackrental | -| tsys9 | Comet Lake 2020 | 24 GB | Infra + control plane + compute | compute (growable) | - -## 5. DDR3 memory policy - -**Do not blindly add RAM to tsys6/7.** They idle at 6%/9% actual usage with -117/171 GB free — they are RAM-saturated in the wrong direction (over-provisioned, -not starved). The fleet's RAM pressure is on the *infra* hosts (tsys1/5/9 -over-allocated), and that is solved by **migrating VMs** to tsys6/7's existing -free RAM, not by installing sticks. - -Compatibility is decisive and is documented per-host in -[`AUDIT-2026-07-29.md`](AUDIT-2026-07-29.md) §5. Short version: tsys5/6/7 require -**Registered ECC (RDIMM)**; only RDIMM sticks will POST. Consumer UDIMM -(Patriot / most Kingston) fits essentially nowhere useful here. - ---- - -## 6. Local storage policy: ephemeral scratch, not OS drives - -**Principle:** Local disk storage is a **scratch tier**, not a boot/OS tier. -VM OS drives stay on NFS (tsys4/tsys5) where redundancy is handled by spreading -across physical disks and -01/-02 HA pairs. Local storage is used for ephemeral -workloads where data loss on node failure is acceptable by definition. - -### 6.1 Why OS drives stay on NFS - -Individual Proxmox nodes have **no local disk redundancy** (single disk, no -RAID, no ZFS mirror). If a host's local disk dies, every VM on it is gone. -Redundancy is instead achieved at the **storage-server + application layer**: - -- **-01/-02 HA pairs** (netinfra, UCS): each half on a different NFS server -- **k8s spreading**: cnode/wnode disks spread across D2/D5/S2/S3/S4 so no - single disk or server failure takes down the whole cluster -- **PBS backups**: all VMs are backed up to Proxmox Backup Server regardless of - disk tier - -This is deliberate. Moving OS drives to local storage would trade NFS network -latency for single-disk-failure risk. Not worth it for persistent workloads. - -### 6.2 What local storage IS good for: ephemeral scratch - -Workloads that produce or consume temporary data that can be discarded on node -failure. These are exactly the workloads where NFS latency hurts most and local -IOPS help most: - -| Use case | Why local scratch helps | Examples | -|----------|------------------------|---------| -| **Container image pulls** | Image layers write to local disk during pull; NFS-over-HDD is seek-bound and slow | RackRental labs, any pod startup | -| **Build artifacts** | Cross-compile / firmware builds write hundreds of small files; local SSD/NVMe is 10-100x faster than NFS-HDD | ETL, cross-compile, CI pipelines | -| **Temporary processing** | Intermediate data that doesn't need to survive a reboot | Weather/GIS transform scratch, log rotation | -| **kubelet ephemeral storage** | `emptyDir` volumes, container writable layers | All k8s pods | - -### 6.3 Per-host local storage inventory and suitability - -| Host | Local storage | Type | Available | Scratch-suitable? | Notes | -|------|--------------|------|-----------|-------------------|-------| -| **tsys2** | NVMe 960 PRO + SSD 850 EVO | NVMe + SSD | 512 GB + 1 TB | **Yes — best in fleet** | Incoming (post-rebuild). No NFS needed at all | -| **tsys3** | local-lvm (Samsung PM961) | NVMe | 349 GB | **Yes — excellent** | Fastest existing local storage. 0% used | -| **tsys9** | local-lvm (PNY CS900) | SSD | 136 GB | **Yes — excellent** | Low latency. 0% used | -| **tsys5** | local-lvm (Hitachi) | HDD | ~1.7 TB | Yes (bulk) | Large capacity, HDD speed. Storage host — scratch is secondary | -| **tsys1** | local-lvm | HDD | ~90 GB | Marginal (small) | Barely worth it; tsys1 is infra-loaded anyway | -| **tsys4** | local-lvm | HDD | ~62 GB | No | PBS host; minimal free space | -| **tsys6** | local-lvm (WD My Passport) | **USB 2.0 HDD** | 1.7 TB | **NO — 30 MB/s** | Slower than NFS. Do not use for VMs | -| **tsys7** | local-lvm (WD portable) | **USB 2.0 HDD** | 1.7 TB | **NO — 30 MB/s** | Slower than NFS. Do not use for VMs | - -**tsys6/7 are the hard problem.** They are the RackRental hosts — exactly the -workload that benefits most from local scratch (fast container image pulls) — -but their local storage is USB 2.0 portable HDD, which is slower than -NFS-over-HDD. Options for the future: - -1. **Install internal drives** (tsys6 has a disabled SAS controller; tsys7 has - 5 empty SATA ports). This is the real fix — gives both RackRental hosts proper - local scratch. Deferred to a physical-hardware session. -2. **Accept NFS for image pulls** and use local-lvm only for swap/temp that - tolerates 30 MB/s. Functional but sluggish for lab spin-up. - -### 6.4 How this maps to k8s StorageClasses (future, post-Friday) - -The scratch tier maps to two k8s concepts: - -``` -StorageClass: local-fast - provisioner: local-path - where: tsys3 (NVMe), tsys9 (SSD), tsys2 (NVMe+SSD), tsys5 (HDD) - use: emptyDir replacement, build artifacts, container runtime cache - redundancy: none — ephemeral, dies with the pod - -StorageClass: nfs-hdd (existing concept) - provisioner: nfs-subdir-external - where: D2/D5 (tsys4), S1/S2/S3/S4 (tsys5) - use: persistent bulk data, VM OS drives, anything that must survive restarts - redundancy: cross-server spreading + PBS backup -``` - -A wnode with local scratch gets a second disk (or a partition on local-lvm) -mounted as `/scratch`. ETL jobs write intermediate data to `/scratch` -(local-fast) and final output to `/data` (nfs-hdd). Pods that need ephemeral -fast storage request `local-fast` PVCs; pods that need persistence request -`nfs-hdd`. - -**This is forward-looking.** The concrete implementation (formatting local-lvm, -adding to Proxmox as VM disk targets, defining StorageClasses, labeling nodes) -happens after the Friday maintenance window when the SSD/NVMe tiers are -physically installed and the storage architecture is finalized. +> **Documentation moved to Discourse — the canonical source of truth.** +> +> **Capacity targets + workload placement model** +> +> **Read it here:** https://community.turnsys.com/t/304 +> +> *Migrated 2026-08-06. This file is kept as a pointer for git-browsing context. +> Do not update content here — edit the Discourse wiki topic instead.* diff --git a/proxmox/docs/DRIFT-2026-07-30.md b/proxmox/docs/DRIFT-2026-07-30.md index 3f3fd6a..f68a097 100644 --- a/proxmox/docs/DRIFT-2026-07-30.md +++ b/proxmox/docs/DRIFT-2026-07-30.md @@ -1,175 +1,10 @@ -# Fleet Drift Report +# proxmox/docs/DRIFT-2026-07-30.md -> **Data gathered:** 2026-07-30 ~04:15 UTC (live, via `tests/remote.sh` from all 7 hosts) -> **Probe script:** `/tmp/pfv-probe-drift.sh` (read-only, all 7 hosts in parallel) -> **Report generated:** 2026-07-30 - ---- - -## Executive Briefing - -### What needs your decision now - -| # | Issue | Impact | Recommendation | -|---|-------|--------|---------------| -| **1** | **lldpd INACTIVE on tsys1** | No LLDP neighbor data from tsys1 — blind spot in network topology map | Enable: `systemctl enable --now lldpd` | -| **2** | **tsys9 missing 2 SSH keys** | Only 3 authorized_keys vs 5 on every other host — possible access gap | Verify which keys should be present; add missing | -| **3** | **iperf3 missing on tsys9** | Can't run throughput tests from the best compute host | Install: `apt install iperf3` | -| **4** | **net-tools missing on tsys1/6/7** | `ifconfig`/`netstat` unavailable (minor — `ip` command works) | Install for consistency: `apt install net-tools` | -| **5** | **sysstat missing on tsys5** | No `iostat`/`mpstat`/`sar` on a storage host | Install: `apt install sysstat` | -| **6** | **nvme-cli missing on tsys4/5** | Can't query NVMe health/SMART on the storage servers that have NVMe | Install: `apt install nvme-cli` | -| **7** | **tsys4 tuning drift** | rmem/wmem=16MB (fleet: 128MB), tcp_max_syn_backlog=1024 (fleet: 2048+), netdev_max_backlog=5000 (fleet: 250000), tuned=throughput-performance (should be network-throughput) | Run `apply-tunings.sh --apply` on tsys4 | -| **8** | **noatime only on tsys5 root** | All other hosts use relatime on root fs — minor perf difference | Standardize (noatime preferred for storage hosts) | -| **9** | **rsyslog inactive fleet-wide** | No syslog forwarding/receiving anywhere | Saturday OAM Day: set up centralized syslog | -| **10** | **snmpd inactive fleet-wide** | No SNMP polling on hosts (switches have it) | Saturday OAM Day: wire snmpd for LibreNMS | -| **11** | **beszel not deployed** | No metrics agent on any host yet | Saturday OAM Day: deploy Beszel agent fleet-wide | - -### Consistent across the fleet (good) - -These items are the SAME on all 7 hosts — no drift: - -- **PVE version:** 9.2.5 / kernel 7.0.14-6-pve (except tsys4: 9.1.5 / 6.17.9 — known, separate upgrade task) -- **Debian version:** all on trixie/13 -- **tcp_congestion_control:** bbr everywhere -- **CPUs scaling_governor:** performance everywhere (except tsys5: no cpufreq driver — Westmere, expected) -- **ethtool:** same version everywhere -- **nfs-common:** same version everywhere -- **tcpdump:** same version everywhere -- **smartmontools:** 7.5 everywhere, smartd active everywhere -- **tuned:** installed everywhere -- **SSHD config:** PermitRootLogin=yes, KbdInteractiveAuthentication=no everywhere (pubkey+2FA via PAM) -- **PasswordAuthentication:** not explicitly set (default) — relies on 2FA PAM module - -### Quick severity summary - -- **Fix now (5 min each, no risk):** lldpd on tsys1, iperf3 on tsys9, sysstat on tsys5, nvme-cli on tsys4/5 -- **Fix with tuning pass:** tsys4 rmem/wmem/backlog/tuned-profile (via existing `apply-tunings.sh --apply`) -- **Saturday OAM Day:** rsyslog, snmpd, beszel fleet-wide deployment -- **Investigate:** tsys9 SSH key count (2 keys missing) - ---- - -## Appendix A: Package Presence Matrix - -> MISSING = not installed. Version shown = installed version (truncated). - -| Package | tsys1 | tsys3 | tsys4 | tsys5 | tsys6 | tsys7 | tsys9 | -|---------|-------|-------|-------|-------|-------|-------|-------| -| lldpd | **MISSING** | 1.0.18 | 1.0.18 | 1.0.18 | 1.0.18 | 1.0.18 | 1.0.18 | -| smartmontools | 7.5 | 7.5 | 7.5 | 7.5 | 7.5 | 7.5 | 7.5 | -| nfs-common | 2.8.3 | 2.8.3 | 2.8.3 | 2.8.3 | 2.8.3 | 2.8.3 | 2.8.3 | -| nfs-kernel-server | — | — | 2.8.3 | 2.8.3 | — | — | — | -| iperf3 | 3.18 | 3.18 | 3.18 | 3.18 | 3.18 | 3.18 | **MISSING** | -| tcpdump | 4.99.5 | 4.99.5 | 4.99.5 | 4.99.5 | 4.99.5 | 4.99.5 | 4.99.5 | -| rsyslog | installed | installed | installed | installed | installed | installed | installed | -| qemu-guest-agent | — | — | — | — | — | — | — | -| snmpd | **MISSING** | **MISSING** | **MISSING** | **MISSING** | **MISSING** | **MISSING** | **MISSING** | -| net-tools | **MISSING** | 2.10 | 2.10 | 2.10 | **MISSING** | **MISSING** | **MISSING** | -| ethtool | 6.14.2 | 6.14.2 | 6.14.2 | 6.14.2 | 6.14.2 | 6.14.2 | 6.14.2 | -| sysstat | 12.7.5 | 12.7.5 | 12.7.5 | **MISSING** | 12.7.5 | 12.7.5 | 12.7.5 | -| nvme-cli | 2.13 | 2.13 | **MISSING** | **MISSING** | 2.13 | 2.13 | 2.13 | -| conman | — | — | 0.3.1 | — | — | — | — | -| ser2net | 4.6.4 | — | 4.6.4 | — | — | — | — | -| nut-server | 2.8.1 | — | — | — | — | — | — | -| tuned | 2.25.1 | 2.25.1 | 2.25.1 | 2.25.1 | 2.25.1 | 2.25.1 | 2.25.1 | - -> **Note:** conman on tsys1, ser2net on tsys1, and nut-server on tsys1 are -> expected — tsys1 hosts the UPS (NUT) and has ser2net from a previous config. -> conman/ser2net on tsys4 is expected (console server). These are not drift. - -## Appendix B: Service State Matrix - -| Service | tsys1 | tsys3 | tsys4 | tsys5 | tsys6 | tsys7 | tsys9 | -|---------|-------|-------|-------|-------|-------|-------|-------| -| lldpd | **inactive** | active | active | active | active | active | active | -| smartd | active | active | active | active | active | active | active | -| snmpd | **inactive** | **inactive** | **inactive** | **inactive** | **inactive** | **inactive** | **inactive** | -| rsyslog | **inactive** | **inactive** | **inactive** | **inactive** | **inactive** | **inactive** | **inactive** | -| beszel | — | — | — | — | — | — | — | - -> snmpd, rsyslog, and beszel are inactive on ALL hosts. These are Saturday OAM -> Day items, not drift — they haven't been deployed yet. - -## Appendix C: Sysctl Tuning Matrix - -| Setting | tsys1 | tsys3 | tsys4 | tsys5 | tsys6 | tsys7 | tsys9 | -|---------|-------|-------|-------|-------|-------|-------|-------| -| rmem_max | 128MB | 128MB | **16MB** | 128MB | 128MB | 128MB | 128MB | -| wmem_max | 128MB | 128MB | **16MB** | 128MB | 128MB | 128MB | 128MB | -| tcp_congestion | bbr | bbr | bbr | bbr | bbr | bbr | bbr | -| swappiness | 10 | 10 | 1 | 1 | 10 | 10 | 10 | -| tcp_max_syn_backlog | 2048 | 2048 | **1024** | 4096 | 4096 | 4096 | 2048 | -| netdev_max_backlog | 250000 | 250000 | **5000** | 250000 | 250000 | 250000 | 250000 | -| governor | perf | perf | perf | N/A | perf | perf | perf | - -> **tsys4 is the outlier** on 4 settings: rmem/wmem (16MB vs 128MB), -> tcp_max_syn_backlog (1024 vs 2048+), netdev_max_backlog (5000 vs 250000). -> These directly affect NFS throughput — the 16MB TCP buffers cap per-connection -> window size, and the low backlog values can cause packet drops under load. -> swappiness=1 on tsys4/5 is intentional (storage hosts). - -## Appendix D: Tuned Profile Matrix - -| Host | Profile | Expected | Match? | -|------|---------|----------|--------| -| tsys1 | virtual-host | virtual-host | ✓ | -| tsys3 | virtual-host | virtual-host | ✓ | -| **tsys4** | **throughput-performance** | **network-throughput** | **✗** | -| tsys5 | network-throughput | network-throughput | ✓ | -| tsys6 | virtual-host | virtual-host | ✓ | -| tsys7 | virtual-host | virtual-host | ✓ | -| tsys9 | virtual-host | virtual-host | ✓ | - -## Appendix E: SSH + Security State - -### Authorized keys (root) - -| Host | Key count | Notes | -|------|-----------|-------| -| tsys1 | 5 | Standard | -| tsys3 | 5 | Standard | -| tsys4 | 5 | Standard | -| tsys5 | 5 | Standard | -| tsys6 | 5 | Standard | -| tsys7 | 5 | Standard | -| **tsys9** | **3** | **2 fewer keys than rest of fleet — investigate** | - -### SSHD config (consistent across fleet) - -| Setting | Value | All hosts | -|---------|-------|-----------| -| PermitRootLogin | yes | ✓ (all) | -| PasswordAuthentication | (default — not set) | ✓ (all) | -| KbdInteractiveAuthentication | no | ✓ (all) | - -> 2FA is enforced via PAM module (`secharden-2fa`), not via sshd's -> KbdInteractiveAuthentication. The PAM approach is consistent. - -## Appendix F: Filesystem Mount Options - -| Host | root fs mount option | Notes | -|------|---------------------|-------| -| tsys1 | relatime | Default | -| tsys3 | relatime | Default | -| tsys4 | relatime | Default | -| **tsys5** | **noatime** | **Only host with noatime on root** | -| tsys6 | relatime | Default | -| tsys7 | relatime | Default | -| tsys9 | relatime | Default | - -> Minor: noatime reduces metadata writes (slight improvement on HDD). -> Storage hosts (tsys4/5) would benefit most from noatime. - -## Appendix G: Raw Data Location - -All raw drift probe output is stored in `/tmp/-drift.txt` on this -workstation (not committed — ephemeral). Re-gather any time with: - -```bash -for h in pfv-tsys1 pfv-tsys3 pfv-tsys4 pfv-tsys5 pfv-tsys6 pfv-tsys7 pfv-tsys9; do - PROX_HOST=$h bash tests/remote.sh prox-file /tmp/pfv-probe-drift.sh > /tmp/$h-drift.txt -done -``` - -The probe script should be saved to the repo as `perf/scripts/probe-drift.sh` -for reuse. It is read-only and portable. +> **Documentation moved to Discourse — the canonical source of truth.** +> +> **Fleet drift report 2026-07-30** +> +> **Read it here:** https://community.turnsys.com/t/298 +> +> *Migrated 2026-08-06. This file is kept as a pointer for git-browsing context. +> Do not update content here — edit the Discourse wiki topic instead.* diff --git a/proxmox/docs/HOST-REFERENCE.md b/proxmox/docs/HOST-REFERENCE.md index 8ef23a2..ac79a78 100644 --- a/proxmox/docs/HOST-REFERENCE.md +++ b/proxmox/docs/HOST-REFERENCE.md @@ -1,173 +1,10 @@ -# Host Reference +# proxmox/docs/HOST-REFERENCE.md -> **One-page-per-host quick reference for every Proxmox node.** Hardware, -> network, storage, role, and running VMs. Data gathered 2026-07-30. +> **Documentation moved to Discourse — the canonical source of truth.** > -> Companion docs: [`NETWORK-TOPOLOGY.md`](NETWORK-TOPOLOGY.md), -> [`STORAGE-ARCHITECTURE.md`](STORAGE-ARCHITECTURE.md), -> [`VM-INVENTORY.md`](VM-INVENTORY.md). - ---- - -## Fleet summary - -```mermaid -graph TB - subgraph "RACK 5 -- Core / Data / Infra" - T1["tsys1 -- Infra
i7-4770 | 32GB | 9 VMs
57% RAM"] - T3["tsys3 -- Compute
Xeon E3-1535M v5 | 32GB | 1 VM
10% RAM"] - T4["tsys4 -- Storage+PBS
Xeon E3-1246 v3 | 16GB | 1 VM
15% RAM"] - T5["tsys5 -- Storage+Sandbox
Xeon E5620 | 96GB | 4 VMs
15% RAM"] - T9["tsys9 -- Infra+Compute
i5-10500 | 24GB | 6 VMs
69% RAM"] - end - - subgraph "RACK 3 -- Compute / RackRental" - T6["tsys6 -- RackRental
2x Xeon E5530 | 126GB | 2 VMs
12% RAM"] - T7["tsys7 -- RackRental
2x Xeon E5-2630 v2 | 189GB | 5 VMs
7% RAM"] - end -``` - -| Host | CPU (era) | Threads | RAM | PVE | Kernel | Role | Running VMs | RAM actual% | -|------|-----------|---------|-----|-----|--------|------|-------------|-------------| -| tsys1 | i7-4770 Haswell (2013) | 8 | 32 GB | 9.2.5 | 7.0.14 | Infra | 9 | 57% | -| tsys3 | Xeon E3-1535M v5 Skylake (2015) | 8 | 32 GB | 9.2.5 | 7.0.14 | Compute | 1 | 10% | -| tsys4 | Xeon E3-1246 v3 Haswell (2013) | 8 | 16 GB | **9.1.5** | **6.17.9** | Storage + PBS | 1 | 15% | -| tsys5 | Xeon E5620 Westmere (2010) | 8 | 96 GB | 9.2.5 | 7.0.14 | Storage + Sandbox | 4 | 15% | -| tsys6 | 2x Xeon E5530 Nehalem (2009) | 16 | 126 GB | 9.2.5 | 7.0.14 | RackRental | 2 | 12% | -| tsys7 | 2x Xeon E5-2630 v2 Ivy Bridge (2013) | 24 | 189 GB | 9.2.5 | 7.0.14 | RackRental | 5 | 7% | -| tsys9 | i5-10500 Comet Lake (2020) | 12 | 24 GB | 9.2.5 | 7.0.14 | Infra + Compute | 6 | 69% | - -**Retired:** tsys2 (Win10, pending rebuild), tsys8 (permanently retired) - ---- - -## pfv-tsys1 -- Infrastructure host - -| Attribute | Value | -|-----------|-------| -| **Role** | Infrastructure: Home Assistant, CA, netinfra-01, LibreNMS, PDM, k8s cnode3, UCS-01, kali, workbench | -| **Hardware** | Dell, i7-4770 Haswell (2013), 8 threads, 32 GB RAM (maxed) | -| **PVE** | 9.2.5 / kernel 7.0.14-6-pve | -| **mgmt IP** | 192.168.3.11 (vmbr0, DHCP) | -| **Tailscale** | 100.121.189.98 | -| **Storage IP** | 10.100.100.1 (datanet via nic1, USB dongle, no bond) | -| **NFS mounts** | S1, S2, S3, S4, D2 (all nconnect=4) | -| **Local storage** | local-lvm (~90 GB, HDD, marginal) | -| **Tuning** | governor=performance, swappiness=10, bbr, rmem/wmem=128MB, tuned=virtual-host | -| **Bond** | None (single USB NIC for storage) | -| **Special services** | NUT (UPS), powerman (PDU) | -| **Running VMs** | 100, 101, 103, 104, 105, 106, 108, 114, 117 | - ---- - -## pfv-tsys3 -- Compute host - -| Attribute | Value | -|-----------|-------| -| **Role** | Compute: k8s wnode-tsys3 (28 GB RAM, 8 vCPU) | -| **Hardware** | Xeon E3-1535M v5 Skylake (2015), 8 threads, 32 GB RAM | -| **PVE** | 9.2.5 / kernel 7.0.14-6-pve | -| **mgmt IP** | 192.168.2.5 (vmbr0, DHCP) | -| **Storage IP** | 10.100.100.3 (datanet via enx USB dongle, no bond) | -| **NFS mounts** | D2, D5, S2, S4, T5-SSD (all nconnect=4) | -| **Local storage** | local-lvm (Samsung PM961 NVMe, 349 GB, 0% used) -- **best existing local storage** | -| **Tuning** | governor=performance, swappiness=10, bbr, rmem/wmem=128MB, tuned=virtual-host | -| **Bond** | None (single USB NIC for storage) | -| **Running VMs** | 313 | - ---- - -## pfv-tsys4 -- Storage server + PBS - -| Attribute | Value | -|-----------|-------| -| **Role** | NFS storage server (D1/D2/D3/D5) + Proxmox Backup Server + console management (conman/ser2net) | -| **Hardware** | Xeon E3-1246 v3 Haswell (2013), 8 threads, 16 GB RAM | -| **PVE** | **9.1.5** / kernel **6.17.9-1-pve** (BEHIND fleet) | -| **mgmt IP** | 192.168.3.251 (vmbr0, DHCP) | -| **Tailscale** | 100.70.77.93 | -| **Storage IP** | 10.100.100.4 (datanet via enx8cae4ccda926 USB dongle, bond0 with 1 slave) | -| **NFS mounts** | None (NFS server only) | -| **Local storage** | local-lvm (WDC WD10EZRX 1TB, ~62 GB free) | -| **NFS exports** | D1 (WD 1TB USB), D2 (WDC Red 3TB), D3 (Seagate 500GB USB), D5 (Hitachi 2TB) | -| **Other disks** | /mnt/albert (Hitachi 2TB, not exported), /mnt/backup (WD 5TB USB) | -| **Tuning** | governor=performance, swappiness=1, bbr, **rmem/wmem=16MB** (should be 128MB), **tuned=throughput-performance** (should be network-throughput) | -| **Bond** | bond0 (1 USB slave), hash=layer2+3 | -| **Special services** | ser2net + conman (7 switch consoles on :7890) | -| **Running VMs** | 400 (PBS) | -| **Notes** | USB NIC bottleneck. Needs PVE upgrade + tuning alignment. | - ---- - -## pfv-tsys5 -- Storage server + sandboxes - -| Attribute | Value | -|-----------|-------| -| **Role** | NFS storage server (S1/S2) + sectestbed sandboxes + ultix streaming | -| **Hardware** | Dell T7500, Xeon E5620 Westmere (2010) **single CPU** (CPU2 socket empty), 8 threads, 96 GB RAM | -| **PVE** | 9.2.5 / kernel 7.0.14-6-pve | -| **mgmt IP** | 192.168.3.179 (vmbr0, DHCP) | -| **Storage IP** | 10.100.100.5 (storagenet via bond0, **1 of 2 slaves active** -- nic2 DOWN) | -| **NFS mounts** | D2, D3, D5 from tsys4 (**missing nconnect**) | -| **NFS exports** | S1 (IronWolf 1TB), S2 (IronWolf 1TB), S3 (stale), SSD (Samsung 860 PRO) | -| **Local storage** | local-lvm (Hitachi 2TB), local-nonprod (=S1), SSD dir (Samsung 860 PRO 256GB) | -| **Unconfigured disks** | **NVMe CT500P1 500GB** (not mounted), **SK hynix SC300B 512GB SSD** (not mounted) | -| **Tuning** | governor=N/A (no cpufreq driver on Westmere), swappiness=1, bbr, rmem/wmem=128MB, tuned=network-throughput | -| **Bond** | bond0 (1 of 2 slaves), hash=layer2+3 | -| **Running VMs** | 509, 515, 5111, 5112 (+ 22 stopped sandboxes) | -| **Notes** | Bond degraded (needs 2nd cable). NVMe + SSD need mounting/config. SSD 93% full. Single CPU. | - ---- - -## pfv-tsys6 -- RackRental host - -| Attribute | Value | -|-----------|-------| -| **Role** | RackRental: k8s wnode-tsys6 (64 GB RAM) + AWX | -| **Hardware** | 2x Xeon E5530 Nehalem (2009), 16 threads, 126 GB RAM | -| **PVE** | 9.2.5 / kernel 7.0.14-6-pve | -| **mgmt IP** | 192.168.3.169 (vmbr0, DHCP) | -| **Tailscale** | 100.73.35.111 | -| **Storage IP** | 10.100.100.6 (storagenet via bond0, **2 of 2 slaves active**, 2G LACP) | -| **NFS mounts** | D1, D2, D3, D5, S1, S2, S3, S4, T5-SSD (9 mounts, all nconnect=4) | -| **Local storage** | local-lvm (WD My Passport USB 2.0 HDD, 1.7 TB, **30 MB/s -- do not use for VMs**) | -| **Tuning** | governor=performance, swappiness=10, bbr, rmem/wmem=128MB, tuned=virtual-host | -| **Bond** | bond0 (2 of 2 slaves), **hash=layer3+4** (correct) | -| **Running VMs** | 100 (wnode-tsys6, 64GB), 600 (tsys-awx) | -| **Notes** | Best storage bond in fleet (2G LACP). Local storage is USB 2.0 -- unusable for scratch. | - ---- - -## pfv-tsys7 -- RackRental host - -| Attribute | Value | -|-----------|-------| -| **Role** | RackRental: k8s wnode-tsys7 + hfnoc-uisp + rr-middleware + TCTC + k8s cnode2 | -| **Hardware** | 2x Xeon E5-2630 v2 Ivy Bridge (2013), 24 threads, 189 GB RAM | -| **PVE** | 9.2.5 / kernel 7.0.14-6-pve | -| **mgmt IP** | 192.168.3.55 (vmbr0, DHCP) | -| **Storage IP** | 10.100.100.7 (datanet via bond0, **2 of 2 slaves active**, 2G LACP) | -| **NFS mounts** | D1, D2, D3, D5, S1, S2, S3, S4, T5-SSD (9 mounts, all nconnect=4) | -| **Local storage** | local-lvm (WD portable USB 2.0 HDD, 1.7 TB, **30 MB/s -- do not use for VMs**) | -| **Tuning** | governor=performance, swappiness=10, bbr, rmem/wmem=128MB, tuned=virtual-host | -| **Bond** | bond0 (2 of 2 slaves), **hash=layer3+4** (correct) | -| **Running VMs** | 701, 702, 703, 704, 705 | -| **Notes** | Best storage bond in fleet (2G LACP). Most RAM in fleet (189 GB). Local storage USB 2.0. | - ---- - -## pfv-tsys9 -- Infra + Compute - -| Attribute | Value | -|-----------|-------| -| **Role** | Infra + Compute: k8s cnode1 + wnode-tsys9 + siem + UCS-02 + netinfra-02 + kali | -| **Hardware** | i5-10500 Comet Lake (2020), 12 threads, 24 GB RAM | -| **PVE** | 9.2.5 / kernel 7.0.14-6-pve | -| **mgmt IP** | 192.168.3.58 (vmbr0, DHCP) | -| **Storage IP** | 10.100.100.9 (datanet via enx9c69d36a5b6c USB dongle, no bond) | -| **NFS mounts** | D2, D5, S2, S3, S4, T5-SSD (all nconnect=4) | -| **Local storage** | local-lvm (PNY CS900 SSD, 136 GB, 0% used) -- **excellent for scratch** | -| **Tuning** | governor=performance, swappiness=10, bbr, rmem/wmem=128MB, tuned=virtual-host | -| **Bond** | None (single USB NIC for storage) | -| **Running VMs** | 901, 902, 903, 904, 905, 906 | -| **Notes** | Newest CPU (best IPC in fleet). **69% RAM -- approaching 80% goal.** Target for compute growth (after siem migrates off). | +> **Per-host hardware/network/storage/tuning/role reference** +> +> **Read it here:** https://community.turnsys.com/t/304 +> +> *Migrated 2026-08-06. This file is kept as a pointer for git-browsing context. +> Do not update content here — edit the Discourse wiki topic instead.* diff --git a/proxmox/docs/K8S.md b/proxmox/docs/K8S.md index 6c69a8a..1d1ca15 100644 --- a/proxmox/docs/K8S.md +++ b/proxmox/docs/K8S.md @@ -1,485 +1,10 @@ -# K8S.md -- Kubernetes Architecture Deep-Dive - -**Date:** 2026-07-27 -**Purpose:** Detailed kubernetes architecture plan for the pfv-k8s cluster. -Companion to [`PROJECT.md`](PROJECT.md) (which has the fleet-wide assessment). -**Status:** For discussion in a future session. No changes made. - ---- - -## Table of Contents - -1. [Workload Profile](#1-workload-profile) -2. [Current State](#2-current-state) -3. [Target Architecture](#3-target-architecture) -4. [Control Plane (Cnodes)](#4-control-plane-cnodes) -5. [Worker Nodes (Wnodes)](#5-worker-nodes-wnodes) -6. [Storage Class Design](#6-storage-class-design) -7. [ETL/HPC Considerations](#7-etlhpc-considerations) -8. [Migration Plan](#8-migration-plan) - ---- - -## 1. Workload Profile - -This cluster runs **R&D and RackRental (containerlab) workloads** via -Kubernetes. Production (Gitea, RustFS, Redmine, websites) lives on a VPS in -Reston, VA running Cloudron. - -**Workload types expected:** - -| Type | Description | Storage need | RAM need | Examples | -|------|------------|-------------|----------|---------| -| **ETL (weather/GIS)** | Batch processing of large geospatial datasets. Sequential reads, transform, sequential writes. | High capacity (100s of GB), moderate IOPS | Medium (8-32 GB per job) | GRIB/NetCDF processing, raster reprojection | -| **HPC (hardware startup)** | Compute-intensive simulations, firmware build pipelines, hardware-in-the-loop testing. | Low capacity, moderate IOPS | High (32-128 GB per job) | RTL simulation, PCB thermal analysis | -| **RackRental (containerlab)** | Rapid deployment/teardown of network lab topologies. Many containers, short-lived. | Low capacity, high IOPS (container image pulls) | Low-Medium (4-16 GB) | Network topology testing, protocol validation | - -**Key storage insight:** ETL workloads need bulk capacity (NFS-HDD is fine -- -sequential I/O). HPC and containerlab need low-latency random I/O (local -SSD/NVMe is essential). The tiered StorageClass design (section 6) serves both. - ---- - -## 2. Current State - -### 2.1 pfv-k8s nodes and their storage - -| Node | Type | Host | Storage | Disk type | Status | -|------|------|------|---------|-----------|--------| -| cnode1 (107) | control | tsys1 | D5 (tsys4) | NFS-HDD | running | -| cnode2 (603) | control | tsys6 | D2 (tsys4) | NFS-HDD | running | -| cnode3 (106) | control | tsys1 | D2 (tsys4) | NFS-HDD | running | -| cnode4 (601) | control | tsys6 | D2 (tsys4) | NFS-HDD | running | -| cnode5 (602) | control | tsys6 | D5 (tsys4) | NFS-HDD | running | -| wnode-tsys3 (313) | worker | tsys3 | D5 (tsys4) | NFS-HDD | running | -| wnode-tsys5 (509) | worker | tsys5 | D2 (tsys4) | NFS-HDD | running | -| wnode-tsys6 (100) | worker | tsys6 | D5 (tsys4) | NFS-HDD | **STOPPED** | -| wnode-tsys7 (701) | worker | tsys7 | D5 (tsys4) | NFS-HDD | running | -| wnode-tsys9 (905) | worker | tsys9 | S3 (tsys5) | NFS-HDD | running | - -### 2.2 Problems - -1. **100% of cnodes on tsys4 NFS.** D2 disk failure loses 3 of 5 cnodes = - etcd quorum lost. -2. **90% of all k8s nodes on tsys4 NFS.** tsys4 failure kills the cluster. -3. **Zero nodes use SSD or NVMe.** All on NFS-over-HDD. -4. **Zero nodes use local-lvm.** tsys3/6/7/9 all have empty local storage - (349 GB / 1.7 TB / 1.7 TB / 136 GB SSD respectively). -5. **wnode-tsys6 is stopped.** Reduces cluster capacity. -6. **3 cnodes on tsys6** -- should be on lighter hosts to free tsys6 for workers. - ---- - -## 3. Target Architecture - -### 3.1 Design principles - -1. **Cnodes on lightweight hosts** (tsys1, tsys9, tsys3) -- frees tsys6/7 for - heavy workers. -2. **Cnode storage split across tsys4 and tsys5** -- etcd survives either - storage server failing. -3. **Wnode boot disks on local storage** -- eliminates NFS latency for - container runtime and kubelet. -4. **Wnode data disks on NFS-HDD** -- bulk capacity for ETL/weather/GIS. -5. **tsys5 NVMe dedicated to wnode-tsys5** -- fastest tier for HPC jobs. -6. **One wnode per hypervisor host** -- maximize total cluster capacity. - -### 3.2 Target node-host-storage matrix - -| Node | Type | Host | Boot disk | Data disk | Disk type | -|------|------|------|-----------|-----------|-----------| -| cnode1 | control | tsys1 | D5 (tsys4) | -- | NFS-HDD | -| cnode2 | control | tsys9 | D2 (tsys4) | -- | NFS-HDD | -| cnode3 | control | tsys1 | S2 (tsys5) | -- | NFS-HDD | -| cnode4 | control | tsys9 | D5 (tsys4) | -- | NFS-HDD | -| cnode5 | control | tsys3 | S3 (tsys5) | -- | NFS-HDD | -| wnode-tsys1 | worker | tsys1 | D5 (tsys4) | -- | NFS-HDD (small) | -| wnode-tsys2 | worker | tsys2 | **NVMe (960 PRO 512GB)** | **SATA SSD (850 EVO 1TB)** | **NVMe + SSD -- no NFS needed** | -| wnode-tsys3 | worker | tsys3 | **local-lvm (NVMe PM961)** | S3 (NFS) | **LOCAL-NVMe** | -| wnode-tsys5 | worker | tsys5 | **NVMe (local, Friday)** | local-nonprod (HDD) | **NVMe** | -| wnode-tsys6 | worker | tsys6 | D2 (tsys4 NFS) | -- | NFS-HDD (local-lvm is USB 2.0 -- do not use) | -| wnode-tsys7 | worker | tsys7 | D5 (tsys4 NFS) | -- | NFS-HDD (local-lvm is USB 2.0 -- do not use) | -| wnode-tsys9 | worker | tsys9 | **local-lvm (SSD)** | S2 (NFS) | **LOCAL-SSD** | - -### 3.3 Storage server distribution after changes - -| Storage server | cnodes | wnodes (boot) | wnodes (data) | -|---------------|--------|---------------|---------------| -| tsys4 (D2) | cnode2 | wnode-tsys6 | wnode-tsys7 | -| tsys4 (D5) | cnode1, cnode4 | wnode-tsys1 | -- | -| tsys5 (S2) | cnode3 | wnode-tsys9 | -- | -| tsys5 (S3) | cnode5 | -- | wnode-tsys3 | - -**Note:** wnode-tsys2 needs no NFS (1.5 TB local SSD). wnode-tsys6/7 stay on -NFS by design -- their local-lvm is USB 2.0 portable HDD (~30 MB/s), slower -than NFS-HDD, and the user has chosen not to install internal drives. - -**No single disk or server is a quorum-losing failure point.** - ---- - -## 4. Control Plane (Cnodes) - -### 4.1 Cnode sizing - -Each cnode: 4 cores, 4 GB RAM, 32 GB disk. This is sufficient for etcd + -kubernetes control plane components (API server, scheduler, controller-manager). - -### 4.2 Cnode host placement rationale - -| Host | cnodes | RAM for cnodes | Total host RAM | Remaining for other VMs | -|------|--------|---------------|---------------|------------------------| -| tsys1 | 2 (cnode1, cnode3) | 8 GB | 32 GB | ~24 GB (but 11 infra VMs consume most) | -| tsys9 | 2 (cnode2, cnode4) | 8 GB | 24 GB | ~16 GB (4 infra VMs + 1 wnode) | -| tsys3 | 1 (cnode5) | 4 GB | 32 GB | ~28 GB (1 wnode at 20 GB = 8 GB headroom) | - -**tsys6 and tsys7 have ZERO cnodes** -- fully dedicated to heavy worker nodes. - -### 4.3 Cnode storage placement rationale - -The 5 cnodes are split 3-on-tsys4 / 2-on-tsys5: - -| Disk | cnodes | Rationale | -|------|--------|-----------| -| D5 (tsys4 HDD) | cnode1, cnode4 | Spread load across 2 disks on tsys4 | -| D2 (tsys4 HDD) | cnode2 | Only 1 cnode on D2 (was 3 -- reduces blast radius) | -| S2 (tsys5 HDD) | cnode3 | tsys5 storage for quorum diversity | -| S3 (tsys5 HDD) | cnode5 | tsys5 storage, different disk than S2 | - -**If D2 fails:** cnode2 dies. 4 of 5 survive. Quorum OK. -**If D5 fails:** cnode1 + cnode4 die. 3 of 5 survive. Quorum OK. -**If tsys4 fails:** cnode1, cnode2, cnode4 die. cnode3 + cnode5 survive on -tsys5. **Only 2 of 5 -- QUORUM LOST.** - -Wait -- that is a problem. If tsys4 goes completely offline, we lose 3 -cnodes and only have 2 on tsys5. That loses quorum (need 3). - -**Revision needed:** Move 1 more cnode to tsys5 storage. Target: 2 on tsys4, -3 on tsys5. But that means tsys5 failure (3 cnodes die) leaves only 2 on -tsys4. Same problem inverted. - -The fundamental issue: with 5 cnodes and 2 storage servers, the best split is -3/2. The server holding 3 cnodes is a quorum-loss risk if it fails. The server -holding 2 cnodes is safe (3 survive). - -**Proper solution: 3 cnodes on the "less likely to fail" server, 2 on the -other.** After Friday's hardware work: -- tsys4 will have a new PCIe NIC + 64 GB RAM -- more reliable -- tsys5 will have bond0 fixed + NVMe -- more reliable - -Either way, 3/2 split means one server failure could lose quorum. **To truly -solve this, use a 3rd storage target.** Options: -- Use tsys9 local SSD for 1 cnode (breaks the 2-server model, adds a 3rd - independent failure domain) -- Use local-lvm on the cnode's own host (etcd data is local to the VM's host, - no NFS dependency at all) - -**Best option: put cnode boot disks on local-lvm where available.** This -eliminates NFS entirely for the control plane. Each cnode's etcd data lives on -its own host's local disk -- no shared dependency. - -| cnode | Host | **Recommended storage** | Type | -|-------|------|------------------------|------| -| cnode1 | tsys1 | **local-lvm** (if space) or D5 (tsys4) | LOCAL-HDD or NFS-HDD | -| cnode2 | tsys9 | **local-lvm (SSD)** | **LOCAL-SSD** | -| cnode3 | tsys1 | **S2 (tsys5)** | NFS-HDD | -| cnode4 | tsys9 | **local-lvm (SSD)** | **LOCAL-SSD** | -| cnode5 | tsys3 | **local-lvm** | LOCAL-HDD | - -With this layout, a tsys4 failure takes down 0 cnodes. A tsys5 failure takes -down 1 (cnode3). A host failure takes down at most 2 cnodes. Quorum always -survives. - -**This is the recommended approach.** Local storage for cnodes wherever -possible. NFS only as fallback. - -### 4.4 etcd performance on local vs NFS - -| Storage | Typical fsync latency | etcd commit latency | Impact | -|---------|----------------------|--------------------|--------| -| NFS-HDD (via USB dongle on tsys4) | 5-15 ms | 10-30 ms | Slow API responses, sluggish pod scheduling | -| NFS-HDD (via PCIe NIC, post-Friday) | 2-8 ms | 5-15 ms | Better but still network-bound | -| Local HDD (tsys1/3/6/7 local-lvm) | 1-5 ms | 3-10 ms | No network hop, moderate improvement | -| Local SSD (tsys9 PNY CS900) | 0.1-0.5 ms | 0.5-2 ms | **10-30x faster than NFS-HDD** | -| NVMe (tsys5, Friday) | 0.02-0.1 ms | 0.1-0.5 ms | **100x faster than NFS-HDD** | - -etcd is the heartbeat of the kubernetes control plane. Every API call, every -pod schedule, every controller reconciliation involves an etcd write. Cutting -etcd commit latency from 15 ms to 1 ms makes the entire cluster feel 15x more -responsive. **This is the single highest-impact change for k8s performance.** - ---- - -## 5. Worker Nodes (Wnodes) - -### 5.1 One wnode per hypervisor host - -| Host | wnode | Boot disk | Data disk | Total RAM | wnode RAM | Role | -|------|-------|-----------|-----------|-----------|-----------|------| -| tsys1 | wnode-tsys1 | D5 (tsys4 NFS) | -- | 32 GB | 4-8 GB | Small worker, infra co-tenant | -| tsys2 | wnode-tsys2 | **NVMe (960 PRO 512GB)** | **SATA SSD (850 EVO 1TB)** | 32 GB | 16-24 GB | **Best storage of any worker -- 1.5TB local SSD, no NFS needed** | -| tsys3 | wnode-tsys3 | **local-lvm (349 GB)** | S3 (NFS) | 32 GB | 20 GB | General worker | -| tsys5 | wnode-tsys5 | **NVMe (local)** | local-nonprod (HDD) | 96 GB | 32-64 GB | **HPC/ETL powerhouse** | -| tsys6 | wnode-tsys6 | D2 (tsys4 NFS) | -- | 128 GB | 64-96 GB | **Heavy worker, max RAM.** local-lvm is USB 2.0 -- stays on NFS | -| tsys7 | wnode-tsys7 | D5 (tsys4 NFS) | -- | 192 GB | 96-128 GB | **Heavy worker, max RAM.** local-lvm is USB 2.0 -- stays on NFS | -| tsys9 | wnode-tsys9 | **local-lvm SSD (136 GB)** | S2 (NFS) | 24 GB | 4-8 GB | Small worker, SSD boot | - -### 5.2 Why boot disks on local-lvm - -Current: all wnodes boot from NFS. Every container image pull, every kubelet -log write, every ephemeral volume traverses the NFS network path. - -With local-lvm boot disks: -- **Container image pulls** write to local disk (100-150 MB/s HDD, no network - hop) instead of NFS-HDD (80-120 MB/s with network latency) -- **kubelet logs** stay local (no NFS writes for log rotation) -- **ephemeral storage** (emptyDir volumes) uses local disk by default -- **NFS server failure does not kill the wnode** -- the VM stays running, only - the data disk (if mounted) goes away - -### 5.3 Wnode sizing guidance - -| Host | Recommended wnode config | Rationale | -|------|------------------------|-----------| -| tsys7 (192 GB) | 8-12 cores, 96-128 GB RAM, NFS boot | Largest host -- run the heaviest ETL/HPC jobs here. local-lvm is USB 2.0 | -| tsys6 (128 GB) | 8 cores, 64-96 GB RAM, NFS boot | Second-largest -- parallel heavy jobs. local-lvm is USB 2.0 | -| tsys5 (96 GB + NVMe) | 4 cores, 32-64 GB RAM, NVMe boot + HDD data | NVMe makes this fastest for I/O-bound HPC | -| tsys3 (32 GB) | 4 cores, 20 GB RAM, local-lvm boot | General-purpose worker | -| tsys2 (32 GB, NVMe+SSD, incoming) | 4 cores, 16-24 GB RAM, **NVMe boot + SSD data** | **Fastest storage worker** -- HPC with I/O bounds | -| tsys1 (32 GB) | 2 cores, 4-8 GB RAM | Small worker, don't starve infra VMs | -| tsys9 (24 GB) | 2-4 cores, 4-8 GB RAM | Small worker, SSD boot is the advantage | - -### 5.4 Tainting and labeling strategy - -Label wnodes by capability so the k8s scheduler can target them: - -```yaml -# Heavy RAM hosts (ETL/HPC) -wnode-tsys6: workload=heavy, ram=128g -wnode-tsys7: workload=heavy, ram=192g - -# NVMe host (I/O-intensive HPC) -wnode-tsys5: workload=hpc, storage=nvme - -# SSD boot host (low-latency) -wnode-tsys9: workload=light, storage=ssd - -# General workers -wnode-tsys3: workload=general -wnode-tsys2: workload=storage-fast, storage=nvme -wnode-tsys1: workload=light -``` - -Then use nodeSelector or nodeAffinity in job specs: - -```yaml -# Weather/GIS ETL job -- needs lots of RAM -spec: - nodeSelector: - workload: heavy - -# Firmware build -- needs fast storage -spec: - nodeSelector: - storage: nvme -``` - ---- - -## 6. Storage Class Design - -### 6.1 Proposed StorageClasses - -| StorageClass | Provisioner | Where | Speed | Use case | -|-------------|------------|-------|-------|----------| -| `local-fast` | local-path (k8s) | wnode local-lvm / NVMe | 100-3500 MB/s | Container runtime, scratch, databases | -| `nfs-hdd` | nfs-subdir-external-provisioner | tsys4 D2/D5, tsys5 S1-S4 | 80-120 MB/s | Bulk data, weather/GIS datasets | -| `nfs-ssd` | nfs-subdir-external-provisioner | tsys4 D3, tsys5 T5-SSD | 200-400 MB/s | Latency-sensitive persistent data | - -### 6.2 How this maps to wnode disk topology - -Each wnode has: -- **Disk 1 (boot/OS):** local-lvm or NVMe. Contains the OS, kubelet, container - runtime. k8s `local-fast` StorageClass provisioner points here. -- **Disk 2 (bulk data, optional):** NFS mount. Mounted inside the VM as a - second block device or filesystem. k8s `nfs-hdd` provisioner points here. - -Inside k8s, pods request storage via PVC: - -```yaml -# ETL job: needs bulk storage for weather data -apiVersion: v1 -kind: PersistentVolumeClaim -spec: - storageClassName: nfs-hdd - accessModes: [ReadWriteMany] # NFS allows RWX - resources: - requests: - storage: 500Gi - -# HPC job: needs fast scratch -spec: - storageClassName: local-fast - accessModes: [ReadWriteOnce] - resources: - requests: - storage: 50Gi -``` - -### 6.3 NFS-SSD tier (D3 and T5-SSD -- both on tsys5 after Friday) - -**Storage philosophy (user directive): NVMe/SSD is EXCLUSIVELY for k8s worker -scratch space, with the exception of ultix-streaming which stays on T5-SSD. -Spinning rust hosts all other infrastructure VMs** (UCS, netinfra, LibreNMS, -SIEM, etc.). - -The SSD NFS exports: -- **D3 (tsys5 SAS, 445 GB free):** k8s scratch exclusively (etcd, container - cache, ephemeral volumes). Currently 99% empty. -- **T5-SSD (tsys5 SAS, 140 GB free after ultix-streaming):** ultix-streaming - occupies 83 GB. Remaining 140 GB available for k8s use. - -**tsys5 is the fast-tier hub:** NVMe (local) + D3 SSD + T5-SSD all on one host. -This simplifies the StorageClass design -- latency-sensitive k8s PVCs target -tsys5 SSD exports, bulk PVCs target either server. - -### 6.4 NFS data distribution across storage servers - -To avoid re-creating the "everything on tsys4" problem, distribute NFS data -disks across both servers: - -| wnode | Boot (local) | Bulk data (NFS) | NFS server | -|-------|-------------|-----------------|------------| -| wnode-tsys3 | local-lvm | S3 | tsys5 | -| wnode-tsys5 | NVMe | local-nonprod | local (no NFS) | -| wnode-tsys6 | D2 (tsys4 NFS) | -- | tsys4 | -| wnode-tsys7 | D5 (tsys4 NFS) | -- | tsys4 | -| wnode-tsys9 | local-lvm (SSD) | S2 | tsys5 | - -This balances: 2 wnodes using tsys4 for bulk data, 2 using tsys5. - ---- - -## 7. ETL/HPC Considerations - -### 7.1 Weather/GIS ETL pipeline - -Typical flow: download GRIB/NetCDF files -> process (reproject, aggregate) -> -store results. - -| Stage | Storage class | Why | -|-------|-------------|-----| -| Download raw data | `nfs-hdd` | Large sequential writes. NFS-HDD handles this well. | -| Processing scratch | `local-fast` | Random access during transform. Local disk avoids NFS latency. | -| Store results | `nfs-hdd` | Large sequential writes. Persistent. | - -**Recommendation:** Deploy a `local-fast` PV mount as `/scratch` on every -wnode. ETL jobs use `/scratch` for intermediate processing and write final -output to the NFS-mounted `/data`. - -### 7.2 HPC workloads (hardware startup) - -Use cases: RTL simulation, PCB thermal analysis, firmware build pipelines. - -| Workload | Best wnode | Why | -|----------|-----------|-----| -| RTL simulation (CPU-bound, high RAM) | tsys7 (192 GB) | Most RAM, most cores (24t) | -| Firmware builds (I/O-bound, moderate RAM) | tsys5 (NVMe) | Fastest storage for compile I/O | -| Hardware-in-the-loop (latency-sensitive) | tsys9 (local SSD) | Lowest latency storage | -| Parallel batch jobs | tsys6 + tsys7 | Distribute across both heavy hosts | - -### 7.3 RackRental/containerlab - -Rapid container deployment. Key need: fast container image pulls. - -This is where **local-lvm boot disks** shine. Currently, every container image -pull writes through NFS to a spinning disk -- slow. With local-lvm, images -cache on local disk (even HDD is 2-3x faster than NFS-HDD for random I/O). -On tsys9 (SSD) and tsys5 (NVMe), image pulls are near-instant. - -### 7.4 Data locality for ETL - -For weather/GIS data that is read repeatedly (e.g., climate reanalysis), cache -it on local-lvm of the heavy hosts: - -``` -tsys3 local-lvm (NVMe 349 GB): /data/cache/weather/ -- fastest cache tier -tsys5 NVMe (local): /data/cache/gis/ -- fastest cache tier -``` - -**Note:** tsys6/7 local-lvm is USB 2.0 portable HDD (~30 MB/s) -- cannot -be used for caching. Pre-populate weather/GIS data on D2/D5 (NFS) instead. - -This avoids re-reading the same data from the same NFS export on every job -if the data is already cached in the page cache. - ---- - -## 8. Migration Plan - -**Key enabler:** The hosts are standalone Proxmox installs, but **Proxmox -Datacenter Manager (PDM)** manages them collectively and supports VM migration -between nodes. Storage migration can be done via the PDM/Proxmox UI rather -than manual disk copies -- the destination node just needs access to the target -storage (which all nodes have for NFS exports, and local storage can be -migrated through the UI's "Storage Migrate" function). - -### 8.1 Phase 1: Friday (after hardware work) - -After tsys5 cable + NVMe and tsys4 NIC + RAM: - -1. **Format tsys5 NVMe** as local directory storage (e.g., `nvme-local`) -2. **Restart wnode-tsys6** (VM 100). Keep on NFS (D5). local-lvm is USB 2.0 -- - do not use for VM storage. Recreate on D2 or D5 NFS. -3. **Move wnode-tsys9** (VM 905) disk from S3 (NFS) to local-lvm (SSD). - -### 8.2 Phase 2: Cnode rebalance (maintenance window) - -These changes require creating new VMs on target hosts and migrating disks. -Plan for a maintenance window with the k8s cluster briefly down. - -1. Create cnode2 on tsys9 (local-lvm SSD if possible, or D2 NFS). -2. Create cnode4 on tsys9 (D5 NFS or local-lvm SSD). -3. Create cnode5 on tsys3 (S3 NFS or local-lvm). -4. Move cnode3 disk from D2 to S2 (tsys4 to tsys5). -5. Join new cnodes to etcd cluster, drain old cnode2/4/5, remove. - -### 8.3 Phase 3: Wnode local storage migration (maintenance window) - -1. Recreate wnode-tsys3 with boot disk on local-lvm (349 GB). -2. wnode-tsys6 stays on NFS (local-lvm is USB 2.0 HDD -- not suitable). -3. wnode-tsys7 stays on NFS (same reason). -4. Recreate wnode-tsys5 with boot disk on NVMe. -5. Add data disks (NFS) as second SCSI devices where applicable. - -### 8.4 Phase 4: tsys2 integration (when rebuilt) - -1. Install Proxmox on tsys2. -2. Run `scripts/check.sh` to inventory. -3. Run `scripts/apply-tunings.sh --apply`. -4. Create wnode-tsys2 with **boot disk on NVMe (960 PRO)** and **data disk on SATA SSD (850 EVO)**. No NFS needed -- 1.5 TB local SSD is the most local storage of any worker. -5. Join to k8s cluster. - -### 8.5 Phase 5: Critical VM relocation - -1. Move netinfra-02 (VM 904) from D2 to S3 (tsys5 HDD). -2. Move ucs-02 (VM 902) from D5 to S2 (tsys5 HDD). -3. (No change to T5-SSD -- ultix-streaming stays.) - ---- - -## Open questions for next session - -1. **Are the hosts a Proxmox cluster (pvecm) or standalone?** This determines - whether live migration is available (huge simplification) or we need manual - disk migration. Check `pvecm status` on each host. -2. **What k8s distribution is in use?** (k3s, kubeadm, RKE2?) This affects how - nodes are joined/drain and how StorageClasses are configured. -3. **Container runtime?** (containerd, cri-o?) Affects local storage layout. -4. **Is there a container image registry mirror in the cluster?** Or do all - pulls go to Docker Hub / external? A local registry on D3 SSD would speed - up all pulls. -5. **What specific ETL tools?** (GDAL, PostGIS, xarray, Dask?) This affects - whether jobs need shared (RWX) or exclusive (RWO) storage. -6. **HPC job scheduler?** (plain k8s Jobs, Argo Workflows, Volcano?) Affects - how we label and taint nodes. +# proxmox/docs/K8S.md + +> **Documentation moved to Discourse — the canonical source of truth.** +> +> **Legacy k8s storage/host analysis** +> +> **Read it here:** https://community.turnsys.com/t/305 +> +> *Migrated 2026-08-06. This file is kept as a pointer for git-browsing context. +> Do not update content here — edit the Discourse wiki topic instead.* diff --git a/proxmox/docs/NAMING-POLICY.md b/proxmox/docs/NAMING-POLICY.md index fef09e9..c7de54c 100644 --- a/proxmox/docs/NAMING-POLICY.md +++ b/proxmox/docs/NAMING-POLICY.md @@ -1,25 +1,10 @@ -# Naming and DNS Policy +# proxmox/docs/NAMING-POLICY.md -> **Canonical naming policy for the PFVCluster fleet.** -> Established: 2026-07-31 - -## Rule - -**The Proxmox VM name is the canonical identifier.** Everything else -derives from it: - -1. **VM name** (set via `qm set --name `) is the source of truth -2. **DNS records** (Technitium, forward + reverse) must match the VM name exactly -3. **Tailscale node name** must match the VM name -4. **Monitoring/alerting** (Beszel, Uptime Kuma, LibreNMS) uses the VM name -5. **Inventory/k8s** references use the VM name - -Any mismatch between VM name and DNS/Tailscale name is a **critical bug** -that must be fixed. The Technitium DNS API key is stored at -`~/.secrets/api-keys.env` for programmatic DNS management. - -## Tools - -- Technitium DNS API key: `~/.secrets/api-keys.env` -- Uptime Kuma API key: `~/.secrets/api-keys.env` -- DNS audit: `perf/scripts/probe-vm-dns.sh` +> **Documentation moved to Discourse — the canonical source of truth.** +> +> **Canonical naming policy for VMs/DNS/Tailscale** +> +> **Read it here:** https://community.turnsys.com/t/304 +> +> *Migrated 2026-08-06. This file is kept as a pointer for git-browsing context. +> Do not update content here — edit the Discourse wiki topic instead.* diff --git a/proxmox/docs/NETWORK-TOPOLOGY.md b/proxmox/docs/NETWORK-TOPOLOGY.md index 01ee9c0..a868cf9 100644 --- a/proxmox/docs/NETWORK-TOPOLOGY.md +++ b/proxmox/docs/NETWORK-TOPOLOGY.md @@ -1,365 +1,10 @@ -# Network Topology +# proxmox/docs/NETWORK-TOPOLOGY.md -> **Complete physical + logical network reference for the PFVCluster.** -> Data gathered 2026-07-30 via SNMP, LLDP, conman console captures, and direct -> host probes. All data is live ground truth -- not documentation inherited. +> **Documentation moved to Discourse — the canonical source of truth.** > -> Companion docs: [`STORAGE-ARCHITECTURE.md`](STORAGE-ARCHITECTURE.md), -> [`HOST-REFERENCE.md`](HOST-REFERENCE.md), [`VM-INVENTORY.md`](VM-INVENTORY.md). -> Findings/analysis: [`AUDIT-2026-07-30.md`](AUDIT-2026-07-30.md). - ---- - -## 1. Physical topology diagram - -```mermaid -graph TB - subgraph "RACK 5 - Core / Data / Infra" - CORE["pfv-core-sw01
PowerConnect 5448
192.168.0.12
(mgmt + storage VLANs)"] - - TSYS1["pfv-tsys1
i7-4770 Haswell
Infra host"] - TSYS3["pfv-tsys3
Xeon E3-1535M v5
Compute"] - TSYS4["pfv-tsys4
Xeon E3-1246 v3
NFS Storage + PBS"] - TSYS5["pfv-tsys5
Xeon E5620 Westmere
NFS Storage + Sandboxes"] - TSYS9["pfv-tsys9
i5-10500 Comet Lake
Infra + Compute"] - - TSYS4_CONSOLE["conman + ser2net
7 switch consoles
Tailscale 100.70.77.93:7890"] - TSYS1_PDU["Cyclades PM10i PDU
powerman on tsys1"] - TSYS1_UPS["APC Smart-UPS C 1500
NUT on tsys1"] - end - - subgraph "RACK 3 - Compute / RackRental" - TOR3STOR["pfv-tor3-stor
PowerConnect 5324
192.168.0.9
storage VLAN"] - TOR3MGMT["pfv-tor3-mgmt
PowerConnect 5324
192.168.0.7
mgmt VLAN"] - - TSYS6["pfv-tsys6
2x Xeon E5530 Nehalem
RackRental"] - TSYS7["pfv-tsys7
2x Xeon E5-2630 v2
RackRental"] - end - - %% Cross-rack trunk (two one-way dotted arrows with labels) - CORE -. "ch1: 3x1G active (g13-15)
hash=layer-2-3
mode=on" .-> TOR3STOR - TOR3STOR -. "ch1: 4x1G (g20-23)
hash=layer-2-3-4
mode=on" .-> CORE - - %% Rack 5 connections to core - CORE --> TSYS1 - CORE --> TSYS3 - CORE --> TSYS4 - CORE --> TSYS5 - CORE --> TSYS9 - - %% Rack 3 connections - TOR3STOR --> TSYS6 - TOR3STOR --> TSYS7 - TOR3MGMT --> TSYS6 - TOR3MGMT --> TSYS7 -``` - -### How to read this diagram - -The dashed line between core-sw01 and tor3-stor is the **cross-rack trunk** -- -the most critical link in the cluster. It carries all NFS storage traffic -between rack-3 compute hosts (tsys6/7) and rack-5 storage hosts (tsys4/5). -The trunk is a 4-port static LAG, but only 3 links are active due to a cable -mismatch (see §3 below). - ---- - -## 2. Subnet / VLAN map - -| VLAN | Subnet | Purpose | Where it lives | -|------|--------|---------|----------------| -| **1** (default) | 192.168.0.0/22 | Management network | All switches, all hosts (vmbr0 DHCP) | -| **1000** | 10.100.100.0/24 | Storage network (NFS) | core-sw01 + tor3-stor only (trunked) | -| -- (Tailscale) | 100.x.y.z/32 (CGNAT) | Remote access / overlay | All hosts + select VMs | - -**Management IPs (VLAN 1, via DHCP reservations):** - -| Host | mgmt IP (vmbr0) | Tailscale IP | -|------|-----------------|--------------| -| tsys1 | 192.168.3.11 | 100.121.189.98 | -| tsys3 | 192.168.2.5 | (connected) | -| tsys4 | 192.168.3.251 | 100.70.77.93 | -| tsys5 | 192.168.3.179 | (connected) | -| tsys6 | 192.168.3.169 | 100.73.35.111 | -| tsys7 | 192.168.3.55 | (connected) | -| tsys9 | 192.168.3.58 | (connected) | - -**Storage IPs (VLAN 1000, static):** - -| Host | Storage IP | Interface | -|------|-----------|-----------| -| tsys1 | 10.100.100.1 | datanet (nic1, USB dongle) | -| tsys3 | 10.100.100.3 | datanet (enx8cae4ccda774, USB dongle) | -| tsys4 | 10.100.100.4 | datanet (enx8cae4ccda926, USB dongle) | -| tsys5 | 10.100.100.5 | storagenet (bond0 → nic1, single link) | -| tsys6 | 10.100.100.6 | storagenet (bond0 → nic1+nic2, 2G LACP) | -| tsys7 | 10.100.100.7 | datanet (bond0 → nic1+nic2, 2G LACP) | -| tsys9 | 10.100.100.9 | datanet (enx9c69d36a5b6c, USB dongle) | - -**Switch IPs (VLAN 1, static DHCP reservations):** - -| Switch | IP | Model | Console | -|--------|----|-------|---------| -| pfv-core-sw01 | 192.168.0.12 | PowerConnect 5448 | conman port 2001 | -| pfv-tor3-stor | 192.168.0.9 | PowerConnect 5324 (Neyland 24T) | conman port 2003 | -| pfv-tor3-mgmt | 192.168.0.7 | PowerConnect 5324 (Neyland 24T) | conman port 2002 | -| (unidentified) | 192.168.0.8 | PowerConnect 5324 (Neyland 24T) | conman port 2005? | - -**SNMP community:** `kn3lmgmt` (read-only, all switches respond to SNMPv2c) - ---- - -## 3. Cross-rack trunk detail (ch1) -- the critical link - -This trunk carries ALL NFS storage traffic between rack 3 (tsys6/7) and -rack 5 (tsys4/5). It is the single most important network path in the cluster. - -```mermaid -graph LR - subgraph "core-sw01 (rack5)" - G13["g13
1G UP"] - G14["g14
1G UP"] - G15["g15
1G UP"] - G16["g16
1G UP
NOT in LAG!"] - G17["g17
DOWN
in config but
no cable"] - end - - subgraph "tor3-stor (rack3)" - G20["g20
1G UP"] - G21["g21
1G UP"] - G22["g22
1G UP"] - G23["g23
1G UP"] - end - - G13 --- G20 - G14 --- G21 - G15 --- G22 - G16 -.- G23 - G17 -.->|no cable| G23 -``` - -### Configuration comparison - -| Property | core-sw01 (rack5) | tor3-stor (rack3) | Match? | -|----------|-------------------|--------------------|--------| -| Physical ports configured | g(13-15,17) | g(20-23) | **No -- asymmetric** | -| Physical ports UP | g13, g14, g15, g16 | g20, g21, g22, g23 | -- | -| Active in LAG | g13, g14, g15 (3) | g20, g21, g22, g23 (4) | **No -- 3 vs 4** | -| Load-balance hash | layer-2-3 (MAC+IP) | layer-2-3-4 (MAC+IP+Port) | **No -- asymmetric** | -| Channel mode | on (static) | on (static) | Yes (both should be LACP) | -| VLAN | access vlan 1000 | access vlan 1000 | Yes | - -### Problems - -1. **g16/g17 cable mismatch.** The 4th physical cable from tor3-stor lands on - core-sw01 g16, but the config expects it on g17. g16 is UP (1G, full duplex) - but is NOT in the LAG group. g17 is configured in the LAG but has no cable - (DOWN). Result: **only 3 of 4 links are active**. - -2. **Hash policy asymmetry.** core-sw01 hashes on layer-2-3 (src/dst MAC + IP). - tor3-stor hashes on layer-2-3-4 (src/dst MAC + IP + TCP/UDP port). With - layer-2-3 on core-sw01, **all traffic between a given host pair hashes to - one link** regardless of how many TCP connections (nconnect) are used. - -3. **Static mode (`on`).** Neither switch runs LACP. There is no link-failure - detection -- a dead cable stays in the LAG until manually discovered. - ---- - -## 4. Complete switch port map - -### pfv-core-sw01 (PowerConnect 5448, 48 ports) -- rack 5 - -| Port | Speed | Status | VLAN | Description / Connected device | -|------|-------|--------|------|-------------------------------| -| g1 | 1000 | UP | 1 | pfvsvrpi (Raspberry Pi) | -| g4 | 100 | UP | 1 | pfv-tsys4 (mgmt, secondary) | -| g5 | -- | UP | 1 | pfv-tsys1 (mgmt) | -| g6 | 100 | UP | 1 | pfvsvrpi (secondary) | -| g8 | 1000 | UP | 1 | pfv-tsys4-mgmt | -| g9 | 1000 | UP | 1 | AP-Wallmount (UAP-AC-LR) **⚠ 2902 inErrors** | -| g10 | 1000 | UP | 1 | (unknown) | -| g11 | 1000 | UP | 1 | (unknown) | -| g12 | 1000 | UP | 1 | (unknown) | -| **g13** | **1000** | **UP** | **1000 (ch1)** | **cross-rack trunk → tor3-stor g20** | -| **g14** | **1000** | **UP** | **1000 (ch1)** | **cross-rack trunk → tor3-stor g21** | -| **g15** | **1000** | **UP** | **1000 (ch1)** | **cross-rack trunk → tor3-stor g22** | -| **g16** | **1000** | **UP** | **1000** | **cross-rack trunk cable (NOT in LAG -- should be g17)** | -| **g17** | -- | **DOWN** | **1000 (ch1)** | **in LAG config but no cable** | -| g19 | 1000 | UP | 1 | pfv-tsys5-mgmt | -| g21 | 1000 | UP | 1 | (unknown) | -| g23 | 100 | UP | 1 | pfv-tsys9 (mgmt) | -| g26 | 1000 | UP | 1 | AP-Tabletop **⚠ 73 inErrors** | -| g27 | 1000 | UP | 1000 | pfv-tsys3-stor (datanet) | -| g31 | 1000 | UP | 1000 | pfv-tsys4-stor (datanet, USB) | -| g32 | -- | -- | -- | pfv-tsys4-storage (configured, DOWN) | -| g33 | -- | -- | -- | pfv-tsys5-stor (ch2, DOWN/Not Present) | -| g34 | 1000 | UP | 1000 (ch3) | pfv-tsys5-stor (bond0, 1 link active) | -| g38 | 1000 | UP | 1000 | pfv-tsys9-stor (datanet) **⚠ 590 inErrors** | -| g40 | 1000 | UP | 1000 | pfv-tsys1-stor (datanet) | -| g41 | 1000 | UP | 1 | (unknown) | -| g43 | 1000 | UP | 1 | pfv-tsys3 (mgmt) | -| g44 | 1000 | UP | 1 | (unknown) | -| g45-g48 | -- | DOWN | -- | (Combo ports, unused) | - -**Port-channels on core-sw01:** - -| Channel | Speed | Status | Members | VLAN | Notes | -|---------|-------|--------|---------|------|-------| -| ch1 | 1000 | UP | g13-15 (active), g17 (inactive) | 1000 | Cross-rack trunk to tor3-stor | -| ch2 | -- | DOWN | g32-33 (non-candidate) | -- | Unused | -| ch3 | 1000 | UP | g34 (active) | 1000 | tsys5 storage bond (degraded, 1 of 2 links) | - -**LAG hash:** `port-channel load-balance layer-2-3` - -### pfv-tor3-stor (PowerConnect 5324 / Neyland 24T, 24 ports) -- rack 3 - -| Port | Speed | Status | VLAN | Description / Connected device | -|------|-------|--------|------|-------------------------------| -| g1 | 1000 | UP | 1 | (unknown -- uplink to tor3-mgmt or router) | -| g2-g7 | 1000 | UP | 1 | (unused but UP) | -| **g8** | **1000** | **UP** | **1000 (ch2)** | **tsys6 stor nic1** | -| **g9** | **1000** | **UP** | **1000 (ch2)** | **tsys6 stor nic2** | -| **g10** | **1000** | **UP** | **1000 (ch3)** | **tsys7 stor nic1** | -| **g11** | **1000** | **UP** | **1000 (ch3)** | **tsys7 stor nic2** | -| **g13** | **1000** | **UP** | **--** | **tsys8-stor (STALE -- tsys8 retired)** | -| **g14** | **1000** | **UP** | **--** | **tsys8-stor (STALE -- tsys8 retired)** | -| g15-g19 | 1000 | UP | 1000 | (unused storage ports) | -| **g20** | **1000** | **UP** | **1000 (ch1)** | **cross-rack trunk → core-sw01 g13** | -| **g21** | **1000** | **UP** | **1000 (ch1)** | **cross-rack trunk → core-sw01 g14** | -| **g22** | **1000** | **UP** | **1000 (ch1)** | **cross-rack trunk → core-sw01 g15** | -| **g23** | **1000** | **UP** | **1000 (ch1)** | **cross-rack trunk → core-sw01 g16** | -| g24 | -- | DOWN | 1 | (spanning-tree disabled, portfast) | - -**Port-channels on tor3-stor:** - -| Channel | Speed | Status | Members | VLAN | Notes | -|---------|-------|--------|---------|------|-------| -| ch1 | 1000 | UP | g20-23 (`mode on`) | 1000 | Cross-rack trunk to core-sw01 (4x1G) | -| ch2 | 1000 | UP | g8-9 (`mode auto`) | 1000 | tsys6 storage bond (2x1G LACP) | -| ch3 | 1000 | UP | g10-11 (`mode auto`) | 1000 | tsys7 storage bond (2x1G LACP) | -| ch4 | 1000 | UP | g13-14 (`mode on`) | 1000 | **tsys8-stor (STALE -- retired host)** | - -**LAG hash:** `port-channel load-balance layer-2-3-4` - -### pfv-tor3-mgmt (PowerConnect 5324 / Neyland 24T, 24 ports) -- rack 3 - -| Port | Speed | Status | Connected device | -|------|-------|--------|-----------------| -| g3 | -- | UP | tsys7 (mgmt nic0) | -| g7 | -- | UP | tsys6 (mgmt nic0) | - -(Full port map not captured -- this switch carries only VLAN 1 management -traffic. Low priority for storage performance.) - ---- - -## 5. Per-host network interface reference - -| Host | mgmt NIC | Switch port | Storage NIC | Switch port | Storage bond | Bond hash | Storage link speed | -|------|----------|-------------|-------------|-------------|-------------|-----------|-------------------| -| tsys1 | nic0 | core g5 | nic1 (onboard) | core g40 | **No bond** (single NIC) | -- | 1G | -| tsys3 | nic0 | core g43 | enx8cae4ccda774 (USB) | core g27 | **No bond** (USB dongle) | -- | 1G | -| tsys4 | nic0 | core g8 | enx8cae4ccda926 (USB) | core g31 | bond0 (1 slave, USB) | layer2+3 | 1G | -| tsys5 | nic0 | core g19 | nic1 + nic2 | core ch3 (g34) | bond0 (1 of 2 slaves) | layer2+3 | **1G (degraded)** | -| tsys6 | nic0 | tor3-mgmt g7 | nic1 + nic2 | tor3-stor g8/g9 (ch2) | bond0 (2 of 2 slaves) | **layer3+4** | **2G LACP** | -| tsys7 | nic0 | tor3-mgmt g3 | nic1 + nic2 | tor3-stor g10/g11 (ch3) | bond0 (2 of 2 slaves) | **layer3+4** | **2G LACP** | -| tsys9 | nic0 | core g23 | enx9c69d36a5b6c (USB) | core g38 | **No bond** (USB dongle) | -- | 1G | - -### Key observations - -- **tsys6/7 are the only hosts with working 2G LACP bonds** (layer3+4 hash, 2 - active slaves). All NFS traffic from rack 3 uses these bonds. -- **tsys1/3/4/9 use USB dongles** for storage network -- single 1G, no redundancy. -- **tsys5 bond0 is degraded** -- nic2 is DOWN (no cable connected). Only nic1 is - active via core-sw01 ch3 (g34). This is the LACP problem the operator has been - troubleshooting. -- **Bond hash inconsistency:** tsys6/7 use layer3+4 (optimal for nconnect NFS), - tsys4/5 use layer2+3 (suboptimal -- same src/dst IP pair always hashes to one - link even with nconnect's multiple TCP connections). - ---- - -## 6. NFS data flow diagram - -```mermaid -graph TB - subgraph "Storage Servers (rack 5)" - T4D2["tsys4 D2
WDC Red 3TB
Tier 2"] - T4D5["tsys4 D5
Hitachi 2TB
Tier 2"] - T4D1["tsys4 D1
WD 1TB USB
Tier 3"] - T4D3["tsys4 D3
Seagate 500GB USB
Tier 3"] - T5S1["tsys5 S1
IronWolf 1TB
Tier 2"] - T5S2["tsys5 S2
IronWolf 1TB
Tier 2"] - T5SSD["tsys5 SSD
Samsung 860 PRO
Tier 1 (93% full)"] - end - - subgraph "VLAN 1000 storage network" - NFS["NFS v4.2
nconnect=4
rsize/wsize=1MB"] - end - - subgraph "NFS Clients" - T1["tsys1 (9 mounts)"] - T3["tsys3 (5 mounts)"] - T5["tsys5 (3 mounts, NO nconnect)"] - T6["tsys6 (9 mounts)"] - T7["tsys7 (9 mounts)"] - T9["tsys9 (5 mounts)"] - end - - T4D2 --> NFS - T4D5 --> NFS - T4D1 --> NFS - T4D3 --> NFS - T5S1 --> NFS - T5S2 --> NFS - T5SSD --> NFS - - NFS --> T1 - NFS --> T3 - NFS --> T5 - NFS --> T6 - NFS --> T7 - NFS --> T9 -``` - -### NFS mount inventory (live) - -| Server | Export | tsys1 | tsys3 | tsys4 | tsys5 | tsys6 | tsys7 | tsys9 | -|--------|--------|-------|-------|-------|-------|-------|-------|-------| -| tsys4 | D1 | -- | -- | -- | -- | ✓ | ✓ | -- | -| tsys4 | **D2** | ✓ | ✓ | -- | ✓ | ✓ | ✓ | ✓ | -| tsys4 | D3 | -- | -- | -- | ✓ | ✓ | ✓ | -- | -| tsys4 | D5 | -- | ✓ | -- | ✓ | ✓ | ✓ | ✓ | -| tsys5 | S1 | ✓ | -- | -- | -- | ✓ | ✓ | -- | -| tsys5 | S2 | ✓ | ✓ | -- | -- | ✓ | ✓ | -- | -| tsys5 | S3 | ✓ | -- | -- | -- | ✓ | ✓ | ✓ | -| tsys5 | S4 | ✓ | ✓ | -- | -- | ✓ | ✓ | ✓ | -| tsys5 | T5-SSD | -- | ✓ | -- | -- | ✓ | ✓ | ✓ | - -(✓ = NFS client mount active. tsys4 = NFS server only, no client mounts. -tsys5 = NFS server + 3 client mounts from tsys4, **missing nconnect**.) - ---- - -## 7. Switch management access - -| Method | Details | -|--------|---------| -| **SNMP** | Community `kn3lmgmt` (read-only). All 4 switches respond on VLAN 1 IPs. | -| **Console (conman)** | 7 serial consoles via ser2net + conman on pfv-tsys4. Conman server at `pfv-tsys4:7890` (Tailscale). Escape: `&.` | -| **Console tool** | `perf/scripts/conman-console.py` (PTY-based driver, portable) | -| **SNMP audit tool** | `perf/scripts/snmp-switch-audit.py` (config-driven) | -| **Direct telnet** | `telnet pfv-tsys4 200X` -- ONLY when conmand stopped (conflicts). Use conman instead. | - -### Console port assignments - -| Port | Console name | Switch | ID_PATH (USB port) | -|------|-------------|--------|---------------------| -| 2001 | pfv-core-sw01 | Dell PowerConnect 5448 (core, rack5) | usb-0:1.5.4.4 | -| 2002 | pfv-tor3-mgmt | Neyland 24T (mgmt TOR, rack3) | usb-0:1.6.3.1 | -| 2003 | pfv-tor3-stor | Neyland 24T (storage TOR, rack3) | usb-0:1.6.3.3.2 | -| 2004 | pfv-rrinfra-rtr | Cisco router (rrinfra) | usb-0:1.6.3.3.1 | -| 2005 | pfv-r2-tor-top | Rack 2 top-of-rack switch | usb-0:1.6.3.3.3 | -| 2006 | subodev-torsw | Suborbital device TOR switch | usb-0:1.5.4.1 | -| 2007 | pfv-r2-sw | Rack 2 old Dell switch | usb-0:1.6.3.2 | +> **Network reference: topology diagrams, switch ports, VLAN map, NFS data-flow** +> +> **Read it here:** https://community.turnsys.com/t/299 +> +> *Migrated 2026-08-06. This file is kept as a pointer for git-browsing context. +> Do not update content here — edit the Discourse wiki topic instead.* diff --git a/proxmox/docs/PHYSICAL-SYSTEMS.md b/proxmox/docs/PHYSICAL-SYSTEMS.md index d09d548..c931590 100644 --- a/proxmox/docs/PHYSICAL-SYSTEMS.md +++ b/proxmox/docs/PHYSICAL-SYSTEMS.md @@ -1,70 +1,10 @@ -# Physical Systems (Pi's + other hardware) +# proxmox/docs/PHYSICAL-SYSTEMS.md -> **Access patterns and locations for physical infrastructure devices.** -> Updated: 2026-07-31 - -## Raspberry Pi systems - -All accessible via Tailscale. Use `localuser` (passwordless sudo to root) -except where noted. - -| Name | Tailscale IP | LAN IP | Access | Purpose | -|------|-------------|--------|--------|---------| -| pfvsvrpi | 100.91.151.113 | 192.168.1.230 | `localuser` | Server Pi (general purpose) | -| subopi-dev-3 | 100.64.231.65 | 192.168.1.232 | via subodev jump | Suborbital dev Pi 3 | -| subopi-dev-4 | 100.65.224.85 | 192.168.1.231 | via subodev jump | Suborbital dev Pi 4 | - -> **Note from operator:** "use subodev to get into subopi-*" — subodev is -> a jump host for the suborbital Pi's. - -## Network devices discovered (non-Proxmox) - -| IP | Device | Type | Identified via | -|----|--------|------|---------------| -| 192.168.0.12 | pfv-core-sw01 | Dell PowerConnect 5448 | SNMP | -| 192.168.0.7 | pfv-tor3-mgmt | Neyland 24T (PC5324) | SNMP | -| 192.168.0.8 | (unidentified) | Neyland 24T (PC5324) | SNMP | -| 192.168.0.9 | pfv-tor3-stor | Neyland 24T (PC5324) | SNMP | -| 192.168.1.166 | pfv-stor1 | Linux 4.14.22 (Synology?) | SNMP | -| 192.168.1.182 | UAP-AC-LR | UniFi AP (FW 6.8.2.15592) | SNMP | -| 192.168.1.230 | pfvsvrpi | Linux 6.12.62 (RPi) | SNMP | -| 192.168.1.231 | subopi-dev-4 | Linux 6.12.25 (RPi) | SNMP | -| 192.168.1.232 | subopi-dev-3 | Linux 6.12.9 (RPi) | SNMP | -| 192.168.1.217 | subopi3 | Linux 6.12.25 (RPi) | SNMP | - -## Other identified hosts (SSH open, unidentified) - -| IP | Notes | -|----|-------| -| 192.168.0.72 | SSH open, no SNMP — investigate | -| 192.168.0.250 | SSH open, no SNMP — investigate | -| 192.168.1.138 | SSH open — investigate | -| 192.168.1.181 | SSH open — investigate | -| 192.168.1.192 | SSH open — investigate | -| 192.168.1.201 | SSH open — investigate | -| 192.168.1.205 | SSH open — investigate | -| 192.168.1.214 | SSH open — investigate | -| 192.168.1.223 | SSH open — investigate | -| 192.168.1.226 | SSH open — investigate | -| 192.168.1.228 | SSH open — investigate | -| 192.168.2.5 | SSH open — investigate (tsys3 vmbr0) | -| 192.168.3.1 | SSH open — investigate (gateway/router?) | -| 192.168.3.78 | SSH open — investigate | - -## 192.168.3.12-24 range - -These 13 hosts respond to ping but have no SSH/SNMP. Likely: -- IoT devices -- Cameras -- Phones (VoIP) -- Other DHCP clients - -## UniFi Access Points - -| AP | Port (core-sw01) | Model | FW | -|----|-----------------|-------|-----| -| UAP-AC-LR | g9 (2902 inErrors!) | UAP-AC-LR | 6.8.2.15592 | -| UAP-AC-Lite | g38 (590 inErrors!) | UAP-AC-Lite | (unknown) | - -Both managed by UniFi controller (likely on Home Assistant VM 100 or -pfv-stor1). UniPoller exports metrics to Prometheus. +> **Documentation moved to Discourse — the canonical source of truth.** +> +> **Physical infrastructure: Pis, network devices, UniFi APs** +> +> **Read it here:** https://community.turnsys.com/t/304 +> +> *Migrated 2026-08-06. This file is kept as a pointer for git-browsing context. +> Do not update content here — edit the Discourse wiki topic instead.* diff --git a/proxmox/docs/PROJECT.md b/proxmox/docs/PROJECT.md index 7ba3d50..1becfda 100644 --- a/proxmox/docs/PROJECT.md +++ b/proxmox/docs/PROJECT.md @@ -1,857 +1,10 @@ -# Proxmox Cluster Project Report - -**Date:** 2026-07-27 (re-audited) -**Prepared by:** Performance Optimization Engagement -**Status:** Comprehensive fleet assessment with VM placement and redundancy analysis - -**Data freshness:** All 7 hosts re-audited at 21:50 CDT 2026-07-27 via -`deploy-check.sh`. VM placements reflect live state after user's PDM -migrations. This is ground truth. - ---- - -## Table of Contents - -1. [Executive Summary](#1-executive-summary) -2. [Host Fleet](#2-host-fleet) -3. [Storage Architecture](#3-storage-architecture) -4. [VM Fleet Inventory](#4-vm-fleet-inventory) -5. [Kubernetes Node Distribution](#5-kubernetes-node-distribution) -6. [Storage Redundancy Analysis](#6-storage-redundancy-analysis) -7. [Local SSD/NVMe Opportunity](#7-local-ssdnvme-opportunity) -8. [Role Alignment Audit](#8-role-alignment-audit) -9. [Network Findings](#9-network-findings) -10. [Recommendations](#10-recommendations) -11. [Hardware End-of-Support Exposure](#11-hardware-end-of-support-exposure) -12. [Open Items](#12-open-items) - ---- - -## 1. Executive Summary - -The cluster consists of 7 active Proxmox hosts and 1 incoming (pfv-tsys2), -running 43 VMs across two NFS storage servers (tsys4, tsys5). Host-side -performance tunings are complete on 5 of 7 hosts. Two hosts (tsys4, tsys5) -are blocked on physical hardware work scheduled for Friday. - -**Progress since initial audit:** The user has been actively rebalancing k8s -nodes via PDM. Storage distribution improved from 90%/10% (tsys4/tsys5) to -73%/27%. One cnode now uses tsys5 storage (cnode5 on S2). More migration -needed for etcd quorum survival. - -The VM-layer assessment reveals: - -| # | Finding | Severity | Status | -|---|---------|----------|--------| -| 1 | **4 of 5 cnodes still store disks on tsys4 NFS.** cnode5 moved to tsys5. Still need 1-2 more moves for quorum survival. | **CRITICAL** | Improving | -| 2 | **Both -01/-02 infrastructure pairs (netinfra, UCS) on tsys4 NFS only.** | **HIGH** | TODO today | -| 3 | **No k8s node uses SSD or NVMe yet.** tsys3 has 349 GB unused local NVMe; tsys9 has 136 GB local SSD. | **HIGH** | Deferred to k8s session | -| 4 | **D3 SSD (tsys4, USB) is 99% empty (445 GB free).** Moving to tsys5 SAS Friday. | **MEDIUM** | Friday | - ---- - -## 2. Host Fleet - -### 2.1 Inventory - -| Host | Model | CPU (year) | Cores | RAM | Local Disk | Role (intended) | Tuning | -|------|-------|-----------|-------|-----|-----------|-----------------|--------| -| pfv-tsys1 | OptiPlex 9020 | i7-4770 Haswell (2013) | 4c/8t | 32 GB DDR3 | HDD (LVM-thin) | **Infrastructure** | Done | -| pfv-tsys2 | Precision 5520 | i7-7820HQ Kaby Lake (2017) | 4c/8t | 32 GB (max) | **NVMe 512GB + SATA SSD 1TB** | **Kubernetes** | Incoming (Win10) | -| pfv-tsys3 | Precision 7510 | Xeon E3-1535M v5 Skylake (2015) | 4c/8t | 32 GB DDR4 | HDD (LVM-thin) | **Kubernetes** | Done | -| pfv-tsys4 | Precision T1700 | Xeon E3-1246 v3 Haswell (2013) | 4c/8t | 16 GB DDR3 | 6 disks (HDD+SSD+SMR) | **Storage (NFS+PBS)** | Blocked (NIC+RAM) | -| pfv-tsys5 | Precision T7500 | Xeon E5620 Westmere (2010) | 4c/8t | 96 GB DDR3 | 6 disks (HDD+SSD) | **Storage (NFS+VMs)** | Blocked (cable) | -| pfv-tsys6 | PowerEdge R610 | 2x Xeon E5530 Nehalem (2009) | 8c/16t | 128 GB DDR3 | HDD (LVM-thin) | **Kubernetes** | Done | -| pfv-tsys7 | PowerEdge R620 | 2x Xeon E5-2630 v2 Ivy Bridge (2013) | 12c/24t | 192 GB DDR3 | HDD (LVM-thin) | **Kubernetes** | Done | -| pfv-tsys9 | OptiPlex 7080 | i5-10500 Comet Lake (2020) | 6c/12t | 24 GB DDR4 | **250 GB SSD** (PNY CS900) | **Infrastructure** | Done | - -### 2.2 Role taxonomy (per user directive) - -| Role | Hosts | Workload | -|------|-------|----------| -| **Infrastructure + k8s control** | tsys1, tsys9 | Infra VMs (netinfra, UCS, PBS, CA, HA) + pfv-k8s cnodes (control plane) + small wnodes | -| **Kubernetes workers** | tsys2, tsys3, tsys6, tsys7 | pfv-k8s wnodes (heavy workers) -- these hosts have the RAM (32-192 GB) for ETL/HPC | -| **Storage** | tsys4, tsys5 | NFS server + PBS backup target. tsys5 also runs sectestbed/preprod VMs | - -**Design rationale:** cnodes (control plane) are lightweight (4 cores, 4 GB -RAM each) and are weighted toward tsys1/tsys9 to keep the heavy RAM/CPU hosts -(tsys6 with 128 GB, tsys7 with 192 GB) free for large worker nodes. wnodes -run one per hypervisor host across the fleet to maximize total cluster capacity. - -pfv-k8s runs all R&D and RackRental (containerlab) workloads via Kubernetes. -Production (Gitea, RustFS, Redmine, websites) lives on a VPS in Reston, VA -running Cloudron -- not in this cluster. - ---- - -## 3. Storage Architecture - -### 3.1 NFS exports from tsys4 (primary storage server) - -| Export | Disk model | Type | Bus | Total | Used | Free | Use% | -|--------|-----------|------|-----|-------|------|------|------| -| D2 | WDC WD30EFRX Red | HDD (7200rpm) | SATA | 2.7 TB | 187 GB | **2.4 TB** | 8% | -| ~~D3~~ | ~~SK hynix SC300~~ | ~~SSD~~ | ~~USB~~ | — | — | — | **moving to tsys5 Friday** | -| D5 | Hitachi HDS72302 | HDD (7200rpm) | SATA | 1.8 TB | 236 GB | **1.5 TB** | 14% | - -Non-exported disks on tsys4: -- sda (Hitachi 1.8T) at /mnt/albert -- not NFS shared, 1.7 TB free -- sdd (WDC 1T) -- **idle, unmounted, removable** (free up for other use) -- sdf (WDC 4.5T SMR) at /mnt/backup -- **PBS backup target**, 4.3 TB free - -**D3 migration (Friday):** The SK hynix SC300 SSD is currently USB-attached on -tsys4 (via a "ThinkPad SSD" USB adapter). It is moving to a tsys5 SAS port, -eliminating the USB bottleneck. tsys4's 4 SATA ports are all occupied (sda/sdb -/sdc/sdd), so tsys5 is the better target. See section 3.2. - -### 3.2 NFS exports from tsys5 (secondary storage -- becoming the fast-tier hub) - -| Export | Disk model | Type | Bus | Total | Used | Free | Use% | -|--------|-----------|------|-----|-------|------|------|------| -| S1 | Seagate ST1000VN | HDD | SAS | 916 GB | 60 GB | 810 GB | 7% | -| S2 | Seagate ST1000VN | HDD | SAS | 916 GB | **6.9 GB** | **863 GB** | **1%** | -| S3 | Seagate ST1000VN | HDD | SAS | 916 GB | 7.0 GB | **863 GB** | **1%** | -| S4 | Toshiba DT01ACA050 | HDD | SAS | 458 GB | 2 MB | **435 GB** | **0%** | -| T5-SSD | Samsung 860 PRO | **SSD** | SAS | 234 GB | **122 GB** | **101 GB** | **55%** | -| **D3** (Friday) | SK hynix SC300 | **SSD** | **SAS** | **469 GB** | **2 MB** | **445 GB** | **0%** | - -**tsys5 storage controllers (plenty of free ports):** -- LSI SAS1068E (SAS 6/iR): 8 ports, 3 used (Samsung SSD, Hitachi, Seagate), - **5 free** -- Intel ICH10 SATA #1 (4-port): 2 used (Seagate S3, Toshiba S4), **2 free** -- Intel ICH10 SATA #2 (2-port): **status unknown, likely free** -- 2x Renesas USB 3.0 xHCI controllers (real USB 3.0, unlike tsys6/7) - -**Key finding: S2 and S3 now have k8s node disks.** S2 holds cnode5 + -wnode-tsys1 + wnode-tsys9 (6.9 GB used). S3 has wnode-tsys9's old disk -(unused, 7 GB). S4 still 99% empty (435 GB free). - -T5-SSD grew to 55% used (122 GB) -- ultix-streaming is the primary consumer. - -**Friday additions:** -1. **D3 (SK hynix SSD)** moves from tsys4 USB to tsys5 SAS port. Eliminates - USB 2.0 bottleneck. Becomes the second SSD-tier NFS export. -2. **PCI NVMe drive** (local-only, not NFS-exported). Used for wnode-tsys5 - boot disk and HPC scratch. The fastest tier in the fleet. - -After Friday, **tsys5 consolidates all fast storage**: NVMe (local) + 2 SSD -NFS exports (D3 + T5-SSD) + 4 HDD NFS exports (S1-S4). This makes tsys5 the -natural home for latency-sensitive workloads and the k8s StorageClass design -center. - -### 3.3 Local storage tiers (per host, with utilization) - -| Host | Storage ID | Disk type | Bus | Total | Used | Free | Used by VMs? | -|------|-----------|-----------|-----|-------|------|------|-------------| -| tsys1 | local-lvm | HDD | SATA | ~90 GB | low | ~90 GB | No (all VMs on NFS) | -| **tsys3** | **local-lvm** | **NVMe (Samsung PM961)** | **NVMe** | **349 GB** | **0 GB** | **349 GB** | **No (all VMs on NFS)** | -| tsys4 | local-lvm | HDD | SATA | ~94 GB | PBS VM | ~62 GB | Yes (PBS VM 400) | -| tsys5 | local-lvm | HDD (Hitachi 1.8T) | SATA | 1.7 TB | 40 MB | **1.7 TB** | No | -| tsys5 | local-nonprod | HDD (Seagate 1T, =S1) | SATA | 916 GB | 53 GB | **856 GB** | Yes (sectestbed suite) | -| **tsys6** | **local-lvm** | **HDD (WD My Passport)** | **USB 2.0** | **1.7 TB** | **0 GB** | **1.7 TB** | **No -- DO NOT USE for VM storage** | -| **tsys7** | **local-lvm** | **HDD (WD portable)** | **USB 2.0** | **1.7 TB** | **0 GB** | **1.7 TB** | **No -- DO NOT USE for VM storage** | -| **tsys9** | **local-lvm** | **SSD (PNY CS900)** | **SATA** | **136 GB** | **0 GB** | **136 GB** | **No (all VMs on NFS)** | -| **tsys2** | **NVMe** (Samsung 960 PRO) | **NVMe** | **NVMe** | **512 GB** | (Win10) | **512 GB** | **Incoming -- fastest boot tier after tsys5 NVMe** | -| **tsys2** | **SATA SSD** (Samsung 850 EVO) | **SSD** | **SATA** | **1 TB** | (Win10) | **1 TB** | **Incoming -- bulk data on SSD, not rust** | - -**CRITICAL WARNING: tsys6 and tsys7 local-lvm is USB 2.0 portable HDD.** - -The entire Proxmox OS, swap, and local-lvm on both R610 and R620 run on a -single **USB 2.0-attached WD My Passport portable HDD** (tsys6: "My Passport -260D"; tsys7: "Drive 2657"). Both servers' only USB controllers are EHCI -(USB 2.0, ~480 Mbps). There is **no USB 3.0/xHCI** on either host. - -**USB 2.0 practical throughput is ~30-35 MB/s.** This is 3-4x SLOWER than -NFS-over-HDD (~80-120 MB/s). Moving wnode boot disks to local-lvm on these -hosts would **decrease** performance. local-lvm on tsys6/7 must NOT be used -for VM storage. - -Additionally, both servers have completely empty internal drive bays: -- **tsys6**: SAS controller present but **DISABLED** in BIOS. No internal -drives. -- **tsys7**: 6-port SATA AHCI controller present, **5 ports EMPTY** (only -DVD-ROM on port 5). No internal drives. - -This is a reliability risk beyond performance: the entire host OS boots -from a consumer-grade portable USB drive not designed for 24/7 server use. - -**tsys3 correction:** Previously documented as HDD. Actually boots from a -**Samsung PM961 NVMe 512GB SSD** -- the fastest existing local storage in the -fleet. Its 349 GB of local-lvm is excellent for wnode boot disk use. - -**Critical observation: every k8s host has 0% used local-lvm.** tsys3 -(Samsung PM961 **NVMe**, 349 GB), tsys6 (WD My Passport **USB 2.0** HDD, -1.7 TB), tsys7 (WD portable **USB 2.0** HDD, 1.7 TB) all have unused local -storage. - -**However, only tsys3's local-lvm is suitable for VM storage.** tsys6 and -tsys7 local-lvm is USB 2.0 portable HDD (~30-35 MB/s) -- slower than -NFS-over-HDD and unsuitable for wnode boot disks. - -### 3.4 Disk speed tiers summary - -| Tier | Where | Speed class | Best for | -|------|-------|------------|----------| -| **NVMe** | tsys3 (Samsung PM961), tsys5 (Friday addition), **tsys2 (Samsung 960 PRO 512GB)** | 2000-3500 MB/s | HPC scratch, ETL staging, container runtime, wnode boot, etcd | -| **Local SSD** | tsys9 (PNY CS900, 136 GB), **tsys2 (Samsung 850 EVO 1TB)** | 500 MB/s | wnode boot disk, etcd | -| **NFS-SSD** | tsys5 D3 (SK hynix, **SAS post-Friday**), tsys5 T5-SSD (Samsung) | 200-400 MB/s over NFS | **k8s worker scratch only** (etcd, container cache, ephemeral volumes) | -| **NFS-HDD** | tsys4 D2/D5, tsys5 S1-S4 | 80-120 MB/s over NFS | Bulk data, large disks, non-critical VMs, **wnode boot on tsys6/7** | -| **Local SATA HDD** | tsys1 local-lvm | 100-150 MB/s | Host OS only | -| **USB 2.0 HDD** | tsys6/7 local-lvm (WD My Passport) | **~30-35 MB/s** | **NOTHING -- slower than NFS, do not use for VMs** | - -### 3.5 Storage tier characterization per host - -| Host | Storage profile | Detail | -|------|----------------|--------| -| **tsys2** | **SSD/NVMe only** | 960 PRO NVMe 512GB + 850 EVO SATA SSD 1TB. No spinning disk. | -| **tsys3** | **NVMe only** | Samsung PM961 NVMe 512GB. No spinning disk. | -| **tsys5** | **Hybrid** (fast-tier hub) | NVMe (local, Friday) + D3 SSD + T5-SSD + S1-S4 HDD | -| **tsys4** | **Bulk/spinning disk only** | D2 HDD 3TB + D5 HDD 2TB. D3 SSD leaving Friday. PBS target on SMR HDD. | -| **tsys9** | **Local SSD + NFS** | PNY CS900 SSD 136GB local + NFS client | -| **tsys1** | **Local HDD + NFS** | Small local-lvm + NFS client | -| **tsys6/7** | **NFS only** | local-lvm is USB 2.0 HDD (unusable for VMs). All VMs on NFS. | - -### 3.6 All exports are single-disk with no redundancy - -Every NFS export is a single physical disk formatted ext4. No RAID, no ZFS -mirror, no mdraid. A single disk failure takes down every VM whose disk lives -on that export. This applies to **both storage servers** and to the **PBS -backup target** (a single 4.5T SMR drive). - ---- - -## 4. VM Fleet Inventory - -### 4.1 Complete VM roster (running VMs only, 40 VMs across 7 hosts) - -#### tsys1 (Infrastructure) -- 11 running VMs - -| VMID | Name | Cores | RAM (MB) | Disk | Storage | Tier | -|------|------|-------|----------|------|---------|------| -| 100 | pfv-bms (HomeAssistant) | 2 | 4096 | 32 GB | D2 (tsys4 HDD) | NFS | -| 101 | tsys-ca | 2 | 2048 | 32 GB | D2 (tsys4 HDD) | NFS | -| 103 | **pfv-netinfra-01** | 2 | 2048 | 32 GB | D5 (tsys4 HDD) | NFS | -| 104 | tsys-librenms | 2 | 2048 | 50 GB | D2 (tsys4 HDD) | NFS | -| 105 | tsys-proxmox-datacenter | 2 | 2048 | 32 GB | D2 (tsys4 HDD) | NFS | -| 106 | **pfv-k8s-cnode3** | 2 | 4096 | 32 GB | D2 (tsys4 HDD) | NFS | -| 107 | **pfv-k8s-cnode1** | 2 | 4096 | 32 GB | D5 (tsys4 HDD) | NFS | -| 108 | **tsys-ucs-01** | 2x2 | 8000 | 32 GB | D2 (tsys4 HDD) | NFS | -| 109 | tailscale-router | 2 | 2048 | 25 GB | D2 (tsys4 HDD) | NFS | -| 114 | kali-tsys | 2 | 2048 | 32 GB | D2 (tsys4 HDD) | NFS | -| 117 | tsys-secure-workbench | 2 | 4000 | 32 GB | D2 (tsys4 HDD) | NFS | - -#### tsys3 (Kubernetes) -- 1 running VM - -| VMID | Name | Cores | RAM (MB) | Disk | Storage | Tier | -|------|------|-------|----------|------|---------|------| -| 313 | **pfv-k8s-wnode-tsys3** | 4x2 | 20000 | 32 GB | D5 (tsys4 HDD) | NFS | - -#### tsys4 (Storage) -- 1 running VM - -| VMID | Name | Cores | RAM (MB) | Disk | Storage | Tier | -|------|------|-------|----------|------|---------|------| -| 400 | pfv-proxmox-backup-server | 2 | 2048 | 32 GB | local-lvm | LOCAL | - -#### tsys5 (Storage) -- 15 running VMs - -| VMID | Name | Cores | RAM (MB) | Disk | Storage | Tier | -|------|------|-------|----------|------|---------|------| -| 509 | **pfv-k8s-wnode-tsys5** | 2x4 | 32000 | 32 GB | D2 (tsys4 HDD) | NFS | -| 5101 | sectestbed-siem | 2x2 | 10000 | 132 GB | local-nonprod | LOCAL | -| 5105 | sectestbed-awx | 2x2 | 4096 | 288 GB | local-nonprod | LOCAL | -| 5106 | sectestbed-k8s-cnode | 2x2 | 4096 | 32 GB | local-nonprod | LOCAL | -| 5107 | sectestbed-k8s-wnode | 2x2 | 4096 | 32 GB | local-nonprod | LOCAL | -| 5108 | sectestbed-librenms | 2x2 | 4096 | 32 GB | local-nonprod | LOCAL | -| 5109 | sectestbed-netinfra | 2x2 | 4096 | 32 GB | local-nonprod | LOCAL | -| 5111 | ultix-streaming | 2x2 | 9000 | 288 GB | T5-SSD (tsys5 SSD) | NFS-SSD | -| 5112 | ultix-offstage | 2x2 | 6000 | 288 GB | local-lvm | LOCAL | -| 6000 | sectestbed-sandbox | 2x2 | 4096 | 32 GB | local-nonprod | LOCAL | -| 51010 | sectestbed-tctc | 2x2 | 4096 | 32 GB | local-nonprod | LOCAL | -| 51011 | sectestbed-cloudron | 2x2 | 4096 | 32 GB | local-nonprod | LOCAL | -| 51012 | sectestbed-hfnoc | 2x2 | 4096 | 32 GB | local-nonprod | LOCAL | -| 51013 | sectestbed-rancherplatform | 2x2 | 4096 | 32 GB | local-nonprod | LOCAL | -| 53100 | tsys-preprod-awx | 2x2 | 9000 | 160 GB | local-nonprod | LOCAL | -| 53101 | tsys-preprod-siem | 2x2 | 12000 | 32 GB | local-nonprod | LOCAL | -| 53102 | tsys-preprod-rancherplatform | 2x2 | 8000 | 32 GB | local-nonprod | LOCAL | - -#### tsys6 (Kubernetes) -- 3 running VMs (1 wnode stopped) - -| VMID | Name | Cores | RAM (MB) | Disk | Storage | Tier | -|------|------|-------|----------|------|---------|------| -| 100 | pfv-k8s-wnode-tsys6 | 2x2 | 32000 | 32 GB | D5 (tsys4 HDD) | NFS -- **STOPPED** | -| 600 | tsys-awx | 2x2 | 12000 | 32 GB | D2 (tsys4 HDD) | NFS -- **STOPPED** | -| 601 | **pfv-k8s-cnode4** | 4 | 4096 | 32 GB | D2 (tsys4 HDD) | NFS | -| 602 | **pfv-k8s-cnode5** | 4 | 4096 | 32 GB | D5 (tsys4 HDD) | NFS | -| 603 | **pfv-k8s-cnode2** | 4 | 4096 | 32 GB | D2 (tsys4 HDD) | NFS | - -#### tsys7 (Kubernetes) -- 4 running VMs - -| VMID | Name | Cores | RAM (MB) | Disk | Storage | Tier | -|------|------|-------|----------|------|---------|------| -| 701 | **pfv-k8s-wnode-tsys7** | 4 | 32000 | 32 GB | D5 (tsys4 HDD) | NFS | -| 702 | hfnoc-uisp | 2x2 | 8000 | 100 GB | D2 (tsys4 HDD) | NFS | -| 703 | rr-middleware | 2 | 2048 | 32 GB | D2 (tsys4 HDD) | NFS | -| 704 | TCTC | 4 | 6000 | 32 GB | D2 (tsys4 HDD) | NFS | - -#### tsys9 (Infrastructure) -- 5 running VMs - -| VMID | Name | Cores | RAM (MB) | Disk | Storage | Tier | -|------|------|-------|----------|------|---------|------| -| 901 | tsys-siem | 2 | 8000 | 132 GB | D2 (tsys4 HDD) | NFS | -| 902 | **tsys-ucs-02** | 2x2 | 8000 | 50 GB | D5 (tsys4 HDD) | NFS | -| 903 | kali-rd | 2 | 2048 | 32 GB | D5 (tsys4 HDD) | NFS | -| 904 | **pfv-netinfra-02** | 2 | 4000 | 32 GB | D2 (tsys4 HDD) | NFS | -| 905 | **pfv-k8s-wnode-tsys9** | 4 | 4096 | 32 GB | S3 (tsys5 HDD) | NFS | - -### 4.2 Storage concentration summary - -| Storage target | # of running VMs | % of fleet | -|---------------|-----------------|------------| -| **D2 (tsys4 WDC Red 3TB HDD)** | **18** | **45%** | -| D5 (tsys4 Hitachi 2TB HDD) | 9 | 23% | -| local-nonprod (tsys5 local HDD) | 10 | 25% | -| S3 (tsys5 Seagate 1TB HDD) | 1 | 3% | -| T5-SSD (tsys5 Samsung SSD) | 1 | 3% | -| local-lvm (tsys4 local) | 1 | 3% | - -**27 of 40 running VMs (68%) store their disks on tsys4 NFS exports.** -If tsys4 goes offline, two-thirds of the fleet loses its storage. - ---- - -## 5. Kubernetes Node Distribution (re-audited 21:50 CDT) - -### 5.1 pfv-k8s cnode (control plane) placement -- CURRENT - -| VMID | Name | Hypervisor | Storage | NFS Server | Changed? | -|------|------|------------|---------|-----------|----------| -| 906 | cnode1 | **tsys9** | D5 | tsys4 | **MOVED from tsys1** | -| 705 | cnode2 | **tsys7** | D2 | tsys4 | **MOVED from tsys6** | -| 106 | cnode3 | tsys1 | D2 | tsys4 | no change | -| 601 | cnode4 | tsys6 | D2 | tsys4 | no change | -| 706 | cnode5 | **tsys7** | **S2** | **tsys5** | **MOVED from tsys6, storage moved D5→S2** | - -**Storage distribution:** - -| Storage server | cnodes | Quorum impact if it fails | -|---------------|--------|--------------------------| -| tsys4 (D2+D5) | **4** (cnode1,2,3,4) | Only cnode5 survives = **QUORUM LOST** | -| tsys5 (S2) | **1** (cnode5) | 4 survive = quorum OK | - -**Progress:** cnode5 is now on tsys5 (was all 5 on tsys4). But 4-of-5 on tsys4 -still means a tsys4 failure loses quorum. **Need 2 more cnodes on tsys5.** - -**Host distribution:** cnodes spread across 4 hosts (tsys1, tsys6, tsys7, -tsys9) -- good host diversity. - -### 5.2 pfv-k8s wnode (worker) placement -- CURRENT - -| VMID | Name | Hypervisor | Storage | NFS Server | RAM | Status | Changed? | -|------|------|------------|---------|-----------|-----|--------|----------| -| 102 | wnode-tsys1 | tsys1 | S2 | tsys5 | 4 GB | **STOPPED** | **NEW** | -| 313 | wnode-tsys3 | tsys3 | D5 | tsys4 | **28 GB** | running | **RAM bumped 20→28** | -| 509 | wnode-tsys5 | tsys5 | D2 | tsys4 | 32 GB | running | no change | -| 100 | wnode-tsys6 | tsys6 | D5 | tsys4 | 32 GB | running | **NOW RUNNING** | -| 701 | wnode-tsys7 | tsys7 | D5 | tsys4 | 32 GB | running | no change | -| 905 | wnode-tsys9 | tsys9 | **S2** | tsys5 | 4 GB | running | **Storage moved S3→S2** | - -**One wnode per host achieved** (tsys1,3,5,6,7,9). wnode-tsys1 is created but -stopped. wnode-tsys6 restarted. - -**Storage distribution:** - -| Storage server | wnodes | Notes | -|---------------|--------|-------| -| tsys4 (D2+D5) | 4 (tsys3,5,6,7) | Still concentrated | -| tsys5 (S2) | 2 (tsys1,tsys9) | Improving | - -### 5.3 Summary: k8s node storage distribution - -| Storage server | cnodes | wnodes | Total k8s nodes | -|---------------|--------|--------|-----------------| -| **tsys4 NFS** | **4 (80%)** | **4 (67%)** | **8 (73%)** | -| **tsys5 NFS** | **1 (20%)** | **2 (33%)** | **3 (27%)** | -| Local SSD/NVMe | 0 | 0 | 0 (0%) | - -**Was 90%/10%. Now 73%/27%.** Improving but still tsys4-heavy. Target: 3 -cnodes on each storage server (60/40 or better) so either server failing -leaves quorum intact. - -### 5.4 Remaining cnode migration needed for etcd quorum survival - -To survive a tsys4 failure with quorum (3 of 5 alive), at least 3 cnodes must -be on tsys5: - -| Action | Effect | -|--------|--------| -| Move cnode3 (D2→S3 on tsys5) | 3 cnodes on tsys5, 2 on tsys4. tsys4 fail = 3 survive | -| Move cnode4 (D2→S2 on tsys5) | Same result, different disk | -| Leave cnode1 and one other on tsys4 | tsys5 fail = 4 survive (OK) | - -**Simplest path:** migrate cnode3 and cnode4 storage to tsys5 (S3 and S2) via -PDM. Then tsys4 failure leaves cnode5 + cnode3 + cnode4 = 3 of 5 = quorum OK. - -### 5.5 Future k8s architecture (next session -- see [K8S.md](K8S.md)) - -The k8s layer will be tackled soon. Key requirements from user: - -- **Platform:** vcluster + Rancher for multi-tenant management -- **Auth:** OIDC to Keycloak (running on Cloudron in Reston, VA production) -- **Workload isolation (vcluster per tenant):** - - RackRental workloads (containerlab network labs) - - Suborbital ITAR (compliance-restricted) - - Suborbital non-ITAR - - Starting Line Productions customer workloads -- **Solar-aware scale-out:** PowerEdge 19xx and 2950 systems (older hardware) - will be brought online during peak solar production for burst capacity. - These older cores/ram supplement the main fleet when power is abundant. -- **WNode sizing:** every Proxmox node will have a wnode. Some nodes will host - both cnodes + wnodes. Worker sizes will vary from small (4 GB, fitting into - leftover host capacity) to large (28-32 GB, consuming most of a host). -- **Friday final audit:** tsys2 will be loaded with Proxmox on Friday, and a - full final audit will be performed at that time (post-NVMe install on tsys5, - post-D3 SSD relocation, post-tsys4 NIC+RAM). - -### 5.4 sectestbed k8s nodes (separate from pfv-k8s) - -tsys5 also hosts a separate sectestbed kubernetes stack using local storage: - -| VMID | Name | Storage | -|------|------|---------| -| 5106 | sectestbed-k8s-cnode | local-nonprod (local HDD) | -| 5107 | sectestbed-k8s-wnode | local-nonprod (local HDD) | - -These are on local storage (good -- no NFS dependency) but on a single host's -single local disk (no redundancy). They are isolated from the pfv-k8s cluster. - ---- - -## 6. Storage Redundancy Analysis - -### 6.1 -01/-02 infrastructure pair audit - -Two -01/-02 pairs exist in the fleet: - -**Pair 1: pfv-netinfra (network infrastructure)** - -| Role | VMID | Host | Storage | NFS Server | -|------|------|------|---------|-----------| -| -01 | 103 | tsys1 | D5 | **tsys4** | -| -02 | 904 | tsys9 | D2 | **tsys4** | - -**Verdict: HOST redundancy OK (different hosts), STORAGE redundancy FAILED.** -Both halves depend on tsys4. If tsys4 goes down, both netinfra VMs lose their -disks. The -02 half should be on an S2/S3/S4 export from tsys5. - -**Pair 2: tsys-ucs (Univention Corporate Server)** - -| Role | VMID | Host | Storage | NFS Server | -|------|------|------|---------|-----------| -| -01 | 108 | tsys1 | D2 | **tsys4** | -| -02 | 902 | tsys9 | D5 | **tsys4** | - -**Verdict: HOST redundancy OK (different hosts), STORAGE redundancy FAILED.** -Same issue. Both halves on tsys4. The -02 half should be on tsys5 storage. - -### 6.2 Redundancy principle for paired VMs - -For any -01/-02 pair to survive a single storage server failure: - -``` --01 VM disk -> tsys4 NFS export (D2/D3/D5) --02 VM disk -> tsys5 NFS export (S2/S3/S4/T5-SSD) -``` - -This ensures that losing either tsys4 or tsys5 takes down only one half of -the pair. Currently, **both pairs fail this test** because both halves are on -tsys4. - -### 6.3 NFS server failure blast radius - -If **tsys4** goes offline (USB NIC failure, disk failure, reboot): - -| Impact | Count | -|--------|-------| -| k8s cnodes that lose storage | 5 of 5 (**etcd quorum lost**) | -| k8s wnodes that lose storage | 4 of 5 | -| Infrastructure VMs that lose storage | 11 of 12 on tsys1 (all on D2/D5) | -| Total VMs that lose storage | **27 of 40 (68%)** | - -If **tsys5** goes offline: - -| Impact | Count | -|--------|-------| -| k8s cnodes that lose storage | 0 of 5 | -| k8s wnodes that lose storage | 1 of 5 | -| Total VMs that lose storage | 1 of 40 (3%) | - -**tsys4 is a massive blast-radius liability. tsys5 is barely used.** -Rebalancing VM storage across both servers dramatically reduces risk. - ---- - -## 7. Local SSD/NVMe Opportunity - -### 7.1 Available fast tiers (currently unused by k8s) - -| Host | Device | Type | Size | Available for VMs? | Currently used by k8s? | -|------|--------|------|------|--------------------|-----------------------| -| tsys4 | D3 (SK hynix SC300, USB) | SSD | 512 GB | Yes (via NFS) | **No** | -| tsys5 | T5-SSD (Samsung 860 PRO) | SSD | 256 GB | Yes (via NFS) | **No** (used by ultix-streaming) | -| tsys5 | **New NVMe (Friday)** | **NVMe** | TBD | **Yes (local or NFS)** | **No** | -| tsys9 | local-lvm (PNY CS900) | SSD | 137 GB free | Yes (local) | **No** | - -### 7.2 Why local storage matters for k8s nodes - -Kubernetes nodes are latency-sensitive in two specific areas: - -1. **etcd (control plane):** etcd writes are synchronous and latency-critical. - On NFS over HDD, every etcd write traverses: VM -> virtio-scsi -> NFS - client -> TCP -> USB dongle (on tsys4) -> ext4 -> spinning disk. Typical - latency: 2-10 ms per write. On local SSD: 0.1-0.5 ms. On NVMe: 0.02-0.1 ms. - This directly affects k8s API responsiveness and pod scheduling speed. - -2. **Container image pulls:** Worker nodes pull container images frequently. - On NFS-over-HDD, image layer extraction is seek-bound and slow. Local SSD - eliminates the network hop and reduces seek time. This matters most for - RackRental/containerlab workloads that spin up containers rapidly. - -### 7.3 Current waste: tsys9 local SSD - -tsys9 has a 250 GB PNY CS900 SSD with 137 GB of LVM-thin space available. -**Zero VMs use it.** All 5 VMs on tsys9 boot from NFS. The local SSD sits -idle. wnode-tsys9 (VM 905) would benefit significantly from local SSD -- -its disk is currently on S3 (tsys5 NFS over a Seagate HDD). - -### 7.4 Upcoming opportunity: tsys5 NVMe (Friday) - -The PCI NVMe being added to tsys5 will be the fastest storage tier in the -fleet. Two placement options: - -**Option A: NFS-export the NVMe (shared).** All hosts can use it. Good for -VMs that might need migration. Adds the NFS/network overhead back. - -**Option B: Local-only on tsys5.** VMs on tsys5 get full NVMe speed with no -network overhead. Best for k8s wnode-tsys5 and sectestbed VMs. Cannot be -accessed from other hosts. - -**Recommendation:** Option B (local-only). k8s worker nodes do not need -shared storage -- pods are ephemeral and reschedule on failure. The NVMe -should be formatted as a Proxmox directory storage (or LVM-thin) on tsys5 -and used for local VM images. - ---- - -## 8. Role Alignment Audit - -Per the user's intended role taxonomy: tsys1/9 = infrastructure + k8s control -plane; tsys2/3/6/7 = k8s workers; tsys4/5 = storage. Cnodes on tsys1/9 is -**correct by design** (keeps heavy hosts free for workers). - -### 8.1 VMs that need to move - -| VMID | Name | Current host | Issue | Target | -|------|------|-------------|-------|--------| -| 509 | pfv-k8s-wnode-tsys5 | tsys5 (storage) | Worker on storage host | tsys7 or tsys2 (when online) | -| 905 | pfv-k8s-wnode-tsys9 | tsys9 (infra) | Can stay if small; user decides | tsys9 OK if small wnode | - -### 8.2 Host capacity for k8s nodes - -| Host | Role | Current k8s nodes | k8s RAM used | RAM total | Headroom | -|------|------|-------------------|-------------|-----------|----------| -| tsys1 | Infra+k8s ctrl | 2 cnodes | 8 GB | 32 GB | ~12 GB (after 11 infra VMs) | -| tsys3 | K8s worker | 1 wnode | 20 GB | 32 GB | ~12 GB | -| tsys6 | K8s worker | 3 cnodes + 1 wnode (stopped) | 12 GB | 128 GB | **~116 GB** | -| tsys7 | K8s worker | 1 wnode | 32 GB | 192 GB | **~160 GB** | -| tsys9 | Infra+k8s ctrl | 1 wnode | 4 GB | 24 GB | ~12 GB (after 4 infra VMs) | -| tsys2 | K8s worker | 0 (incoming) | 0 | 32 GB | ~32 GB | - -**tsys6 and tsys7 are dramatically underutilized** -- 116 GB and 160 GB of -free RAM respectively. They should be the primary targets for heavy worker -nodes and ETL/HPC workloads. - -### 8.3 tsys6 wnode-tsys6 is stopped - -VM 100 (pfv-k8s-wnode-tsys6) is stopped on tsys6. Its disk is on D5 (tsys4 -NFS). This wnode should be restarted (or recreated on local-lvm) to restore -cluster capacity. - ---- - -## 9. Network Findings - -### 9.1 tsys9 storage NIC is a USB dongle (new finding) - -Validating tsys9 revealed that its storage network interface -(`enx9c69d36a5b6c`) is USB-attached (`parentbus usb`). This is the same -anti-pattern as tsys4. The onboard Intel NIC (`enp0s31f6`) is used for -management; storage uses the USB adapter. - -**Impact:** Same as tsys4 -- achieves line rate but is susceptible to cable -wobble, ESD, and USB controller resets. For an infrastructure host with 5 -VMs, this is a reliability risk. - -**Mitigation:** tsys9 is an OptiPlex 7080 SFF -- it has PCIe slots. A -PCIe NIC would eliminate this risk (same recommendation as tsys4). - -### 9.2 tsys4 and tsys5 still blocked (Friday hardware work) - -| Host | Blocker | Staged fix | -|------|---------|-----------| -| tsys4 | USB cdc_ncm storage NIC | PCIe NIC install + RAM upgrade (16 to 64 GB) | -| tsys5 | bond0 broken (1 of 2 slaves) | Plug 2nd ethernet cable + apply layer3+4 hash | - -### 9.3 LACP resolved on tsys6/tsys7 - -tsys6 to tsys7 storage path now measures **1.83 Gbps** (was 943 Mbps). -The switch LACP hash change took effect after renegotiation. The 56-106K -retransmits on this path are confirmed to be non-lossy multi-flow TCP-over- -LACP overhead. See `RESULTS.md` (not yet created) for the full analysis. - -### 9.4 NFS nconnect=4 + noatime confirmed active - -All hosts (including tsys9) show `nconnect=4,noatime` in their NFS mount -options. Each host maintains 4 TCP connections per NFS mount to each storage -server. This was the Tier 0 tuning item from the performance optimization -engagement and is confirmed working cluster-wide. - ---- - -## 10. Recommendations - -**No changes have been made. These are assessment-only recommendations.** -See `K8S.md` for the detailed kubernetes architecture deep-dive. - -### 10.1 CRITICAL: Critical infrastructure VM placement (netinfra, UCS) - -These are the most critical production VMs in the fleet. They must survive -any single-point failure (host, storage server, or disk). - -**Design principle for -01/-02 HA pairs:** -- Different hypervisors (already satisfied: tsys1 vs tsys9) -- Different storage servers (currently FAILED: all on tsys4) -- Prefer SSD for latency-sensitive services - -**Recommended placement:** - -| VM | Host | Storage | Tier | Free space | Rationale | -|----|------|---------|------|-----------|-----------| -| **netinfra-01** (103) | tsys1 | **D5 (tsys4 HDD)** | NFS-HDD | 1.5 TB | DNS/DHCP/NTP = minimal I/O. Stays put. | -| **netinfra-02** (904) | tsys9 | **S3 (tsys5 HDD)** | NFS-HDD | 870 GB | Move from D2. Cross-server redundancy. Minimal I/O. | -| **ucs-01** (108) | tsys1 | **D2 (tsys4 HDD)** | NFS-HDD | 2.4 TB | Stays put. LDAP/AD does not need SSD. | -| **ucs-02** (902) | tsys9 | **S2 (tsys5 HDD)** | NFS-HDD | 870 GB | Move from D5. Cross-server redundancy. No SSD needed. | - -**Failure survival matrix (all single-point failures):** - -| Failure | netinfra-01 | netinfra-02 | ucs-01 | ucs-02 | Result | -|---------|-------------|-------------|--------|--------|--------| -| tsys4 dies | dies (D5) | **alive** (S3) | dies (D2) | **alive** (S2) | netinfra-02 + ucs-02 alive | -| tsys5 dies | **alive** (D5) | dies (S3) | **alive** (D2) | dies (S2) | netinfra-01 + ucs-01 alive | -| tsys1 dies | dies | **alive** | dies | **alive** | -02 pair survives | -| tsys9 dies | **alive** | dies | **alive** | dies | -01 pair survives | -| Any single disk | **all 4 on different disks/servers** | **all 4 alive** | | | | - -**Why this works:** Every row has at least one netinfra and one UCS alive. -The network (DNS/DHCP) and directory (AD/LDAP) services never go fully dark. - -**D3 SSD repurposed:** With UCS staying on HDD, the D3 SSD (moving to tsys5 -Friday) is freed for latency-sensitive workloads that actually benefit from -SSD -- sectestbed k8s nodes, CI/CD artifact cache, or a container image -registry mirror. Not infrastructure VMs. - -### 10.2 CRITICAL: Cnode (control plane) storage split - -**Problem:** All 5 cnodes store disks on tsys4. D2 disk failure loses etcd -quorum (3 of 5 cnodes share D2). - -**Target: cnodes weighted toward tsys1/tsys9 (lightweight hosts), freeing -tsys6/tsys7 for heavy workers. Storage splits across tsys4 and tsys5.** - -| cnode | Current host | **Target host** | Current storage | **Target storage** | Rationale | -|-------|-------------|----------------|----------------|-------------------|-----------| -| cnode1 (107) | tsys1 | **tsys1** (stays) | D5 (tsys4) | **D5 (tsys4)** -- no change | Already correct | -| cnode3 (106) | tsys1 | **tsys1** (stays) | D2 (tsys4) | **S2 (tsys5)** -- **MOVE disk** | Split storage to tsys5 | -| cnode2 (603) | tsys6 | **tsys9** | D2 (tsys4) | **D2 (tsys4)** -- no disk change | Free tsys6 for heavy workers | -| cnode4 (601) | tsys6 | **tsys9** | D2 (tsys4) | **D5 (tsys4)** -- spread disk | Free tsys6; spread off D2 | -| cnode5 (602) | tsys6 | **tsys3** | D5 (tsys4) | **S3 (tsys5)** -- **MOVE disk** | Free tsys6; split storage to tsys5 | - -**Result after changes:** - -| Host | cnodes | Storage server | -|------|--------|---------------| -| tsys1 | cnode1 (D5), cnode3 (S2) | tsys4 + tsys5 | -| tsys9 | cnode2 (D2), cnode4 (D5) | tsys4 | -| tsys3 | cnode5 (S3) | tsys5 | - -- 3 cnodes on tsys4 storage, 2 on tsys5. Either storage server can fail and - etcd keeps quorum (3 of 5 survive). -- D2 has 1 cnode (was 3). D5 has 2. S2 and S3 have 1 each. No single disk - holds more than 2 cnodes. -- Cnodes now on 3 hosts (tsys1, tsys9, tsys3). Any single host failure leaves - at least 3 cnodes alive. -- **tsys6 and tsys7 are fully freed** for heavy worker nodes. - -Note: tsys1 RAM is tight (32 GB, 11 infra VMs). Adding 0 new cnodes (keeping -the 2 already there) is feasible with KSM. tsys9 (24 GB) has room for 2 -cnodes (8 GB). tsys3 (32 GB) has room for 1 cnode (4 GB) alongside its wnode. - -### 10.3 HIGH: Wnode distribution -- one per host, tiered storage - -**Target: one wnode per hypervisor host, using local storage where possible -and NFS-HDD for bulk data.** - -| wnode | Host | Boot disk (OS+containers) | Data disk (bulk/ETL) | Rationale | -|-------|------|--------------------------|---------------------|-----------| -| wnode-tsys1 | tsys1 | D5 (tsys4 NFS) | -- | Small wnode on infra host. Minimal capacity. | -| wnode-tsys3 | tsys3 | **local-lvm (349 GB NVMe)** | S3 (NFS) | Move from NFS to **NVMe** (Samsung PM961). Fastest boot disk after tsys5/2. | -| wnode-tsys6 | tsys6 | D2 (tsys4 NFS) | -- | **Stays on NFS.** local-lvm is USB 2.0 HDD (~30 MB/s) -- slower than NFS. | -| wnode-tsys7 | tsys7 | D5 (tsys4 NFS) | -- | **Stays on NFS.** local-lvm is USB 2.0 HDD (~30 MB/s) -- slower than NFS. | -| wnode-tsys9 | tsys9 | **local-lvm (136 GB SSD)** | S2 (NFS) | Move from NFS to local SSD. Fast boot, NFS for bulk. | -| wnode-tsys2 | tsys2 | **NVMe (Samsung 960 PRO 512GB)** | **SATA SSD (Samsung 850 EVO 1TB)** | **Best storage of any wnode.** No NFS needed -- 1.5 TB local SSD. | -| wnode-tsys5 | tsys5 | **new NVMe (local)** | local-nonprod (HDD) | **Fastest wnode in fleet.** HPC/ETL workloads land here. | - -**Storage tiering strategy per wnode:** - -Each wnode gets two disk tiers mapped to k8s StorageClasses: - -1. **Boot + container runtime** (local-lvm or NVMe): OS, kubelet, container - images, ephemeral storage. This is where local SSD/NVMe shines -- container - image pulls and layer extraction are seek-bound and benefit enormously from - low-latency storage. - -2. **Bulk data** (NFS-HDD via D2/D5/S2/S3): weather/GIS datasets, ETL staging - areas, large files that do not fit on local storage. Mounted as a second - disk in the VM and exposed to k8s as a StorageClass. - -This maps to two k8s StorageClasses: -- `local-storage`: bound to the wnode's boot/local disk (fast, ephemeral) -- `nfs-bulk`: bound to NFS exports (slow, persistent, large capacity) - -### 10.4 HIGH: Dedicate D3 SSD exclusively to k8s scratch - -**Storage philosophy (user directive): NVMe/SSD is for k8s worker scratch -space and ultix-streaming (developer workstation running "cluster of 1" -pre-production jobs). Spinning rust hosts all other infrastructure VMs** -(UCS, netinfra, LibreNMS, SIEM, etc.). - -SSD allocation after Friday: -- **D3 (tsys5 SAS, 445 GB free, 0% used)** -- dedicated to k8s scratch via - the `nfs-ssd` StorageClass (etcd, container cache, ephemeral volumes). -- **T5-SSD (tsys5 SAS, 140 GB free)** -- ultix-streaming (VM 5111) stays here - (developer workstation, runs single-node test jobs before k8s). Remaining - 140 GB available for k8s. - -**Deep-dive on exact k8s scratch allocation is deferred to the next session** -(K8S.md) once we know the k8s distribution, job scheduler, and workload mix. - -### 10.5 MEDIUM: Restart wnode-tsys6 - -VM 100 (pfv-k8s-wnode-tsys6) is stopped on tsys6. Recreate on local-lvm -(1.7 TB free) instead of D5 NFS. This restores cluster capacity and moves -the boot disk to local storage simultaneously. - -### 10.6 MEDIUM: tsys5 NVMe placement (Friday) - -**Recommendation: local-only on tsys5, formatted as Proxmox LVM-thin or -directory storage.** - -Use for: -- wnode-tsys5 boot disk (primary beneficiary -- HPC/ETL workloads) -- sectestbed VMs that need fast scratch space -- Not NFS-exported (avoid adding network overhead to the fastest tier) - -### 10.7 LOW: Add PCIe NIC to tsys9 - -tsys9's storage NIC is a USB dongle. tsys9 is an OptiPlex 7080 with PCIe -slots. A $150 PCIe NIC eliminates the USB reliability risk. - -### 10.8 LOW: Standardize PVE/kernel versions - -tsys3 is on PVE kernel 7.0.14; others on 6.17.x. PVE-manager versions vary -(9.1.1 / 9.1.5 / 9.2.5). Standardize in a maintenance window. - ---- - -## 11. Hardware End-of-Support Exposure - -| Host | EOS date | Years past | Form factor | -|------|----------|-----------|-------------| -| pfv-tsys6 (R610) | 2013-05 | 13.2 | 1U server | -| pfv-tsys5 (T7500) | 2014-12 | 11.7 | Workstation | -| pfv-tsys4 (T1700) | 2018-03 | 8.4 | Workstation | -| pfv-tsys7 (R620) | 2019-03 | 7.4 | 1U server | -| pfv-tsys1 (9020) | 2019-07 | 7.0 | SFF desktop | -| pfv-tsys3 (7510) | 2020-07 | 6.0 | Laptop | -| pfv-tsys2 (5520) | TBD | -- | Laptop | -| **pfv-tsys9 (7080)** | **2024-02** | **2.4** | **SFF desktop (only supported)** | - -**6 of 8 hosts are past vendor end-of-support.** Only tsys9 is still covered. -The two actual rack servers (R610, R620) are the most overdue for replacement. -This belongs in the next budget cycle. - ---- - -## 12. Open Items - -### 12.1 Immediate (do today via PDM) - -1. **Migrate ucs-02** (VM 902) from D5 (tsys4) to S2 (tsys5) for cross-server - redundancy. UCS stays on HDD. -2. **Migrate netinfra-02** (VM 904) from D2 (tsys4) to S3 (tsys5). -3. **Migrate cnode3** (VM 106) from D2 (tsys4) to S3 (tsys5) -- etcd quorum. -4. **Migrate cnode4** (VM 601) from D2 (tsys4) to S2 (tsys5) -- etcd quorum. -5. **Start wnode-tsys1** (VM 102) if the cluster needs the capacity. - -### 12.2 Friday maintenance window (user action) - -1. **tsys4:** Install PCIe NIC (replace USB dongle), add RAM (16 to 64 GB), - reconfigure `/etc/network/interfaces`, reboot. -2. **tsys5:** Plug 2nd ethernet cable, verify bond0, apply layer3+4 hash, - install PCI NVMe, relocate D3 SSD from tsys4 USB to tsys5 SAS port, - format NVMe as local storage, reboot. -3. **tsys2:** Load Proxmox (replacing Windows 10). -4. **Final audit:** Re-run `deploy-check.sh` across all hosts including tsys2. - -### 12.3 Post-Friday validation - -1. Re-run iperf matrix: `./iperf-full-matrix.sh` -2. Validate tsys4 and tsys5: `./validate-fixes.sh pfv-tsys4 && ./validate-fixes.sh pfv-tsys5` -3. Run `scripts/check.sh` on tsys2 once Proxmox is loaded. -4. Update PROJECT.md with post-hardware numbers. - -### 12.4 Future: Kubernetes deep-dive (see [K8S.md](K8S.md)) - -Next major workstream. Requirements captured: -- **vcluster + Rancher** for multi-tenant k8s management -- **OIDC auth** to Keycloak (on Cloudron, Reston VA production) -- **Workload isolation** via separate vclusters: - - RackRental (containerlab) - - Suborbital ITAR - - Suborbital non-ITAR - - Starting Line Productions customer workloads -- **Solar-aware scale-out:** PowerEdge 19xx + 2950 systems brought online - during peak solar production for burst capacity -- **Every host gets a wnode** (variable sizing: small 4 GB to large 32 GB) -- **SSD/NVMe reserved for k8s scratch** (plus ultix-streaming exception) -- **Spinning rust for all infrastructure VMs** - -### 12.5 Data gaps - -| Gap | How to close | -|-----|--------------| -| tsys5 SDR/parallel-port workload dependency | Confirm what uses the SDR | -| tsys2 post-Proxmox baseline | Run `check.sh` after Friday install | -| tsys5 NVMe size and model | Confirm after Friday installation | -| tsys3 thermal state (laptop in rack) | Check `sensors` on next maintenance | -| PowerEdge 19xx/2950 inventory | When solar scale-out is planned | - -### 12.6 Scripts and tools available - -| Script | Purpose | -|--------|---------| -| `scripts/check.sh` | Read-only data collector (run on hosts) | -| `scripts/apply-tunings.sh` | Apply all Tier 0 tunings (dry-run/apply/rollback) | -| `scripts/fix-bond-nfs.sh` | Fix NFS options + bond hash | -| `validate-fixes.sh` | Read-only validation of all applied changes | -| `iperf-full-matrix.sh` | Full iperf test suite (mgmt + storage) | -| `deploy-check.sh` | Deploy check.sh to all hosts via SSH (now includes tsys9) | +# proxmox/docs/PROJECT.md + +> **Documentation moved to Discourse — the canonical source of truth.** +> +> **Comprehensive fleet report** +> +> **Read it here:** https://community.turnsys.com/t/304 +> +> *Migrated 2026-08-06. This file is kept as a pointer for git-browsing context. +> Do not update content here — edit the Discourse wiki topic instead.* diff --git a/proxmox/docs/STORAGE-ARCHITECTURE.md b/proxmox/docs/STORAGE-ARCHITECTURE.md index f284099..1389ec6 100644 --- a/proxmox/docs/STORAGE-ARCHITECTURE.md +++ b/proxmox/docs/STORAGE-ARCHITECTURE.md @@ -1,196 +1,10 @@ -# Storage Architecture +# proxmox/docs/STORAGE-ARCHITECTURE.md -> **Complete storage reference for the PFVCluster.** Covers physical disks, -> NFS exports/mounts, Proxmox storage IDs, tier definitions, and per-spindle -> VM placement. Data gathered 2026-07-30 via live host probes. +> **Documentation moved to Discourse — the canonical source of truth.** > -> Companion docs: [`NETWORK-TOPOLOGY.md`](NETWORK-TOPOLOGY.md), -> [`HOST-REFERENCE.md`](HOST-REFERENCE.md), [`VM-INVENTORY.md`](VM-INVENTORY.md). -> Findings/analysis: [`AUDIT-2026-07-30.md`](AUDIT-2026-07-30.md). - ---- - -## 1. Storage tier definitions - -| Tier | Definition | Typical speed | Reliability | Use case | -|------|-----------|---------------|-------------|----------| -| **Tier 1** | NVMe or SSD on internal SATA/SAS/PCIe | 200-3000 MB/s | High | OS drives, high-IOPS workloads, build scratch | -| **Tier 2** | HDD on internal SATA/SAS | 80-150 MB/s | Moderate | Production VM OS drives, general-purpose storage | -| **Tier 3** | HDD/SSD on USB adapter / "toaster" | 30-60 MB/s | Low (USB) | Bulk, backup, scratch only. Never for production OS drives. | - -**Policy (from [`CAPACITY-GOALS.md`](CAPACITY-GOALS.md) §6):** Local disk -storage is a scratch tier, not a boot/OS tier. VM OS drives stay on NFS -(tsys4/tsys5) where redundancy is achieved by spreading across physical disks -and HA pairs. Local storage is for ephemeral data where loss on node failure -is acceptable. - ---- - -## 2. Physical disk inventory - -### tsys4 -- NFS storage server + PBS (7 disks) - -| Device | Model | Size | RPM | Interface | Mount | NFS ID | Tier | SMART | Used | -|--------|-------|------|-----|-----------|-------|--------|------|-------|------| -| sda | Hitachi HDS723020BLA642 | 2.0 TB | 7200 | SATA | /mnt/albert | (not exported) | **Tier 2** | PASSED, 0 reallocated | 0% | -| sdb | HITACHI HUA723020ALA640 | 2.0 TB | 7200 | SATA | /mnt/tsys4/D5 | **D5** | Tier 2 | PASSED, 0 reallocated | 13% (220 GB) | -| sdc | WDC WD30EFRX (Red) | 3.0 TB | 5400 | SATA | /mnt/tsys4/D2 | **D2** | Tier 2 | PASSED, 0 reallocated | 7% (170 GB) | -| sdd | WDC WD10EZRX | 1.0 TB | -- | SATA | (pve boot/swap/lvm) | -- | Tier 2 | PASSED, 1 reallocated (sdf has 1) | OS disk | -| sde | ST500LM021 (Seagate) | 500 GB | 7200 | **USB** | /mnt/tsys4/D3 | **D3** | **Tier 3** | PASSED | 0% | -| sdf | WDC WD10EARS | 1.0 TB | -- | **USB** | /mnt/tsys4/D1 | **D1** | **Tier 3** | PASSED, 1 reallocated sector | 0% | -| sdg | WDC WD50NDZW (easystore) | 5.0 TB | 5400 | **USB** | /mnt/backup | (not exported) | **Tier 3** | PASSED | 0% | - -**Changes from previous audit:** -- **D3 changed:** was SK hynix SC300 512GB SSD → now Seagate ST500LM021 500GB HDD (still USB). Demoted from unreliable-SSD to Tier 3 HDD. -- **/mnt/albert added:** Hitachi 2TB internal SATA. Not exported, not in Proxmox. Unused Tier 2. -- **/mnt/backup added:** WD 5TB USB easystore. Bulk backup only. -- **D7 removed:** Previously held DellOpenManageEnterprise. Now gone. - -### tsys5 -- NFS storage server + sandboxes (7 disks) - -| Device | Model | Size | RPM | Interface | Mount | NFS/Local ID | Tier | SMART | Used | -|--------|-------|------|-----|-----------|-------|-------------|------|-------|------| -| sda | ST1000VN002 (IronWolf) | 1.0 TB | 5900 | SATA | **(not mounted)** | was S3 | Tier 2 | PASSED | -- | -| sdb | SK hynix SC300B | 512 GB | SSD | SATA | **(not mounted)** | -- | **Tier 1** | PASSED | **NEW, not configured** | -| sdc | HITACHI HUS724020ALA640 | 2.0 TB | 7200 | SAS | (pve boot/swap/lvm) | -- | Tier 2 | PASSED | OS disk | -| sdd | Samsung SSD 860 PRO | 256 GB | SSD | SAS | /mnt/pfv-tsys5/ssd | `ssd` / `SSD` dir (local) | **Tier 1** | PASSED | **93% full!** | -| sde | ST1000VN002 (IronWolf) | 1.0 TB | 5900 | SAS | /mnt/pfv-tsys5/S1 | **S1** + `local-nonprod` | Tier 2 | PASSED | 11% (96 GB) | -| sdf | ST1000VN002 (IronWolf) | 1.0 TB | 5900 | SAS | /mnt/pfv-tsys5/S2 | **S2** | Tier 2 | PASSED | 4% (32 GB) | -| **nvme0n1** | **CT500P1SSD8 (Crucial)** | **500 GB** | **NVMe** | **PCIe** | **(not mounted)** | -- | **Tier 1** | PASSED, 0 integrity errors, 34K hrs | **NEW, not configured** | - -**Changes from previous audit:** -- **NVMe added:** Crucial CT500P1 500GB. True NVMe -- highest-performance local storage in the fleet. Not mounted, not in fstab, not in storage.cfg. -- **SK hynix SSD relocated:** Moved from tsys4 USB enclosure to tsys5 internal SATA. Not mounted or configured. -- **S3 unmounted:** sda (was S3) is present but fstab entry is commented out. Stale export remains in /etc/exports. -- **S4 gone:** Old Toshiba 458GB no longer present. fstab entry commented. storage.cfg has broken `dir: S4` entry. -- **S1/S2 drives changed:** Both are now ST1000VN002 IronWolf 1TB (5900 rpm). -- **SSD nearly full:** Samsung 860 PRO at 93% (206/234 GB) -- needs attention. -- **Boot disk changed:** HITACHI HUS724020ALA640 2TB (was Hitachi 2TB in old audit, now confirmed model). - ---- - -## 3. NFS export → Proxmox storage ID map - -```mermaid -graph LR - subgraph "tsys4 -- NFS Server" - D2["D2: /mnt/tsys4/D2
WDC Red 3TB
Tier 2 | 7% used"] - D5["D5: /mnt/tsys4/D5
Hitachi 2TB
Tier 2 | 13% used"] - D1["D1: /mnt/tsys4/D1
WD 1TB USB
Tier 3 | 0% used"] - D3["D3: /mnt/tsys4/D3
Seagate 500GB USB
Tier 3 | 0% used"] - end - - subgraph "tsys5 -- NFS Server" - S1["S1: /mnt/pfv-tsys5/S1
IronWolf 1TB
Tier 2 | 11% used"] - S2["S2: /mnt/pfv-tsys5/S2
IronWolf 1TB
Tier 2 | 4% used"] - S3stale["S3: (stale export)
disk not mounted"] - end - - subgraph "All hosts -- NFS Clients" - MNT["/mnt/pve/D1 D2 D3 D5
/mnt/pve/S1 S2 S3 S4
/mnt/pve/T5-SSD"] - end - - D2 -->|NFS v4.2| MNT - D5 -->|NFS v4.2| MNT - D1 -->|NFS v4.2| MNT - D3 -->|NFS v4.2| MNT - S1 -->|NFS v4.2| MNT - S2 -->|NFS v4.2| MNT - S3stale -.->|broken| MNT -``` - -### NFS export details - -| Server | Export path | NFS ID | Protocol | Options (client-side) | -|--------|-------------|--------|----------|----------------------| -| tsys4 | /mnt/tsys4/D1 | D1 | NFSv4.2 | nconnect=4,noatime,rsize=1048576,wsize=1048576,hard | -| tsys4 | /mnt/tsys4/D2 | D2 | NFSv4.2 | nconnect=4,noatime,rsize=1048576,wsize=1048576,hard | -| tsys4 | /mnt/tsys4/D3 | D3 | NFSv4.2 | nconnect=4,noatime,rsize=1048576,wsize=1048576,hard | -| tsys4 | /mnt/tsys4/D5 | D5 | NFSv4.2 | nconnect=4,noatime,rsize=1048576,wsize=1048576,hard | -| tsys5 | /mnt/pfv-tsys5/S1 | S1 | NFSv4.2 | nconnect=4,noatime,rsize=1048576,wsize=1048576,hard | -| tsys5 | /mnt/pfv-tsys5/S2 | S2 | NFSv4.2 | nconnect=4,noatime,rsize=1048576,wsize=1048576,hard | -| tsys5 | /mnt/pfv-tsys5/S3 | S3 | NFSv4.2 | **stale** (disk not mounted on server) | -| tsys5 | /mnt/pfv-tsys5/ssd | T5-SSD | NFSv4.2 | nconnect=4,noatime,rsize=1048576,wsize=1048576,hard | -| tsys5 | /mnt/pfv-tsys5/S4 | S4 | -- | **broken** (disk gone, storage.cfg entry stale) | - -> **Note:** tsys5 is the only host where NFS client mounts (D2/D3/D5 from -> tsys4) are **missing nconnect=4**. All other client hosts have nconnect on -> all mounts. - ---- - -## 4. Per-spindle VM placement (running VMs only) - -> Shows every running VM's primary disk and the physical spindle it lives on. -> This is the IOPS distribution map -- the key to storage balancing. - -```mermaid -graph TB - subgraph "tsys4 spindles" - D2["D2 -- WDC Red 3TB
*** 13 VMs -- WORST HOTSPOT"] - D5["D5 -- Hitachi 2TB
4 VMs"] - D1["D1 -- WD 1TB USB
0 VMs (empty)"] - D3["D3 -- Seagate 500GB USB
0 VMs (empty)"] - ALB["/mnt/albert -- Hitachi 2TB
0 VMs (not in Proxmox)"] - end - - subgraph "tsys5 spindles" - S1["S1 -- IronWolf 1TB
1 NFS VM + 12 local stopped"] - S2["S2 -- IronWolf 1TB
5 VMs"] - SSD["SSD -- Samsung 860 PRO
1 VM (93% full!)"] - NVME["NVMe -- Crucial CT500P1
NOT CONFIGURED"] - HYNIX["SK hynix SSD 512GB
NOT CONFIGURED"] - end - - D2 --- D2vms["100 101 104 105 108
114 117 509 600 702
704 901 904"] - D5 --- D5vms["106 313 600(d0) 903"] - S1 --- S1vms["702 hfnoc-uisp"] - S2 --- S2vms["103 703 705 902 905"] - SSD --- SSDvms["5111 ultix-streaming"] -``` - -### Detailed per-spindle load - -| Spindle | Server | NFS ID | Tier | Running VMs | VMIDs | Notes | -|---------|--------|--------|------|-------------|-------|-------| -| **D2** (WDC Red 3TB) | tsys4 | D2 | Tier 2 | **13** | 100,101,104,105,108,114,117,509,600,702,704,901,904 | **Worst IOPS hotspot** | -| D5 (Hitachi 2TB) | tsys4 | D5 | Tier 2 | 4 | 106,313,600(disk0),903 | Moderate | -| S1 (IronWolf 1TB) | tsys5 | S1 + local-nonprod | Tier 2 | 1 (NFS) + 0 (local, 12 stopped) | 702 | Low; 12 stopped sandboxes on local-nonprod | -| S2 (IronWolf 1TB) | tsys5 | S2 | Tier 2 | 5 | 103,703,705,902,905 | Moderate | -| ssd (Samsung 860 PRO) | tsys5 | SSD dir | Tier 1 | 1 | 5111 | **93% full** | -| D1 (WD 1TB USB) | tsys4 | D1 | Tier 3 | 0 | -- | Empty | -| D3 (Seagate 500GB USB) | tsys4 | D3 | Tier 3 | 0 | -- | Empty | -| /mnt/albert (Hitachi 2TB) | tsys4 | (none) | Tier 2 | 0 | -- | Not in Proxmox | -| NVMe (CT500P1 500GB) | tsys5 | (none) | Tier 1 | 0 | -- | **Not configured** | -| SK hynix SSD (512GB) | tsys5 | (none) | Tier 1 | 0 | -- | **Not configured** | - -### Local-only VM disks (not on NFS) - -| Host | Storage ID | Disk | VMs | Notes | -|------|-----------|------|-----|-------| -| tsys4 | local-lvm | sdd (1TB internal) | 400 (PBS) | Proxmox boot/OS disk | -| tsys5 | local-lvm | sdc (2TB internal) | 5102,5103,5104,5112 | Mixed: stopped sandboxes + ultix-offstage | -| tsys5 | local-nonprod (=S1) | sde (IronWolf 1TB) | 5101,5105,5106-5109,515,53100-53103,6000,51010-51014 | All STOPPED sandboxes. Same physical disk as S1 NFS. | -| tsys5 | SSD dir | sdd (Samsung 860 PRO) | 5111 | ultix-streaming. 93% full. | - ---- - -## 5. Storage capacity summary - -| Spindle | Size | Used | Available | % Used | -|---------|------|------|-----------|--------| -| D2 (WDC Red 3TB) | 2.7 TB | 170 GB | 2.4 TB | 7% | -| D5 (Hitachi 2TB) | 1.8 TB | 220 GB | 1.5 TB | 13% | -| D1 (WD 1TB USB) | 916 GB | 2 MB | 870 GB | 0% | -| D3 (Seagate 500GB USB) | 458 GB | 2 MB | 435 GB | 0% | -| /mnt/albert (Hitachi 2TB) | 1.8 TB | 2 MB | 1.7 TB | 0% | -| /mnt/backup (WD 5TB USB) | 4.6 TB | 2 MB | 4.3 TB | 0% | -| S1 (IronWolf 1TB) | 916 GB | 96 GB | 774 GB | 11% | -| S2 (IronWolf 1TB) | 916 GB | 32 GB | 838 GB | 4% | -| SSD (Samsung 860 PRO) | 234 GB | 206 GB | **16 GB** | **93%** | -| NVMe (CT500P1) | 466 GB | -- | -- | Not configured | -| SK hynix SSD | 477 GB | -- | -- | Not configured | - -**Total Tier 2 available (NFS production):** ~5.5 TB unused across D2/D5/S1/S2 -**Total Tier 3 available (USB):** ~6 TB unused across D1/D3/backup -**Total Tier 1 available (unconfigured):** ~940 GB across NVMe + SK hynix SSD +> **Storage reference: tiers, disk inventory, NFS exports, VM placement** +> +> **Read it here:** https://community.turnsys.com/t/300 +> +> *Migrated 2026-08-06. This file is kept as a pointer for git-browsing context. +> Do not update content here — edit the Discourse wiki topic instead.* diff --git a/proxmox/docs/TODO.md b/proxmox/docs/TODO.md index 2a5eba3..c14f1d1 100644 --- a/proxmox/docs/TODO.md +++ b/proxmox/docs/TODO.md @@ -1,242 +1,10 @@ -# TODO.md — Pending User Actions +# proxmox/docs/TODO.md -**Date:** 2026-07-27 -**Items needing user input or physical action.** - ---- - -## 1. tsys2 Windows hardware inventory (run on the Windows host) - -pfv-tsys2 is currently Windows 10. Before rebuilding it as Proxmox, gather -hardware data so the architecture plan can account for it. - -### Option A: PowerShell (recommended — single command, copy-paste output) - -Open **PowerShell as Administrator** and run: - -```powershell -# Full hardware inventory in one shot -Write-Output "=== COMPUTER ===" -Get-CimInstance Win32_ComputerSystem | Select-Object Manufacturer, Model, SystemType, TotalPhysicalMemory | Format-List - -Write-Output "`n=== CPU ===" -Get-CimInstance Win32_Processor | Select-Object Name, NumberOfCores, NumberOfLogicalProcessors, MaxClockSpeed | Format-List - -Write-Output "`n=== MEMORY STICKS ===" -Get-CimInstance Win32_PhysicalMemory | Select-Object Manufacturer, PartNumber, Capacity, Speed, ConfiguredClockSpeed, DeviceLocator, FormFactor | Format-Table -AutoSize - -Write-Output "`n=== DISKS ===" -Get-PhysicalDisk | Select-Object FriendlyName, MediaType, BusType, Size, SpindleSpeed | Format-Table -AutoSize - -Write-Output "`n=== DISK PARTITIONS ===" -Get-Disk | Select-Object Number, FriendlyName, Size, PartitionStyle, OperationalStatus | Format-Table -AutoSize - -Write-Output "`n=== NETWORK ADAPTERS ===" -Get-NetAdapter | Select-Object Name, InterfaceDescription, Status, LinkSpeed, MacAddress | Format-Table -AutoSize - -Write-Output "`n=== GPU(s) ===" -Get-CimInstance Win32_VideoController | Select-Object Name, AdapterRAM, DriverVersion, VideoProcessor | Format-List - -Write-Output "`n=== PCIe SLOTS ===" -Get-CimInstance Win32_SystemSlot | Select-Object SlotDesignation, CurrentUsage, Status | Format-Table -AutoSize - -Write-Output "`n=== USB DEVICES (storage + network only) ===" -Get-PnpDevice -PresentOnly | Where-Object { $_.Class -in @('DiskDrive','Net','USB') } | Select-Object Class, FriendlyName, Status | Format-Table -AutoSize -``` - -Copy the full output into a file (e.g., `tsys2-hardware.txt`) or paste it -directly into the chat. - -### Option B: Command Prompt (cmd.exe) fallbacks - -If PowerShell is unavailable for some reason, these cmd commands give a -subset: - -```cmd -:: Computer model and serial -wmic computersystem get manufacturer,model -wmic bios get serialnumber - -:: CPU -wmic cpu get name,numberofcores,numberoflogicalprocessors,maxclockspeed - -:: RAM (total) -wmic computersystem get totalphysicalmemory - -:: RAM sticks (per-slot detail) -wmic memorychip get manufacturer,capacity,speed,partnumber,devicelocator - -:: Disks -wmic diskdrive get model,size,interfacetype,mediatype - -:: Network adapters -wmic nic where netenabled=true get name,speed,macaddress - -:: GPU -wmic path win32_videocontroller get name,adapterram,driverversion -``` - -### What I'm looking for - -- **Disk inventory**: Are there any SSDs/NVMe available locally? (Determines - whether wnode-tsys2 can use local storage like the other wnodes.) -- **Network adapters**: How many onboard NICs? Model? (Determines whether - tsys2 needs a USB dongle for storage network like tsys4/9, or has a real - onboard NIC available.) -- **RAM layout**: Is all 32 GB in 1 stick, 2 sticks, or 4 sticks? (Affects - memory bandwidth for HPC workloads — dual-channel matters.) -- **GPU detail**: Confirm the Quadro M1200 model and VRAM for passthrough - planning. -- **PCIe slots**: Is there a free PCIe slot for adding a NIC or HBA? -- **Service tag confirmation**: `GH1XZG2` (already on file from spreadsheet). - -### STATUS: Collected 2026-07-27 - -**Disk inventory — RESOLVED:** -- Disk 0: Samsung SSD 960 PRO **512 GB NVMe** (best local storage in fleet) -- Disk 1: Samsung SSD 850 EVO **1 TB SATA SSD** -- Both SSDs, no spinning rust. 1.5 TB total local SSD. - -**Network adapters — RESOLVED (concerning):** -- StorageNetwork: **ASIX USB to Gigabit Ethernet** (dongle, D4-81-D7-3E-0D-5E) -- Ethernet: **Realtek USB GbE Family Controller** (also USB, 18-FD-CB-00-D2-CA) -- Wi-Fi: Intel 8265 (disconnected) -- **Both wired NICs are USB-attached.** Same anti-pattern as tsys4/9. - Unavoidable on this laptop form factor — no onboard PCIe NIC available. - -**GPU — RESOLVED:** -- Intel HD Graphics 630 (integrated, 1 GB) -- NVIDIA Quadro M1200 (4 GB, confirmed for passthrough) - -**PCIe slots — RESOLVED:** -- Slots 3/6/7/8 report "Available" but these are laptop M.2/WWAN slots, not - user-accessible full PCIe. **Cannot add a PCIe NIC.** NVMe slot occupied - by 960 PRO. - -**RAM — PARTIAL:** -- Total 32 GB confirmed (34,097,573,888 bytes). -- Per-stick detail failed to run (PowerShell line-break split - `Format-T` + `able`). Re-run the command below if bandwidth planning - needs stick-level detail: - ```powershell - Get-CimInstance Win32_PhysicalMemory | Select-Object Manufacturer, PartNumber, Capacity, Speed, ConfiguredClockSpeed, DeviceLocator, FormFactor | Format-Table -AutoSize - ``` - -**Service tag — CONFIRMED:** `GH1XZG2` (Precision 5520). - ---- - -## 2. Friday maintenance window (physical hardware) - -### tsys4 — install PCIe NIC + add RAM - -1. Power down tsys4 (graceful shutdown via Proxmox UI or `shutdown -h now`). -2. Install the **PCIe NIC** (Intel i350-T2 or similar 1 GbE dual-port). -3. Add **RAM**: 16 GB → 64 GB DDR3 ECC. -4. Power on, then update `/etc/network/interfaces` to replace - `enx8cae4ccda926` (USB dongle) with the new PCIe NIC device name. -5. Reboot to activate new NIC and NFS nconnect. -6. Run `validate-fixes.sh pfv-tsys4` to confirm. - -### tsys5 — plug storage cable + install NVMe + relocate D3 SSD - -1. Plug the **second ethernet cable** into tsys5's dedicated storage NIC. -2. Verify bond0 recovery: `cat /proc/net/bonding/bond0` — look for - "Number of ports: 2" and a real partner MAC (not all zeros). -3. Apply bond hash fix (same as tsys6/7): - ```bash - echo "layer3+4" > /sys/class/net/bond0/bonding/xmit_hash_policy - ``` -4. **Relocate D3 SSD** from tsys4 USB to tsys5 SAS port: - - Power down tsys4 - - Remove the SK hynix SC300 SSD from its USB enclosure on tsys4 - - Install it on a free SAS port on tsys5 (5 ports free on LSI SAS1068E) - - On tsys5: mount as `/mnt/pfv-tsys5/D3`, add to `/etc/exports` - - Update `/etc/pve/storage.cfg` cluster-wide: repoint D3 `server` from - `pfv-tsys4-nfs-stor` to `pfv-tsys5-nfs-stor`, update `export` path - - Copy any existing D3 data from tsys4 first (currently ~2 MB, essentially - empty, so minimal migration) -5. Install the **PCI NVMe drive** (uses a PCI slot, not a SATA/SAS port). -6. Format NVMe as local directory storage (see TODO section 3 below). -7. Reboot tsys5 to activate NFS nconnect. -8. Run `validate-fixes.sh pfv-tsys5` to confirm. - ---- - -## 3. tsys5 NVMe format/mount decision (after Friday install) - -**Recommendation: local-only, not NFS-exported.** Format as Proxmox directory -storage so it shows up as a VM image target in the Proxmox UI. - -After the NVMe is physically installed and visible in Proxmox: - -1. Identify the device: `lsblk` or `ls /dev/nvme*` -2. Format and add to Proxmox: - ```bash - # Option A: LVM-thin (thin provisioning, snapshots) - pvcreate /dev/nvme0n1 - vgcreate nvme-pool /dev/nvme0n1 - lvcreate -l 100%FREE -T nvme-pool/data - # Then in Proxmox UI: Datacenter > Storage > Add > LVM-Thin - # ID: nvme-local - # Volume Group: nvme-pool - # Thin Pool: data - # Content: Disk image, Container template - - # Option B: Directory (simpler, no thin provisioning) - mkfs.ext4 /dev/nvme0n1 - mkdir -p /mnt/nvme - mount /dev/nvme0n1 /mnt/nvme - # Add to /etc/fstab for persistence - # Then in Proxmox UI: Datacenter > Storage > Add > Directory - # ID: nvme-local - # Directory: /mnt/nvme - # Content: Disk image, Container template - ``` - -3. Use for wnode-tsys5 boot disk (highest impact) and sectestbed VM scratch. - ---- - -## 4. Post-hardware validation (run after Friday work) - -1. Re-run iperf matrix: `./iperf-full-matrix.sh` -2. Validate tsys4 and tsys5: `./validate-fixes.sh pfv-tsys4 && ./validate-fixes.sh pfv-tsys5` -3. Update PROJECT.md with post-hardware iperf numbers. - ---- - -## 5. UCS storage migration to spinning disk (do today) - -UCS (Univention Corporate Server / open-source AD) does not need SSD. Both -UCS VMs should stay on spinning disk (HDD) and be split across storage -servers for redundancy. - -| VM | Current | Target | Action | -|----|---------|--------|--------| -| ucs-01 (108) | D2 (tsys4 HDD) | **D2 (tsys4 HDD) -- no change** | Already correct | -| ucs-02 (902) | D5 (tsys4 HDD) | **S2 (tsys5 HDD)** | Move for cross-server redundancy | - -**To migrate ucs-02 to S2 (use PDM/Proxmox UI):** -1. In Proxmox Datacenter or the node UI, select VM 902 on tsys9 -2. Use "Migrate" or "Storage Migrate" to move the disk from D5 to S2 - (both are NFS exports visible to tsys9, so this is a storage-only migration) -3. Verify VM 902 boots and LDAP/AD services are healthy after migration - -Note: both VMs are currently on tsys4 HDD, which is fine for UCS. Only -ucs-02 needs to move -- it should be on a different storage server than -ucs-01 so a tsys4 failure doesn't take down both halves of the AD pair. - ---- - -## 6. Open questions for next session - -- Are the hosts a Proxmox cluster (`pvecm status`) or standalone installs? - Determines whether live migration is available. -- What k8s distribution is in use? (k3s, kubeadm, RKE2?) -- Container runtime? (containerd, cri-o?) -- Is there a local container image registry mirror? -- What specific ETL tools? (GDAL, PostGIS, xarray, Dask?) -- HPC job scheduler? (plain k8s Jobs, Argo Workflows, Volcano?) -- What uses tsys5's SDR + parallel port before planning tsys5 role changes? -- tsys3 thermal state (laptop in rack for years) — check `sensors`. +> **Documentation moved to Discourse — the canonical source of truth.** +> +> **Pending hardware work — folded into Operations Status** +> +> **Read it here:** https://community.turnsys.com/t/298 +> +> *Migrated 2026-08-06. This file is kept as a pointer for git-browsing context. +> Do not update content here — edit the Discourse wiki topic instead.* diff --git a/proxmox/docs/VM-INVENTORY.md b/proxmox/docs/VM-INVENTORY.md index 3f72551..0221ea2 100644 --- a/proxmox/docs/VM-INVENTORY.md +++ b/proxmox/docs/VM-INVENTORY.md @@ -1,225 +1,10 @@ -# VM Inventory +# proxmox/docs/VM-INVENTORY.md -> **Every VM in the cluster, with host placement, disk location, CPU/RAM -> allocation, and optimization status.** Data gathered 2026-07-30. +> **Documentation moved to Discourse — the canonical source of truth.** > -> Companion docs: [`NETWORK-TOPOLOGY.md`](NETWORK-TOPOLOGY.md), -> [`STORAGE-ARCHITECTURE.md`](STORAGE-ARCHITECTURE.md), -> [`HOST-REFERENCE.md`](HOST-REFERENCE.md). - ---- - -## 1. VM placement diagram - -```mermaid -graph TB - subgraph tsys1["tsys1 -- Infra (57% RAM)"] - VM100["100 pfv-bms
2c/4G D2"] - VM101["101 tsys-ca
2c/2G D2"] - VM103["103 netinfra-01
2c/4G S2"] - VM104["104 librenms
2c/2G D2"] - VM105["105 proxmox-dc
2c/2G D2"] - VM106["106 cnode3
4c/4G D5"] - VM108["108 ucs-01
2c/8G D2"] - VM114["114 kali-tsys
2c/2G D2"] - VM117["117 workbench
2c/4G D2"] - end - - subgraph tsys3["tsys3 -- Compute (10% RAM)"] - VM313["313 wnode-tsys3
8c/28G D5"] - end - - subgraph tsys4["tsys4 -- Storage (15% RAM)"] - VM400["400 PBS
2c/2G local-lvm"] - end - - subgraph tsys5["tsys5 -- Storage+Sandbox (15% RAM)"] - VM509["509 wnode-tsys5
2c/32G D2"] - VM515["515 hfnoc-uisp-preprod
2c/2G local-nonprod"] - VM5111["5111 ultix-streaming
4c/9G SSD"] - VM5112["5112 ultix-offstage
4c/6G local-lvm"] - end - - subgraph tsys6["tsys6 -- RackRental (12% RAM)"] - VM100b["100 wnode-tsys6
2c/64G S4"] - VM600["600 tsys-awx
2c/12G D5"] - end - - subgraph tsys7["tsys7 -- RackRental (7% RAM)"] - VM701["701 wnode-tsys7
4c/32G S3"] - VM702["702 hfnoc-uisp
2c/8G S1"] - VM703["703 rr-middleware
2c/2G S2"] - VM704["704 TCTC
4c/6G D2"] - VM705["705 cnode2
4c/4G S2"] - end - - subgraph tsys9["tsys9 -- Infra+Compute (69% RAM)"] - VM901["901 tsys-siem
2c/8G D2"] - VM902["902 ucs-02
2c/8G S2"] - VM903["903 kali-rd
2c/2G D5"] - VM904["904 netinfra-02
2c/4G D2"] - VM905["905 wnode-tsys9
4c/4G S2"] - VM906["906 cnode1
2c/4G S3"] - end -``` - ---- - -## 2. Complete running VM inventory (28 running VMs) - -### Infrastructure (netinfra, UCS, CA, LibreNMS, PDM) - -| VMID | Name | Host | vCPU | CPU type | RAM | Disk store | Spindle | Disk type | Cache | iothread | qga | Notes | -|------|------|------|------|----------|-----|-----------|---------|-----------|-------|----------|-----|-------| -| 100 | pfv-bms (HomeAssistant) | tsys1 | 2 | host | 4G | D2 (raw) | D2 | raw | -- | -- | ✓ | discard=on, ssd=1. HA for power/temp monitoring. | -| 101 | tsys-ca | tsys1 | 2 | v2-AES | 2G | D2 (virtio) | D2 | qcow2 | -- | ✓ | -- | no nested-virt flag. Certificate Authority. | -| 103 | pfv-netinfra-01 | tsys1 | 2 | v2-AES | 4G | S2 | S2 | qcow2 | -- | ✓ | -- | DNS primary (Technitium + Pi-hole). | -| 104 | tsys-librenms | tsys1 | 2 | host | 2G | D2 | D2 | qcow2 | -- | ✓ | ✓ | SNMP monitoring for switches. | -| 105 | tsys-proxmox-datacenter | tsys1 | 2 | host | 2G | D2 | D2 | qcow2 | -- | ✓ | ✓ | PDM (Proxmox Datacenter Manager). | -| 108 | tsys-ucs-01 | tsys1 | 2 | host | 8G | D2 | D2 | qcow2 | **writethrough** | ✓ | ✓ | Univention Corporate Server half 1. | -| 902 | tsys-ucs-02 | tsys9 | 2 | host | 8G | S2 | S2 | qcow2 | -- | ✓ | ✓ | UCS half 2 (HA pair with 108). | -| 904 | pfv-netinfra-02 | tsys9 | 2 | v2-AES | 4G | D2 | D2 | qcow2 | -- | ✓ | -- | DNS secondary. HA pair with 103. | - -### k8s control plane (cnode1/2/3) - -| VMID | Name | Host | vCPU | CPU type | RAM | Disk store | Spindle | Disk type | Cache | iothread | qga | Notes | -|------|------|------|------|----------|-----|-----------|---------|-----------|-------|----------|-----|-------| -| 106 | pfv-k8s-cnode3 | tsys1 | 4 | host | 4G | D5 | D5 | qcow2 | **writethrough** | ✓ | ✓ | etcd member. | -| 705 | pfv-k8s-cnode2 | tsys7 | 4 | host | 4G | S2 | S2 | qcow2 | **writethrough** | ✓ | ✓ | etcd member. | -| 906 | pfv-k8s-cnode1 | tsys9 | 2 | host | 4G | S3 | S3 | qcow2 | -- | ✓ | -- | etcd member. | - -### k8s workers (wnodes) - -| VMID | Name | Host | vCPU | CPU type | RAM | Disk store | Spindle | Disk type | Cache | iothread | qga | Notes | -|------|------|------|------|----------|-----|-----------|---------|-----------|-------|----------|-----|-------| -| 313 | pfv-k8s-wnode-tsys3 | tsys3 | 8 | host | 28G | D5 | D5 | qcow2 | **writethrough** | ✓ | ✓ | Compute worker. | -| 509 | pfv-k8s-wnode-tsys5 | tsys5 | 2 | host | 32G | D2 (virtio) | D2 | qcow2 | -- | ✓ | -- | Undersized vCPU (2). | -| 100 | pfv-k8s-wnode-tsys6 | tsys6 | 2 | host | 64G | S4 | S4 | qcow2 | -- | ✓ | -- | Undersized vCPU (2), high RAM. | -| 701 | pfv-k8s-wnode-tsys7 | tsys7 | 4 | host | 32G | S3 (virtio) | S3 | qcow2 | -- | ✓ | -- | RackRental worker. | -| 905 | pfv-k8s-wnode-tsys9 | tsys9 | 4 | v2-AES | 4G | S2 | S2 | qcow2 | -- | ✓ | ✓ | Target: grow to 6c/8G. | - -### Application VMs - -| VMID | Name | Host | vCPU | CPU type | RAM | Disk store | Spindle | Disk type | Cache | iothread | qga | Notes | -|------|------|------|------|----------|-----|-----------|---------|-----------|-------|----------|-----|-------| -| 114 | kali-tsys | tsys1 | 2 | host | 2G | D2 | D2 | qcow2 | **writethrough** | ✓ | ✓ | Kali Linux pentest. | -| 117 | tsys-secure-workbench | tsys1 | 2 | host | 4G | D2 | D2 | qcow2 | **writethrough** | ✓ | ✓ | Secure workbench. | -| 400 | pfv-proxmox-backup-server | tsys4 | 2 | -- | 2G | local-lvm | (local) | raw | -- | -- | -- | PBS -- all VM backups. | -| 515 | hfnoc-uisp-preprod | tsys5 | 2 | host | 2G | local-nonprod | S1 | qcow2 | -- | ✓ | -- | Preprod. Locked to tsys5. | -| 5111 | ultix-streaming | tsys5 | 4 | -- | 9G | SSD dir | SSD | qcow2 | -- | -- | -- | 288GB on Samsung 860 PRO (**93% full**). | -| 5112 | ultix-offstage | tsys5 | 4 | -- | 6G | local-lvm | (local) | raw | -- | -- | -- | 288GB. Locked to tsys5. | -| 600 | tsys-awx | tsys6 | 2 | host | 12G | D5 | D5 | qcow2 | **writethrough** | ✓ | ✓ | Ansible AWX. | -| 702 | hfnoc-uisp | tsys7 | 2 | host | 8G | S1 | S1 | qcow2 | -- | ✓ | ✓ | 100GB disk. UISP network monitoring. | -| 703 | rr-middleware | tsys7 | 2 | host | 2G | S2 | S2 | qcow2 | **writethrough** | ✓ | ✓ | RackRental middleware. | -| 704 | TCTC | tsys7 | 4 | host | 6G | D2 (virtio) | D2 | qcow2 | -- | ✓ | -- | | -| 901 | tsys-siem | tsys9 | 2 | host | 8G | D2 | D2 | qcow2 | **writethrough** | ✓ | ✓ | 132GB disk. SIEM. | -| 903 | kali-rd | tsys9 | 2 | host | 2G | D5 | D5 | qcow2 | **writethrough** | ✓ | ✓ | Kali R&D. | - ---- - -## 3. Optimization findings - -### 3.1 Cache mode audit - -`cache=writethrough` forces synchronous writes to the NFS server on every write -operation. For NFS-backed qcow2, this is unnecessarily slow -- `writeback` or -`none` is recommended (qcow2 already provides journaling integrity). - -| Cache mode | Count | VMs | -|-----------|-------|-----| -| **writethrough** (slow) | **10** | 106, 108, 114, 117, 313, 600, 703, 705, 901, 903 | -| none/default (optimal) | 18 | all others | - -**Fix:** Change `cache=writethrough` → `cache=writeback` or remove (defaults to -writeback for qcow2). Requires VM stop/start. - -### 3.2 CPU type audit - -`cpu: host` passes the host's full CPU instruction set to the VM (best -performance). `x86-64-v2-AES` is a conservative baseline (safe for migration -but lacks some host instructions). - -| CPU type | Count | VMs | -|----------|-------|-----| -| **host** | 21 | most production VMs | -| **x86-64-v2-AES** | 5 | 101, 103, 509, 904, 905 | -| not set | 2 | 400, 5111, 5112 | - -**Note:** Mixed CPU types prevent live migration between hosts with different -CPU generations. K8s cnodes/wnodes should ideally match within each tier. - -### 3.3 Disk controller audit - -| Controller | Count | Notes | -|-----------|-------|-------| -| **scsi0** (virtio-scsi) | 22 | Optimal for Proxmox | -| **virtio0** | 4 | 101, 509, 701, 704 -- also good | -| raw (no virtio) | 2 | 100 (pfv-bms), 500 (stopped) | - -All running VMs use virtio-based disk controllers. No IDE/SATA legacy disks. - -### 3.4 Network audit - -All running VMs use **virtio-net**. Firewall enabled on most (`firewall=1`). -No legacy e1000/rtl8139 adapters. - -### 3.5 Guest agent (qga) audit - -| Status | Count | VMs without qga | -|--------|-------|-----------------| -| **enabled** | 20 | -- | -| **not set** | 8 | 101, 103, 509, 100(tsys6), 701, 704, 904, 906 | - -**Fix:** Enable `qm set --agent 1` on the 8 VMs missing it. No VM -stop/start required (takes effect on next guest reboot, but the setting itself -is immediate). - -### 3.6 k8s wnode sizing - -| VMID | Name | vCPU | RAM | Tier target | Issue | -|------|------|------|-----|-------------|-------| -| 509 | wnode-tsys5 | 2 | 32G | compute | **Undersized vCPU** (2 for a compute worker) | -| 100 | wnode-tsys6 | 2 | 64G | rackrental | **Undersized vCPU** (2 for 64G RAM) | -| 701 | wnode-tsys7 | 4 | 32G | rackrental | OK for idle labs | -| 905 | wnode-tsys9 | 4 | 4G | compute | **Target: grow to 6c/8G** after siem migrates off | -| 313 | wnode-tsys3 | 8 | 28G | compute | OK (best compute worker) | - ---- - -## 4. Stopped VMs (not running, for reference) - -### Sectestbed / sandboxes (all on tsys5, all stopped) - -| VMID | Name | RAM | Disk | Notes | -|------|------|-----|------|-------| -| 5100 | sectestbed-template | 4G | local-lvm | Base template | -| 5101 | sectestbed-siem | 10G | local-nonprod (S1) | 132G disk | -| 5102 | sectestbed-proxmox-pve | 4G | local-lvm | | -| 5103 | sectestbed-proxmox-datacenter | 4G | local-lvm | | -| 5104 | sectestbed-proxmox-pbs | 4G | local-lvm | | -| 5105 | sectestbed-awx | 4G | local-nonprod (S1) | 288G disk | -| 5106 | sectestbed-k8s-cnode | 4G | local-nonprod (S1) | | -| 5107 | sectestbed-k8s-wnode | 4G | local-nonprod (S1) | | -| 5108 | sectestbed-librenms | 4G | local-nonprod (S1) | | -| 5109 | sectestbed-netinfra | 4G | local-nonprod (S1) | | -| 51010 | sectestbed-tctc | 4G | local-nonprod (S1) | | -| 51011 | sectestbed-cloudron | 4G | local-nonprod (S1) | | -| 51012 | sectestbed-hfnoc | 4G | local-nonprod (S1) | | -| 51013 | sectestbed-rancherplatform | 4G | local-nonprod (S1) | | -| 51014 | sectestbed-proxmox-mailgw | 4G | local-nonprod (S1) | | -| 53100 | tsys-preprod-awx | 9G | local-nonprod (S1) | 160G disk | -| 53101 | tsys-preprod-siem | 12G | local-nonprod (S1) | | -| 53102 | tsys-preprod-rancherplatform | 8G | local-nonprod (S1) | | -| 53103 | tsys-preprod-proxmoxmailgw | 4G | local-nonprod (S1) | | -| 6000 | sectestbed-sandbox | 4G | local-nonprod (S1) | Has 3 state snapshots | - -### Other stopped VMs - -| VMID | Name | Host | RAM | Disk | Notes | -|------|------|------|-----|------|-------| -| 102 | pfv-k8s-wnode-tsys1 | tsys1 | 4G | (no disk) | Stopped, no boot disk | -| 500 | DellOpenManageEnterprise | tsys5 | 20G | **D7 (gone!)** | **Would fail -- disk export removed** | -| 5500 | RestoreTemplate | tsys5 | 2G | D2 | Template | -| 1000 | KNELTemplate | tsys1 | 2G | D2 | Base template | -| 3000 | KNELTemplate | tsys3 | 2G | D2 | Base template copy | -| 4000 | KNELTemplate | tsys4 | 2G | (local) | Base template copy | +> **Complete VM catalog with disk/CPU/RAM/config** +> +> **Read it here:** https://community.turnsys.com/t/304 +> +> *Migrated 2026-08-06. This file is kept as a pointer for git-browsing context. +> Do not update content here — edit the Discourse wiki topic instead.* diff --git a/tests/README.md b/tests/README.md index 2a7488e..730e84e 100644 --- a/tests/README.md +++ b/tests/README.md @@ -1,176 +1,10 @@ -# PFVCluster Test Suite +# tests/README.md -## Overview - -This testing framework provides comprehensive validation for the PFVCluster infrastructure. It includes unit tests, integration tests, security tests, and system validation. - -## Test Categories - -### 1. Unit Tests (`unit/`) -- **Purpose:** Test individual framework functions and components -- **Scope:** Framework includes, helper functions, syntax validation -- **Example:** `framework-functions.sh` - Tests logging, pretty print, and error handling functions - -### 2. Integration Tests (`integration/`) -- **Purpose:** Test complete workflows and module interactions -- **Scope:** End-to-end deployment scenarios, module integration -- **Future:** Module interaction testing, deployment workflow validation - -### 3. Security Tests (`security/`) -- **Purpose:** Validate security configurations and practices -- **Scope:** HTTPS enforcement, deployment security, SSH hardening -- **Example:** `https-enforcement.sh` - Validates all URLs use HTTPS - -### 4. Validation Tests (`validation/`) -- **Purpose:** System compatibility and pre-flight checks -- **Scope:** System requirements, network connectivity, permissions -- **Example:** `system-requirements.sh` - Validates minimum system requirements - -## Usage - -### Run All Tests -```bash -./run-tests.sh -``` - -### Run Specific Test Categories -```bash -./run-tests.sh unit # Unit tests only -./run-tests.sh integration # Integration tests only -./run-tests.sh security # Security tests only -./run-tests.sh validation # Validation tests only -``` - -### Run Individual Tests -```bash -./validation/system-requirements.sh -./security/https-enforcement.sh -./unit/framework-functions.sh -``` - -## Test Results - -- **Console Output:** Real-time test results with color-coded status -- **JSON Reports:** Detailed test reports saved to `logs/tests/` -- **Exit Codes:** 0 for success, 1 for failures - -## Configuration Validation - -The validation framework performs pre-flight checks to ensure system compatibility: - -### System Requirements -- **Memory:** Minimum 2GB RAM -- **Disk Space:** Minimum 10GB available -- **OS Compatibility:** Ubuntu/Debian (tested), others (may work) - -### Network Connectivity -- Tests connection to required download sources -- Validates HTTPS endpoints are accessible -- Checks for firewall/proxy issues - -### Command Dependencies -- Verifies required tools are installed (`curl`, `wget`, `git`, `systemctl`, `apt-get`) -- Checks for proper versions where applicable - -### Permissions -- Validates write access to system directories -- Checks for required administrative privileges - -## Adding New Tests - -### Test File Structure -```bash -#!/bin/bash -set -euo pipefail - -function test_something() { - echo "🔍 Testing something..." - - if [[ condition ]]; then - echo "✅ Test passed" - return 0 - else - echo "❌ Test failed" - return 1 - fi -} - -function main() { - echo "🧪 Running Test Suite Name" - echo "==========================" - - local total_failures=0 - test_something || ((total_failures++)) - - echo "==========================" - if [[ $total_failures -eq 0 ]]; then - echo "✅ All tests passed" - exit 0 - else - echo "❌ $total_failures tests failed" - exit 1 - fi -} - -if [[ "${BASH_SOURCE[0]}" == "${0}" ]]; then - main "$@" -fi -``` - -### Test Categories Guidelines - -- **Unit Tests:** Focus on individual functions, fast execution -- **Integration Tests:** Test module interactions, longer execution -- **Security Tests:** Validate security configurations -- **Validation Tests:** Pre-flight system checks - -## Continuous Integration - -The testing framework is designed to integrate with CI/CD pipelines: - -```bash -# Example CI script -./run-tests.sh all -test_exit_code=$? - -if [[ $test_exit_code -eq 0 ]]; then - echo "All tests passed - deployment approved" -else - echo "Tests failed - deployment blocked" - exit 1 -fi -``` - -## Test Development Best Practices - -1. **Clear Test Names:** Use descriptive function names -2. **Proper Exit Codes:** Return 0 for success, 1 for failure -3. **Informative Output:** Use emoji and clear messages -4. **Timeout Protection:** Use timeout for network operations -5. **Cleanup:** Remove temporary files and resources -6. **Error Handling:** Use `set -euo pipefail` for strict error handling - -## Troubleshooting - -### Common Issues - -- **Permission Denied:** Run tests with appropriate privileges -- **Network Timeouts:** Check firewall and proxy settings -- **Missing Dependencies:** Install required tools before testing -- **Script Errors:** Validate syntax with `bash -n script.sh` - -### Debug Mode -```bash -# Enable debug output -export DEBUG=1 -./run-tests.sh -``` - -## Contributing - -When adding new functionality to PFVCluster: - -1. Add corresponding tests in appropriate category -2. Run full test suite before committing -3. Update documentation for new test cases -4. Ensure tests pass in clean environment \ No newline at end of file +> **Documentation moved to Discourse — the canonical source of truth.** +> +> **Test suite + VM validation harness** +> +> **Read it here:** https://community.turnsys.com/t/302 +> +> *Migrated 2026-08-06. This file is kept as a pointer for git-browsing context. +> Do not update content here — edit the Discourse wiki topic instead.*