KNEL/PFVCluster
5
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: migrate all documentation to Discourse wiki topics

Browse Source 
All knowledge docs (architecture, runbooks, references, audits, policies)
have been migrated to community.turnsys.com as wiki topics in the VP
TechOps category. Discourse is now the canonical source of truth for
documentation; git edit history no longer serves as the doc changelog.

37 .md files converted to short pointers linking to their Discourse
topics. AGENTS.md updated with new documentation workflow policy.
Code (scripts, configs, playbooks) remains authoritative in git.
This commit is contained in:
reachableceo
2026-08-06 14:08:34 -05:00
parent d6988bb35f
commit 88faf35ec3
37 changed files with 378 additions and 8239 deletions
Download Patch File Download Diff File Expand all files Collapse all files
dcinfra/console/README.md
+10 -132
View File
@@ -1,132 +1,10 @@
# Console Management (ser2net + conman)
> **Redmine:** [#360](https://projects.knownelement.com/issues/360) (deployment, closed) · [#373](https://projects.knownelement.com/issues/373) (pfv-r2-sw fix, open)
Network-accessible serial console management for all production network
switches and routers, running on **pfv-tsys4** (storage server).
## Architecture
```
USB-DB9 adapters → udev symlinks (/dev/consoles/<name>) → ser2net telnet(rfc2217) TCP → conman (logging + multiplexing)
```
ser2net owns the physical serial devices and exposes them on TCP ports
using the **telnet(rfc2217) protocol** bound to the **Tailscale interface
only** (`100.70.77.93:200X`). conman connects to those TCP ports via
telnet for session logging, output capture, and multi-user console
sharing.
**Why telnet(rfc2217)?** The serial devices send `
␍` (LF+CR) line
endings instead of standard `
`. Raw TCP transport caused conman's
telnet NVT to strip bare CR characters, producing stair-stepped output.
With telnet(rfc2217) on both sides, binary mode is negotiated and CR/LF
translation is handled correctly by the telnet layer.
**conman and ser2net do NOT share ports** — only one process can open a
serial device at a time. ser2net owns the physical device; conman connects
over TCP.
## The USB Enumeration Problem (SOLVED)
The 9 Prolific USB-to-DB9 adapters (`067b:2303`) on pfv-tsys4 have **no
unique USB serial numbers** and get assigned `/dev/ttyUSB0-8` based on
enumeration order, which shifts on every boot. This made the old
`/root/conmap` + manual `screen` workflow break after every reboot.
**Fix:** udev rules pin each adapter by its **ID_PATH** (physical USB port
topology), which is stable across reboots regardless of enumeration order.
Each adapter gets a named symlink in `/dev/consoles/` that never changes.
The udev rules are generated from `mapping.txt`, which maps each adapter's
ID_PATH to a console name and TCP port. To re-map after physically moving
an adapter, update `mapping.txt` and re-run `setup.sh`.
**Fallback:** if udev trigger doesn't create symlinks for already-discovered
devices (common on first run), `setup.sh` creates them manually by matching
ID_PATH. On subsequent boots, udev creates them automatically.
## Port Assignments
| TCP Port | Console Name | ID_PATH | Description |
|----------|-------------|---------|-------------|
| 2001 | pfv-core-sw01 | usb-0:1.5.4.4 | Dell PowerConnect 5448 (core switch) |
| 2002 | pfv-tor3-mgmt | usb-0:1.6.3.1 | Rack 3 management TOR switch |
| 2003 | pfv-tor3-stor | usb-0:1.6.3.3.2 | Rack 3 storage TOR switch |
| 2004 | pfv-rrinfra-rtr | usb-0:1.6.3.3.1 | Cisco router (rrinfra) |
| 2005 | pfv-r2-tor-top | usb-0:1.6.3.3.3 | Rack 2 top-of-rack switch |
| 2006 | subodev-torsw | usb-0:1.5.4.1 | Suborbital device TOR switch |
| 2007 | pfv-r2-sw | usb-0:1.6.3.2 | Rack 2 old Dell switch |
All ports listen on the Tailscale IP (`100.70.77.93`) using telnet(rfc2217).
## Scripts
| Script | Purpose |
|--------|---------|
| [`mapping.txt`](mapping.txt) | Source of truth: TCP port ↔ ID_PATH ↔ name ↔ baud |
| [`generate-config.sh`](generate-config.sh) | Generates udev rules, ser2net.yaml, conman.conf from mapping.txt |
| [`setup.sh`](setup.sh) | Full deploy: generate configs, create symlinks, restart services |
| [`discover.sh`](discover.sh) | Read-only discovery of USB adapters, existing config, services |
## Usage
### Connect to a console
**Primary method — conman client (with logging + multiplexing):**
```bash
# From any Tailscale-connected workstation:
conman -d pfv-tsys4:7890 -f pfv-core-sw01 # connect to console
conman -d pfv-tsys4:7890 -q # list all consoles
```
Escape sequence: `&.` to disconnect, `&?` for help.
**Direct telnet (emergency only — conflicts with conman):**
```bash
# Direct telnet to ser2net works ONLY when conmand is stopped, because
# conmand maintains persistent connections to all 7 TCP ports. Use:
ssh pfv-tsys4 'systemctl stop conmand'
telnet pfv-tsys4 2001 # pfv-core-sw01
ssh pfv-tsys4 'systemctl start conmand' # restart when done
```
**Do NOT use telnet while conmand is running** — conmand will reconnect
and kick your telnet session immediately ("Connection closed by foreign host").
The correct workflow is conman client → conmand → ser2net → device.
### Re-deploy after changing mapping.txt
```bash
PROX_HOST=pfv-tsys4 bash tests/remote.sh prox 'bash /root/console/setup.sh'
```
### Find the ID_PATH for a new adapter
```bash
PROX_HOST=pfv-tsys4 bash tests/remote.sh prox-file console/discover.sh
```
Then match the new adapter's ID_PATH to its physical location and add a line
to `mapping.txt`.
## Files on pfv-tsys4
| File | Purpose |
|------|---------|
| `/etc/udev/rules.d/99-console-ports.rules` | Stable symlinks by ID_PATH |
| `/etc/ser2net.yaml` | ser2net config (telnet rfc2217 TCP ports → serial symlinks) |
| `/etc/conman.conf` | conman config (CONSOLE entries between markers) |
| `/etc/systemd/system/conmand.service` | systemd unit for conmand |
| `/root/console/mapping.txt` | Copy of the source-of-truth mapping |
| `/root/console/setup.sh` | Setup script (re-runnable) |
| `/root/console/generate-config.sh` | Config generator |
## Old workflow (replaced)
The old `/root/conmap` file and manual `screen` sessions are no longer
needed. The new setup is fully automated and survives reboots.
# dcinfra/console/README.md
> **Documentation moved to Discourse — the canonical source of truth.**
>
> **Serial console management (ser2net + conman)**
>
> **Read it here:** https://community.turnsys.com/t/301
>
> *Migrated 2026-08-06. This file is kept as a pointer for git-browsing context.
> Do not update content here — edit the Discourse wiki topic instead.*
dcinfra/powerman/README.md
+9 -112
View File
@@ -1,113 +1,10 @@
# Powerman PDU Management
# dcinfra/powerman/README.md
> **Redmine:** [#359](https://projects.knownelement.com/issues/359) (deployment, closed) · [#374](https://projects.knownelement.com/issues/374) (outlet renaming, open)
Centralized power management for the Cyclades AlterPath PM10i PDU via
[Powerman](https://github.com/chaos/powerman), running on pfv-tsys1.
## Hardware
| Component | Details |
|-----------|---------|
| **PDU** | Cyclades AlterPath PM10i (10 controllable AC outlets) |
| **Firmware** | v1.9.0 (Aug 4, 2006) |
| **Connection** | USB-to-DB9 adapter (Prolific pl2303, serial BJAAb144J07) |
| **Host** | pfv-tsys1 (OptiPlex 9020, Proxmox) |
| **Serial** | 9600 baud, 8N1, raw mode |
| **Credentials** | Factory defaults: `admin` / `pm8` (in cyclades-pm10.dev) |
| **Network access** | powermand listens on `127.0.0.1:10101` (local) + `100.121.189.98:10101` (Tailscale) |
## Device mapping
```
USB adapter (067b:23a3, serial BJAAb144J07)
└─ pl2303 driver → /dev/ttyUSB1
└─ udev symlink → /dev/cyclades-pm10 (stable across reboots)
└─ powermand reads/writes serial → Cyclades PM10i
└─ 10 outlets (factory default names: 1-10)
```
The udev rule (`/etc/udev/rules.d/99-cyclades-pdu.rules`) pins the adapter
by its USB serial number, so the symlink survives replugs and reboots.
## Scripts
All scripts run on the target host (pfv-tsys1) via `tests/remote.sh`:
```bash
# Setup (idempotent — safe to re-run):
PROX_HOST=pfv-tsys1 bash tests/remote.sh prox-file powerman/setup.sh
# Validate PDU control (cycles outlet 10 off → on):
PROX_HOST=pfv-tsys1 bash tests/remote.sh prox-file powerman/test-pdu.sh
# Status check:
PROX_HOST=pfv-tsys1 bash tests/remote.sh prox-file powerman/status.sh
```
### Customizing for other hosts/PDUs
The setup script accepts environment overrides:
```bash
PDU_SERIAL=XXXX PDU_VENDOR=067b PDU_OUTLETS=20 PDU_TYPE=pm20 \
PROX_HOST=other-host bash tests/remote.sh prox-file powerman/setup.sh
```
## Usage (daily operations)
From pfv-tsys1 (or any host with network access to port 10101):
```bash
# List all outlets
powerman -l
# Query status (all outlets)
powerman -q
# Turn outlet off
powerman -0 outlet-10
# Turn outlet on
powerman -1 outlet-10
# Cycle outlet (off → 4s delay → on)
powerman -c outlet-10
# Query a specific outlet
powerman -q outlet-10
```
### Remote access from other hosts
powermand listens on `0.0.0.0:10101`. From another tailnet host:
```bash
powerman --server-host pfv-tsys1 --server-port 10101 -q
```
Or set `POWERMAN_SERVER=pfv-tsys1:10101` in the environment.
## Configuration files on pfv-tsys1
| File | Purpose |
|------|---------|
| `/etc/udev/rules.d/99-cyclades-pdu.rules` | Stable symlink for USB-DB9 adapter |
| `/etc/powerman/powerman.conf` | Device definition + 10 outlet nodes |
| `/etc/powerman/cyclades-pm10.dev` | Cyclades PM10 protocol spec (shipped with powerman) |
## Validation results
2026-07-28: All 8 checks passed.
Outlet 10 turned OFF (confirmed), turned ON (confirmed), then cycled.
## TODO (Friday onsite)
- [ ] **Rename outlets** in `/etc/powerman/powerman.conf` to match the
physical devices plugged into each outlet (e.g., `node "tsys4-psu"
"cyclades-pm10" "3"`). Currently all outlets are generically named
`outlet-1` through `outlet-10`.
- [ ] **Change PDU admin password** from factory default (`pm8`) if
security-sensitive. Update `/etc/powerman/cyclades-pm10.dev` login
script to match.
- [ ] **Verify all 10 outlets** individually once device mapping is known.
> **Documentation moved to Discourse — the canonical source of truth.**
>
> **Cyclades PM10i PDU management via powerman**
>
> **Read it here:** https://community.turnsys.com/t/301
>
> *Migrated 2026-08-06. This file is kept as a pointer for git-browsing context.
> Do not update content here — edit the Discourse wiki topic instead.*
dcinfra/ups/README.md
+9 -211
View File
@@ -1,212 +1,10 @@
# UPS Management (NUT — Network UPS Tools)
# dcinfra/ups/README.md
> **Redmine:** [#340](https://projects.knownelement.com/issues/340) (APC, closed) · [#372](https://projects.knownelement.com/issues/372) (Tripp Lite, open)
Centralized UPS monitoring for the server room via
[NUT](https://networkupstools.org/), running on **pfv-tsys1**. USB HID UPS
units feed one `upsd` network server; Home Assistant polls it over Tailscale for
real-time power/load/runtime tracking, and a local `upsmon` shuts the hypervisor
down gracefully when battery is low.
> **Why NUT (not apcupsd)?** Two different UPS brands (APC + Tripp Lite) must be
> covered. `apcupsd` only supports APC, so it would require a second daemon
> stack. NUT's `usbhid-ups` driver speaks to **both** via the USB HID Power
> Device class, and Home Assistant ships a first-class NUT integration.
## Hardware
| UPS | Model | VID:PID | USB Serial | Status |
|-----|-------|---------|------------|--------|
| **APC** | Smart-UPS C 1500 (FW 02.2) | `051d:0003` | `AS1213210423` | **LIVE** |
| **Tripp Lite** | UPS (HID PDC) | `09ae:3016` | `2352CVLSM871900694` | **Blocked** — see below |
## Current State (2026-07-30)
### APC Smart-UPS C 1500 — OPERATIONAL
Fully reporting via `usbhid-ups` + `APC HID 0.100` subdriver. Data validated:
```
battery.charge: 100 battery.runtime: 1800 battery.voltage: 27.4
ups.status: OL ups.load: (via HA) ups.model: Smart-UPS C 1500
```
### Tripp Lite UPS — BLOCKED (hardware issue)
The driver finds the device, matches the `TrippLite HID 0.85` subdriver, claims
the interface, and reads the HID descriptor — but **fails reading the 878-byte
HID Report Descriptor** (`Resource temporarily unavailable` / EAGAIN after 5s).
The driver is masked to prevent restart-loop spam.
USB descriptors (manufacturer, product, serial) are readable via `lsusb -v` and
`nut-scanner`, but the bulk control transfer for the full report descriptor
times out. Likely causes:
1. **USB hub** — the Tripp Lite is behind a Genesys Logic hub (`05e3:0608`).
Try plugging directly into a motherboard USB port.
2. **USB cable** — try a high-quality data cable (not charge-only).
3. **UPS firmware** — the USB controller may not properly implement all HID
endpoints.
**To retry after physical reseat:**
```bash
# On pfv-tsys1:
systemctl unmask nut-driver@tripp-lite-ups
systemctl start nut-driver@tripp-lite-ups
upsc tripp-lite-ups@localhost
```
## Architecture
```
pfv-tsys1 (192.168.3.11 / Tailscale 100.121.189.98)
├─ APC Smart-UPS C 1500 ──┐
└─ Tripp Lite UPS (masked) ──┤ USB HID
▼
nut-driver@apc-smartups-c1500 (usbhid-ups)
▼
upsd :3493 (LISTEN 127.0.0.1 + Tailscale + LAN)
▼ ▼
upsmon (local) Home Assistant (NUT integration)
graceful shutdown via LAN 192.168.3.11 (HAOS can't
route to Tailscale IPs)
```
- **Driver layer** — `usbhid-ups` process, pinned by USB serial. Debian uses
templated `nut-driver@<upsname>.service` units managed by
`nut-driver-enumerator`.
- **Server layer** — `upsd` exposes UPS data on TCP 3493 (localhost + Tailscale
+ LAN). Clients authenticate via `upsd.users`.
- **Monitor layer** — `upsmon` runs locally as `master` to trigger
`SHUTDOWNCMD` (`/sbin/shutdown -h now`) when a UPS reports `LOWBATT`.
- **Home Assistant** — native NUT integration connects to `upsd` over Tailscale
and exposes `ups.load`, `battery.runtime`, `ups.status`, etc. as sensors.
### Key deployment lesson: udev must cover raw USB devices
The `usbhid-ups` driver opens `/dev/bus/usb/BBB/DDD` (raw USB device files),
**not** `/dev/hidraw*`. After calling `setuid(111)` to drop to the `nut` user,
it needs write access to those raw USB files. The udev rule must match
`SUBSYSTEM=="usb"` by vendor/product ID to set `GROUP="nut"` — matching only
`hidraw` is insufficient. See `/etc/udev/rules.d/99-nut-ups.rules`.
## Scripts
NUT host scripts run on pfv-tsys1 via `tests/remote.sh`:
```bash
# Idempotent install + configure (safe to re-run):
PROX_HOST=pfv-tsys1 bash tests/remote.sh prox-file ups/setup.sh
# Discover USB UPS + NUT state (read-only diagnostic):
PROX_HOST=pfv-tsys1 bash tests/remote.sh prox-file ups/discover.sh
# Query UPS data + service health:
PROX_HOST=pfv-tsys1 bash tests/remote.sh prox-file ups/status.sh
```
The HA integration script runs from your workstation (needs HA API access):
```bash
# Add the NUT integration to Home Assistant (idempotent):
bash ups/setup-ha-nut.sh
```
`setup.sh` accepts environment overrides for serials/VIDs/PIDs/usernames, so it
can be repurposed for other hosts or UPS units. Passwords for `monuser` and
`homeassistant` are auto-generated on first run and reused on subsequent runs
(stored in `/etc/nut/upsd.users`).
Set `TRIPP_ENABLED=0` to skip the Tripp Lite entirely (useful if it's physically
unplugged).
## Configuration files on pfv-tsys1
| File | Purpose |
|------|---------|
| `/etc/udev/rules.d/99-nut-ups.rules` | Grant nut group rw on raw USB + hidraw devices (both subsystems) |
| `/etc/nut/ups.conf` | `usbhid-ups` device(s), pinned by serial + subdriver |
| `/etc/nut/upsd.conf` | `LISTEN 127.0.0.1` + `LISTEN <tailscale>` + `LISTEN <lan>` on port 3493 |
| `/etc/nut/upsd.users` | `monuser` (master) + `homeassistant` (read-only) credentials |
| `/etc/nut/upsmon.conf` | Local master monitor + `SHUTDOWNCMD` |
| `/etc/nut/nut.conf` | `MODE=netserver` |
## Home Assistant integration
The NUT integration is added automatically by `setup-ha-nut.sh`, which drives
HA's REST config-flow API. It is idempotent (skips if the entry exists).
```bash
# Prerequisites: create token + password files (one-time):
mkdir -p ~/.config/pfvcluster
# HA → Profile → Long-Lived Access Tokens → Create Token:
echo -n 'YOUR_HA_TOKEN' > ~/.config/pfvcluster/ha-token
# Password is in /etc/nut/upsd.users on pfv-tsys1 (the homeassistant user):
echo -n 'YOUR_NUT_PASS' > ~/.config/pfvcluster/nut-password
chmod 600 ~/.config/pfvcluster/{ha-token,nut-password}
# Run:
bash ups/setup-ha-nut.sh
```
### Why LAN IP, not Tailscale
upsd listens on **both** the Tailscale IP (`100.121.189.98`) **and** the LAN IP
(`192.168.3.11`). The HA NUT integration uses the **LAN IP** because HAOS runs
Tailscale as an isolated add-on container — the HA core container cannot route
to Tailscale IPs. Since pfv-bms (HA, `192.168.3.12`) and pfv-tsys1 (`192.168.3.11`)
share the same vmbr0 bridge, LAN connectivity is instant and reliable.
### Manual UI alternative
In Home Assistant → **Settings → Devices & Services → Add Integration → NUT**:
| Field | Value |
|-------|-------|
| Host | `192.168.3.11` (LAN — HAOS can't reach Tailscale IPs from the HA container) |
| Port | `3493` |
| Username | `homeassistant` |
| Password | *(stored in `/etc/nut/upsd.users` on pfv-tsys1)* |
| UPS | `apc-smartups-c1500` |
### Live sensors
HA exposes UPS data as sensors (prefix `sensor.apc_smartups_c1500_`):
`battery_charge`, `status` (Online/On Battery), `status_data` (OL/OB/DISCHRG).
Additional sensors (load, runtime, voltage) populate as the UPS reports them.
## Daily operations
From pfv-tsys1 (or any tailnet host with NUT client installed):
```bash
# List UPS units served by upsd
upsc -l pfv-tsys1
# Full variable dump for one UPS
upsc apc-smartups-c1500@pfv-tsys1
# Battery runtime (the only runtime/charge data this UPS exposes)
upsc apc-smartups-c1500@pfv-tsys1 battery.runtime
```
## Notes
- **No USB passthrough to the HA VM.** Keeping the UPS on the host preserves
hypervisor graceful-shutdown capability and matches the `powerman/` pattern
(PDU managed on the host where the adapter physically lives).
- **No `ups.load` / `ups.realpower` on this UPS (FW 02.2, mfg 2012):** The
APC Smart-UPS C 1500 does not expose load or power data over USB HID.
Both NUT `usbhid-ups` and `apcupsd` (USB mode, tested 2026-07-30) read the
same HID descriptor — the variable simply isn't there. This means the HA
NUT integration provides **battery/runtime/status sensors only**, not
wattage for the Energy Dashboard.
- **apcupsd test note:** Debian's `apcupsd` package conflicts with
`nut-server` (mutually exclusive). apcupsd USB mode returned `COMMLOST`
even before we could check load. The APC Smart Serial protocol (serial
cable, AP940-1524C, ~$30) DOES report load%, but this requires a serial
port on the UPS and on the host.
- **Energy Dashboard path:** A smart plug (Shelly Plug S / TP-Link Kasa,
~$15-25) on the UPS output reports real watts natively and feeds the
Energy Dashboard with zero UPS-driver hacking. The NUT sensors remain
valuable for outage detection and graceful-shutdown automations.
> **Documentation moved to Discourse — the canonical source of truth.**
>
> **UPS management (NUT) for APC Smart-UPS C 1500**
>
> **Read it here:** https://community.turnsys.com/t/301
>
> *Migrated 2026-08-06. This file is kept as a pointer for git-browsing context.
> Do not update content here — edit the Discourse wiki topic instead.*