KNEL/PFVCluster
5
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: migrate all documentation to Discourse wiki topics

Browse Source 
All knowledge docs (architecture, runbooks, references, audits, policies)
have been migrated to community.turnsys.com as wiki topics in the VP
TechOps category. Discourse is now the canonical source of truth for
documentation; git edit history no longer serves as the doc changelog.

37 .md files converted to short pointers linking to their Discourse
topics. AGENTS.md updated with new documentation workflow policy.
Code (scripts, configs, playbooks) remains authoritative in git.
This commit is contained in:
reachableceo
2026-08-06 14:08:34 -05:00
parent d6988bb35f
commit 88faf35ec3
37 changed files with 378 additions and 8239 deletions
Download Patch File Download Diff File Expand all files Collapse all files
dcinfra/ups/README.md
+9 -211
View File
@@ -1,212 +1,10 @@
# UPS Management (NUT — Network UPS Tools)
# dcinfra/ups/README.md
> **Redmine:** [#340](https://projects.knownelement.com/issues/340) (APC, closed) · [#372](https://projects.knownelement.com/issues/372) (Tripp Lite, open)
Centralized UPS monitoring for the server room via
[NUT](https://networkupstools.org/), running on **pfv-tsys1**. USB HID UPS
units feed one `upsd` network server; Home Assistant polls it over Tailscale for
real-time power/load/runtime tracking, and a local `upsmon` shuts the hypervisor
down gracefully when battery is low.
> **Why NUT (not apcupsd)?** Two different UPS brands (APC + Tripp Lite) must be
> covered. `apcupsd` only supports APC, so it would require a second daemon
> stack. NUT's `usbhid-ups` driver speaks to **both** via the USB HID Power
> Device class, and Home Assistant ships a first-class NUT integration.
## Hardware
| UPS | Model | VID:PID | USB Serial | Status |
|-----|-------|---------|------------|--------|
| **APC** | Smart-UPS C 1500 (FW 02.2) | `051d:0003` | `AS1213210423` | **LIVE** |
| **Tripp Lite** | UPS (HID PDC) | `09ae:3016` | `2352CVLSM871900694` | **Blocked** — see below |
## Current State (2026-07-30)
### APC Smart-UPS C 1500 — OPERATIONAL
Fully reporting via `usbhid-ups` + `APC HID 0.100` subdriver. Data validated:
```
battery.charge: 100 battery.runtime: 1800 battery.voltage: 27.4
ups.status: OL ups.load: (via HA) ups.model: Smart-UPS C 1500
```
### Tripp Lite UPS — BLOCKED (hardware issue)
The driver finds the device, matches the `TrippLite HID 0.85` subdriver, claims
the interface, and reads the HID descriptor — but **fails reading the 878-byte
HID Report Descriptor** (`Resource temporarily unavailable` / EAGAIN after 5s).
The driver is masked to prevent restart-loop spam.
USB descriptors (manufacturer, product, serial) are readable via `lsusb -v` and
`nut-scanner`, but the bulk control transfer for the full report descriptor
times out. Likely causes:
1. **USB hub** — the Tripp Lite is behind a Genesys Logic hub (`05e3:0608`).
Try plugging directly into a motherboard USB port.
2. **USB cable** — try a high-quality data cable (not charge-only).
3. **UPS firmware** — the USB controller may not properly implement all HID
endpoints.
**To retry after physical reseat:**
```bash
# On pfv-tsys1:
systemctl unmask nut-driver@tripp-lite-ups
systemctl start nut-driver@tripp-lite-ups
upsc tripp-lite-ups@localhost
```
## Architecture
```
pfv-tsys1 (192.168.3.11 / Tailscale 100.121.189.98)
├─ APC Smart-UPS C 1500 ──┐
└─ Tripp Lite UPS (masked) ──┤ USB HID
▼
nut-driver@apc-smartups-c1500 (usbhid-ups)
▼
upsd :3493 (LISTEN 127.0.0.1 + Tailscale + LAN)
▼ ▼
upsmon (local) Home Assistant (NUT integration)
graceful shutdown via LAN 192.168.3.11 (HAOS can't
route to Tailscale IPs)
```
- **Driver layer** — `usbhid-ups` process, pinned by USB serial. Debian uses
templated `nut-driver@<upsname>.service` units managed by
`nut-driver-enumerator`.
- **Server layer** — `upsd` exposes UPS data on TCP 3493 (localhost + Tailscale
+ LAN). Clients authenticate via `upsd.users`.
- **Monitor layer** — `upsmon` runs locally as `master` to trigger
`SHUTDOWNCMD` (`/sbin/shutdown -h now`) when a UPS reports `LOWBATT`.
- **Home Assistant** — native NUT integration connects to `upsd` over Tailscale
and exposes `ups.load`, `battery.runtime`, `ups.status`, etc. as sensors.
### Key deployment lesson: udev must cover raw USB devices
The `usbhid-ups` driver opens `/dev/bus/usb/BBB/DDD` (raw USB device files),
**not** `/dev/hidraw*`. After calling `setuid(111)` to drop to the `nut` user,
it needs write access to those raw USB files. The udev rule must match
`SUBSYSTEM=="usb"` by vendor/product ID to set `GROUP="nut"` — matching only
`hidraw` is insufficient. See `/etc/udev/rules.d/99-nut-ups.rules`.
## Scripts
NUT host scripts run on pfv-tsys1 via `tests/remote.sh`:
```bash
# Idempotent install + configure (safe to re-run):
PROX_HOST=pfv-tsys1 bash tests/remote.sh prox-file ups/setup.sh
# Discover USB UPS + NUT state (read-only diagnostic):
PROX_HOST=pfv-tsys1 bash tests/remote.sh prox-file ups/discover.sh
# Query UPS data + service health:
PROX_HOST=pfv-tsys1 bash tests/remote.sh prox-file ups/status.sh
```
The HA integration script runs from your workstation (needs HA API access):
```bash
# Add the NUT integration to Home Assistant (idempotent):
bash ups/setup-ha-nut.sh
```
`setup.sh` accepts environment overrides for serials/VIDs/PIDs/usernames, so it
can be repurposed for other hosts or UPS units. Passwords for `monuser` and
`homeassistant` are auto-generated on first run and reused on subsequent runs
(stored in `/etc/nut/upsd.users`).
Set `TRIPP_ENABLED=0` to skip the Tripp Lite entirely (useful if it's physically
unplugged).
## Configuration files on pfv-tsys1
| File | Purpose |
|------|---------|
| `/etc/udev/rules.d/99-nut-ups.rules` | Grant nut group rw on raw USB + hidraw devices (both subsystems) |
| `/etc/nut/ups.conf` | `usbhid-ups` device(s), pinned by serial + subdriver |
| `/etc/nut/upsd.conf` | `LISTEN 127.0.0.1` + `LISTEN <tailscale>` + `LISTEN <lan>` on port 3493 |
| `/etc/nut/upsd.users` | `monuser` (master) + `homeassistant` (read-only) credentials |
| `/etc/nut/upsmon.conf` | Local master monitor + `SHUTDOWNCMD` |
| `/etc/nut/nut.conf` | `MODE=netserver` |
## Home Assistant integration
The NUT integration is added automatically by `setup-ha-nut.sh`, which drives
HA's REST config-flow API. It is idempotent (skips if the entry exists).
```bash
# Prerequisites: create token + password files (one-time):
mkdir -p ~/.config/pfvcluster
# HA → Profile → Long-Lived Access Tokens → Create Token:
echo -n 'YOUR_HA_TOKEN' > ~/.config/pfvcluster/ha-token
# Password is in /etc/nut/upsd.users on pfv-tsys1 (the homeassistant user):
echo -n 'YOUR_NUT_PASS' > ~/.config/pfvcluster/nut-password
chmod 600 ~/.config/pfvcluster/{ha-token,nut-password}
# Run:
bash ups/setup-ha-nut.sh
```
### Why LAN IP, not Tailscale
upsd listens on **both** the Tailscale IP (`100.121.189.98`) **and** the LAN IP
(`192.168.3.11`). The HA NUT integration uses the **LAN IP** because HAOS runs
Tailscale as an isolated add-on container — the HA core container cannot route
to Tailscale IPs. Since pfv-bms (HA, `192.168.3.12`) and pfv-tsys1 (`192.168.3.11`)
share the same vmbr0 bridge, LAN connectivity is instant and reliable.
### Manual UI alternative
In Home Assistant → **Settings → Devices & Services → Add Integration → NUT**:
| Field | Value |
|-------|-------|
| Host | `192.168.3.11` (LAN — HAOS can't reach Tailscale IPs from the HA container) |
| Port | `3493` |
| Username | `homeassistant` |
| Password | *(stored in `/etc/nut/upsd.users` on pfv-tsys1)* |
| UPS | `apc-smartups-c1500` |
### Live sensors
HA exposes UPS data as sensors (prefix `sensor.apc_smartups_c1500_`):
`battery_charge`, `status` (Online/On Battery), `status_data` (OL/OB/DISCHRG).
Additional sensors (load, runtime, voltage) populate as the UPS reports them.
## Daily operations
From pfv-tsys1 (or any tailnet host with NUT client installed):
```bash
# List UPS units served by upsd
upsc -l pfv-tsys1
# Full variable dump for one UPS
upsc apc-smartups-c1500@pfv-tsys1
# Battery runtime (the only runtime/charge data this UPS exposes)
upsc apc-smartups-c1500@pfv-tsys1 battery.runtime
```
## Notes
- **No USB passthrough to the HA VM.** Keeping the UPS on the host preserves
hypervisor graceful-shutdown capability and matches the `powerman/` pattern
(PDU managed on the host where the adapter physically lives).
- **No `ups.load` / `ups.realpower` on this UPS (FW 02.2, mfg 2012):** The
APC Smart-UPS C 1500 does not expose load or power data over USB HID.
Both NUT `usbhid-ups` and `apcupsd` (USB mode, tested 2026-07-30) read the
same HID descriptor — the variable simply isn't there. This means the HA
NUT integration provides **battery/runtime/status sensors only**, not
wattage for the Energy Dashboard.
- **apcupsd test note:** Debian's `apcupsd` package conflicts with
`nut-server` (mutually exclusive). apcupsd USB mode returned `COMMLOST`
even before we could check load. The APC Smart Serial protocol (serial
cable, AP940-1524C, ~$30) DOES report load%, but this requires a serial
port on the UPS and on the host.
- **Energy Dashboard path:** A smart plug (Shelly Plug S / TP-Link Kasa,
~$15-25) on the UPS output reports real watts natively and feeds the
Energy Dashboard with zero UPS-driver hacking. The NUT sensors remain
valuable for outage detection and graceful-shutdown automations.
> **Documentation moved to Discourse — the canonical source of truth.**
>
> **UPS management (NUT) for APC Smart-UPS C 1500**
>
> **Read it here:** https://community.turnsys.com/t/301
>
> *Migrated 2026-08-06. This file is kept as a pointer for git-browsing context.
> Do not update content here — edit the Discourse wiki topic instead.*